Repository: ESMCI/cime

Issue:
Should we support one or more scripts for quickly parsing output from large test suites?
We're trying to get more CTSM developers to run the full aux_clm test suite. One aspect of making this easier is providing a robust, easy-to-use mechanism for parsing the results of a large test suite to determine what has failed.

I wrote a simple script a few years ago that I use all the time for this purpose (https://github.com/billsacks/bin/blob/master/filter_cs_status). My workflow is (1) Run the appropriate cs.status script(s) for the test suite I ran; (2) Run `filter_cs_status` on the output from each. (I like having these two steps separated, because (1) can take a while on large test suites, and I often rerun (2) a few times for each test suite, excluding various things each time.) [Update 2018-08-14: cs.status seems to run fast enough that these don't really need to be separated. And for the sake of new users, it would be nice if this were built into cs.status.]

It's basically a glorified `grep -v`, with the added benefits that (1) There are a few hard-coded patterns that it always ignores (lines with 'Overall...' and 'PASS'; I also generally ignore TPUTCOMP and MEMCOMP, but that wouldn't need to be the case); (2) It prints counts of the number of lines matching each ignore pattern (so, for example, you can see if _everything_ had a TPUTCOMP failure, or just a few tests).

I'm wondering if I should put this script in cime. (I'd be happy to rewrite it in python.) Would others find this useful? If not, is there another, similar script that is already supported or should be supported for this purpose? There's the old cimeteststatus script, but my sense is that this no longer works with cime5. (Should we delete the old cimeteststatus script?)


Discussion:
By the way, here's some example output from `filter_cs_status`:

```
[cheyenne4:/glade/scratch/sacks/tests_0804_2122c]$ filter_cs_status < cs.i
========================================================================
0804_2122c_i
    FAIL ERI_N2_Ld9.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-default BASELINE ctsm1.0.dev006
    FAIL ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart RUN time=105
    FAIL ERP_D_Ld5.f09_g17.I2000Clm50Vic.cheyenne_intel.clm-vrtlay RUN time=21
    FAIL SMS_Lm13.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-cropMonthOutput BASELINE ctsm1.0.dev006
========================================================================
Ignored lines:
   ^\s*$ : 2
   \(Overall:.*details: : 156
   ^ *PASS : 2140
    TPUTCOMP  : 1
    MEMCOMP  : 8
[cheyenne4:/glade/scratch/sacks/tests_0804_2122c]$ filter_cs_status BASELINE < cs.i
========================================================================
0804_2122c_i
    FAIL ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart RUN time=105
    FAIL ERP_D_Ld5.f09_g17.I2000Clm50Vic.cheyenne_intel.clm-vrtlay RUN time=21
========================================================================
Ignored lines:
   ^\s*$ : 2
   \(Overall:.*details: : 156
   ^ *PASS : 2140
    TPUTCOMP  : 1
    MEMCOMP  : 8
   BASELINE : 2
```
There's also this script in ctsm that sounds like it does essentially the same thing:

https://github.com/ESCOMP/ctsm/blob/master/parse_cime.cs.status

It's in perl, but's small enough it could be changed.
And here's some sample output that it creates for summary mode (which is how I normally use it):

```
scripts/cases> ../../../parse_cime.cs.status cs.status.ctsm10dev008ch* -s
================================================================================
Test summary
191 Total tests
188 Tests passed
36 Tests compare different to baseline
0 Tests are new where there is no baseline
1 Tests pending
2 Tests failed
================================================================================
================================================================================
These tests passed
================================================================================
DAE_N2_D_Lh12.f10_f10_musgs.I2000Clm50BgcCropGs.cheyenne_intel.clm-DA_multidrv			
ERI_D_Ld9.T31_g37.I2000Clm50Sp.cheyenne_intel.clm-SNICARFRC			
ERI_D_Ld9.f09_g16.I1850Clm45BgcCruGs.cheyenne_intel.clm-default			
ERI_D_Ld9.f09_g16.I1850Clm50Sp.cheyenne_intel.clm-default			
.
.
.			
ERP_D_Ld5.f09_g17.I2000Clm50Vic.cheyenne_intel.clm-vrtlay			FAILED PREVIOUSLY
ERP_D_Ld5.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-ciso_flexCN_FUN			
ERP_D_Ld5.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_intel.clm-NoVSNoNI			
ERP_D_Ld5.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_intel.clm-rootlit			
.
.
.	
SSP_D_Ld4.f09_g17.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_rtmColdSSP			
SSP_Ld10.f19_g17.I1850Clm50Bgc.cheyenne_intel.clm-rtmColdSSP			
================================================================================
These tests compare different to the baseline
================================================================================
ERI_D_Ld9.f10_f10_musgs.I1850Clm50Bgc.cheyenne_gnu.clm-default
ERI_Ld9.f09_g17.I1850Clm50Bgc.cheyenne_intel.clm-drydepnomegan
ERP_D_Ld5.f09_g17.I2000Clm50Vic.cheyenne_intel.clm-vrtlay
ERP_D_Ld5.f19_g17_gl4.I1850Clm50BgcCrop.cheyenne_intel.clm-glcMEC_changeFlags
ERP_D_Ld5.f19_g17_gl4.I1850Clm50BgcCropG.cheyenne_intel.clm-glcMEC_changeFlags
.
.
.
SMS_Lm1.f19_g17_gl4.I1850Clm50Bgc.cheyenne_intel.clm-clm50dynroots
SSP_D_Ld10.f19_g17.I1850Clm50Bgc.cheyenne_intel.clm-rtmColdSSP
SSP_D_Ld4.f09_g17.I1850Clm50BgcCrop.cheyenne_intel.clm-ciso_rtmColdSSP
SSP_Ld10.f19_g17.I1850Clm50Bgc.cheyenne_intel.clm-rtmColdSSP
================================================================================
These tests are pending (some tests may fail in the pending state)
================================================================================
ERS_Lm40_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropGs.cheyenne_gnu.clm-monthly		
================================================================================
These tests failed
================================================================================
ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart		EXPECTED
ERS_Lm20_Mmpi-serial.1x1_smallvilleIA.I2000Clm50BgcCropGs.cheyenne_gnu.clm-monthly		EXPECTED

```
Don't forget CIME's wait_for_tests script/library.
Okay, as I should have realized, we all have different workflows and scripts that work well for our workflow and the way we want to see test results summarized. Do people have any feelings as to whether I move my `filter_cs_status` script into cime/scripts/Tools, or maintain it outside of cime? (My preference would be to move it into cime, and I would add some documentation of this as one option for parsing test results. But I'm okay with keeping it out of cime if others would prefer that cime not get cluttered in this way.)
@billsacks , I'm no expert on filter_cs_status, but I would encourage you to at least think about wait_for_tests since it's officially supported in CIME (unit and regr tested).

You can use it as a library:
```
def wait_for_tests_impl(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False):
```
^ all you need to pass in is a glob that captures all the TestStatus files that you are interested in. You probably want to pass in no_wait=True since you aren't interested in the waiting. It returns a list of tuples [(test_name, test_path, overall_test_status)].
It looks like you want to get the phase that caused the fail; I could add that as a fourth item in that tuple. I think that would be a lot easier than re-writing your entire script in python.
I've given this more thought, and also looked at tools used by @ekluzek and @mnlevy1981 , and I'd like to propose the following requirements:

1. **This parsing / filtering ability should be built in to cs.status.** We've been having trouble getting CTSM scientists to adopt the test suite, so I want to make this as easy as possible. Since cs.status already appears in the test directory, and already has the basic functionality we need, this seems like the simplest option for new users, rather than requiring them to know about yet another script.

2. **It should be possible to see, at a glance, everything that has failed (not having to visually sift through output).** What I find easiest to work with, as a starting point, is a list of just the non-passing lines from all of the relevant TestStatus files. (There can be more than one such line for a given test, e.g., if there are NLCOMP failures and BASELINE failures.) So if you have a fully-passing test suite, you should have very minimal (or no) output. Others seem to prefer to see more information, but to have passes listed separately from failures; that's okay with me, too.

3. **It should be easy to filter out particular failures, but with these filtered failures still appearing in some way.** For example, if baselines are expected to fail, it should be easy to either (a) just see a count of baseline failures, or (b) have baseline failures listed separately from other failures.

As a starting point, I'd like to propose the following implementation, which I feel would be fairly simple to implement and maintain:

1. Add an option to cs.status like `-f` / `--fails-only` (maybe a misnomer, because this really would include PENDs as well as FAILs; I'm not tied to this name). If provided, this would only output lines from the TestStatus files that do not have a PASS status.

   I think this could be implemented simply by having an optional argument to ts.phase_statuses_dump (skip_passes=True), which would avoid giving output for any phase with a PASS status.

2. Add another option to cs.status, `-c PHASE` / `--count-fails PHASE`, where PHASE is any of the phases in `ALL_PHASES` in `test_status.py`. This can be provided multiple times. For any of the given phases: We do not include the line-by-line output from this phase, but instead give a summary count of the number of tests that failed this phase.

   I think this could be implemented fairly easily, by having a bit more logic in `phase_status_dump`, and then having a new routine in `test_status.py`: `phase_status_count_failures(self, counts)`, where counts is a dictionary of counts for each phase of interest; each failure in a phase of interest increments the given count. Then, at the bottom of the cs.status output, we'd print the counts for each listed phase.

3. Add an option to cs.status, `--count-comp-fails`. This is a short-hand for `--count-failures MEMCOMP --count-failures TPUTCOMP`, because this is something we commonly do for the CTSM test suite (since those failures are often false positives).


With all that, a typical CTSM user would run:

`./cs.status.0804_2122c_i --count-comp-fails`, and would get output like:

```
    FAIL ERI_N2_Ld9.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-default BASELINE ctsm1.0.dev006
    FAIL ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart RUN time=105
    FAIL ERP_D_Ld5.f09_g17.I2000Clm50Vic.cheyenne_intel.clm-vrtlay RUN time=21
    FAIL SMS_Lm13.f19_g17.I2000Clm50BgcCrop.cheyenne_intel.clm-cropMonthOutput BASELINE ctsm1.0.dev006
========================================================================
Additional failure counts:
    TPUTCOMP  : 1
    MEMCOMP  : 8
```


@jgfouca and others: If you're okay with my proposal in the comment I just made, I'm happy to take this on myself (I'm not asking you or anyone else to do this).
@billsacks , I think that looks like a good idea.
@jgfouca thanks. I'll take a stab at this soon.

Resolving patch:
diff --git a/scripts/lib/CIME/cs_status.py b/scripts/lib/CIME/cs_status.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/cs_status.py
@@ -0,0 +1,82 @@
+"""
+Implementation of the cs.status script, which prints the status of all
+of the tests in one or more test suites
+"""
+
+from __future__ import print_function
+from CIME.XML.standard_module_setup import *
+from CIME.test_status import TestStatus
+import os
+import sys
+from collections import defaultdict
+
+def cs_status(test_paths, summary=False, fails_only=False,
+              count_fails_phase_list=None, out=sys.stdout):
+    """Print the test statuses of all tests in test_paths. The default
+    is to print to stdout, but this can be overridden with the 'out'
+    argument.
+
+    If summary is True, then only the overall status of each test is printed
+
+    If fails_only is True, then only test failures are printed (this
+    includes PENDs as well as FAILs).
+
+    If count_fails_phase_list is provided, it should be a list of phases
+    (from the phases given by test_status.ALL_PHASES). For each phase in
+    this list: do not give line-by-line output; instead, just report the
+    total number of tests that have not PASSed this phase (this includes
+    PENDs and FAILs). (This is typically used with the fails_only
+    option, but it can also be used without that option.)
+    """
+    expect(not (summary and fails_only),
+           "Cannot have both summary and fails_only")
+    expect(not (summary and count_fails_phase_list),
+           "Cannot have both summary and count_fails_phase_list")
+    if count_fails_phase_list is None:
+        count_fails_phase_list = []
+    non_pass_counts = dict.fromkeys(count_fails_phase_list, 0)
+    test_id_output = defaultdict(str)
+    test_id_counts = defaultdict(int)
+    for test_path in test_paths:
+        test_dir=os.path.dirname(test_path)
+        ts = TestStatus(test_dir=test_dir)
+        test_id = os.path.basename(test_dir).split(".")[-1]
+        if summary:
+            output = _overall_output(ts, "  {status} {test_name}\n")
+        else:
+            if fails_only:
+                output = ''
+            else:
+                output = _overall_output(ts, "  {test_name} (Overall: {status}) details:\n")
+            output += ts.phase_statuses_dump(prefix="    ",
+                                             skip_passes=fails_only,
+                                             skip_phase_list=count_fails_phase_list)
+            if count_fails_phase_list:
+                ts.increment_non_pass_counts(non_pass_counts)
+
+        test_id_output[test_id] += output
+        test_id_counts[test_id] += 1
+
+    for test_id in sorted(test_id_output):
+        count = test_id_counts[test_id]
+        print("{}: {} test{}".format(test_id, count, 's' if count > 1 else ''), file=out)
+        print(test_id_output[test_id], file=out)
+        print(' ', file=out)
+
+    if count_fails_phase_list:
+        print(72*'=', file=out)
+        print('Non-PASS results for select phases:', file=out)
+        for phase in count_fails_phase_list:
+            print('{} non-passes: {}'.format(phase, non_pass_counts[phase]), file=out)
+
+def _overall_output(ts, format_str):
+    """Returns a string giving the overall test status
+
+    Args:
+    ts: TestStatus object
+    format_str (string): string giving the format of the output; must
+        contain place-holders for status and test_name
+    """
+    test_name = ts.get_name()
+    status = ts.get_overall_test_status()
+    return format_str.format(status=status, test_name=test_name)
diff --git a/scripts/lib/CIME/cs_status_creator.py b/scripts/lib/CIME/cs_status_creator.py
--- a/scripts/lib/CIME/cs_status_creator.py
+++ b/scripts/lib/CIME/cs_status_creator.py
@@ -7,7 +7,7 @@
 import os
 import stat
 
-def create_cs_status(test_root, test_id, filename=None):
+def create_cs_status(test_root, test_id, extra_args='', filename=None):
     """Create a test suite-specific cs.status file from the template
 
     Arguments:
@@ -19,6 +19,8 @@ def create_cs_status(test_root, test_id, filename=None):
         this too general: for example, ending this with '*' will pick up
         the *.ref1 directories for ERI and other tests, which is NOT
         what you want.
+    extra_args (string): extra arguments to the cs.status command
+        (If there are multiple arguments, these should be in a space-delimited string.)
     filename (string): name of the generated cs.status file. If not
         given, this will be built from the test_id.
     """
@@ -28,6 +30,7 @@ def create_cs_status(test_root, test_id, filename=None):
     template = open(template_file, "r").read()
     template = template.replace("<PATH>",
                                 os.path.join(cime_root,"scripts","Tools")).replace\
+                                ("<EXTRA_ARGS>", extra_args).replace\
                                 ("<TESTID>", test_id).replace\
                                 ("<TESTROOT>", test_root)
     if not os.path.exists(test_root):
diff --git a/scripts/lib/six_additions.py b/scripts/lib/six_additions.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/six_additions.py
@@ -0,0 +1,13 @@
+"""Additions to the six library needed for python 2/3 compatibility"""
+
+import six
+
+if six.PY3:
+    # This is only available in python3.2 and later, so this code won't
+    # run with python3 versions prior to 3.2
+    _assertNotRegex = "assertNotRegex"
+else:
+    _assertNotRegex = "assertNotRegexpMatches"
+
+def assertNotRegex(self, *args, **kwargs):
+    return getattr(self, _assertNotRegex)(*args, **kwargs)


Test patch:
diff --git a/scripts/lib/CIME/test_status.py b/scripts/lib/CIME/test_status.py
--- a/scripts/lib/CIME/test_status.py
+++ b/scripts/lib/CIME/test_status.py
@@ -226,15 +226,24 @@ def get_status(self, phase):
     def get_comment(self, phase):
         return self._phase_statuses[phase][1] if phase in self._phase_statuses else None
 
-    def phase_statuses_dump(self, prefix=''):
+    def phase_statuses_dump(self, prefix='', skip_passes=False, skip_phase_list=None):
         """
         Args:
             prefix: string printed at the start of each line
+            skip_passes: if True, do not output lines that have a PASS status
+            skip_phase_list: list of phases (from the phases given by
+                ALL_PHASES) for which we skip output
         """
+        if skip_phase_list is None:
+            skip_phase_list = []
         result = ""
         if self._phase_statuses:
             for phase, data in self._phase_statuses.items():
+                if phase in skip_phase_list:
+                    continue
                 status, comments = data
+                if skip_passes and status == TEST_PASS_STATUS:
+                    continue
                 if not comments:
                     result += "{}{} {} {}\n".format(prefix, status, self._test_name, phase)
                 else:
@@ -242,6 +251,21 @@ def phase_statuses_dump(self, prefix=''):
 
         return result
 
+    def increment_non_pass_counts(self, non_pass_counts):
+        """
+        Increment counts of the number of times given phases did not pass
+
+        non_pass_counts is a dictionary whose keys are phases of
+        interest and whose values are running counts of the number of
+        non-passes. This method increments those counts based on results
+        in the given TestStatus object.
+        """
+        for phase in non_pass_counts:
+            if phase in self._phase_statuses:
+                status, _ = self._phase_statuses[phase]
+                if status != TEST_PASS_STATUS:
+                    non_pass_counts[phase] += 1
+
     def flush(self):
         if self._phase_statuses and not self._no_io:
             with open(self._filename, "w") as fd:
diff --git a/scripts/lib/CIME/tests/test_cs_status.py b/scripts/lib/CIME/tests/test_cs_status.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/tests/test_cs_status.py
@@ -0,0 +1,315 @@
+#!/usr/bin/env python
+
+import unittest
+import shutil
+import os
+import tempfile
+import re
+import six
+import six_additions
+from CIME.cs_status import cs_status
+from CIME import test_status
+
+# ========================================================================
+# Custom assertions
+# ========================================================================
+
+class CustomAssertions(unittest.TestCase):
+
+    def assert_status_of_phase(self, output, status, phase, test_name):
+        """Asserts that 'output' contains a line showing the given
+        status for the given phase for the given test_name"""
+        expected = re.compile(r'^ *{} +'.format(re.escape(status)) +
+                              self._test_name_and_phase_regex(test_name, phase),
+                              flags=re.MULTILINE)
+
+        six.assertRegex(self, output, expected)
+
+    def assert_phase_absent(self, output, phase, test_name):
+        """Asserts that 'output' does not contain a status line for the
+        given phase and test_name"""
+        expected = re.compile(r'^.* +' +
+                              self._test_name_and_phase_regex(test_name, phase),
+                              flags=re.MULTILINE)
+
+        six_additions.assertNotRegex(self, output, expected)
+
+    def assert_core_phases(self, output, test_name, fails):
+        """Asserts that 'output' contains a line for each of the core test
+        phases for the given test_name. All results should be PASS
+        except those given by the fails list, which should be FAILS.
+        """
+        for phase in test_status.CORE_PHASES:
+            if phase in fails:
+                status = test_status.TEST_FAIL_STATUS
+            else:
+                status = test_status.TEST_PASS_STATUS
+            self.assert_status_of_phase(output=output,
+                                        status=status,
+                                        phase=phase,
+                                        test_name=test_name)
+
+    @staticmethod
+    def _test_name_and_phase_regex(test_name, phase):
+        """Returns a regex matching the portion of a TestStatus line
+        containing the test name and phase"""
+        # The main purpose of extracting this into a shared method is:
+        # assert_phase_absent could wrongly pass if the format of the
+        # TestStatus output changed without that method's regex
+        # changing. By making its regex shared as much as possible with
+        # the regex in assert_status_of_phase, we decrease the chances
+        # of these false passes.
+        return r'{} +{} *$'.format(re.escape(test_name), re.escape(phase))
+
+class TestCustomAssertions(CustomAssertions):
+
+    @staticmethod
+    def output_line(status, test_name, phase):
+        return status + ' ' + test_name + ' ' + phase + '\n'
+
+    # Note: No explicit tests of assert_status_of_phase: this is covered
+    # sufficiently via tests of assert_core_phases
+
+    def test_assert_phase_absent_passes(self):
+        """assert_phase_absent should pass when the phase is absent for
+        the given test_name"""
+        test_name1 = 'my.test.name1'
+        test_name2 = 'my.test.name2'
+        output = self.output_line('PASS', test_name1, 'PHASE1')
+        output += self.output_line('PASS', test_name2, 'PHASE2')
+
+        self.assert_phase_absent(output, 'PHASE2', test_name1)
+        self.assert_phase_absent(output, 'PHASE1', test_name2)
+
+    def test_assert_phase_absent_fails(self):
+        """assert_phase_absent should fail when the phase is present for
+        the given test_name"""
+        test_name = 'my.test.name'
+        output = self.output_line('PASS', test_name, 'PHASE1')
+
+        with self.assertRaises(AssertionError):
+            self.assert_phase_absent(output, 'PHASE1', test_name)
+
+    def test_assert_core_phases_passes(self):
+        """assert_core_phases passes when it should"""
+        output = ''
+        fails = [test_status.CORE_PHASES[1]]
+        test_name = 'my.test.name'
+        for phase in test_status.CORE_PHASES:
+            if phase in fails:
+                status = test_status.TEST_FAIL_STATUS
+            else:
+                status = test_status.TEST_PASS_STATUS
+            output = output + self.output_line(status, test_name, phase)
+
+        self.assert_core_phases(output, test_name, fails)
+
+    def test_assert_core_phases_missing_phase_fails(self):
+        """assert_core_phases fails if there is a missing phase"""
+        output = ''
+        test_name = 'my.test.name'
+        for phase in test_status.CORE_PHASES:
+            if phase != test_status.CORE_PHASES[1]:
+                output = output + self.output_line(test_status.TEST_PASS_STATUS, test_name, phase)
+
+        with self.assertRaises(AssertionError):
+            self.assert_core_phases(output, test_name, fails=[])
+
+    def test_assert_core_phases_wrong_status_fails(self):
+        """assert_core_phases fails if a phase has the wrong status"""
+        output =''
+        test_name = 'my.test.name'
+        for phase in test_status.CORE_PHASES:
+            output = output + self.output_line(test_status.TEST_PASS_STATUS, test_name, phase)
+
+        with self.assertRaises(AssertionError):
+            self.assert_core_phases(output, test_name, fails=[test_status.CORE_PHASES[1]])
+
+    def test_assert_core_phases_wrong_name_fails(self):
+        """assert_core_phases fails if the test name is wrong"""
+        output =''
+        test_name = 'my.test.name'
+        for phase in test_status.CORE_PHASES:
+            output = output + self.output_line(test_status.TEST_PASS_STATUS, test_name, phase)
+
+        with self.assertRaises(AssertionError):
+            self.assert_core_phases(output, 'my.test', fails=[])
+
+# ========================================================================
+# Actual test class
+# ========================================================================
+
+class TestCsStatus(CustomAssertions):
+
+    # ------------------------------------------------------------------------
+    # Test helper functions
+    # ------------------------------------------------------------------------
+
+    # An arbitrary phase we can use when we want to work with a non-core phase
+    _NON_CORE_PHASE = test_status.MEMLEAK_PHASE
+
+    # Another arbitrary phase if we need two different non-core phases
+    _NON_CORE_PHASE2 = test_status.BASELINE_PHASE
+
+    def setUp(self):
+        self._testroot = tempfile.mkdtemp()
+        self._output = six.StringIO()
+
+    def tearDown(self):
+        self._output.close()
+        shutil.rmtree(self._testroot, ignore_errors=True)
+
+    def create_test_dir(self, test_dir):
+        """Creates the given test directory under testroot.
+
+        Returns the full path to the created test directory.
+        """
+        fullpath = os.path.join(self._testroot, test_dir)
+        os.makedirs(fullpath)
+        return fullpath
+
+    @staticmethod
+    def create_test_status_core_passes(test_dir_path, test_name):
+        """Creates a TestStatus file in the given path, with PASS status
+        for all core phases"""
+        with test_status.TestStatus(test_dir=test_dir_path,
+                                    test_name=test_name) as ts:
+            for phase in test_status.CORE_PHASES:
+                ts.set_status(phase, test_status.TEST_PASS_STATUS)
+
+    def set_last_core_phase_to_fail(self, test_dir_path, test_name):
+        """Sets the last core phase to FAIL
+
+        Returns the name of this phase"""
+        fail_phase = test_status.CORE_PHASES[-1]
+        self.set_phase_to_status(test_dir_path=test_dir_path,
+                                 test_name=test_name,
+                                 phase=fail_phase,
+                                 status=test_status.TEST_FAIL_STATUS)
+        return fail_phase
+
+    @staticmethod
+    def set_phase_to_status(test_dir_path, test_name, phase, status):
+        """Sets the given phase to the given status for this test"""
+        with test_status.TestStatus(test_dir=test_dir_path,
+                                    test_name=test_name) as ts:
+            ts.set_status(phase, status)
+
+    # ------------------------------------------------------------------------
+    # Begin actual tests
+    # ------------------------------------------------------------------------
+
+    def test_single_test(self):
+        """cs_status for a single test should include some minimal expected output"""
+        test_name = 'my.test.name'
+        test_dir = 'my.test.name.testid'
+        test_dir_path = self.create_test_dir(test_dir)
+        self.create_test_status_core_passes(test_dir_path, test_name)
+        cs_status([os.path.join(test_dir_path, 'TestStatus')],
+                  out=self._output)
+        self.assert_core_phases(self._output.getvalue(), test_name, fails=[])
+
+    def test_two_tests(self):
+        """cs_status for two tests (one with a FAIL) should include some minimal expected output"""
+        test_name1 = 'my.test.name1'
+        test_name2 = 'my.test.name2'
+        test_dir1 = test_name1 + '.testid'
+        test_dir2 = test_name2 + '.testid'
+        test_dir_path1 = self.create_test_dir(test_dir1)
+        test_dir_path2 = self.create_test_dir(test_dir2)
+        self.create_test_status_core_passes(test_dir_path1, test_name1)
+        self.create_test_status_core_passes(test_dir_path2, test_name2)
+        test2_fail_phase = self.set_last_core_phase_to_fail(test_dir_path2, test_name2)
+        cs_status([os.path.join(test_dir_path1, 'TestStatus'),
+                   os.path.join(test_dir_path2, 'TestStatus')],
+                  out=self._output)
+        self.assert_core_phases(self._output.getvalue(), test_name1, fails=[])
+        self.assert_core_phases(self._output.getvalue(), test_name2, fails=[test2_fail_phase])
+
+    def test_fails_only(self):
+        """With fails_only flag, only fails and pends should appear in the output"""
+        test_name = 'my.test.name'
+        test_dir = 'my.test.name.testid'
+        test_dir_path = self.create_test_dir(test_dir)
+        self.create_test_status_core_passes(test_dir_path, test_name)
+        fail_phase = self.set_last_core_phase_to_fail(test_dir_path, test_name)
+        pend_phase = self._NON_CORE_PHASE
+        self.set_phase_to_status(test_dir_path, test_name,
+                                 phase=pend_phase,
+                                 status=test_status.TEST_PEND_STATUS)
+        cs_status([os.path.join(test_dir_path, 'TestStatus')],
+                  fails_only=True,
+                  out=self._output)
+        self.assert_status_of_phase(output=self._output.getvalue(),
+                                    status=test_status.TEST_FAIL_STATUS,
+                                    phase=fail_phase,
+                                    test_name=test_name)
+        self.assert_status_of_phase(output=self._output.getvalue(),
+                                    status=test_status.TEST_PEND_STATUS,
+                                    phase=pend_phase,
+                                    test_name=test_name)
+        for phase in test_status.CORE_PHASES:
+            if phase != fail_phase:
+                self.assert_phase_absent(output=self._output.getvalue(),
+                                         phase=phase,
+                                         test_name=test_name)
+        six_additions.assertNotRegex(self, self._output.getvalue(), r'Overall:')
+
+    def test_count_fails(self):
+        """Test the count of fails with three tests
+
+        For first phase of interest: First test FAILs, second PASSes,
+        third FAILs; count should be 2, and this phase should not appear
+        individually for each test.
+
+        For second phase of interest: First test PASSes, second PASSes,
+        third FAILs; count should be 1, and this phase should not appear
+        individually for each test.
+        """
+        # Note that this test does NOT cover:
+        # - combining count_fails_phase_list with fails_only: currently,
+        #   this wouldn't cover any additional code/logic
+        # - ensuring that PENDs are also counted: currently, this
+        #   wouldn't cover any additional code/logic
+        phase_of_interest1 = self._NON_CORE_PHASE
+        phase_of_interest2 = self._NON_CORE_PHASE2
+        statuses1 = [test_status.TEST_FAIL_STATUS,
+                     test_status.TEST_PASS_STATUS,
+                     test_status.TEST_FAIL_STATUS]
+        statuses2 = [test_status.TEST_PASS_STATUS,
+                     test_status.TEST_PASS_STATUS,
+                     test_status.TEST_FAIL_STATUS]
+        test_paths = []
+        test_names = []
+        for testnum in range(3):
+            test_name = 'my.test.name' + str(testnum)
+            test_names.append(test_name)
+            test_dir = test_name + '.testid'
+            test_dir_path = self.create_test_dir(test_dir)
+            self.create_test_status_core_passes(test_dir_path, test_name)
+            self.set_phase_to_status(test_dir_path, test_name,
+                                     phase=phase_of_interest1,
+                                     status=statuses1[testnum])
+            self.set_phase_to_status(test_dir_path, test_name,
+                                     phase=phase_of_interest2,
+                                     status=statuses2[testnum])
+            test_paths.append(os.path.join(test_dir_path, 'TestStatus'))
+
+        cs_status(test_paths,
+                  count_fails_phase_list=[phase_of_interest1, phase_of_interest2],
+                  out=self._output)
+
+        for testnum in range(3):
+            self.assert_phase_absent(output=self._output.getvalue(),
+                                     phase=phase_of_interest1,
+                                     test_name=test_names[testnum])
+            self.assert_phase_absent(output=self._output.getvalue(),
+                                     phase=phase_of_interest2,
+                                     test_name=test_names[testnum])
+        count_regex1 = r'{} +non-passes: +2'.format(re.escape(phase_of_interest1))
+        six.assertRegex(self, self._output.getvalue(), count_regex1)
+        count_regex2 = r'{} +non-passes: +1'.format(re.escape(phase_of_interest2))
+        six.assertRegex(self, self._output.getvalue(), count_regex2)
+
+if __name__ == '__main__':
+    unittest.main()