Repository: ESMCI/cime

Issue:
TestStatus: Still would like clearer distinction between test comparisons and baseline comparisons
Inspired by @gold2718 's comment here:

https://github.com/ESMCI/cime/pull/392#issuecomment-239626725
- where he suggests knowing ahead of time which test phases are expected: I realized that it would still be good to have a clearer distinction made between history file comparisons done as an integral part of a test (e.g., comparing .base with .rest files for an ERS test) vs. baseline comparisons. Even though these are both about comparing history files, they are fundamentally different in their meaning. 

cime4 test reporting made a clear distinction between these, as:

```
--- Test Functionality  ---:
PASS ERS_D_Ld10.T31_g37_gl5.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.clm2.h0.nc : test compare clm2.h0 (.base and .rest files)
<snip>
--- Baseline Comparison ---:
PASS ERS_D_Ld10.T31_g37_gl5.IGHISTCLM45.yellowstone_intel.clm-glcMEC_decrease.cism.h.nc : baseline compare cism.h (baseline: compare .base file with clm4_5_9_r185 file)
```

and I feel it would be good to maintain that clear distinction in cime5.

@jgfouca made some steps towards this, by adding suffixes to the single 'COMPARE' phase. But it seems like it would be good to explicitly split this into two phases - a TEST_COMPARE phase and a BASELINE_COMPARE phase - both in the code and in the test reporting. In the test reporting, this will make it easier to do greps to include / exclude particular aspects of the tests. In the code, this will make it easier to add new functionality, such as the functionality proposed by @gold2718 .



Discussion:
@billsacks the handling of COMPARE phases in CIME is already awkward and having another phase like COMPARE is not something I'm excited about. If you can describe a specific need/capability that would require this change, would you add a comment hear describing it? I'm going to tentatively close this for now. Feel free to reopen if you think we should reconsider.

We do need to distinguish between "compare with baseline fail" and "compare within a test fail".  The latter is much more serious.  In fact, the test should just stop and not bother with baseline compare if the primary test compare failed.

@rljacob I have a different PR going in soon that will mark a test with an "internal" diff (non-baseline) as FAIL instead of DIFF.

I agree with @rljacob here: These phases need to be more clearly distinguished. I'm going to reopen this issue. Yes, they both do "comparisons", but they are doing fundamentally different things, and I feel it's important to clearly distinguish between them in both the code and the test reporting, rather than treating them as flavors of the same thing. Distinguishing between these is just as important as distinguishing between (say) a RUN failure and a BUILD failure.

Maybe I'm missing something, though: I don't understand what difficulty it would cause to have two clearly-distinguished phases, something like BASELINE_COMPARE and TEST_COMPARE.

If you have too much on your plate and don't feel this is at a high level of priority, I can understand that. But I would like to keep this issue open until it is resolved.

@billsacks I just realized that a test has at most one baseline compare phase. That makes this easier than I thought. 


Resolving patch:
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -48,7 +48,6 @@ def _get_latest_hist_files(testcase, model, from_dir, suffix=""):
         histlist.append(latest_files[key])
     return histlist
 
-
 def copy(case, suffix):
     """Copy the most recent batch of hist files in a case, adding the given suffix.
 


Test patch:
diff --git a/scripts/Testing/testreporter.pl b/scripts/Testing/testreporter.pl
--- a/scripts/Testing/testreporter.pl
+++ b/scripts/Testing/testreporter.pl
@@ -305,7 +305,7 @@ sub getTestStatus
         $teststatushash{$testcase}{'comment'} .= " @testsummary_arr[3..$#testsummary_arr]";
 
         #Compare to baseline
-        my @testsummarylines = grep { /COMPARE_baseline/} @statuslines;
+        my @testsummarylines = grep { /BASELINE/} @statuslines;
         my @testsummary_arr = (split(/\s+/, $testsummarylines[0]));
         $teststatushash{$testcase}{'compare'} = ($#testsummarylines > -1) ? @testsummary_arr[0] : "----";
         $teststatushash{$testcase}{'comment'} .= " @testsummary_arr[3..$#testsummary_arr]";
diff --git a/utils/python/CIME/SystemTests/system_tests_common.py b/utils/python/CIME/SystemTests/system_tests_common.py
--- a/utils/python/CIME/SystemTests/system_tests_common.py
+++ b/utils/python/CIME/SystemTests/system_tests_common.py
@@ -339,7 +339,7 @@ def _compare_baseline(self):
             append_status(comments, sfile="TestStatus.log")
             status = TEST_PASS_STATUS if success else TEST_FAIL_STATUS
             ts_comments = comments if "\n" not in comments else None
-            self._test_status.set_status("%s_baseline" % COMPARE_PHASE, status, comments=ts_comments)
+            self._test_status.set_status(BASELINE_PHASE, status, comments=ts_comments)
             basecmp_dir = os.path.join(self._case.get_value("BASELINE_ROOT"), self._case.get_value("BASECMP_CASE"))
 
             # compare memory usage to baseline
diff --git a/utils/python/CIME/test_status.py b/utils/python/CIME/test_status.py
--- a/utils/python/CIME/test_status.py
+++ b/utils/python/CIME/test_status.py
@@ -16,7 +16,7 @@
 ALL_PHASE_STATUSES = [TEST_PEND_STATUS, TEST_PASS_STATUS, TEST_FAIL_STATUS]
 
 # Special statuses that the overall test can be in
-TEST_DIFF_STATUS     = "DIFF"   # Implies a failure in one of the COMPARE phases
+TEST_DIFF_STATUS     = "DIFF"   # Implies a failure in the BASELINE phase
 NAMELIST_FAIL_STATUS = "NLFAIL" # Implies a failure in the NLCOMP phase
 
 # Special strings that can appear in comments, indicating particular types of failures
@@ -33,7 +33,8 @@
 THROUGHPUT_PHASE      = "TPUTCOMP"
 MEMCOMP_PHASE         = "MEMCOMP"
 MEMLEAK_PHASE         = "MEMLEAK"
-COMPARE_PHASE         = "COMPARE" # This is one special, real phase will be COMPARE_$WHAT
+COMPARE_PHASE         = "COMPARE" # This is one special, real phase will be COMPARE_$WHAT, this is for internal test comparisons, there could be multiple variations of this phase in one test
+BASELINE_PHASE        = "BASELINE"
 GENERATE_PHASE        = "GENERATE"
 
 ALL_PHASES = [CREATE_NEWCASE_PHASE,
@@ -44,6 +45,7 @@
               MODEL_BUILD_PHASE,
               RUN_PHASE,
               COMPARE_PHASE,
+              BASELINE_PHASE,
               THROUGHPUT_PHASE,
               MEMCOMP_PHASE,
               MEMLEAK_PHASE,
@@ -123,17 +125,18 @@ def set_status(self, phase, status, comments=""):
         ...     ts.set_status(XML_PHASE, "PASS")
         ...     ts.set_status(SETUP_PHASE, "FAIL")
         ...     ts.set_status(SETUP_PHASE, "PASS")
-        ...     ts.set_status("%s_baseline" % COMPARE_PHASE, "FAIL")
+        ...     ts.set_status("%s_base_rest" % COMPARE_PHASE, "FAIL")
         ...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
         >>> ts._phase_statuses
-        OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42')), ('COMPARE_baseline', ('FAIL', '')), ('MODEL_BUILD', ('PEND', ''))])
+        OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42')), ('COMPARE_base_rest', ('FAIL', '')), ('MODEL_BUILD', ('PEND', ''))])
 
         >>> with TestStatus(test_dir="/", test_name="ERS.foo.A", no_io=True) as ts:
         ...     ts.set_status(CREATE_NEWCASE_PHASE, "PASS")
         ...     ts.set_status(XML_PHASE, "PASS")
         ...     ts.set_status(SETUP_PHASE, "FAIL")
         ...     ts.set_status(SETUP_PHASE, "PASS")
-        ...     ts.set_status("%s_baseline" % COMPARE_PHASE, "FAIL")
+        ...     ts.set_status(BASELINE_PHASE, "PASS")
+        ...     ts.set_status("%s_base_rest" % COMPARE_PHASE, "FAIL")
         ...     ts.set_status(SHAREDLIB_BUILD_PHASE, "PASS", comments='Time=42')
         ...     ts.set_status(SETUP_PHASE, "PASS")
         >>> ts._phase_statuses
@@ -206,11 +209,11 @@ def _parse_test_status(self, file_contents):
         ... PASS ERS.foo.A CREATE_NEWCASE
         ... PASS ERS.foo.A XML
         ... FAIL ERS.foo.A SETUP
-        ... PASS ERS.foo.A COMPARE_baseline
+        ... PASS ERS.foo.A COMPARE_base_rest
         ... PASS ERS.foo.A SHAREDLIB_BUILD Time=42
         ... '''
         >>> _test_helper1(contents)
-        OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('FAIL', '')), ('COMPARE_baseline', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42'))])
+        OrderedDict([('CREATE_NEWCASE', ('PASS', '')), ('XML', ('PASS', '')), ('SETUP', ('FAIL', '')), ('COMPARE_base_rest', ('PASS', '')), ('SHAREDLIB_BUILD', ('PASS', 'Time=42'))])
         """
         for line in file_contents.splitlines():
             line = line.strip()
@@ -266,9 +269,9 @@ def get_overall_test_status(self, wait_for_run=False, check_throughput=False, ch
         'PASS'
         >>> _test_helper2('PASS ERS.foo.A COMPARE_1\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
         'FAIL'
-        >>> _test_helper2('FAIL ERS.foo.A COMPARE_baseline\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
+        >>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nPASS ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
         'DIFF'
-        >>> _test_helper2('FAIL ERS.foo.A COMPARE_baseline\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
+        >>> _test_helper2('FAIL ERS.foo.A BASELINE\nFAIL ERS.foo.A NLCOMP\nFAIL ERS.foo.A COMPARE_2\nPASS ERS.foo.A RUN')
         'FAIL'
         >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD')
         'PASS'
@@ -304,7 +307,7 @@ def get_overall_test_status(self, wait_for_run=False, check_throughput=False, ch
                     if (rv == TEST_PASS_STATUS):
                         rv = NAMELIST_FAIL_STATUS
 
-                elif (rv in [NAMELIST_FAIL_STATUS, TEST_PASS_STATUS] and phase == "COMPARE_baseline"):
+                elif (rv in [NAMELIST_FAIL_STATUS, TEST_PASS_STATUS] and phase == BASELINE_PHASE):
                     rv = TEST_DIFF_STATUS
 
                 else: