Repository: ESMCI/cime

Issue:
If history files are missing from the baseline directory, BFAIL should be printed
(This was tested with cime5.2.0-alpha.7, but as far as I can tell, this behavior hasn't been changed since then.)

If I run a test with baseline comparisons, and the baseline directory is present but contains no history files, this should be reported as a BFAIL via the test status comment. Instead, no comment is given on the BASELINE line of TestStatus.

A BFAIL is correctly reported if the baseline directory is missing entirely, but in the case where the baseline directory is present but has no history files, I see, in TestStatus:

```
FAIL ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default BASELINE
```

despite having this in TestStatus.log:

```
2016-11-08 22:33:21: Comparing hists for case 'ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p' dir1='/glade/scratch/sacks/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p/run', suffix1='',  dir2='/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r201/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default' suffix2=''
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    File '/glade/scratch/sacks/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p/run/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p.clm2.h0.0003-01-09-00000.nc' had no counterpart in '/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r201/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default' with suffix ''
    File '/glade/scratch/sacks/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p/run/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p.clm2.h1.0003-01-09-00000.nc' had no counterpart in '/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r201/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default' with suffix ''
  comparing model 'sice'
    no hist files found for model sice
  comparing model 'socn'
    no hist files found for model socn
  comparing model 'rtm'
    no hist files found for model rtm
  comparing model 'sglc'
    no hist files found for model sglc
  comparing model 'swav'
    no hist files found for model swav
  comparing model 'cpl'
    File '/glade/scratch/sacks/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p/run/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default.GC.1108-1358.40.p.cpl.hi.0003-01-09-00000.nc' had no counterpart in '/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r201/ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default' with suffix ''
Did not compare any hist files! Missing baselines?
```

What I want is something in TestStatus like what's printed if the baseline directory is missing entirely:

```
FAIL ERP_Ly5_Mmpi-serial.1x1_numaIA.ICRUCLM50BGCCROP.hobart_nag.clm-monthly BASELINE ERROR BFAIL baseline directory '/fs/cgd/csm/ccsm_baselines/clm4_5_12_r201/ERP_Ly5_Mmpi-serial.1x1_numaIA.ICRUCLM50BGCCROP.hobart_nag.clm-monthly' does not exist
```

but now it should say something like:

```
FAIL ERI_D_Ld9.f19_g16.I1850CN.yellowstone_pgi.clm-40default BASELINE ERROR BFAIL Did not compare any hist files! Missing baselines?
```

However, I'd say that this BFAIL should _only_ appear if _no_ history files were found in the baseline directory. Particularly important is to _not_ have a BFAIL if some history files differed and others were missing - so that a `grep -v BFAIL` wouldn't filter out those results.

My reason for wanting this is that these missing baselines generally indicate a very different problem from a standard BASELINE failure: namely, something went wrong in the previous creation of the baselines.


Discussion:
This is still an issue in the latest cime.

Tested with:

```
[yslogin1:/glade/scratch/sacks/cesm_baselines/cesm2_0_alpha06i_test3]$ cp -R /glade/p/cesmdata/cseg/ccsm_baselines/cesm2_0_alpha06i/ERS_Ld3.f19_g16.X.yellowstone_intel .
[yslogin1:/glade/scratch/sacks/cesm_baselines/cesm2_0_alpha06i_test3/ERS_Ld3.f19_g16.X.yellowstone_intel]$ rm cpl.hi.0001-01-04-00000.nc

[yslogin1:/glade/p/work/sacks/cime_for_comparison/scripts]$ submit_test_short ./create_test --test-root . --walltime 0:30 --compare cesm2_0_alpha06i_test3 --baseline-root $scratch/cesm_baselines ERS_Ld3.f19_g16.X.yellowstone_intel
```
This is still an issue. It bit me tonight, as I spent time trying to figure out why a test was giving different results only to eventually realize that the baselines were missing.

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -10,6 +10,9 @@
 logger = logging.getLogger(__name__)
 
 BLESS_LOG_NAME = "bless_log"
+NO_COMPARE     = "had no compare counterpart"
+NO_ORIGINAL    = "had no original counterpart"
+DIFF_COMMENT   = "did NOT match"
 
 def _iter_model_file_substrs(case):
     models = case.get_compset_components()
@@ -211,10 +214,11 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2="", outfile_s
 
         one_not_two, two_not_one, match_ups = _hists_match(model, hists1, hists2, suffix1, suffix2)
         for item in one_not_two:
-            comments += "    File '{}' had no counterpart in '{}' with suffix '{}'\n".format(item, from_dir2, suffix2)
+            comments += "    File '{}' {} in '{}' with suffix '{}'\n".format(item, NO_COMPARE, from_dir2, suffix2)
             all_success = False
+
         for item in two_not_one:
-            comments += "    File '{}' had no counterpart in '{}' with suffix '{}'\n".format(item, from_dir1, suffix1)
+            comments += "    File '{}' {} in '{}' with suffix '{}'\n".format(item, NO_ORIGINAL, from_dir1, suffix1)
             all_success = False
 
         num_compared += len(match_ups)
@@ -226,7 +230,7 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2="", outfile_s
             if success:
                 comments += "    {} matched {}\n".format(hist1, hist2)
             else:
-                comments += "    {} did NOT match {}\n".format(hist1, hist2)
+                comments += "    {} {} {}\n".format(hist1, DIFF_COMMENT, hist2)
                 comments += "    cat " + cprnc_log_file + "\n"
                 expected_log_file = os.path.join(casedir, os.path.basename(cprnc_log_file))
                 if not (os.path.exists(expected_log_file) and filecmp.cmp(cprnc_log_file, expected_log_file)):
@@ -455,3 +459,42 @@ def generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False):
                                                get_timestamp(timestamp_format="%Y-%m-%d_%H:%M:%S")))
 
     return True, comments
+
+def get_ts_synopsis(comments):
+    r"""
+    Reduce case diff comments down to a single line synopsis so that we can put
+    something in the TestStatus file. It's expected that the comments provided
+    to this function came from compare_baseline, not compare_tests.
+
+    >>> get_ts_synopsis('')
+    ''
+    >>> get_ts_synopsis('big error')
+    'big error'
+    >>> get_ts_synopsis('big error\n')
+    'big error'
+    >>> get_ts_synopsis('stuff\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
+    'ERROR BFAIL some baseline files were missing'
+    >>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\nPass\n')
+    'DIFF'
+    >>> get_ts_synopsis('stuff\n    File foo did NOT match bar with suffix baz\n    File foo had no compare counterpart in bar with suffix baz\nPass\n')
+    'DIFF'
+    """
+    if not comments:
+        return ""
+    elif "\n" not in comments.strip():
+        return comments.strip()
+    else:
+        has_bfails = False
+        has_real_fails = False
+        for line in comments.splitlines():
+            if NO_COMPARE in line:
+                has_bfails = True
+            elif DIFF_COMMENT in line:
+                has_real_fails = True
+
+        if has_real_fails:
+            return "DIFF"
+        elif has_bfails:
+            return "ERROR {} some baseline files were missing".format(TEST_NO_BASELINES_COMMENT)
+        else:
+            return ""


Test patch:
diff --git a/scripts/lib/CIME/SystemTests/system_tests_common.py b/scripts/lib/CIME/SystemTests/system_tests_common.py
--- a/scripts/lib/CIME/SystemTests/system_tests_common.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_common.py
@@ -482,7 +482,7 @@ def _compare_baseline(self):
             append_testlog(comments)
             status = TEST_PASS_STATUS if success else TEST_FAIL_STATUS
             baseline_name = self._case.get_value("BASECMP_CASE")
-            ts_comments = (os.path.dirname(baseline_name) + ": " + comments) if "\n" not in comments else os.path.dirname(baseline_name)
+            ts_comments = os.path.dirname(baseline_name) + ": " + get_ts_synopsis(comments)
             self._test_status.set_status(BASELINE_PHASE, status, comments=ts_comments)
 
     def _generate_baseline(self):
diff --git a/scripts/lib/CIME/compare_test_results.py b/scripts/lib/CIME/compare_test_results.py
--- a/scripts/lib/CIME/compare_test_results.py
+++ b/scripts/lib/CIME/compare_test_results.py
@@ -1,7 +1,7 @@
 import CIME.compare_namelists, CIME.simple_compare
 from CIME.utils import expect, append_status, EnvironmentContext
 from CIME.test_status import *
-from CIME.hist_utils import compare_baseline
+from CIME.hist_utils import compare_baseline, get_ts_synopsis
 from CIME.case import Case
 
 import os, glob, logging
@@ -142,11 +142,7 @@ def compare_test_results(baseline_name, baseline_root, test_root, compiler, test
                             compare_result = TEST_FAIL_STATUS
                             all_pass_or_skip = False
 
-                        # Following the logic in SystemTestsCommon._compare_baseline:
-                        # We'll print the comment if it's a brief one-liner; otherwise
-                        # the comment will only appear in the log file
-                        if "\n" not in detailed_comments:
-                            compare_comment = detailed_comments
+                        compare_comment = get_ts_synopsis(detailed_comments)
 
             brief_result = ""
             if not hist_only: