Repository: ESMCI/cime

Issue:
Report baseline fails differently if just because baseline directory DNE (both Overall and detailed)
In cime4 this was called a "BFAIL" if the baseline directory didn't exist. I don't know that's the best name, but we need a way to easily figure out that the baseline comparison failed -- just because it didn't have a baseline directory to compare to. I'd like to see this in the Overall test status, as well as the detailed test status.

@jedwards4b @mvertens @billsacks @bandre-ucar 



Discussion:
The TestStatus file has something like this:

```
FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default COMPARE_baseline ERROR baseline directory '/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r194.cime5/SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default' does not exist
```

but this 'does not exist' information doesn't get transferred to the cs.status output

This can be addressed as part of #564 

Instead of BFAIL we could call it N/A.

> Instead of BFAIL we could call it N/A.

I don't really like N/A because it implies (to me anyway) that there's no problem. To me, a BFAIL is generally a problem, so I like having 'FAIL' in its name.

Personally I like BFAIL, although I also can live with the current 'FAIL... does not exist'. The big point here is that, whatever we choose, it should show up in cs.status.

So you would be okay with cs.status reporting
FAIL SMS_Ln9.f09_f09.F2000_DONOTUSE.yellowstone_intel.cam-outfrq9s COMPARE_baseline ERROR baseline directory '/glade/p/cesmdata/cseg/ccsm_baselines/cesm2_0_alpha01g/SMS_Ln9.f09_f09.F2000_DONOTUSE.yellowstone_intel.cam-outfrq9s' does not exist

Personally, yes; I can't speak for @ekluzek and others, though.

I assume that all of the text you pasted there would appear on a single line, as it does in TestStatus? (That's what I would want.)

The main desire is to have a simple short string that you can grep on, so you can purposely either remove those lines or consciously look for them. I agree with Bill that N/A isn't descriptive enough. I'm used to BFAIL, even though it's not the best name either. I guess I can live with grepping on "does not exist", but I like having a short name in caps, so I know it's not going to change. How about

FAIL_BDNE

for FAIL Baselines (directory or individual files) Do Not Exist

Also I'd like to add that you can also have a fail because a file is missing rather than the whole directory. For either case we'd like it to report the issue.

I pushed back on @billsacks so I'll push back here too. I'm very reluctant to start adding new states to the TestStatus file in order to help keep the tools simple. The state of the test is a very important and widely used concept in CIME, so changes could have wide impact.

I see both points here. Would an acceptable compromise be to keep the status as FAIL but change the comment a bit to include some all-caps, very distinct string? i.e., somehow making it clear that this string is an important supplemental test status that should not be changed lightly, and not just any old comment. It could help to make this string a named constant in test_status.py, similar to these:

```
TEST_PENDING_STATUS  = "PEND"
TEST_PASS_STATUS     = "PASS"
TEST_FAIL_STATUS     = "FAIL"
```

e.g., add

```
TEST_NO_BASELINES_COMMENT = "FAIL_BDNE"
```

Again, that code would just appear in the comments, not replacing the existing FAIL status.

That could work.

I'm perfectly fine will Bill's idea. But, what about just changing the output from cs_status to BFAIL, and leave the TestStatus file alone?  

That would be fine too.

I'd pretty strongly prefer to have the cs.status output match the TestStatus output as closely as possible. I find it confusing when they differ, so that I need to use different tools / techniques to parse each of them.

@ekluzek and @fischer-ncar : @fischer-ncar  talked to me a bit ago about what text should appear when baseline directories do not exist. After waffling a bit on this, I recommended the verbose:

BASELINES_DO_NOT_EXIST

I like that this is clear, explicit, and unlikely to appear anywhere else. The problem with this is that it might be less easily memorable / more prone to typos than something like BFAIL.

This leads me to suggest one further enhancement: A --help (or -h) option to cs.status that prints out all possible codes that might appear - both in terms of the various test statuses (PASS, FAIL, PEND), and this BASELINES_DO_NOT_EXIST code. (It should use the constants defined in test_status.py for this purpose.)

I should also say that I'm not tied to

BASELINES_DO_NOT_EXIST

if others prefer a shorter name.

Okay, I've already been making typos with BASELINES_DO_NOT_EXIST, so I suggest we go with BFAIL.  So the status line will look something like this.  All on one line of course, having troubles with Linux Firefox wrapping lines.

FAIL SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default COMPARE_baseline ERROR: BFAIL, baseline directory '/glade/p/cesmdata/cseg/ccsm_baselines/clm4_5_12_r194.cime /SSP_Ld10.f19_g16.I1850CLM45BGC.yellowstone_intel.clm-default' does not exist

This also has the advantage that it explains what BFAIL is.

I'm happy with that.


Resolving patch:
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -3,6 +3,7 @@
 """
 
 from CIME.XML.standard_module_setup import *
+from CIME.test_status import TEST_NO_BASELINES_COMMENT
 
 import logging, glob, os, shutil, re
 logger = logging.getLogger(__name__)
@@ -277,7 +278,7 @@ def compare_baseline(case, baseline_dir=None):
 
     for bdir in dirs_to_check:
         if not os.path.isdir(bdir):
-            return False, "ERROR baseline directory '%s' does not exist" % bdir
+            return False, "ERROR %s baseline directory '%s' does not exist" % TEST_NO_BASELINES_COMMENT,bdir
 
     return _compare_hists(case, rundir, basecmp_dir)
 


Test patch:
diff --git a/utils/python/CIME/test_status.py b/utils/python/CIME/test_status.py
--- a/utils/python/CIME/test_status.py
+++ b/utils/python/CIME/test_status.py
@@ -19,6 +19,9 @@
 TEST_DIFF_STATUS     = "DIFF"   # Implies a failure in one of the COMPARE phases
 NAMELIST_FAIL_STATUS = "NLFAIL" # Implies a failure in the NLCOMP phase
 
+# Special strings that can appear in comments, indicating particular types of failures
+TEST_NO_BASELINES_COMMENT = "BFAIL" # Implies baseline directory is missing in the baseline comparison phase
+
 # The valid phases
 INITIAL_PHASE         = "INIT"
 CREATE_NEWCASE_PHASE  = "CREATE_NEWCASE"
@@ -121,24 +124,22 @@ def get_status(self, phase):
     def get_comment(self, phase):
         return self._phase_statuses[phase][1] if phase in self._phase_statuses else None
 
-    def phase_statuses_dump(self):
+    def phase_statuses_dump(self, fd):
+        """
+        Args:
+            fd: file open for writing
+        """
         if self._phase_statuses:
             for phase, data in self._phase_statuses.iteritems():
                 status, comments = data
                 if not comments:
-                    logging.info("%s %s %s" % (status, self._test_name, phase))
+                    fd.write("%s %s %s\n" % (status, self._test_name, phase))
                 else:
-                    logging.info("%s %s %s %s" % (status, self._test_name, phase, comments))
+                    fd.write("%s %s %s %s\n" % (status, self._test_name, phase, comments))
 
     def flush(self):
-        if self._phase_statuses:
-            with open(self._filename, "w") as fd:
-                for phase, data in self._phase_statuses.iteritems():
-                    status, comments = data
-                    if not comments:
-                        fd.write("%s %s %s\n" % (status, self._test_name, phase))
-                    else:
-                        fd.write("%s %s %s %s\n" % (status, self._test_name, phase, comments))
+        with open(self._filename, "w") as fd:
+            self.phase_statuses_dump(fd)
 
     def _parse_test_status(self, file_contents):
         """