Repository: ESMCI/cime

Issue:
RMS failures in tests should be reported *concisely* in TestStatus.log
In cime4, RMS diffs were reported in TestStatus.log. That functionality should be brought back.



Discussion:
Never mind, this functionality is still there.

I am reopening this with a different name: Apparently the current behavior is to echo the entire cprnc output file to TestStatus.log. I feel it would be more useful to go back to the old behavior where just the RMS lines were echoed to the TestStatus.log file, to avoid overwhelming that file and making it hard to skim through.

Ideally it would also have the last few lines of the cprnc output, which give the final summary of results.

The point of this functionality, as I see it, is for tests where you just expect a few fields to differ - so for this main use case, it's better to have just a concise summary for the sake of keeping TestStatus.log human-readable.

Tagging @mvertens @ekluzek and @fischer-ncar 

In my testing with cime5.1.1 I don't see any part of the cprnc output echoed to the TestStatus.log file. And although it reports on what files it's comparing it doesn't seem to report if a comparison was an unexpected difference.

@ekluzek can you point me to a case where you expect something in TestStatus.log but it's not there.

For example this test on hobart...

/fs/cgd/data0/erik/clm_esmciport/cime/scripts/ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.C.20160921_014124

TestStatus shows...

FAIL ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest COMPARE_baseline

In TestStatus it reports on files it's comparing as in...

  comparing model 'clm'
    /scratch/cluster/erik/ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.C.20160921_014124/run/ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.C.20160921_014124.clm2.h0.0001-01-01-00000.nc.base matched /scratch/cluster/erik/ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.C.20160921_014124/run/ERS_D_Ld5.f10_f10.ICLM45ED.hobart_nag.clm-edTest.C.20160921_014124.clm2.h0.0001-01-01-00000.nc.rest
  comparing model 'sice'
    no hist files found for model sice
  comparing model 'socn'
    no hist files found for model socn

But, it doesn't report if the comparison was successful or not. I'd like to have it report the result of the comparison.

OK, I thought I wasn't seeing the cprnc.out results, but after a careful look through all my cases, the only reason for my comparison fail's is because of missing baseline files. So I take back my report that the cprnc output isn't there. It looks like it's not there when the comparison is exact. I would still like it to report on the results of the comparison it does do, including saying that a comparison was identical. But, I don't have an example of one with a difference where it didn't include the cprnc output. Sorry about that.

OK, I now have a test with different answers, and it has the entire cprnc.out difference in the TestStatus.log as advertised. This helped me to easily find the difference. I do still agree with Bill, that I'd rather have a summary of the difference, showing the RMS differences and the tail summary of cprnc. And I'd like a confirmation that files were compared and are identical when that happens as well.


Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/pre.py b/scripts/lib/CIME/SystemTests/pre.py
--- a/scripts/lib/CIME/SystemTests/pre.py
+++ b/scripts/lib/CIME/SystemTests/pre.py
@@ -60,7 +60,7 @@ def _case_two_setup(self):
         self._case.set_value("PAUSE_N", pausen)
         comps = [ x.lower() for x in self._case.get_values("COMP_CLASSES") ]
         pcl = self._case.get_value("PAUSE_COMPONENT_LIST")
-        expect(pcl == "all" or set(pcl.split(':')).issubset(comps), 
+        expect(pcl == "all" or set(pcl.split(':')).issubset(comps),
                "PRE ERROR: Invalid PAUSE_COMPONENT_LIST, '%s'"%pcl)
 
         self._case.flush()
@@ -101,7 +101,7 @@ def run_phase(self):
             expect((len(restart_files_2) == 1),
                    "Missing case2 restart file, %s", glob_str)
             rfile2 = restart_files_2[0]
-            ok, out = cprnc(comp, rfile1, rfile2, self._case, rundir2) # pylint: disable=unused-variable
+            ok = cprnc(comp, rfile1, rfile2, self._case, rundir2)[0]
             logger.warning("CPRNC result for %s: %s"%(os.path.basename(rfile1), "PASS" if (ok == should_match) else "FAIL"))
             compare_ok = compare_ok and (should_match == ok)
 
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -178,6 +178,7 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2="", outfile_s
         expect(suffix1 != suffix2, "Comparing files to themselves?")
 
     testcase = case.get_value("CASE")
+    casedir = case.get_value("CASEROOT")
     all_success = True
     num_compared = 0
     comments = "Comparing hists for case '%s' dir1='%s', suffix1='%s',  dir2='%s' suffix2='%s'\n" % \
@@ -204,20 +205,23 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2="", outfile_s
         num_compared += len(match_ups)
 
         for hist1, hist2 in match_ups:
-            success, cprnc_comments = cprnc(model, hist1, hist2, case, from_dir1,
+            success, cprnc_log_file = cprnc(model, hist1, hist2, case, from_dir1,
                                             multiinst_cpl_compare = multiinst_cpl_compare,
                                             outfile_suffix = outfile_suffix)
             if success:
                 comments += "    %s matched %s\n" % (hist1, hist2)
             else:
                 comments += "    %s did NOT match %s\n" % (hist1, hist2)
-                comments += cprnc_comments + "\n"
+                comments += "    cat " + cprnc_log_file + "\n"
+                shutil.copy(cprnc_log_file, casedir)
                 all_success = False
 
     if num_compared == 0:
         all_success = False
         comments += "Did not compare any hist files! Missing baselines?\n"
 
+    comments += "PASS" if all_success else "FAIL"
+
     return all_success, comments
 
 def compare_test(case, suffix1, suffix2):
@@ -245,7 +249,7 @@ def cprnc(model, file1, file2, case, rundir, multiinst_cpl_compare=False, outfil
     outfile_suffix - if non-blank, then the output file name ends with this
         suffix (with a '.' added before the given suffix)
 
-    returns True if the files matched
+    returns (True if the files matched, log_name)
     """
     cprnc_exe = case.get_value("CCSM_CPRNC")
     basename = os.path.basename(file1)
@@ -276,9 +280,9 @@ def cprnc(model, file1, file2, case, rundir, multiinst_cpl_compare=False, outfil
         # dimensions and so cprnc will indicate that the files seem to be DIFFERENT
         # in this case we only want to check that the fields we are able to compare
         # have no differences.
-        return (cpr_stat == 0 and " 0 had non-zero differences" in out, out)
+        return (cpr_stat == 0 and " 0 had non-zero differences" in out, output_filename)
     else:
-        return (cpr_stat == 0 and "files seem to be IDENTICAL" in out, out)
+        return (cpr_stat == 0 and "files seem to be IDENTICAL" in out, output_filename)
 
 def compare_baseline(case, baseline_dir=None, outfile_suffix=""):
     """


Test patch:
diff --git a/scripts/lib/CIME/wait_for_tests.py b/scripts/lib/CIME/wait_for_tests.py
--- a/scripts/lib/CIME/wait_for_tests.py
+++ b/scripts/lib/CIME/wait_for_tests.py
@@ -152,14 +152,17 @@ def create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_ti
             if (test_status not in [TEST_PASS_STATUS, NAMELIST_FAIL_STATUS]):
                 ts = TestStatus(os.path.dirname(test_path))
 
-                build_status = ts.get_status(MODEL_BUILD_PHASE)
-                run_status   = ts.get_status(RUN_PHASE)
-                if ( build_status == TEST_FAIL_STATUS or run_status == TEST_FAIL_STATUS ):
+                build_status    = ts.get_status(MODEL_BUILD_PHASE)
+                run_status      = ts.get_status(RUN_PHASE)
+                baseline_status = ts.get_status(BASELINE_PHASE)
+                if ( build_status == TEST_FAIL_STATUS or run_status == TEST_FAIL_STATUS or baseline_status == TEST_FAIL_STATUS):
                     param = "EXEROOT" if build_status == TEST_FAIL_STATUS else "RUNDIR"
-                    src_dir = CIME.utils.run_cmd_no_fail("./xmlquery %s -value" % param, from_dir=os.path.dirname(test_path))
+                    log_src_dir = CIME.utils.run_cmd_no_fail("./xmlquery %s -value" % param, from_dir=os.path.dirname(test_path))
                     log_dst_dir = os.path.join(log_dir, "%s_%s_logs" % (test_name, param))
                     os.makedirs(log_dst_dir)
-                    for log_file in glob.glob(os.path.join(src_dir, "*log*")):
+                    for log_file in glob.glob(os.path.join(log_src_dir, "*log*")):
+                        shutil.copy(log_file, log_dst_dir)
+                    for log_file in glob.glob(os.path.join(log_src_dir, "*.cprnc.out*")):
                         shutil.copy(log_file, log_dst_dir)
 
                     need_to_upload = True