Repository: ESMCI/cime

Issue:
Copy TestStatus file to the baseline directory
I often find myself wanting to know if a given test passed or failed (entirely or some specific phase) in a previous run of the test suite, for which we have baselines. For example, this would be a quick way of seeing if a test failed the last time around, or it can help indicate if / why there might be problems with the baselines for a given test.

So it would be helpful if the TestStatus file were copied to the baseline directory when running a test suite with the `--generate` flag.

There are a few things I'm not sure of in terms of how this should be implemented:

1. If we copy the TestStatus file in the generate phase (which seems like a logical time to do it), it would be missing some information, since generate comes before some other phases. Would it be acceptable to move the generate phase to be the very last phase, or would that have some negative repercussions?

2. Alternatively, we could copy the TestStatus file as a separate step (probably not even a named phase) at the very end of system_tests_common:run. Would that be reasonable?

3. Is there a way to ensure that this copying of the TestStatus file happens even if there is a failure in the run or some other phase? e.g., would putting this copying in a `finally` block at the end of system_tests_common:run ensure this?

I'm happy to make and test this change myself, but I'd like to get input from @jgfouca , @jedwards4b or others if you have any immediate thoughts. My inclination would be to have a `finally` block at the end of system_tests_common:run that does this copying, so I'd probably try that approach first unless you have other feelings.


Discussion:
I realized that if a test doesn't even make it to the run phase, my proposed solution won't work. I don't feel a need for something that _guarantees_ the TestStatus file appears, so I might still go with my proposal, but if others have thoughts on how to ensure that the TestStatus file appears in the baseline directory regardless of whether / when the test failed, I'd be happy to hear it.
(And I'm un-assigning myself for the moment, because if someone else has a good idea for how this could be done and would rather just do it than explain it, that's fine, too.)
I'd like to see this as well!
@billsacks , this sounds like an idea that makes more sense in the context of the CESM workflow. I'm fine with putting this is in, but we should probably disable it for E3SM.

You've accurately identified the reasons this will be a little bit tricky. I could put some post-processing into create_test to handle the cases where the test didn't make it to the RUN phase. I could put a bit of extra code into system_tests_common to copy the TestStatus file.
I don't think this would negatively impact our workflow so think its ok to include the TestStatus in our baselines.   I'd rather do that then add more if switches.

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -3,7 +3,7 @@
 """
 
 from CIME.XML.standard_module_setup import *
-from CIME.test_status import TEST_NO_BASELINES_COMMENT
+from CIME.test_status import TEST_NO_BASELINES_COMMENT, TEST_STATUS_FILENAME
 from CIME.utils import get_current_commit, get_timestamp, get_model, safe_copy
 
 import logging, os, re, stat, filecmp
@@ -387,6 +387,20 @@ def get_extension(model, filepath):
 
     return result
 
+def generate_teststatus(testdir, baseline_dir):
+    """
+    CESM stores it's TestStatus file in baselines. Do not let exceptions
+    escape from this function.
+    """
+    if get_model() == "cesm":
+        try:
+            if not os.path.isdir(baseline_dir):
+                os.makedirs(baseline_dir)
+
+            safe_copy(os.path.join(testdir, TEST_STATUS_FILENAME), baseline_dir)
+        except Exception as e:
+            logger.warning("Could not copy {} to baselines, {}".format(os.path.join(testdir, TEST_STATUS_FILENAME), str(e)))
+
 def generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False):
     """
     copy the current test output to baseline result


Test patch:
diff --git a/scripts/lib/CIME/SystemTests/system_tests_common.py b/scripts/lib/CIME/SystemTests/system_tests_common.py
--- a/scripts/lib/CIME/SystemTests/system_tests_common.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_common.py
@@ -177,6 +177,10 @@ def run(self, skip_pnl=False):
         if success and get_model() == "e3sm":
             save_test_time(self._case.get_value("BASELINE_ROOT"), self._casebaseid, time_taken)
 
+        if get_model() == "cesm" and self._case.get_value("GENERATE_BASELINE"):
+            baseline_dir = os.path.join(self._case.get_value("BASELINE_ROOT"), self._case.get_value("BASEGEN_CASE"))
+            generate_teststatus(self._caseroot, baseline_dir)
+
         # We return success if the run phase worked; memleaks, diffs will not be taken into account
         # with this return value.
         return success
diff --git a/scripts/lib/CIME/test_scheduler.py b/scripts/lib/CIME/test_scheduler.py
--- a/scripts/lib/CIME/test_scheduler.py
+++ b/scripts/lib/CIME/test_scheduler.py
@@ -10,12 +10,12 @@
 
 import traceback, stat, threading, time, glob
 from collections import OrderedDict
+
 from CIME.XML.standard_module_setup import *
-import CIME.compare_namelists
-import CIME.utils
 import six
 from get_tests import get_recommended_test_time
-from CIME.utils import append_status, append_testlog, TESTS_FAILED_ERR_CODE, parse_test_name, get_full_test_name, get_model, convert_to_seconds
+from CIME.utils import append_status, append_testlog, TESTS_FAILED_ERR_CODE, parse_test_name, get_full_test_name, get_model, \
+    convert_to_seconds, get_cime_root, get_project, get_timestamp, get_python_libs_root
 from CIME.test_status import *
 from CIME.XML.machines import Machines
 from CIME.XML.generic_xml import GenericXML
@@ -28,7 +28,7 @@
 from CIME.provenance import get_recommended_test_time_based_on_past
 from CIME.locked_files import lock_file
 from CIME.cs_status_creator import create_cs_status
-import CIME.test_utils
+from CIME.hist_utils import generate_teststatus
 
 logger = logging.getLogger(__name__)
 
@@ -122,7 +122,7 @@ def __init__(self, test_names, test_data=None,
                  force_procs=None, force_threads=None, mpilib=None,
                  input_dir=None, pesfile=None, mail_user=None, mail_type=None, allow_pnl=False, non_local=False):
     ###########################################################################
-        self._cime_root       = CIME.utils.get_cime_root()
+        self._cime_root       = get_cime_root()
         self._cime_model      = get_model()
         self._save_timing     = save_timing
         self._queue           = queue
@@ -152,7 +152,7 @@ def __init__(self, test_names, test_data=None,
         self._output_root = output_root
         # Figure out what project to use
         if project is None:
-            self._project = CIME.utils.get_project()
+            self._project = get_project()
             if self._project is None:
                 self._project = self._machobj.get_value("PROJECT")
         else:
@@ -176,7 +176,7 @@ def __init__(self, test_names, test_data=None,
             self._test_root = self._test_root.replace("$PROJECT", self._project)
 
         self._test_root = os.path.abspath(self._test_root)
-        self._test_id   = test_id if test_id is not None else CIME.utils.get_timestamp()
+        self._test_id   = test_id if test_id is not None else get_timestamp()
 
         self._compiler = self._machobj.get_default_compiler() if compiler is None else compiler
 
@@ -412,7 +412,7 @@ def _create_newcase_phase(self, test):
         test_dir = self._get_test_dir(test)
 
         _, case_opts, grid, compset,\
-            machine, compiler, test_mods = CIME.utils.parse_test_name(test)
+            machine, compiler, test_mods = parse_test_name(test)
 
         create_newcase_cmd = "{} --case {} --res {} --compset {}"\
                              " --test".format(os.path.join(self._cime_root, "scripts", "create_newcase"),
@@ -487,7 +487,7 @@ def _create_newcase_phase(self, test):
         else:
             # We need to hard code the queue for this test on cheyenne
             # otherwise it runs in share and fails intermittently
-            test_case = CIME.utils.parse_test_name(test)[0]
+            test_case = parse_test_name(test)[0]
             if test_case == "NODEFAIL":
                 machine = machine if machine is not None else self._machobj.get_machine_name()
                 if machine == "cheyenne":
@@ -514,7 +514,7 @@ def _create_newcase_phase(self, test):
     ###########################################################################
     def _xml_phase(self, test):
     ###########################################################################
-        test_case = CIME.utils.parse_test_name(test)[0]
+        test_case = parse_test_name(test)[0]
 
         # Create, fill and write an envtest object
         test_dir = self._get_test_dir(test)
@@ -565,7 +565,7 @@ def _xml_phase(self, test):
         testnode = config_test.get_test_node(test_case)
         envtest.add_test(testnode)
         # Determine the test_case from the test name
-        test_case, case_opts = CIME.utils.parse_test_name(test)[:2]
+        test_case, case_opts = parse_test_name(test)[:2]
 
         # Determine case_opts from the test_case
         if case_opts is not None:
@@ -857,7 +857,7 @@ def _producer(self):
     def _setup_cs_files(self):
     ###########################################################################
         try:
-            python_libs_root = CIME.utils.get_python_libs_root()
+            python_libs_root = get_python_libs_root()
 
             create_cs_status(test_root=self._test_root,
                              test_id=self._test_id)
@@ -921,6 +921,15 @@ def run_tests(self, wait=False,
 
         expect(threading.active_count() == 1, "Leftover threads?")
 
+        # Copy TestStatus files to baselines for tests that have already failed.
+        if get_model() == "cesm":
+            for test in self._tests:
+                status = self._get_test_data(test)[1]
+                if status not in [TEST_PASS_STATUS, TEST_PEND_STATUS] and self._baseline_gen_name:
+                    basegen_case_fullpath = os.path.join(self._baseline_root,self._baseline_gen_name, test)
+                    test_dir = self._get_test_dir(test)
+                    generate_teststatus(test_dir, basegen_case_fullpath)
+
         wait_handles_report = False
         if not self._no_run and not self._no_batch:
             if wait: