Repository: ESMCI/cime

Issue:
Want confirmation of some change made to system_tests_compare_two in #1904
@jedwards4b can you clarify / confirm this change that you made to system_tests_compare_two.py in #1904 

```diff
+            # Go to case2 for the generate and compare baselines
+            self._activate_case2()
```

I am pretty sure that is NOT what we want to do in general. system_tests_compare_two is set up to put all necessary files for comparison / generation in the case1 directory. I'm pretty sure that some things will break if this is done from case2. Can you clarify what the problem was that you were trying to solve with that change?


Discussion:
I believe this is correct.   The internal comparisons happen above this line and this only affects generation and comparison of baselines.     
Yes, I'm referring to the generation and comparison of baselines, too. Can you please describe the problem you were having before introducing this line?
In the ERR test run1 archives output and so the history files are not necessarily present and thus cannot be copied to the baseline directory.   
> In the ERR test run1 archives output and so the history files are not necessarily present and thus cannot be copied to the baseline directory.

Okay, thanks. Now I understand the problem. I still think that the current fix is problematic, though:

1. I think that scripts/tools/component_generate_baseline and component_compare_baseline won't work on an ERR test now. (This fix doesn't make things any worse, but it also doesn't make things any better in that respect.)

2. I'm pretty sure that tools/cprnc/summarize_cprnc_diffs will not work at all now. (I think I'm the only one who uses this, but I use it heavily.)

3. Have you confirmed that various other things work with this fix? Off the top of my head, I'm thinking about:

a. Whether the TestStatus is appended in the correct case directory: should be put in the case1 directory

b. If a comparison fails, does the cprnc.out file get copied into the correct place?: should be put in the case1 directory (I think this is done in wait_for_tests)

4. It is confusing to users if the cprnc.out files from baseline comparison are sometimes generated in the main case directory (for things not using system_tests_compare_two), but sometimes generated in the case2 directory.


Rather than this fix that affects behavior in such a far-reaching way, I'd prefer a fix more specific to the ERR test. For example, could the ERR test copy the relevant *.nc files from case1 to case2 at the end of case2 for the sake of baseline generation and comparison? I know that's a bit ugly, and maybe there's something better, but I'd rather have some ugliness restricted to this one test type rather than having ugliness applying to all test types.

Or, backing up a bit: I thought that, at least at one point, comparison and generation were done using the '\*.base' files. Am I remembering wrong, or is that no longer true? If that were true, then I'm not seeing why this would be an issue for the ERR test, because I don't think the '\*.base' files should be archived, right?
Actually, I think I see a cleaner solution than this:

> For example, could the ERR test copy the relevant *.nc files from case1 to case2 at the end of case2 for the sake of baseline generation and comparison?

If the problem is that some of the \*.nc files are moved out of case1 into the archive space, but the \*.nc.base files are still present, then: Can we have the ERR test do something like: foreach file named FOO.nc.base, if there is no file named FOO.nc then copy FOO.nc.base to FOO.nc.

why would it be .base and not .rest?   Are we not using the output of the second run in the baseline directory?
> why would it be .base and not .rest? Are we not using the output of the second run in the baseline directory?

It doesn't really matter, I guess, since they should be the same. But up until now we have been using the output of the first (.base) run for baseline generation and comparisons. You can see evidence of this by looking at *.nc.cprnc.out files: these files show that it's the case1 (i.e., main case) files that have been used for baseline comparisons. I'm looking at this with a slightly older version of cime, as is currently used in CLM: e.g., see the *.nc.cprnc.out files in /glade/scratch/sacks/PEM_D_Ld5.ne30_g16.I2000Clm50BgcCru.yellowstone_intel.clm-default.GC.0915-0612y_i/run . I also see this behavior with more recent versions of cime, e.g. /glade/scratch/fischer/ERP_D_Ln9.f19_f19_mg17.QPC6.yellowstone_intel.cam-outfrq9s.GC.c2_0_a7d_int/run

By the way: at a glance, it looks like this may no longer be true:

> Or, backing up a bit: I thought that, at least at one point, comparison and generation were done using the '*.base' files. Am I remembering wrong, or is that no longer true?
I don't think that it was ever true.   Anyway I have another fix for the ERR test, PR coming soon.
> I don't think that it was ever true

I just did a bit of digging to understand the history. It was true when system_tests_compare_two was first put in, at which point we were still using component_compgen_baseline.sh to do generation and comparison: this tool was hard-coded to use .base files. But I think it's no longer true. So it was originally necessary for us to use the .base files from case1 for generation/comparison. Now it shouldn't matter if we use .base or .rest, but it would still be best to do it from case1.

Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/err.py b/scripts/lib/CIME/SystemTests/err.py
--- a/scripts/lib/CIME/SystemTests/err.py
+++ b/scripts/lib/CIME/SystemTests/err.py
@@ -2,6 +2,7 @@
 CIME ERR test  This class inherits from ERS
 ERR tests short term archiving and restart capabilities
 """
+import glob, os, shutil
 from CIME.XML.standard_module_setup import *
 from CIME.SystemTests.restart_tests import RestartTest
 from CIME.case_st_archive import restore_from_archive
@@ -10,18 +11,26 @@
 logger = logging.getLogger(__name__)
 
 class ERR(RestartTest):
-
     def __init__(self, case): # pylint: disable=super-init-not-called
         """
         initialize an object interface to the ERR system test
         """
-        RestartTest.__init__(self, case, # pylint: disable=non-parent-init-called
+        super(ERR, self).__init__(case,
                              separate_builds = False,
                              run_two_suffix = 'rest',
                              run_one_description = 'initial',
                              run_two_description = 'restart',
                              multisubmit = True)
 
+    def _case_one_setup(self):
+        super(ERR, self)._case_one_setup()
+        self._case.set_value("DOUT_S", True)
+
+    def _case_two_setup(self):
+        super(ERR, self)._case_two_setup()
+        self._case.set_value("DOUT_S", False)
+
+
     def _case_two_custom_prerun_action(self):
         dout_s_root = self._case1.get_value("DOUT_S_ROOT")
         rest_root = os.path.abspath(os.path.join(dout_s_root,"rest"))
@@ -29,3 +38,15 @@ def _case_two_custom_prerun_action(self):
         expect(len(restart_list) >= 1, "No restart files found in {}".format(rest_root))
         restore_from_archive(self._case, rest_dir=
             os.path.join(rest_root, restart_list[0]))
+
+    def _case_two_custom_postrun_action(self):
+        # Link back to original case1 name
+        # This is needed so that the necessary files are present for
+        # baseline comparison and generation,
+        # since some of them may have been moved to the archive directory
+        for case_file in glob.iglob(os.path.join(self._case1.get_value("RUNDIR"),
+                                                 "*.nc.{}".format(self._run_one_suffix))):
+            orig_file = case_file[:-(1+len(self._run_one_suffix))]
+            if not os.path.isfile(orig_file):
+                shutil.copyfile(case_file, orig_file)
+


Test patch:
diff --git a/config/config_tests.xml b/config/config_tests.xml
--- a/config/config_tests.xml
+++ b/config/config_tests.xml
@@ -307,7 +307,6 @@ NODEFAIL          Tests restart upon detected node failure. Generates fake failu
     <REST_OPTION>$STOP_OPTION</REST_OPTION>
     <HIST_N>$STOP_N</HIST_N>
     <HIST_OPTION>$STOP_OPTION</HIST_OPTION>
-    <DOUT_S>TRUE</DOUT_S>
     <RESUBMIT>1</RESUBMIT>
     <CONTINUE_RUN>FALSE</CONTINUE_RUN>
   </test>
diff --git a/scripts/lib/CIME/SystemTests/system_tests_compare_two.py b/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
--- a/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
@@ -235,8 +235,6 @@ def run_phase(self, success_change=False):  # pylint: disable=arguments-differ
             self._activate_case1()
             self._link_to_case2_output()
             self._component_compare_test(self._run_one_suffix, self._run_two_suffix, success_change=success_change)
-            # Go to case2 for the generate and compare baselines
-            self._activate_case2()
 
     def copy_case1_restarts_to_case2(self):
         """