Repository: ESMCI/cime

Issue:
Unexpected repetition of case.setup in tests
In doing some investigation for #690 - but also possibly relevant for #689 - I noticed that various pieces of case setup are done over and over again for tests. I picked a random SMS test (which was NOT rerun), and saw (in CaseStatus):

```
[yslogin2:/glade/p/work/sacks/cesm_code/clm_limitirrig/cime/scripts/tests_1018-1324/SMS_Lm1_D.f10_f10.ICLM45BGCCROP.yellowstone_intel.clm-snowlayers_3_monthly.GC.1018-1324.45.i]$ cat CaseStatus
2016-10-18 13:32:57: <command> ./xmlchange --force CLM_BLDNML_OPTS=-fire_emis --append  </command>
2016-10-18 13:33:00: case.setup starting
2016-10-18 13:33:04: <command> ./xmlchange --force CLM_BLDNML_OPTS=-fire_emis --append  </command>
2016-10-18 13:34:59: case.setup complete
2016-10-18 15:12:20: case.setup starting
2016-10-18 15:12:20: case.setup clean complete
2016-10-18 15:12:22: <command> ./xmlchange --force CLM_BLDNML_OPTS=-fire_emis --append  </command>
2016-10-18 15:12:46: case.setup complete
2016-10-18 15:12:46: case.testbuild starting
2016-10-18 15:13:04: case.testbuild complete
2016-10-18 15:13:54: case.setup starting
2016-10-18 15:13:54: case.setup clean complete
2016-10-18 15:13:55: <command> ./xmlchange --force CLM_BLDNML_OPTS=-fire_emis --append  </command>
2016-10-18 15:14:22: case.setup complete
2016-10-18 15:14:22: case.testbuild starting
2016-10-18 15:15:30: case.testbuild complete
2016-10-18 15:35:31: case.setup starting
2016-10-18 15:35:32: case.setup clean complete
2016-10-18 15:35:34: <command> ./xmlchange --force CLM_BLDNML_OPTS=-fire_emis --append  </command>
2016-10-18 15:35:58: case.setup complete
2016-10-18 15:36:19: Run started
2016-10-18 15:43:07: Run SUCCESSFUL
```

I was initially surprised by the two calls to case.testbuild... but maybe that's expected? (One for sharedlib, one for model??). But even setting that aside, I was surprised to see case.setup and the same xmlchange command repeated so many times for a simple SMS test.

@jedwards4b , @jgfouca : is this expected behavior?



Discussion:
By the way: That xmlchange command comes from the testmod

Yes, that's odd. It's not what I'd expect. 

**Edit: I have moved the following comment to #734 - see that issue for the testmods-specific issue**

Regarding the testmod: In cime4, I'm pretty sure that testmods / user_mods were applied at create_newcase time, ensuring that they were applied once and only once. Now it appears that they are applied in some other phase(s) of case setup. The implications are:
1. The xmlchange commands get executed repeatedly. Often this has no effect, but for these `--append` options, it leads to xml variables like this (for the CLM test `ERP_P60x2_Lm36.f10_f10.ICRUCLM50BGCCROP.yellowstone_intel.clm-clm50cropIrrigMonth_interp`):
   
   ``` xml
   <entry id="CLM_BLDNML_OPTS" value="-fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -fire_emis -irrig .true. -bgc bgc -crop">
   ```
2. We have code to check if the contents of a given user_nl file from testmods have already been applied to the user_nl file in the case directory, and if so, don't reapply them. This code wasn't needed in cime4, because testmods were applied once and only once. This new code generally won't cause problems, but it (a) leads to additional complexity, and (b) can be wrong in some edge cases. An example of such an edge case is where testmod 'foo' includes 'bar' which includes 'default'. 'default' sets `x = 17` (among other things), 'bar' sets `x = 42`, then 'foo' sets `x = 17` (as the only thing that it sets). The desired behavior here would be to use the last setting, `x = 17`; but I believe that the current implementation will keep `x = 42`, because it will skip applying the 'foo' testmods.

Is there a reason why cime5 applies testmods in this later stage and repeatedly, rather than following what was done in cime4 and just applying them once at create_newcase time? It looks like apply_user_mods is currently called both from create_newcase and from case.setup; I'm wondering if the call from case.setup can be removed? I believe this would also allow us to remove the code for checking if the contents of a user_nl file have already been applied.

I'm realizing there are really two separate issues here:
1. case.setup is called many times for a test
2. the testmods get applied every time case.setup is run

I'm going to rename this issue to be specific to (1), and open a new issue for (2): #734

@jedwards4b has some thoughts on this

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -799,7 +799,7 @@ def _set_pio_xml(self):
         compset = self.get_value("COMPSET")
         mpilib = self.get_value("MPILIB")
         defaults = pioobj.get_defaults(grid=grid,compset=compset,mach=mach,compiler=compiler, mpilib=mpilib)
-        
+
         for vid, value in defaults.items():
             self.set_value(vid,value)
 
diff --git a/utils/python/CIME/case_setup.py b/utils/python/CIME/case_setup.py
--- a/utils/python/CIME/case_setup.py
+++ b/utils/python/CIME/case_setup.py
@@ -270,10 +270,10 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
                          env >> software_environment.txt")
 
 ###############################################################################
-def case_setup(case, clean=False, test_mode=False, reset=False):
+def case_setup(case, clean=False, test_mode=False, reset=False, no_status=False):
 ###############################################################################
     caseroot, casebaseid = case.get_value("CASEROOT"), case.get_value("CASEBASEID")
-    if case.get_value("TEST"):
+    if case.get_value("TEST") and not no_status:
         test_name = casebaseid if casebaseid is not None else case.get_value("CASE")
         with TestStatus(test_dir=caseroot, test_name=test_name) as ts:
             try:


Test patch:
diff --git a/utils/python/CIME/SystemTests/system_tests_common.py b/utils/python/CIME/SystemTests/system_tests_common.py
--- a/utils/python/CIME/SystemTests/system_tests_common.py
+++ b/utils/python/CIME/SystemTests/system_tests_common.py
@@ -33,7 +33,6 @@ def __init__(self, case, expected=None):
         self._test_status = TestStatus(test_dir=caseroot, test_name=self._casebaseid)
         self._init_environment(caseroot)
         self._init_locked_files(caseroot, expected)
-        self._init_case_setup()
 
     def _init_environment(self, caseroot):
         """
@@ -59,16 +58,19 @@ def _init_locked_files(self, caseroot, expected):
             shutil.copy(os.path.join(caseroot,"env_run.xml"),
                         os.path.join(lockedfiles, "env_run.orig.xml"))
 
-    def _init_case_setup(self):
+    def _resetup_case(self, phase):
         """
-        Do initial case setup needed in __init__
+        Re-setup this case. This is necessary if user is re-running an already-run
+        phase.
         """
-        if self._case.get_value("IS_FIRST_RUN"):
+        # We never want to re-setup if we're doing the resubmitted run
+        phase_status = self._test_status.get_status(phase)
+        if self._case.get_value("IS_FIRST_RUN") and phase_status != TEST_PEND_STATUS:
+
+            logging.warning("Resetting case due to detected re-run of phase %s" % phase)
             self._case.set_initial_test_values()
 
-        case_setup(self._case, reset=True, test_mode=True)
-        self._case.set_value("TEST",True)
-        self._case.flush()
+            case_setup(self._case, reset=True, test_mode=True, no_status=True)
 
     def build(self, sharedlib_only=False, model_only=False):
         """
@@ -80,6 +82,7 @@ def build(self, sharedlib_only=False, model_only=False):
         for phase_name, phase_bool in [(SHAREDLIB_BUILD_PHASE, not model_only),
                                        (MODEL_BUILD_PHASE, not sharedlib_only)]:
             if phase_bool:
+                self._resetup_case(phase_name)
                 with self._test_status:
                     self._test_status.set_status(phase_name, TEST_PEND_STATUS)
 
@@ -132,6 +135,7 @@ def run(self):
         try:
             expect(self._test_status.get_status(MODEL_BUILD_PHASE) == TEST_PASS_STATUS,
                    "Model was not built!")
+            self._resetup_case(RUN_PHASE)
             with self._test_status:
                 self._test_status.set_status(RUN_PHASE, TEST_PEND_STATUS)
 
diff --git a/utils/python/CIME/test_scheduler.py b/utils/python/CIME/test_scheduler.py
--- a/utils/python/CIME/test_scheduler.py
+++ b/utils/python/CIME/test_scheduler.py
@@ -462,6 +462,7 @@ def _xml_phase(self, test):
                            os.path.join(self._output_root,
                                         "sharedlibroot.%s"%self._test_id))
             envtest.set_initial_values(case)
+            case.set_value("TEST", True)
             if self._save_timing:
                 case.set_value("SAVE_TIMING", True)
 
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1107,6 +1107,20 @@ def test_full_system(self):
             self.assertIs(type(test_time), int, msg="get time did not return int for %s" % test_status)
             self.assertTrue(test_time > 0, msg="test time was zero for %s" % test_status)
 
+        # Test that re-running works
+        tests = update_acme_tests.get_test_suite("cime_developer")
+        for test in tests:
+            if test.startswith("ERI"):
+                # TODO: these need to all work in the future but currently do not support re-run
+                pass
+            else:
+                casedir = os.path.join(TEST_ROOT, "%s.%s" % (test, self._baseline_name))
+                run_cmd_assert_result(self, "./case.submit", from_dir=casedir)
+
+        if (self._hasbatch):
+            run_cmd_assert_result(self, "%s/wait_for_tests *%s/TestStatus" % (TOOLS_DIR, self._baseline_name),
+                                  from_dir=self._testroot)
+
 ###############################################################################
 class K_TestCimeCase(TestCreateTestCommon):
 ###############################################################################