Repository: ESMCI/cime

Issue:
ERI test cases cannot be re-run
```
./create_test ERI.f45_g37.X
* wait till completes *
cd $casedir
./case.submit
* wait till fails *
```

Error:  ERROR: (seq_infodata_Init) :: rpointer file read returns an error condition


Discussion:
Correction: It looks like ERR and ERS also do not work.
These additional problems were exposed when  #1014 was fixed.
I'm reopening this, since it appears that the rerunnability of ERI has not actually been resolved. If we come to some broad agreement that this doesn't need to be rerun then we can close it as a WONTFIX, but until then I feel it should remain open.

My impression is that we used to have a requirement that all tests be rerunnable; again, if this requirement is dropped for some tests, then I feel that should be agreed upon by a broader group.

tagging @mvertens in case she has anything to add.
I'll take another look at it and see if it's really as hard as it looked.

Resolving patch:
diff --git a/utils/python/CIME/SystemTests/eri.py b/utils/python/CIME/SystemTests/eri.py
--- a/utils/python/CIME/SystemTests/eri.py
+++ b/utils/python/CIME/SystemTests/eri.py
@@ -12,7 +12,10 @@ def _helper(dout_sr, refdate, refsec, rundir):
     rest_path = os.path.join(dout_sr, "rest", "%s-%s" % (refdate, refsec))
 
     for item in glob.glob("%s/*%s*" % (rest_path, refdate)):
-        os.symlink(item, os.path.join(rundir, os.path.basename(item)))
+        dst = os.path.join(rundir, os.path.basename(item))
+        if os.path.exists(dst):
+            os.remove(dst)
+        os.symlink(item, dst)
 
     for item in glob.glob("%s/*rpointer*" % rest_path):
         shutil.copy(item, rundir)
@@ -189,7 +192,10 @@ def run_phase(self):
         for item in glob.glob("%s/*/hist/*nc" % dout_sr2):
             newfile = "%s" % item.replace(".ref2", "")
             newfile = os.path.basename(newfile)
-            os.symlink(item, os.path.join(rundir, newfile))
+            dst = os.path.join(rundir, newfile)
+            if os.path.exists(dst):
+                os.remove(dst)
+            os.symlink(item, dst)
 
         self._component_compare_copy("hybrid")
 


Test patch:
diff --git a/cime_config/config_tests.xml b/cime_config/config_tests.xml
--- a/cime_config/config_tests.xml
+++ b/cime_config/config_tests.xml
@@ -185,6 +185,16 @@ LII    CLM initial condition interpolation test
     <STOP_OPTION>ndays</STOP_OPTION>
     <STOP_N>22</STOP_N>
     <DOUT_S>FALSE</DOUT_S>
+    <RUN_TYPE>startup</RUN_TYPE>
+    <RUN_REFCASE>case.std</RUN_REFCASE>
+    <RUN_REFDATE>0001-01-01</RUN_REFDATE>
+    <RUN_REFTOD>00000</RUN_REFTOD>
+    <REST_OPTION>ndays</REST_OPTION>
+    <REST_N>$STOP_N</REST_N>
+    <GET_REFCASE>FALSE</GET_REFCASE>
+    <CONTINUE_RUN>FALSE</CONTINUE_RUN>
+    <HIST_OPTION>never</HIST_OPTION>
+    <HIST_N>-999</HIST_N>
   </test>
 
   <test NAME="ERP">
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1113,25 +1113,21 @@ def test_full_system(self):
         # Test that re-running works
         tests = update_acme_tests.get_test_suite("cime_developer")
         for test in tests:
-            if test.startswith("ERI"):
-                # TODO: these need to all work in the future but currently do not support re-run
-                pass
-            else:
-                casedir = os.path.join(TEST_ROOT, "%s.%s" % (test, self._baseline_name))
-
-                # Subtle issue: The run phases of these tests will be in the PASS state until
-                # the submitted case.test script is run, which could take a while if the system is
-                # busy. This potentially leaves a window where the wait_for_tests command below will
-                # not wait for the re-submitted jobs to run because it sees the original PASS.
-                # The code below forces things back to PEND to avoid this race condition. Note
-                # that we must use the MEMLEAK phase, not the RUN phase, because RUN being in a non-PEND
-                # state is how system tests know they are being re-run and must reset certain
-                # case settings.
-                if self._hasbatch:
-                    with TestStatus(test_dir=casedir) as ts:
-                        ts.set_status(MEMLEAK_PHASE, TEST_PEND_STATUS)
-
-                run_cmd_assert_result(self, "./case.submit", from_dir=casedir)
+            casedir = os.path.join(TEST_ROOT, "%s.%s" % (test, self._baseline_name))
+
+            # Subtle issue: The run phases of these tests will be in the PASS state until
+            # the submitted case.test script is run, which could take a while if the system is
+            # busy. This potentially leaves a window where the wait_for_tests command below will
+            # not wait for the re-submitted jobs to run because it sees the original PASS.
+            # The code below forces things back to PEND to avoid this race condition. Note
+            # that we must use the MEMLEAK phase, not the RUN phase, because RUN being in a non-PEND
+            # state is how system tests know they are being re-run and must reset certain
+            # case settings.
+            if self._hasbatch:
+                with TestStatus(test_dir=casedir) as ts:
+                    ts.set_status(MEMLEAK_PHASE, TEST_PEND_STATUS)
+
+            run_cmd_assert_result(self, "./case.submit", from_dir=casedir)
 
         if (self._hasbatch):
             run_cmd_assert_result(self, "%s/wait_for_tests *%s/TestStatus" % (TOOLS_DIR, self._baseline_name),