Repository: ESMCI/cime

Issue:
case_submit.py has an inaccurate test for continuing a run
The test for making sure continue_run makes sense in case_submit.py is not really valid. There's a line that uses successful completion of a previous run as the test instead of checking for valid pointer files. That section of the script looks like:

    # Check if CONTINUE_RUN value makes sense
    if job != "case.test" and case.get_value("CONTINUE_RUN"):
        rundir = case.get_value("RUNDIR")
        caseroot = case.get_value("CASEROOT")
        expect(os.path.isdir(rundir),
               "CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
        expect(len(glob.glob(os.path.join(rundir, "*.nc"))) > 0,
               "CONTINUE_RUN is true but this case does not appear to have been run before (no .nc files in RUNDIR)")
        expect(does_file_have_string(os.path.join(caseroot, "CaseStatus"), "case.run {}".format(CASE_SUCCESS)),
               "CONTINUE_RUN is true but this case does not appear to have ever run successfully")
The last line either should be modified to really check for the requirements of a continued run or removed.

Note that we often start production runs to go many years and drop sets of restart files along the way. If they ever time out, this test fails and makes it difficult to restart, even thought there are valid sets of restart files.


Discussion:
@jedwards4b , any thoughts on this?
I see what he is saying - suppose a job is started to run for 10 years, writing restarts each year but the job fails in year 8.   Without this line the model can restart using the files written at the end of year 7.  With this line the user will need to manually remove  the CaseStatus file before they can submit a new run.
OK, i'll re-think it
I think maybe if look for rpointer files in the run directory as well as *.nc and combine the two expect statements into one with or logic.
 
@jedwards4b , if you think you have a good solution, feel free to take this ticket.
@jonbob raises a good point here: I agree that we should allow CONTINUE_RUN in this case, even though this wasn't a case that we considered when we were discussing the fix to #213 . Checking for rpointer files is reasonable, but I believe it wouldn't have actually helped with the problems that led us to open #213 : in those cases, I believe users manually prestaged the restart file set from the refcase into the run directory before submitting, so there would have been rpointer files there.

Would it work to check whether there is at least one rpointer file whose contents contain the current case name? In a valid restart, I think the rpointer files should point to files whose file name has the current case name (I'm not sure if that's a strict requirement of the system, though), whereas in the cases that led us to open #213 , the rpointer files would have pointed to files from a different case.

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -8,11 +8,11 @@
 """
 from six.moves                      import configparser
 from CIME.XML.standard_module_setup import *
-from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg, CASE_SUCCESS, does_file_have_string, CIMEError
+from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg, CIMEError
 from CIME.locked_files              import unlock_file, lock_file
 from CIME.test_status               import *
 
-import socket, glob
+import socket
 
 logger = logging.getLogger(__name__)
 
@@ -32,13 +32,18 @@ def _submit(case, job=None, no_batch=False, prereq=None, allow_fail=False, resub
     # Check if CONTINUE_RUN value makes sense
     if job != "case.test" and case.get_value("CONTINUE_RUN"):
         rundir = case.get_value("RUNDIR")
-        caseroot = case.get_value("CASEROOT")
         expect(os.path.isdir(rundir),
                "CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
-        expect(len(glob.glob(os.path.join(rundir, "*.nc"))) > 0,
-               "CONTINUE_RUN is true but this case does not appear to have been run before (no .nc files in RUNDIR)")
-        expect(does_file_have_string(os.path.join(caseroot, "CaseStatus"), "case.run {}".format(CASE_SUCCESS)),
-               "CONTINUE_RUN is true but this case does not appear to have ever run successfully")
+        expect(os.path.exists(os.path.join(rundir,"rpointer.drv")),
+               "CONTINUE_RUN is true but this case does not appear to have restart files staged in {}".format(rundir))
+        # Finally we open the rpointer.drv file and check that it's correct
+        casename = case.get_value("CASE")
+        with open(os.path.join(rundir,"rpointer.drv"), "r") as fd:
+            ncfile = fd.readline().strip()
+            expect(ncfile.startswith(casename) and
+                   os.path.exists(os.path.join(rundir,ncfile)),
+                   "File {ncfile} not present or does not match case {casename}".
+                   format(ncfile=os.path.join(rundir,ncfile),casename=casename))
 
     # if case.submit is called with the no_batch flag then we assume that this
     # flag will stay in effect for the duration of the RESUBMITs


Test patch: