Repository: ESMCI/cime

Issue:
Abort if user tries to do CONTINUE_RUN in the first iteration of a case
This isn't critical, but it has bitten two people in the last month, so at today's CSEG meeting we decided we should open an issue on it: Two people recently have tried to set up branch runs with the same case as the run they're branching from, and then set CONTINUE_RUN to TRUE in the first iteration of the branch run (thinking that that was what they were supposed to do, or at least that it was allowed). The scripts allowed this to happen, and then the model aborted in CISM, whose time manager gets confused by this case. (Apparently other components handle this more correctly.)

The conclusion from the CSEG meeting is that this case shouldn't be allowed: you should never set CONTINUE_RUN to TRUE in the first iteration of a run, whether it's a branch run or something else. Ideally, the scripts would catch this.

A simple solution would be to have a counter of the number of times a case has completed, and fail with an error message if you try to set CONTINUE_RUN before the case has successfully completed. Others pointed out that there are a number of cases where this wouldn't work, but in most / all of these cases, the problem would just be that the error message wouldn't be triggered (i.e., you wouldn't typically get false error messages).



Discussion:
This still seems to be an issue in the latest cime.
Wouldn't a better solution be to have a prerun check of the contents of the run directory - there are certain rpointer files that we are expecting for a CONTINUE_RUN case that won't be in a branch or startup case.
@jedwards4b - that wouldn't have helped in the two cases that the original issue was referring to: In those cases, users copied the contents of a restart directory (from the archive space) into their run directory, so the rpointer files were there as well.
Then it would need to be a run time error, I think that the solution you propose is ripe for unintended side effects.
A runtime error would be acceptable.
But that's what you get now...  as I understand it all of the expected rpointer and restart files that are required for a CONTINUE_RUN are there - so how are we going to detect this problem?
You get a very cryptic runtime error right now. The consensus in the CSEG meeting last June was that we should provide a clearer error message in this case.

> so how are we going to detect this problem?

My suggestion is above. What are some of the unintended side effects you see with this? Recall that we had some discussion on this point at the CSEG meeting that I reference in this issue. We could come up with some (unusual) cases where this solution would fail to produce an error message, but nobody came up with a solution where this produced an error message when it shouldn't.


Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -6,13 +6,14 @@
 jobs.
 submit, check_case and check_da_settings are members of class Case in file case.py
 """
-import socket
-from six.moves                   import configparser
+from six.moves                      import configparser
 from CIME.XML.standard_module_setup import *
-from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg
+from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg, CASE_SUCCESS, does_file_have_string
 from CIME.locked_files              import unlock_file, lock_file
 from CIME.test_status               import *
 
+import socket, glob
+
 logger = logging.getLogger(__name__)
 
 def _build_prereq_str(case, prev_job_ids):
@@ -28,11 +29,16 @@ def _submit(case, job=None, no_batch=False, prereq=None, allow_fail=False, resub
     if job is None:
         job = case.get_primary_job()
 
-    rundir = case.get_value("RUNDIR")
-    if job != "case.test":
-        continue_run = case.get_value("CONTINUE_RUN")
-        expect(os.path.isdir(rundir) or not continue_run,
-               " CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
+    # Check if CONTINUE_RUN value makes sense
+    if job != "case.test" and case.get_value("CONTINUE_RUN"):
+        rundir = case.get_value("RUNDIR")
+        caseroot = case.get_value("CASEROOT")
+        expect(os.path.isdir(rundir),
+               "CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
+        expect(len(glob.glob(os.path.join(rundir, "*.nc"))) > 0,
+               "CONTINUE_RUN is true but this case does not appear to have been run before (no .nc files in RUNDIR)")
+        expect(does_file_have_string(os.path.join(caseroot, "CaseStatus"), "case.run {}".format(CASE_SUCCESS)),
+               "CONTINUE_RUN is true but this case does not appear to have ever run successfully")
 
     # if case.submit is called with the no_batch flag then we assume that this
     # flag will stay in effect for the duration of the RESUBMITs
@@ -117,8 +123,8 @@ def submit(self, job=None, no_batch=False, prereq=None, allow_fail=False, resubm
         logger.warning("resubmit_immediate does not work on Mira/Cetus, submitting normally")
         resubmit_immediate = False
 
+    caseroot = self.get_value("CASEROOT")
     if self.get_value("TEST"):
-        caseroot = self.get_value("CASEROOT")
         casebaseid = self.get_value("CASEBASEID")
         # This should take care of the race condition where the submitted job
         # begins immediately and tries to set RUN phase. We proactively assume
@@ -132,7 +138,6 @@ def submit(self, job=None, no_batch=False, prereq=None, allow_fail=False, resubm
 
     # If this is a resubmit check the hidden file .submit_options for
     # any submit options used on the original submit and use them again
-    caseroot = self.get_value("CASEROOT")
     submit_options = os.path.join(caseroot, ".submit_options")
     if resubmit and os.path.exists(submit_options):
         config = configparser.SafeConfigParser()
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1352,7 +1352,6 @@ def does_file_have_string(filepath, text):
     """
     return os.path.isfile(filepath) and text in open(filepath).read()
 
-
 def is_last_process_complete(filepath, expect_text, fail_text ):
     """
     Search the filepath in reverse order looking for expect_text
@@ -1656,6 +1655,8 @@ def indent_string(the_string, indent_level):
 def verbatim_success_msg(return_val):
     return return_val
 
+CASE_SUCCESS = "success"
+CASE_FAILURE = "error"
 def run_and_log_case_status(func, phase, caseroot='.', custom_success_msg_functor=None):
     append_case_status(phase, "starting", caseroot=caseroot)
     rv = None
@@ -1663,11 +1664,11 @@ def run_and_log_case_status(func, phase, caseroot='.', custom_success_msg_functo
         rv = func()
     except:
         e = sys.exc_info()[1]
-        append_case_status(phase, "error", msg=("\n{}".format(e)), caseroot=caseroot)
+        append_case_status(phase, CASE_FAILURE, msg=("\n{}".format(e)), caseroot=caseroot)
         raise
     else:
         custom_success_msg = custom_success_msg_functor(rv) if custom_success_msg_functor else None
-        append_case_status(phase, "success", msg=custom_success_msg, caseroot=caseroot)
+        append_case_status(phase, CASE_SUCCESS, msg=custom_success_msg, caseroot=caseroot)
 
     return rv
 


Test patch: