Repository: ESMCI/cime

Issue:
Unable to rerun a test
With the latest trunk, I'm unable to rerun a test.  When I try to do so, I get the traceback below.  This seems to be because the local TestStatus file does not have write permission.  Only the TestStatus file seems to have this permission issue.  If I chmod u+w to TestStatus, the test reruns fine and the TestStatus permissions are not reset.

Exception during run:
[Errno 13] Permission denied: '/glade/p/work/tcraig/cime.nuopc/scripts/SMS.T31_g37_rx1.A.yellowstone_intel.nBa01/TestStatus'
Traceback (most recent call last):
  File "/glade/p/work/tcraig/cime.nuopc/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 137, in run
    self._test_status.set_status(RUN_PHASE, TEST_PEND_STATUS)
  File "/glade/p/work/tcraig/cime.nuopc/scripts/Tools/../../scripts/lib/CIME/test_status.py", line 129, in __exit__
    self.flush()
  File "/glade/p/work/tcraig/cime.nuopc/scripts/Tools/../../scripts/lib/CIME/test_status.py", line 225, in flush
    with open(self._filename, "w") as fd:
IOError: [Errno 13] Permission denied: '/glade/p/work/tcraig/cime.nuopc/scripts/SMS.T31_g37_rx1.A.yellowstone_intel.nBa01/TestStatus'



Discussion:
This happens if the first run went wrong.  Manually change permissions on the TestStatus file
before submitting the second job.
That seems pretty inconvenient.  I'm not sure what setting the write bit on that file is trying to accomplish, but there must be another way.  How about a couple of env flags or something, one that sets whether the last test went right or wrong, the other whether the user wants the submit to fail if the last test went wrong.  During development, if I have to keep flipping that permission bit, it's going to be very frustrating.
There has been an ongoing problem with detecting and handling submit failures in tests.  This is the solution we have been able to come up with and I agree that it's not ideal - you are invited  to submit a working alternative if you can come up with one.   
Are the requirements or a few use cases defined somewhere?
See #1006 and #1136
I agree with @apcraig that the current behavior is unintuitive and likely to trip people up. 

@jedwards4b , I read through the issue and PR that you referenced, but it's still unclear to me why it is necessary to change the permissions of the TestStatus file - that need isn't discussed anywhere, as far as I can tell.

Is the whole issue here the ERR test? If so, the most intuitive solution to me would be to design the ERR test itself so that it will fail if any of its jobs fail to run, e.g., as @fischer-ncar suggests here https://github.com/ESMCI/cime/issues/416#issuecomment-279089122 . I'd prefer a solution like that rather than the current solution that involves embedding test-specific logic in the production code.
The problem is not specifically the ERR test, it's just the one that most easily demonstrates the problem.   Since @jgfouca implemented this solution, I would like to invite him to the discussion.  
Can someone post a reproducer? I resubmit failed tests all the time without looking at file permissions.
The TestStatus file is set to read-only if an exception occurs during the batch submission. This is so that, if the exception occurred after the job is submitted, the submitted job can't change the TestStatus to PASS.

Admittedly, this solution is very hacky; I was thinking that this type of error would be quite rare but apparently that isn't the case.
Maybe we could treat the batch submission as it's own phase?

PASS Test SUBMIT
A quick short term improvement would be for the submission script to check the permission of TestStatus and to NOT submit if it's read only and to produce a reasonable error message.  Then at least the user could fix this quickly (ie chmod the file) and continue working.  That would help a lot.  Longer term, I agree that this deserves further review.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -357,8 +357,7 @@ def submit_single_job(self, case, job, depid=None, no_batch=False, batch_args=No
             else:
                 # This is what we want longterm
                 function_name = job.replace(".", "_")
-                success = locals()[function_name](case)
-                expect(success, "%s failed" % function_name)
+                locals()[function_name](case)
 
             return
 
diff --git a/scripts/lib/CIME/case_submit.py b/scripts/lib/CIME/case_submit.py
--- a/scripts/lib/CIME/case_submit.py
+++ b/scripts/lib/CIME/case_submit.py
@@ -7,7 +7,7 @@
 """
 import socket
 from CIME.XML.standard_module_setup import *
-from CIME.utils                     import expect, append_testlog, run_and_log_case_status
+from CIME.utils                     import expect, run_and_log_case_status
 from CIME.preview_namelists         import create_namelists
 from CIME.check_lockedfiles         import check_lockedfiles
 from CIME.check_input_data          import check_all_input_data
@@ -66,6 +66,15 @@ def _submit(case, job=None, resubmit=False, no_batch=False, batch_args=None):
     logger.info("Submitted job ids %s" % job_ids)
 
 def submit(case, job=None, resubmit=False, no_batch=False, batch_args=None):
+    if case.get_value("TEST"):
+        caseroot = case.get_value("CASEROOT")
+        casebaseid = case.get_value("CASEBASEID")
+        # This should take care of the race condition where the submitted job
+        # begins immediately and tries to set RUN phase. We proactively assume
+        # a passed SUBMIT phase.
+        with TestStatus(test_dir=caseroot, test_name=casebaseid) as ts:
+            ts.set_status(SUBMIT_PHASE, TEST_PASS_STATUS)
+
     try:
         functor = lambda: _submit(case, job, resubmit, no_batch, batch_args)
         run_and_log_case_status(functor, "case.submit", caseroot=case.get_value("CASEROOT"))
@@ -73,12 +82,8 @@ def submit(case, job=None, resubmit=False, no_batch=False, batch_args=None):
         # If something failed in the batch system, make sure to mark
         # the test as failed if we are running a test.
         if case.get_value("TEST"):
-            caseroot = case.get_value("CASEROOT")
-            casebaseid = case.get_value("CASEBASEID")
-            with TestStatus(test_dir=caseroot, test_name=casebaseid, lock=True) as ts:
-                ts.set_status(RUN_PHASE, TEST_FAIL_STATUS, comments="batch system failure")
-
-            append_testlog("Batch submission failed, TestStatus file changed to read-only", caseroot=caseroot)
+            with TestStatus(test_dir=caseroot, test_name=casebaseid) as ts:
+                ts.set_status(SUBMIT_PHASE, TEST_FAIL_STATUS)
 
         raise
 


Test patch:
diff --git a/scripts/lib/CIME/SystemTests/system_tests_common.py b/scripts/lib/CIME/SystemTests/system_tests_common.py
--- a/scripts/lib/CIME/SystemTests/system_tests_common.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_common.py
@@ -130,8 +130,6 @@ def run(self):
         success = True
         start_time = time.time()
         try:
-            expect(self._test_status.get_status(MODEL_BUILD_PHASE) == TEST_PASS_STATUS,
-                   "Model was not built!")
             self._resetup_case(RUN_PHASE)
             with self._test_status:
                 self._test_status.set_status(RUN_PHASE, TEST_PEND_STATUS)
diff --git a/scripts/lib/CIME/case_test.py b/scripts/lib/CIME/case_test.py
--- a/scripts/lib/CIME/case_test.py
+++ b/scripts/lib/CIME/case_test.py
@@ -62,7 +62,7 @@ def case_test(case, testname=None):
         with TestStatus(test_dir=caseroot) as ts:
             ts.set_status(RUN_PHASE, TEST_FAIL_STATUS, comments="failed to initialize")
         append_testlog(str(sys.exc_info()[1]))
-        raise
+        return False
 
     success = test.run()
 
diff --git a/scripts/lib/CIME/test_status.py b/scripts/lib/CIME/test_status.py
--- a/scripts/lib/CIME/test_status.py
+++ b/scripts/lib/CIME/test_status.py
@@ -28,7 +28,6 @@
 from collections import OrderedDict
 
 import os
-from stat import S_IREAD, S_IRGRP, S_IROTH
 
 TEST_STATUS_FILENAME = "TestStatus"
 
@@ -53,6 +52,7 @@
 NAMELIST_PHASE        = "NLCOMP"
 SHAREDLIB_BUILD_PHASE = "SHAREDLIB_BUILD"
 MODEL_BUILD_PHASE     = "MODEL_BUILD"
+SUBMIT_PHASE          = "SUBMIT"
 RUN_PHASE             = "RUN"
 THROUGHPUT_PHASE      = "TPUTCOMP"
 MEMCOMP_PHASE         = "MEMCOMP"
@@ -67,6 +67,7 @@
               NAMELIST_PHASE,
               SHAREDLIB_BUILD_PHASE,
               MODEL_BUILD_PHASE,
+              SUBMIT_PHASE,
               RUN_PHASE,
               COMPARE_PHASE,
               BASELINE_PHASE,
@@ -81,6 +82,7 @@
                SETUP_PHASE,
                SHAREDLIB_BUILD_PHASE,
                MODEL_BUILD_PHASE,
+               SUBMIT_PHASE,
                RUN_PHASE]
 
 def _test_helper1(file_contents):
@@ -98,7 +100,7 @@ def _test_helper2(file_contents, wait_for_run=False, check_throughput=False, che
 
 class TestStatus(object):
 
-    def __init__(self, test_dir=None, test_name=None, no_io=False, lock=False):
+    def __init__(self, test_dir=None, test_name=None, no_io=False):
         """
         Create a TestStatus object
 
@@ -113,7 +115,6 @@ def __init__(self, test_dir=None, test_name=None, no_io=False, lock=False):
         self._test_name = test_name
         self._ok_to_modify = False
         self._no_io = no_io
-        self._lock = lock
 
         if os.path.exists(self._filename):
             self._parse_test_status_file()
@@ -174,7 +175,7 @@ def set_status(self, phase, status, comments=""):
 
         if phase in CORE_PHASES and phase != CORE_PHASES[0]:
             previous_core_phase = CORE_PHASES[CORE_PHASES.index(phase)-1]
-            #TODO: enamble check below
+            #TODO: enable check below
             #expect(previous_core_phase in self._phase_statuses, "Core phase '%s' was skipped" % previous_core_phase)
 
             if previous_core_phase in self._phase_statuses:
@@ -225,9 +226,6 @@ def flush(self):
             with open(self._filename, "w") as fd:
                 self.phase_statuses_dump(fd)
 
-            if self._lock:
-                os.chmod(self._filename, S_IREAD | S_IRGRP | S_IROTH)
-
     def _parse_test_status(self, file_contents):
         """
         >>> contents = '''
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -388,7 +388,7 @@ def setUp(self):
 
         make_fake_teststatus(os.path.join(self._testdir_with_fail,   "5"), "Test_5", TEST_FAIL_STATUS, RUN_PHASE)
         make_fake_teststatus(os.path.join(self._testdir_unfinished,  "5"), "Test_5", TEST_PEND_STATUS, RUN_PHASE)
-        make_fake_teststatus(os.path.join(self._testdir_unfinished2, "5"), "Test_5", TEST_PASS_STATUS, MODEL_BUILD_PHASE)
+        make_fake_teststatus(os.path.join(self._testdir_unfinished2, "5"), "Test_5", TEST_PASS_STATUS, SUBMIT_PHASE)
 
         # Set up proxy if possible
         self._unset_proxy = setup_proxy()
@@ -794,7 +794,7 @@ def test_c_use_existing(self):
             else:
                 self.assertTrue(test_name == pass_test)
                 self.assertEqual(ts.get_status(CIME.test_scheduler.MODEL_BUILD_PHASE), TEST_PASS_STATUS)
-                self.assertEqual(ts.get_status(CIME.test_scheduler.RUN_PHASE), TEST_PEND_STATUS)
+                self.assertEqual(ts.get_status(CIME.test_scheduler.SUBMIT_PHASE), TEST_PEND_STATUS)
 
         os.environ["TESTBUILDFAIL_PASS"] = "True"
         ct2 = TestScheduler(tests, test_id=test_id, no_batch=NO_BATCH, use_existing=True,
@@ -813,6 +813,7 @@ def test_c_use_existing(self):
         for test_status in test_statuses:
             ts = TestStatus(test_dir=os.path.dirname(test_status))
             self.assertEqual(ts.get_status(CIME.test_scheduler.MODEL_BUILD_PHASE), TEST_PASS_STATUS)
+            self.assertEqual(ts.get_status(CIME.test_scheduler.SUBMIT_PHASE),         TEST_PASS_STATUS)
             self.assertEqual(ts.get_status(CIME.test_scheduler.RUN_PHASE),         TEST_PASS_STATUS)
 
 ###############################################################################