Repository: ESMCI/cime

Issue:
Directories created by scripts_regression_tests should have unique names
I noticed that I was getting inconsistent results in my runs of scripts_regression_tests. I specifically noticed this in `test_createnewcase (__main__.B_TestCreateNewcase)`, which seemed to pass sometimes and fail sometimes. I began to suspect (but have not confirmed) that this was because I was running scripts_regression_tests out of two different cime clones at once (I was running this for my own branch at the same time as I was running it for master).

In looking a little closer, I found that, at least for this test, a directory is created with a hard-coded path, meaning that tests can stomp on each other. I'm guessing that was what happened in my case... but even if not, this opens the door to problems running multiple instances of the test suite at once, so it seems this should be fixed:

``` python
    def setUp(self):
        self._testroot = MACHINE.get_value("CESMSCRATCHROOT")

    def test_createnewcase(self):
        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testcreatenewcase')
        if os.path.exists(testdir):
            shutil.rmtree(testdir)
```

Would it be possible to create a directory with a unique ID in this case? Using https://docs.python.org/2/library/tempfile.html seems like it could be ideal, but maybe not necessary if there's some reason to do something different.



Discussion:
I did try to do it as you suggest with tempfile but ran into several problems - one I remember is that it kept trying to put things into the /tmp directory which is generally not a shared filesystem.  You are welcome to take another stab at it.

@jedwards4b : When you say that /tmp is not a shared filesystem, is this referring to its not being shared between (e.g.) login nodes and compute nodes? I ask because I have introduced the use of tempfile in some of my new unit tests (seeing that it's done for some existing tests)... I'm thinking this is okay in this case, because my new unit tests aren't doing any job submissions, but please let me know if I'm mistaken.

RIght /tmp is not shared between login nodes and compute nodes or between compute nodes.  
It's also quite small in some cases and depending on the test you could easily run out of disk space.  
The testcreatenewcase does not submit anything either - but I was running into problems using tempfile.   I don't recall now what it was - I think I wanted a partial path and could only figure out how to get a full path out of it.     Also scripts_regression_tests will remove directories after tests have successfully completed and leave only directories associated with failed tests behind.  

> RIght /tmp is not shared between login nodes and compute nodes or between compute nodes.
> 
> It's also quite small in some cases and depending on the test you could easily run out of disk space.

Okay, thanks, I think that should still work fine for my unit tests, which generally only create a few bytes of output.

> I think I wanted a partial path and could only figure out how to get a full path out of it.

Would this do what you want?:

``` python
self._testroot = tempfile.mkdtemp(dir=MACHINE.get_value("CESMSCRATCHROOT"))
```

(that will create a unique temporary directory under CESMSCRATCHROOT).

I think that I tried it, you are welcome to try again.

Sounds like the `/tmp` question was resolved, but I just wanted to mention that `/tmp` is often placed in RAM, in which case it is not shared between nodes and also will not persist through a system reboot. For this reason it's usually better to use it just for unit tests, rather than full system tests.


Resolving patch:
diff --git a/utils/python/CIME/XML/env_batch.py b/utils/python/CIME/XML/env_batch.py
--- a/utils/python/CIME/XML/env_batch.py
+++ b/utils/python/CIME/XML/env_batch.py
@@ -276,7 +276,7 @@ def set_job_defaults(self, bjobs, pesize=None, walltime=None, force_queue=None):
             else:
                 task_count = int(task_count)
 
-            queue = force_queue if force_queue is not None else self.select_best_queue(task_count)
+            queue = force_queue if force_queue is not None else self.select_best_queue(task_count, job)
             self.set_value("JOB_QUEUE", queue, subgroup=job)
 
             walltime = self.get_max_walltime(queue) if walltime is None else walltime
@@ -330,6 +330,8 @@ def get_submit_args(self, case, job):
             if name is None:
                 submitargs+=" %s"%flag
             else:
+                if name.startswith("$"):
+                    name = name[1:]
                 val = case.get_value(name,subgroup=job)
                 if val is None:
                     val = case.get_resolved_value(name)
@@ -464,7 +466,7 @@ def get_job_id(self, output):
         jobid = re.search(jobid_pattern, output).group(1)
         return jobid
 
-    def select_best_queue(self, num_pes):
+    def select_best_queue(self, num_pes, job=None):
         # Make sure to check default queue first.
         all_queues = []
         all_queues.append( self.get_default_queue())
@@ -473,8 +475,12 @@ def select_best_queue(self, num_pes):
             if queue is not None:
                 jobmin = queue.get("jobmin")
                 jobmax = queue.get("jobmax")
+                jobname = queue.get("jobname")
+                if jobname is not None:
+                    if job == jobname:
+                        return queue.text
                 # if the fullsum is between the min and max # jobs, then use this queue.
-                if jobmin is not None and jobmax is not None and num_pes >= int(jobmin) and num_pes <= int(jobmax):
+                elif jobmin is not None and jobmax is not None and num_pes >= int(jobmin) and num_pes <= int(jobmax):
                     return queue.text
         return None
 


Test patch:
diff --git a/scripts/Testing/Testcases/config_tests.xml b/cime_config/config_tests.xml
similarity index 100%
rename from scripts/Testing/Testcases/config_tests.xml
rename to cime_config/config_tests.xml
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -113,7 +113,7 @@ def setup_proxy():
     return False
 
 ###############################################################################
-class B_TestCreateNewcase(unittest.TestCase):
+class J_TestCreateNewcase(unittest.TestCase):
 ###############################################################################
     def setUp(self):
         self._testroot = MACHINE.get_value("CESMSCRATCHROOT")
@@ -121,7 +121,7 @@ def setUp(self):
         self._do_teardown = []
 
     def test_createnewcase(self):
-        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testcreatenewcase')
+        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testcreatenewcase.%s'% CIME.utils.get_utc_timestamp())
         if os.path.exists(testdir):
             shutil.rmtree(testdir)
         self._testdirs.append(testdir)
@@ -137,7 +137,7 @@ def test_createnewcase(self):
         self._do_teardown.append(testdir)
 
     def test_user_mods(self):
-        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testusermods')
+        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testusermods.%s'% CIME.utils.get_utc_timestamp())
         if os.path.exists(testdir):
             shutil.rmtree(testdir)
         self._testdirs.append(testdir)
@@ -162,17 +162,17 @@ def tearDown(self):
             elif do_teardown:
                 shutil.rmtree(tfile)
 ###############################################################################
-class D_TestWaitForTests(unittest.TestCase):
+class M_TestWaitForTests(unittest.TestCase):
 ###############################################################################
 
     ###########################################################################
     def setUp(self):
     ###########################################################################
         self._testroot = MACHINE.get_value("CESMSCRATCHROOT")
-        self._testdir_all_pass    = os.path.join(self._testroot, 'scripts_regression_tests.testdir_all_pass')
-        self._testdir_with_fail   = os.path.join(self._testroot, 'scripts_regression_tests.testdir_with_fail')
-        self._testdir_unfinished  = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished')
-        self._testdir_unfinished2 = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished2')
+        self._testdir_all_pass    = os.path.join(self._testroot, 'scripts_regression_tests.testdir_all_pass.%s'% CIME.utils.get_utc_timestamp())
+        self._testdir_with_fail   = os.path.join(self._testroot, 'scripts_regression_tests.testdir_with_fail.%s'% CIME.utils.get_utc_timestamp())
+        self._testdir_unfinished  = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished.%s'% CIME.utils.get_utc_timestamp())
+        self._testdir_unfinished2 = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished2.%s'% CIME.utils.get_utc_timestamp())
         testdirs = [self._testdir_all_pass, self._testdir_with_fail, self._testdir_unfinished, self._testdir_unfinished2]
         for testdir in testdirs:
             if os.path.exists(testdir):
@@ -417,7 +417,7 @@ def tearDown(self):
                     os.remove(file_to_clean)
 
 ###############################################################################
-class C_TestCreateTest(TestCreateTestCommon):
+class N_TestCreateTest(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -472,7 +472,7 @@ def test_create_test_rebless_namelist(self):
         self.simple_test(True, "-c -n -b %s -t %s-%s" % (self._baseline_name, self._baseline_name, CIME.utils.get_utc_timestamp()))
 
 ###############################################################################
-class E_TestTestScheduler(TestCreateTestCommon):
+class O_TestTestScheduler(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -609,7 +609,7 @@ def test_b_full(self):
                     self.assertEqual(ts.get_status(MEMLEAK_PHASE), TEST_PASS_STATUS)
 
 ###############################################################################
-class TestJenkinsGenericJob(TestCreateTestCommon):
+class P_TestJenkinsGenericJob(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -705,7 +705,7 @@ def test_jenkins_generic_job_kill(self):
         assert_dashboard_has_build(self, build_name)
 
 ###############################################################################
-class TestBlessTestResults(TestCreateTestCommon):
+class Q_TestBlessTestResults(TestCreateTestCommon):
 ###############################################################################
 
     _test_name = "TESTRUNDIFF_Mmpi-serial.f19_g16_rx1.A"
@@ -761,7 +761,7 @@ def test_bless_test_results(self):
 
 ###############################################################################
 @unittest.skip("Disabling this test until we figure out how to integrate ACME tests and CIME xml files.")
-class TestUpdateACMETests(unittest.TestCase):
+class R_TestUpdateACMETests(unittest.TestCase):
 ###############################################################################
 
     ###########################################################################
@@ -868,7 +868,7 @@ def test_full_system(self):
             self.assertTrue(test_time > 0, msg="test time was zero for %s" % test_status)
 
 ###############################################################################
-class TestCimeCase(TestCreateTestCommon):
+class K_TestCimeCase(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -910,7 +910,7 @@ def test_cime_case(self):
             self.assertEqual(case.get_value("PIO_TYPENAME"), "netcdf")
 
 ###############################################################################
-class TestSingleSubmit(TestCreateTestCommon):
+class X_TestSingleSubmit(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -933,7 +933,7 @@ def test_single_submit(self):
                          msg="COMMAND SHOULD HAVE WORKED\nwait_for_tests output:\n%s\n\nerrput:\n%s\n\ncode: %d" % (output, errput, stat))
 
 ###############################################################################
-class TestSaveTimings(TestCreateTestCommon):
+class L_TestSaveTimings(TestCreateTestCommon):
 ###############################################################################
 
     ###########################################################################
@@ -954,7 +954,7 @@ def test_save_timings(self):
                              msg="COMMAND SHOULD HAVE WORKED\nwait_for_tests output:\n%s\n\nerrput:\n%s\n\ncode: %d" % (output, errput, stat))
 
 ###############################################################################
-class TestXMLQuery(unittest.TestCase):
+class C_TestXMLQuery(unittest.TestCase):
 # Testing command line scripts
 ###############################################################################
 
@@ -964,7 +964,7 @@ def setUp(self):
         self._testdirs = []
         self._do_teardown = []
 
-        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testscripts')
+        testdir = os.path.join(self._testroot, 'scripts_regression_tests.testscripts.%s'% CIME.utils.get_utc_timestamp())
         machine = 'melvin'
         if os.path.exists(testdir):
             shutil.rmtree(testdir)
@@ -1037,7 +1037,7 @@ def test_subgroup(self):
         self.assertEqual(searchObj.group() , group_out)
 
 ###############################################################################
-class CheckCode(unittest.TestCase):
+class B_CheckCode(unittest.TestCase):
 ###############################################################################
 
     ###########################################################################
@@ -1283,7 +1283,7 @@ def assert_variable_matches(self, var_name, regex, env=None, var=None):
 
 
 ###############################################################################
-class TestMacrosBasic(unittest.TestCase):
+class G_TestMacrosBasic(unittest.TestCase):
 ###############################################################################
 
     """Basic infrastructure tests.
@@ -1317,7 +1317,7 @@ def test_script_rejects_bad_build_system(self):
 
 
 ###############################################################################
-class TestMakeMacros(unittest.TestCase):
+class H_TestMakeMacros(unittest.TestCase):
 ###############################################################################
 
     """Makefile macros tests.
@@ -1592,7 +1592,7 @@ def test_variable_insertion_with_machine_specific_setting(self):
 
 ###############################################################################
 @unittest.skipIf(FAST_ONLY, "Skipping slow CMake tests.")
-class TestCMakeMacros(TestMakeMacros):
+class I_TestCMakeMacros(H_TestMakeMacros):
 ###############################################################################
 
     """CMake macros tests.