Repository: ESMCI/cime

Issue:
Build cprnc as part of create_test
It seems like the python implementation of cprnc (#1031 ) has stalled out (due to no fault of @mfdeakin-sandia 's -- we just hit some roadblocks with this, particularly related to netcdf dependencies). It occurred to me: The primary motivation for that python implementation was to avoid needing to maintain a build of cprnc on all of the systems we test on (users need to pre-build this on their system, we need to keep this up-to-date... which we're often not very good about... etc.).

Now that the cprnc build is more robust, can we simply build cprnc as part of each `create_test` instantiation? I feel that would make the Fortran version of cprnc a sustainable solution.


Discussion:
@billsacks , that's a pretty good idea. What about users who want to use cprnc manually?
I'm not sure if it is, I think that we have some machines/compiler combinations which cprnc doesn't work with and we have a default build of cprnc on those systems that work for all compilers.   I also don't like the added compiication of users having to debug cprnc every time they get a test failure.  How about doing this in create_test iff a default cprnc is not defined in config_machines.xml?
The bigger problem was you needed a build of cprnc for each fortran-netcdf combo on a system.  That's solved by having cprnc be a script that sets the environment for and calls just one combo.
I agree with Jim. The standard build of cprnc should be used unless there isn't one available and then cprnc should be built. Note, that initial versions of CAM testing had as part of it a build of cprnc. This added complexity to the system, if the cprnc build failed, and people didn't know what to do.. And that happened often enough that it was an issue. So there was a purposeful choice to move to a system that didn't rebuild cprnc every time for standard systems. For systems that don't have cprnc built -- I do see adding the capability of building it as part of the system to make it easier to test on new systems. But, for standard systems I think it's better to have a version that's already built for everyone.
I don't feel strongly about this, so I don't want to spend too much energy arguing my point, but I do want to take one more stab at explaining myself.

First, the motivation: I realized that cprnc on hobart hadn't been updated in a long time, and so was marking tests as passed that should have failed (due to a change in dimension sizes of variables). This is definitely not the first time I have had something like this happen. I thought it would be nice if you could rely on the cprnc behavior being stable across machines.

Regarding @jgfouca 's point

> What about users who want to use cprnc manually?

I think this is a very good and important point: I do this a ton myself. I'd still imagine providing a cprnc build somewhere on the system, but automated tests wouldn't use that one.

Regarding @jedwards4b 's points:

> I think that we have some machines/compiler combinations which cprnc doesn't work with and we have a default build of cprnc on those systems that work for all compilers

I imagined we would only build cprnc once for a given create_test invocation (i.e., once per test suite, not once per test), using the default compiler on that machine.

> I also don't like the added compiication of users having to debug cprnc every time they get a test failure. 

Sorry, I don't understand this point.

@rljacob - Sorry, I can't figure out if you're supporting my suggestion or saying it's unnecessary.

@ekluzek I think we can deal with this problem much better now:

> if the cprnc build failed, and people didn't know what to do.. And that happened often enough that it was an issue

We can put a build of cprnc in scripts_regression_tests (it only takes about 5 seconds) to ensure that it will always work on our standard test machines.
@billsacks yeah, I would add a build of cprnc to scripts_regression_tests as it should have already been in there. Then it would be good to hear that a lot of people are running it and not finding any problems. That would be a good test case to show that adding it to create_test would be fine.

There is a valid point that if cprnc isn't built as part of create_test that the version being used will be out of date. And if there is an important feature missing in a new version of cprnc, that can be a problem with create_test. Essentially the current mechanism means that if you update cprnc -- you should create an issue that cprnc should be updated on standard machines. I don't think that step is happening now. And the builds of cprnc are probably happening sporadically. But, better tracking of when the builds are being done could show how off they are and indicate if a build as part of create_test would be better.

The last part of this though is how often does cprnc change in such a way that create_test needs the newer version? I suspect not so often...
> The last part of this though is how often does cprnc change in such a way that create_test needs the newer version? I suspect not so often...

But which is the cause and which is the effect? There are a few changes I've been wanting to make to cprnc for a while, but haven't because of (1) uncertainty about whether we are sticking with the fortran or moving to python, and (2) the pain involved in making any update, particularly one that requires coordination between cprnc code and scripts (as one of my changes does) -- since this involves synchronization across all cesm and e3sm test platforms.
From today's cime call: Feeling is that it could make sense to tackle this. I'll plan to take it at some point.

The main reason not to do this is if we'd be moving ahead with the python version of cprnc. @mfdeakin-sandia was close to having this working, though still needed to do more testing. But this would require having an installation of python's netcdf4 on the system. (See also #1031.) If we started building cprnc as part of the test system, as suggested in the current issue, then I wouldn't see much need for having a python version myself, and there are downsides to the python version, like performance and requiring a netcdf4-python installation. The main upside I see to the python version is that it allows tighter integration between cprnc and the rest of the test system. (I originally thought it would be easier to add new functionality in the python version, but I'm not sure that's really true: you have to be very careful about how you write things in order to maintain decent performance, so it can actually be harder in some ways.) We may wait to make a final call until we've had more discussions about what netcdf python library (if any) we'll allow as a cime dependency (we want something for ctsm, so cime can write netcdf parameter files, but the scipy netcdf would probably be sufficient there: we probably don't need netcdf4 support).

Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -23,6 +23,8 @@ def _get_batch_job_id_for_syslog(case):
             return os.environ["SLURM_JOB_ID"]
         elif mach in ['mira', 'theta']:
             return os.environ["COBALT_JOBID"]
+        elif mach in ['summit']:
+            return os.environ["LSB_JOBID"]
     except:
         pass
 
@@ -195,6 +197,13 @@ def _save_prerun_timing_e3sm(case, lid):
                 full_cmd = cmd + " " + filename
                 run_cmd_no_fail(full_cmd + "." + lid, from_dir=full_timing_dir)
                 gzip_existing_file(os.path.join(full_timing_dir, filename + "." + lid))
+        elif mach == "summit":
+            for cmd, filename in [("bjobs -u all >", "bjobsu_all"),
+                                  ("bjobs -r -u all -o 'jobid slots exec_host' >", "bjobsru_allo"),
+                                  ("bjobs -l -UF %s >" % job_id, "bjobslUF_jobid")]:
+                full_cmd = cmd + " " + filename
+                run_cmd_no_fail(full_cmd + "." + lid, from_dir=full_timing_dir)
+                gzip_existing_file(os.path.join(full_timing_dir, filename + "." + lid))
 
     # copy/tar SourceModes
     source_mods_dir = os.path.join(caseroot, "SourceMods")
@@ -360,6 +369,9 @@ def _save_postrun_timing_e3sm(case, lid):
             globs_to_copy.append("%s*cobaltlog" % job_id)
         elif mach in ["edison", "cori-haswell", "cori-knl"]:
             globs_to_copy.append("%s*run*%s" % (case.get_value("CASE"), job_id))
+        elif mach == "summit":
+            globs_to_copy.append("e3sm.stderr.%s" % job_id)
+            globs_to_copy.append("e3sm.stdout.%s" % job_id)
 
     globs_to_copy.append("logs/run_environment.txt.{}".format(lid))
     globs_to_copy.append(os.path.join(rundir, "e3sm.log.{}.gz".format(lid)))
@@ -425,7 +437,7 @@ def get_recommended_test_time_based_on_past(baseline_root, test, raw=False):
                 return convert_to_babylonian_time(best_walltime)
         except:
             # We NEVER want a failure here to kill the run
-            logger.warning("Failed to read test time: {}".format(sys.exc_info()[0]))
+            logger.warning("Failed to read test time: {}".format(sys.exc_info()[1]))
 
     return None
 
@@ -441,4 +453,4 @@ def save_test_time(baseline_root, test, time_seconds):
                 fd.write("{}\n".format(int(time_seconds)))
         except:
             # We NEVER want a failure here to kill the run
-            logger.warning("Failed to store test time: {}".format(sys.exc_info()[0]))
+            logger.warning("Failed to store test time: {}".format(sys.exc_info()[1]))
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -409,7 +409,7 @@ def run_cmd(cmd, input_str=None, from_dir=None, verbose=None,
         arg_stderr = _convert_to_fd(arg_stdout, from_dir)
 
     if (verbose != False and (verbose or logger.isEnabledFor(logging.DEBUG))):
-        logger.info("RUN: {}".format(cmd))
+        logger.info("RUN: {}\nFROM: {}".format(cmd, os.getcwd() if from_dir is None else from_dir))
 
     if (input_str is not None):
         stdin = subprocess.PIPE
diff --git a/scripts/lib/jenkins_generic_job.py b/scripts/lib/jenkins_generic_job.py
--- a/scripts/lib/jenkins_generic_job.py
+++ b/scripts/lib/jenkins_generic_job.py
@@ -1,10 +1,10 @@
 import CIME.wait_for_tests
-from CIME.utils import expect
+from CIME.utils import expect, run_cmd_no_fail
 from CIME.case import Case
 
-import os, shutil, glob, signal, logging
+import os, shutil, glob, signal, logging, threading, sys, re
 
-###############################################################################
+##############################################################################
 def cleanup_queue(test_root, test_id):
 ###############################################################################
     """
@@ -21,6 +21,99 @@ def cleanup_queue(test_root, test_id):
 
             case.cancel_batch_jobs(jobkills)
 
+###############################################################################
+def delete_old_test_data(mach_comp, test_id_root, scratch_root, test_root, run_area, build_area, archive_area):
+###############################################################################
+    # Remove old dirs
+    for clutter_area in [scratch_root, test_root, run_area, build_area, archive_area]:
+        for old_file in glob.glob("{}/*{}*{}*".format(clutter_area, mach_comp, test_id_root)):
+            logging.info("Removing {}".format(old_file))
+            if (os.path.isdir(old_file)):
+                shutil.rmtree(old_file)
+            else:
+                os.remove(old_file)
+
+###############################################################################
+def scan_for_test_ids(old_test_archive, mach_comp, test_id_root):
+###############################################################################
+    results = set([])
+    test_id_re = re.compile(".+[.]([^.]+)")
+    for item in glob.glob("{}/{}/*{}*{}*".format(old_test_archive, "old_cases", mach_comp, test_id_root)):
+        filename = os.path.basename(item)
+        the_match = test_id_re.match(filename)
+        if the_match:
+            test_id = the_match.groups()[0]
+            results.add(test_id)
+
+    return list(results)
+
+###############################################################################
+def archive_old_test_data(machine, mach_comp, test_id_root, scratch_root, test_root, old_test_archive):
+###############################################################################
+
+    # Remove old cs.status, cs.submit. I don't think there's any value to leaving these around
+    # or archiving them
+    for old_cs_file in glob.glob("{}/cs.*".format(scratch_root)):
+        logging.info("Removing {}".format(old_cs_file))
+        os.remove(old_cs_file)
+
+    # Remove the old CTest XML, same reason as above
+    if (os.path.isdir("Testing")):
+        logging.info("Removing {}".format(os.path.join(os.getcwd(), "Testing")))
+        shutil.rmtree("Testing")
+
+    if not os.path.exists(old_test_archive):
+        os.mkdir(old_test_archive)
+
+    # Archive old data by looking at old test cases
+    for old_case in glob.glob("{}/*{}*{}*".format(test_root, mach_comp, test_id_root)):
+        logging.info("arching case {}".format(old_case))
+        exeroot, rundir, archdir = run_cmd_no_fail("./xmlquery EXEROOT RUNDIR DOUT_S_ROOT --value", from_dir=old_case).split(",")
+
+        for the_dir, target_area in [(exeroot, "old_builds"), (rundir, "old_runs"), (archdir, "old_archives"), (old_case, "old_cases")]:
+            if os.path.exists(the_dir):
+                logging.info("  archiving {} to {}".format(the_dir, os.path.join(old_test_archive, target_area)))
+                if not os.path.exists(os.path.join(old_test_archive, target_area)):
+                    os.mkdir(os.path.join(old_test_archive, target_area))
+
+                os.rename(the_dir, os.path.join(old_test_archive, target_area, os.path.basename(old_case)))
+
+    # Check size of archive
+    bytes_of_old_test_data = int(run_cmd_no_fail("du -sb {}".format(old_test_archive)).split()[0])
+    bytes_allowed = machine.get_value("MAX_GB_OLD_TEST_DATA") * 1000000000
+    if bytes_of_old_test_data > bytes_allowed:
+        logging.info("Too much test data, {}GB (actual) > {}GB (limit)".format(bytes_of_old_test_data / 1000000000, bytes_allowed / 1000000000))
+        old_test_ids = scan_for_test_ids(old_test_archive, mach_comp, test_id_root)
+        for old_test_id in sorted(old_test_ids):
+            logging.info(" Removing old data for test {}".format(old_test_id))
+            for item in ["old_cases", "old_builds", "old_runs", "old_archives"]:
+                for dir_to_rm in glob.glob("{}/{}/*{}*{}*".format(old_test_archive, item, mach_comp, old_test_id)):
+                    logging.info("   Removing {}".format(dir_to_rm))
+                    shutil.rmtree(dir_to_rm)
+
+            bytes_of_old_test_data = int(run_cmd_no_fail("du -sb {}".format(old_test_archive)).split()[0])
+            if bytes_of_old_test_data < bytes_allowed:
+                break
+
+    else:
+        logging.info("Test data is with accepted bounds, {}GB (actual) < {}GB (limit)".format(bytes_of_old_test_data / 1000000000, bytes_allowed / 1000000000))
+
+###############################################################################
+def handle_old_test_data(machine, compiler, test_id_root, scratch_root, test_root):
+###############################################################################
+    run_area = os.path.dirname(os.path.dirname(machine.get_value("RUNDIR"))) # Assumes XXX/$CASE/run
+    build_area = os.path.dirname(os.path.dirname(machine.get_value("EXEROOT"))) # Assumes XXX/$CASE/build
+    archive_area = os.path.dirname(machine.get_value("DOUT_S_ROOT")) # Assumes XXX/archive/$CASE
+    old_test_archive = os.path.join(scratch_root, "old_test_archive")
+
+    mach_comp = "{}_{}".format(machine.get_machine_name(), compiler)
+
+    try:
+        archive_old_test_data(machine, mach_comp, test_id_root, scratch_root, test_root, old_test_archive)
+    except Exception:
+        logging.warning("Archiving of old test data FAILED: {}\nDeleting data instead".format(sys.exc_info()[1]))
+        delete_old_test_data(mach_comp, test_id_root, scratch_root, test_root, run_area, build_area, archive_area)
+
 ###############################################################################
 def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
                         baseline_name,
@@ -28,7 +121,7 @@ def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
                         arg_test_suite,
                         cdash_build_group, baseline_compare,
                         scratch_root, parallel_jobs, walltime,
-                        machine, compiler, real_baseline_name):
+                        machine, compiler, real_baseline_name, baseline_root):
 ###############################################################################
     """
     Return True if all tests passed
@@ -38,7 +131,6 @@ def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
     proxy = machine.get_value("PROXY")
     test_suite = test_suite if arg_test_suite is None else arg_test_suite
     test_root = os.path.join(scratch_root, "J")
-    run_area = os.path.dirname(os.path.dirname(machine.get_value("RUNDIR")))
 
     if (use_batch):
         batch_system = machine.get_value("BATCH_SYSTEM")
@@ -65,20 +157,9 @@ def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
     # the Jenkins jobs with timeouts to avoid this.
     #
 
-    mach_comp = "{}_{}".format(machine.get_machine_name(), compiler)
-
-    # Remove the old CTest XML
-    if (os.path.isdir("Testing")):
-        shutil.rmtree("Testing")
-
-    # Remove old dirs
     test_id_root = "J{}{}".format(baseline_name.capitalize(), test_suite.replace("e3sm_", "").capitalize())
-    for clutter_area in [scratch_root, test_root, run_area]:
-        for old_file in glob.glob("{}/*{}*{}*".format(clutter_area, mach_comp, test_id_root)):
-            if (os.path.isdir(old_file)):
-                shutil.rmtree(old_file)
-            else:
-                os.remove(old_file)
+    archiver_thread = threading.Thread(target=handle_old_test_data, args=(machine, compiler, test_id_root, scratch_root, test_root))
+    archiver_thread.start()
 
     #
     # Set up create_test command and run it
@@ -103,6 +184,9 @@ def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
     if walltime is not None:
         create_test_args.append(" --walltime " + walltime)
 
+    if baseline_root is not None:
+        create_test_args.append(" --baseline-root " + baseline_root)
+
     create_test_cmd = "./create_test " + " ".join(create_test_args)
 
     if (not CIME.wait_for_tests.SIGNAL_RECEIVED):
@@ -135,6 +219,10 @@ def jenkins_generic_job(generate_baselines, submit_to_cdash, no_batch,
                                                  cdash_project=cdash_project,
                                                  cdash_build_group=cdash_build_group)
 
+    logging.info("Waiting for archiver thread")
+    archiver_thread.join()
+    logging.info("Waiting for archiver finished")
+
     if use_batch and CIME.wait_for_tests.SIGNAL_RECEIVED:
         # Cleanup
         cleanup_queue(test_root, test_id)


Test patch:
diff --git a/config/e3sm/tests.py b/config/e3sm/tests.py
--- a/config/e3sm/tests.py
+++ b/config/e3sm/tests.py
@@ -21,6 +21,10 @@
                              ("ERS.f09_g16.I1850CLM45CN","clm-bgcinterface"),
                               "ERS.ne11_oQU240.I20TRCLM45",
                              ("ERS.f19_g16.I1850CNRDCTCBC","clm-rd"),
+                             ("ERS.f19_g16.I1850GSWCNPECACNTBC","clm-eca_f19_g16_I1850GSWCNPECACNTBC"),
+                             ("ERS.f19_g16.I20TRGSWCNPECACNTBC","clm-eca_f19_g16_I20TRGSWCNPECACNTBC"),
+                             ("ERS.f19_g16.I1850GSWCNPRDCTCBC","clm-ctc_f19_g16_I1850GSWCNPRDCTCBC"),
+                             ("ERS.f19_g16.I20TRGSWCNPRDCTCBC","clm-ctc_f19_g16_I20TRGSWCNPRDCTCBC"),
                               "ERS.f09_g16.ICLM45BC")
                              ),
 
@@ -80,7 +84,7 @@
 		           ("SMS_D_Ld1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6","allactive-v1cmip6"),
                            "ERS_Ln9.ne4_ne4.FC5AV1C-L",
                           #"ERT_Ld31.ne16_g37.B1850C5",#add this line back in with the new correct compset
-                           "NCK.ne4_oQU240.A_WCYCL1850",
+                           "NCK.ne11_oQU240.A_WCYCL1850",
                            ("PET.f19_g16.X","allactive-mach-pet"),
                            ("PET.f45_g37_rx1.A","allactive-mach-pet"),
                            ("PET_Ln9_PS.ne30_oECv3_ICG.A_WCYCL1850S","allactive-mach-pet"),
@@ -88,6 +92,7 @@
                            "ERP_Ld3.ne30_oECv3_ICG.A_WCYCL1850S",
                            "SMS.f09_g16_a.MALI",
                            "SMS_D_Ln5.conusx4v1_conusx4v1.FC5AV1C-L",
+                           ("SMS.ne30_oECv3.BGCEXP_BCRC_CNPECACNT_1850","clm-bgcexp"),
                            ("SMS.ne30_oECv3.BGCEXP_BCRC_CNPRDCTC_1850","clm-bgcexp"))
                           ),
     #e3sm tests for extra coverage
diff --git a/scripts/Tools/wait_for_tests b/scripts/Tools/wait_for_tests
--- a/scripts/Tools/wait_for_tests
+++ b/scripts/Tools/wait_for_tests
@@ -43,6 +43,9 @@ formatter_class=argparse.ArgumentDefaultsHelpFormatter
     parser.add_argument("-n", "--no-wait", action="store_true",
                         help="Do not wait for tests to finish")
 
+    parser.add_argument("--no-run", action="store_true",
+                        help="Do not expect run phase to be completed")
+
     parser.add_argument("-t", "--check-throughput", action="store_true",
                         help="Fail if throughput check fails (fail if tests slow down)")
 
@@ -72,12 +75,12 @@ formatter_class=argparse.ArgumentDefaultsHelpFormatter
 
     args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser)
 
-    return args.paths, args.no_wait, args.check_throughput, args.check_memory, args.ignore_namelist_diffs, args.ignore_memleak, args.cdash_build_name, args.cdash_project, args.cdash_build_group, args.timeout, args.force_log_upload
+    return args.paths, args.no_wait, args.check_throughput, args.check_memory, args.ignore_namelist_diffs, args.ignore_memleak, args.cdash_build_name, args.cdash_project, args.cdash_build_group, args.timeout, args.force_log_upload, args.no_run
 
 ###############################################################################
 def _main_func(description):
 ###############################################################################
-    test_paths, no_wait, check_throughput, check_memory, ignore_namelist_diffs, ignore_memleak, cdash_build_name, cdash_project, cdash_build_group, timeout, force_log_upload = \
+    test_paths, no_wait, check_throughput, check_memory, ignore_namelist_diffs, ignore_memleak, cdash_build_name, cdash_project, cdash_build_group, timeout, force_log_upload, no_run = \
         parse_command_line(sys.argv, description)
 
     sys.exit(0 if CIME.wait_for_tests.wait_for_tests(test_paths,
@@ -90,7 +93,8 @@ def _main_func(description):
                                                      cdash_project=cdash_project,
                                                      cdash_build_group=cdash_build_group,
                                                      timeout=timeout,
-                                                     force_log_upload=force_log_upload)
+                                                     force_log_upload=force_log_upload,
+                                                     no_run=no_run)
              else CIME.utils.TESTS_FAILED_ERR_CODE)
 
 ###############################################################################
diff --git a/scripts/lib/CIME/SystemTests/system_tests_common.py b/scripts/lib/CIME/SystemTests/system_tests_common.py
--- a/scripts/lib/CIME/SystemTests/system_tests_common.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_common.py
@@ -3,7 +3,7 @@
 """
 from CIME.XML.standard_module_setup import *
 from CIME.XML.env_run import EnvRun
-from CIME.utils import append_testlog, get_model, safe_copy
+from CIME.utils import append_testlog, get_model, safe_copy, get_timestamp
 from CIME.test_status import *
 from CIME.hist_utils import *
 from CIME.provenance import save_test_time
@@ -601,6 +601,11 @@ def build_phase(self, sharedlib_only=False, model_only=False):
             TESTRUNPASS.build_phase(self, sharedlib_only, model_only)
         else:
             if (not sharedlib_only):
+                blddir = self._case.get_value("EXEROOT")
+                bldlog = os.path.join(blddir, "{}.bldlog.{}".format(get_model(), get_timestamp("%y%m%d-%H%M%S")))
+                with open(bldlog, "w") as fd:
+                    fd.write("BUILD FAIL: Intentional fail for testing infrastructure")
+
                 expect(False, "BUILD FAIL: Intentional fail for testing infrastructure")
 
 class TESTBUILDFAILEXC(FakeTest):
diff --git a/scripts/lib/CIME/test_status.py b/scripts/lib/CIME/test_status.py
--- a/scripts/lib/CIME/test_status.py
+++ b/scripts/lib/CIME/test_status.py
@@ -27,7 +27,7 @@
 
 from collections import OrderedDict
 
-import os
+import os, itertools
 
 TEST_STATUS_FILENAME = "TestStatus"
 
@@ -92,13 +92,23 @@ def _test_helper1(file_contents):
     ts._parse_test_status(file_contents) # pylint: disable=protected-access
     return ts._phase_statuses # pylint: disable=protected-access
 
-def _test_helper2(file_contents, wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False):
-    ts = TestStatus(test_dir="/", test_name="ERS.foo.A")
-    ts._parse_test_status(file_contents) # pylint: disable=protected-access
-    return ts.get_overall_test_status(wait_for_run=wait_for_run,
-                                      check_throughput=check_throughput,
-                                      check_memory=check_memory,
-                                      ignore_namelists=ignore_namelists)
+def _test_helper2(file_contents, wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, no_run=False):
+    lines = file_contents.splitlines()
+    rv = None
+    for perm in itertools.permutations(lines):
+        ts = TestStatus(test_dir="/", test_name="ERS.foo.A")
+        ts._parse_test_status("\n".join(perm)) # pylint: disable=protected-access
+        the_status = ts.get_overall_test_status(wait_for_run=wait_for_run,
+                                                check_throughput=check_throughput,
+                                                check_memory=check_memory,
+                                                ignore_namelists=ignore_namelists,
+                                                no_run=no_run)
+        if rv is not None and the_status != rv:
+            return "{} != {}".format(rv, the_status)
+        else:
+            rv = the_status
+
+    return rv
 
 class TestStatus(object):
 
@@ -137,6 +147,12 @@ def __iter__(self):
         for phase, data in self._phase_statuses.items():
             yield phase, data[0]
 
+    def __eq__(self, rhs):
+        return self._phase_statuses == rhs._phase_statuses # pylint: disable=protected-access
+
+    def __ne__(self, rhs):
+        return not self.__eq__(rhs)
+
     def get_name(self):
         return self._test_name
 
@@ -271,7 +287,54 @@ def _parse_test_status_file(self):
         with open(self._filename, "r") as fd:
             self._parse_test_status(fd.read())
 
-    def get_overall_test_status(self, wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False):
+    def _get_overall_status_based_on_phases(self, phases, wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, no_run=False):
+
+        rv = TEST_PASS_STATUS
+        run_phase_found = False
+        for phase in phases: # ensure correct order of processing phases
+            if phase in self._phase_statuses:
+                data = self._phase_statuses[phase]
+            else:
+                continue
+
+            status = data[0]
+            if phase == RUN_PHASE:
+                run_phase_found = True
+
+            if phase in [SUBMIT_PHASE, RUN_PHASE] and no_run:
+                break
+
+            if (status == TEST_PEND_STATUS):
+                rv = TEST_PEND_STATUS
+
+            elif (status == TEST_FAIL_STATUS):
+                if ( (not check_throughput and phase == THROUGHPUT_PHASE) or
+                     (not check_memory and phase == MEMCOMP_PHASE) or
+                     (ignore_namelists and phase == NAMELIST_PHASE) or
+                     (ignore_memleak and phase == MEMLEAK_PHASE) ):
+                    continue
+
+                if (phase == NAMELIST_PHASE):
+                    if (rv == TEST_PASS_STATUS):
+                        rv = NAMELIST_FAIL_STATUS
+
+                elif (rv in [NAMELIST_FAIL_STATUS, TEST_PASS_STATUS] and phase == BASELINE_PHASE):
+                    rv = TEST_DIFF_STATUS
+
+                elif phase in CORE_PHASES:
+                    return TEST_FAIL_STATUS
+
+                else:
+                    rv = TEST_FAIL_STATUS
+
+        # The test did not fail but the RUN phase was not found, so if the user requested
+        # that we wait for the RUN phase, then the test must still be considered pending.
+        if rv != TEST_FAIL_STATUS and not run_phase_found and wait_for_run:
+            rv = TEST_PEND_STATUS
+
+        return rv
+
+    def get_overall_test_status(self, wait_for_run=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, no_run=False):
         r"""
         Given the current phases and statuses, produce a single results for this test. Preference
         is given to PEND since we don't want to stop waiting for a test
@@ -282,15 +345,17 @@ def get_overall_test_status(self, wait_for_run=False, check_throughput=False, ch
         >>> _test_helper2('PASS ERS.foo.A SHAREDLIB_BUILD\nPEND ERS.foo.A RUN')
         'PEND'
         >>> _test_helper2('FAIL ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN')
-        'PEND'
+        'FAIL'
         >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN')
         'PASS'
         >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP')
         'PASS'
         >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A TPUTCOMP', check_throughput=True)
         'FAIL'
-        >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
+        >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
         'NLFAIL'
+        >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP')
+        'PEND'
         >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A MEMCOMP')
         'PASS'
         >>> _test_helper2('PASS ERS.foo.A RUN\nFAIL ERS.foo.A NLCOMP', ignore_namelists=True)
@@ -317,37 +382,30 @@ def get_overall_test_status(self, wait_for_run=False, check_throughput=False, ch
         'FAIL'
         >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPASS ERS.foo.A RUN', wait_for_run=True)
         'PASS'
+        >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nFAIL ERS.foo.A RUN\nPEND ERS.foo.A COMPARE')
+        'FAIL'
+        >>> _test_helper2('PASS ERS.foo.A MODEL_BUILD\nPEND ERS.foo.A RUN', no_run=True)
+        'PASS'
         """
-        rv = TEST_PASS_STATUS
-        run_phase_found = False
-        for phase, data in self._phase_statuses.items():
-            status = data[0]
-            if phase == RUN_PHASE:
-                run_phase_found = True
-
-            if (status == TEST_PEND_STATUS):
-                rv = TEST_PEND_STATUS
-
-            elif (status == TEST_FAIL_STATUS):
-                if ( (not check_throughput and phase == THROUGHPUT_PHASE) or
-                     (not check_memory and phase == MEMCOMP_PHASE) or
-                     (ignore_namelists and phase == NAMELIST_PHASE) or
-                     (ignore_memleak and phase == MEMLEAK_PHASE) ):
-                    continue
-
-                if (phase == NAMELIST_PHASE):
-                    if (rv == TEST_PASS_STATUS):
-                        rv = NAMELIST_FAIL_STATUS
-
-                elif (rv in [NAMELIST_FAIL_STATUS, TEST_PASS_STATUS] and phase == BASELINE_PHASE):
-                    rv = TEST_DIFF_STATUS
-
-                else:
-                    rv = TEST_FAIL_STATUS
+        # Core phases take priority
+        core_rv = self._get_overall_status_based_on_phases(CORE_PHASES,
+                                                           wait_for_run=wait_for_run,
+                                                           check_throughput=check_throughput,
+                                                           check_memory=check_memory,
+                                                           ignore_namelists=ignore_namelists,
+                                                           ignore_memleak=ignore_memleak,
+                                                           no_run=no_run)
+        if core_rv != TEST_PASS_STATUS:
+            return core_rv
+        else:
+            phase_order = list(CORE_PHASES)
+            phase_order.extend([item for item in self._phase_statuses if item not in CORE_PHASES])
 
-        # The test did not fail but the RUN phase was not found, so if the user requested
-        # that we wait for the RUN phase, then the test must still be considered pending.
-        if rv != TEST_FAIL_STATUS and not run_phase_found and wait_for_run:
-            rv = TEST_PEND_STATUS
+            return self._get_overall_status_based_on_phases(phase_order,
+                                                            wait_for_run=wait_for_run,
+                                                            check_throughput=check_throughput,
+                                                            check_memory=check_memory,
+                                                            ignore_namelists=ignore_namelists,
+                                                            ignore_memleak=ignore_memleak,
+                                                            no_run=no_run)
 
-        return rv
diff --git a/scripts/lib/CIME/wait_for_tests.py b/scripts/lib/CIME/wait_for_tests.py
--- a/scripts/lib/CIME/wait_for_tests.py
+++ b/scripts/lib/CIME/wait_for_tests.py
@@ -51,20 +51,8 @@ def get_test_output(test_path):
         return ""
 
 ###############################################################################
-def create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname):
+def create_cdash_xml_boiler(phase, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit):
 ###############################################################################
-    # We assume all cases were created from the same code repo
-    first_result_case = os.path.dirname(list(results.items())[0][1][0])
-    try:
-        srcroot = run_cmd_no_fail("./xmlquery --value CIMEROOT", from_dir=first_result_case)
-    except:
-        # Use repo containing this script as last resort
-        srcroot = CIME.utils.get_cime_root()
-
-    git_commit = CIME.utils.get_current_commit(repo=srcroot)
-
-    data_rel_path = os.path.join("Testing", utc_time)
-
     site_elem = xmlet.Element("Site")
 
     if ("JENKINS_START_TIME" in os.environ):
@@ -79,40 +67,89 @@ def create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time
     site_elem.attrib["Hostname"] = hostname
     site_elem.attrib["OSVersion"] = "Commit: {}{}".format(git_commit, time_info_str)
 
-    testing_elem = xmlet.SubElement(site_elem, "Testing")
+    phase_elem = xmlet.SubElement(site_elem, phase)
+
+    xmlet.SubElement(phase_elem, "StartDateTime").text = time.ctime(current_time)
+    xmlet.SubElement(phase_elem, "Start{}Time".format("Test" if phase == "Testing" else phase)).text = str(int(current_time))
+
+    return site_elem, phase_elem
+
+###############################################################################
+def create_cdash_config_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit):
+###############################################################################
+    site_elem, config_elem = create_cdash_xml_boiler("Configure", cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit)
+
+    xmlet.SubElement(config_elem, "ConfigureCommand").text = "namelists"
+
+    config_results = []
+    for test_name in sorted(results):
+        test_status = results[test_name][1]
+        config_results.append("{} {} Config {}".format("" if test_status != NAMELIST_FAIL_STATUS else "CMake Warning:\n", test_name, "PASS" if test_status != NAMELIST_FAIL_STATUS else "NML DIFF"))
+
+    xmlet.SubElement(config_elem, "Log").text = "\n".join(config_results)
+
+    xmlet.SubElement(config_elem, "ConfigureStatus").text = "0"
+    xmlet.SubElement(config_elem, "ElapsedMinutes").text = "0" # Skip for now
+
+    etree = xmlet.ElementTree(site_elem)
+    etree.write(os.path.join(data_rel_path, "Configure.xml"))
+
+###############################################################################
+def create_cdash_build_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit):
+###############################################################################
+    site_elem, build_elem = create_cdash_xml_boiler("Build", cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit)
+
+    xmlet.SubElement(build_elem, "ConfigureCommand").text = "case.build"
+
+    build_results = []
+    for test_name in sorted(results):
+        build_results.append(test_name)
+
+    xmlet.SubElement(build_elem, "Log").text = "\n".join(build_results)
+
+    for idx, test_name in enumerate(sorted(results)):
+        test_path = results[test_name][0]
+        test_norm_path = test_path if os.path.isdir(test_path) else os.path.dirname(test_path)
+        if get_test_time(test_norm_path) == 0:
+            error_elem = xmlet.SubElement(build_elem, "Error")
+            xmlet.SubElement(error_elem, "Text").text = test_name
+            xmlet.SubElement(error_elem, "BuildLogLine").text = str(idx)
+            xmlet.SubElement(error_elem, "PreContext").text = test_name
+            xmlet.SubElement(error_elem, "PostContext").text = ""
+            xmlet.SubElement(error_elem, "RepeatCount").text = "0"
 
-    start_date_time_elem = xmlet.SubElement(testing_elem, "StartDateTime")
-    start_date_time_elem.text = time.ctime(current_time)
+    xmlet.SubElement(build_elem, "ElapsedMinutes").text = "0" # Skip for now
 
-    start_test_time_elem = xmlet.SubElement(testing_elem, "StartTestTime")
-    start_test_time_elem.text = str(int(current_time))
+    etree = xmlet.ElementTree(site_elem)
+    etree.write(os.path.join(data_rel_path, "Build.xml"))
+
+###############################################################################
+def create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit):
+###############################################################################
+    site_elem, testing_elem = create_cdash_xml_boiler("Testing", cdash_build_name, cdash_build_group, utc_time, current_time, hostname, git_commit)
 
     test_list_elem = xmlet.SubElement(testing_elem, "TestList")
     for test_name in sorted(results):
-        test_elem = xmlet.SubElement(test_list_elem, "Test")
-        test_elem.text = test_name
+        xmlet.SubElement(test_list_elem, "Test").text = test_name
 
     for test_name in sorted(results):
         test_path, test_status = results[test_name]
-        test_passed = test_status == TEST_PASS_STATUS
+        test_passed = test_status in [TEST_PASS_STATUS, NAMELIST_FAIL_STATUS]
         test_norm_path = test_path if os.path.isdir(test_path) else os.path.dirname(test_path)
 
         full_test_elem = xmlet.SubElement(testing_elem, "Test")
-        if (test_passed):
+        if test_passed:
             full_test_elem.attrib["Status"] = "passed"
-        elif (test_status == NAMELIST_FAIL_STATUS):
+        elif (test_status == TEST_PEND_STATUS):
             full_test_elem.attrib["Status"] = "notrun"
         else:
             full_test_elem.attrib["Status"] = "failed"
 
-        name_elem = xmlet.SubElement(full_test_elem, "Name")
-        name_elem.text = test_name
+        xmlet.SubElement(full_test_elem, "Name").text = test_name
 
-        path_elem = xmlet.SubElement(full_test_elem, "Path")
-        path_elem.text = test_norm_path
+        xmlet.SubElement(full_test_elem, "Path").text = test_norm_path
 
-        full_name_elem = xmlet.SubElement(full_test_elem, "FullName")
-        full_name_elem.text = test_name
+        xmlet.SubElement(full_test_elem, "FullName").text = test_name
 
         xmlet.SubElement(full_test_elem, "FullCommandLine")
         # text ?
@@ -132,21 +169,40 @@ def create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time
             named_measurement_elem.attrib["type"] = type_attr
             named_measurement_elem.attrib["name"] = name_attr
 
-            value_elem = xmlet.SubElement(named_measurement_elem, "Value")
-            value_elem.text = value
+            xmlet.SubElement(named_measurement_elem, "Value").text = value
 
         measurement_elem = xmlet.SubElement(results_elem, "Measurement")
 
         value_elem = xmlet.SubElement(measurement_elem, "Value")
         value_elem.text = ''.join([item for item in get_test_output(test_norm_path) if ord(item) < 128])
 
-    elapsed_time_elem = xmlet.SubElement(testing_elem, "ElapsedMinutes")
-    elapsed_time_elem.text = "0" # Skip for now
+    xmlet.SubElement(testing_elem, "ElapsedMinutes").text = "0" # Skip for now
 
     etree = xmlet.ElementTree(site_elem)
 
     etree.write(os.path.join(data_rel_path, "Test.xml"))
 
+###############################################################################
+def create_cdash_xml_fakes(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname):
+###############################################################################
+    # We assume all cases were created from the same code repo
+    first_result_case = os.path.dirname(list(results.items())[0][1][0])
+    try:
+        srcroot = run_cmd_no_fail("./xmlquery --value CIMEROOT", from_dir=first_result_case)
+    except:
+        # Use repo containing this script as last resort
+        srcroot = CIME.utils.get_cime_root()
+
+    git_commit = CIME.utils.get_current_commit(repo=srcroot)
+
+    data_rel_path = os.path.join("Testing", utc_time)
+
+    create_cdash_config_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)
+
+    create_cdash_build_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)
+
+    create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname, data_rel_path, git_commit)
+
 ###############################################################################
 def create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_time, hostname, force_log_upload):
 ###############################################################################
@@ -165,7 +221,8 @@ def create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_ti
                 test_case_dir = os.path.dirname(test_path)
                 ts = TestStatus(test_case_dir)
 
-                build_status    = ts.get_status(MODEL_BUILD_PHASE)
+                build_status    = ts.get_status(SHAREDLIB_BUILD_PHASE)
+                build_status    = TEST_FAIL_STATUS if build_status == TEST_FAIL_STATUS else ts.get_status(MODEL_BUILD_PHASE)
                 run_status      = ts.get_status(RUN_PHASE)
                 baseline_status = ts.get_status(BASELINE_PHASE)
 
@@ -276,14 +333,14 @@ def create_cdash_xml(results, cdash_build_name, cdash_project, cdash_build_group
     with open("Testing/TAG", "w") as tag_fd:
         tag_fd.write("{}\n{}\n".format(utc_time, cdash_build_group))
 
-    create_cdash_test_xml(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname)
+    create_cdash_xml_fakes(results, cdash_build_name, cdash_build_group, utc_time, current_time, hostname)
 
     create_cdash_upload_xml(results, cdash_build_name, cdash_build_group, utc_time, hostname, force_log_upload)
 
     run_cmd_no_fail("ctest -VV -D NightlySubmit", verbose=True)
 
 ###############################################################################
-def wait_for_test(test_path, results, wait, check_throughput, check_memory, ignore_namelists, ignore_memleak):
+def wait_for_test(test_path, results, wait, check_throughput, check_memory, ignore_namelists, ignore_memleak, no_run):
 ###############################################################################
     if (os.path.isdir(test_path)):
         test_status_filepath = os.path.join(test_path, TEST_STATUS_FILENAME)
@@ -291,39 +348,57 @@ def wait_for_test(test_path, results, wait, check_throughput, check_memory, igno
         test_status_filepath = test_path
 
     logging.debug("Watching file: '{}'".format(test_status_filepath))
+    test_log_path = os.path.join(os.path.dirname(test_status_filepath), ".internal_test_status.log")
 
-    while (True):
-        if (os.path.exists(test_status_filepath)):
-            ts = TestStatus(test_dir=os.path.dirname(test_status_filepath))
-            test_name = ts.get_name()
-            test_status = ts.get_overall_test_status(wait_for_run=True, # Important
-                                                     check_throughput=check_throughput,
-                                                     check_memory=check_memory, ignore_namelists=ignore_namelists,
-                                                     ignore_memleak=ignore_memleak)
-
-            if (test_status == TEST_PEND_STATUS and (wait and not SIGNAL_RECEIVED)):
-                time.sleep(SLEEP_INTERVAL_SEC)
-                logging.debug("Waiting for test to finish")
-            else:
-                results.put( (test_name, test_path, test_status) )
-                break
+    # We don't want to make it a requirement that wait_for_tests has write access
+    # to all case directories
+    try:
+        fd = open(test_log_path, "w")
+        fd.close()
+    except:
+        test_log_path = "/dev/null"
+
+    prior_ts = None
+    with open(test_log_path, "w") as log_fd:
+        while (True):
+            if (os.path.exists(test_status_filepath)):
+                ts = TestStatus(test_dir=os.path.dirname(test_status_filepath))
+                test_name = ts.get_name()
+                test_status = ts.get_overall_test_status(wait_for_run=not no_run, # Important
+                                                         no_run=no_run,
+                                                         check_throughput=check_throughput,
+                                                         check_memory=check_memory, ignore_namelists=ignore_namelists,
+                                                         ignore_memleak=ignore_memleak)
+
+                if prior_ts is not None and prior_ts != ts:
+                    log_fd.write(ts.phase_statuses_dump())
+                    log_fd.write("OVERALL: {}\n\n".format(test_status))
+
+                prior_ts = ts
+
+                if (test_status == TEST_PEND_STATUS and (wait and not SIGNAL_RECEIVED)):
+                    time.sleep(SLEEP_INTERVAL_SEC)
+                    logging.debug("Waiting for test to finish")
+                else:
+                    results.put( (test_name, test_path, test_status) )
+                    break
 
-        else:
-            if (wait and not SIGNAL_RECEIVED):
-                logging.debug("File '{}' does not yet exist".format(test_status_filepath))
-                time.sleep(SLEEP_INTERVAL_SEC)
             else:
-                test_name = os.path.abspath(test_status_filepath).split("/")[-2]
-                results.put( (test_name, test_path, "File '{}' doesn't exist".format(test_status_filepath)) )
-                break
+                if (wait and not SIGNAL_RECEIVED):
+                    logging.debug("File '{}' does not yet exist".format(test_status_filepath))
+                    time.sleep(SLEEP_INTERVAL_SEC)
+                else:
+                    test_name = os.path.abspath(test_status_filepath).split("/")[-2]
+                    results.put( (test_name, test_path, "File '{}' doesn't exist".format(test_status_filepath)) )
+                    break
 
 ###############################################################################
-def wait_for_tests_impl(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False):
+def wait_for_tests_impl(test_paths, no_wait=False, check_throughput=False, check_memory=False, ignore_namelists=False, ignore_memleak=False, no_run=False):
 ###############################################################################
     results = queue.Queue()
 
     for test_path in test_paths:
-        t = threading.Thread(target=wait_for_test, args=(test_path, results, not no_wait, check_throughput, check_memory, ignore_namelists, ignore_memleak))
+        t = threading.Thread(target=wait_for_test, args=(test_path, results, not no_wait, check_throughput, check_memory, ignore_namelists, ignore_memleak, no_run))
         t.daemon = True
         t.start()
 
@@ -360,14 +435,15 @@ def wait_for_tests(test_paths,
                    cdash_project=E3SM_MAIN_CDASH,
                    cdash_build_group=CDASH_DEFAULT_BUILD_GROUP,
                    timeout=None,
-                   force_log_upload=False):
+                   force_log_upload=False,
+                   no_run=False):
 ###############################################################################
     # Set up signal handling, we want to print results before the program
     # is terminated
     set_up_signal_handlers()
 
     with Timeout(timeout, action=signal_handler):
-        test_results = wait_for_tests_impl(test_paths, no_wait, check_throughput, check_memory, ignore_namelists, ignore_memleak)
+        test_results = wait_for_tests_impl(test_paths, no_wait, check_throughput, check_memory, ignore_namelists, ignore_memleak, no_run)
 
     all_pass = True
     for test_name, test_data in sorted(test_results.items()):
diff --git a/scripts/lib/get_tests.py b/scripts/lib/get_tests.py
--- a/scripts/lib/get_tests.py
+++ b/scripts/lib/get_tests.py
@@ -51,6 +51,9 @@
                     "TESTMEMLEAKPASS_P1.f09_g16.X")
                    ),
 
+    "cime_test_all" : ("cime_test_only", "0:10:00",
+                       ("TESTRUNDIFF_P1.f19_g16_rx1.A", )),
+
     "cime_developer" : (None, "0:15:00",
                             ("NCK_Ld3.f45_g37_rx1.A",
                              "ERI.f09_g16.X",
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -43,15 +43,16 @@
 NO_BATCH    = False
 NO_CMAKE    = False
 TEST_ROOT   = None
+NO_TEARDOWN = False
 
 os.environ["CIME_GLOBAL_WALLTIME"] = "0:05:00"
 
 # pragma pylint: disable=protected-access
 ###############################################################################
-def run_cmd_assert_result(test_obj, cmd, from_dir=None, expected_stat=0, env=None):
+def run_cmd_assert_result(test_obj, cmd, from_dir=None, expected_stat=0, env=None, verbose=False):
 ###############################################################################
     from_dir = os.getcwd() if from_dir is None else from_dir
-    stat, output, errput = run_cmd(cmd, from_dir=from_dir, env=env)
+    stat, output, errput = run_cmd(cmd, from_dir=from_dir, env=env, verbose=verbose)
     if expected_stat == 0:
         expectation = "SHOULD HAVE WORKED, INSTEAD GOT STAT %s" % stat
     else:
@@ -129,10 +130,10 @@ def make_fake_teststatus(path, testname, status, phase):
     with TestStatus(test_dir=path, test_name=testname) as ts:
         for core_phase in CORE_PHASES:
             if core_phase == phase:
-                ts.set_status(core_phase, status)
+                ts.set_status(core_phase, status, comments=("time=42" if phase == RUN_PHASE else ""))
                 break
             else:
-                ts.set_status(core_phase, TEST_PASS_STATUS)
+                ts.set_status(core_phase, TEST_PASS_STATUS, comments=("time=42" if phase == RUN_PHASE else ""))
 
 ###############################################################################
 def parse_test_status(line):
@@ -246,7 +247,7 @@ def test_b_cime_f90_unit_tests(self):
 
     @classmethod
     def tearDownClass(cls):
-        do_teardown = len(cls._do_teardown) > 0 and sys.exc_info() == (None, None, None)
+        do_teardown = len(cls._do_teardown) > 0 and sys.exc_info() == (None, None, None) and not NO_TEARDOWN
 
         teardown_root = True
         for tfile in cls._testdirs:
@@ -655,7 +656,7 @@ def test_k_append_config(self):
 
     @classmethod
     def tearDownClass(cls):
-        do_teardown = len(cls._do_teardown) > 0 and sys.exc_info() == (None, None, None)
+        do_teardown = len(cls._do_teardown) > 0 and sys.exc_info() == (None, None, None) and not NO_TEARDOWN
 
         for tfile in cls._testdirs:
             if tfile not in cls._do_teardown:
@@ -672,18 +673,28 @@ class M_TestWaitForTests(unittest.TestCase):
     def setUp(self):
     ###########################################################################
         self._testroot = os.path.join(TEST_ROOT,"TestWaitForTests")
+
+        # basic tests
         self._testdir_all_pass    = os.path.join(self._testroot, 'scripts_regression_tests.testdir_all_pass')
         self._testdir_with_fail   = os.path.join(self._testroot, 'scripts_regression_tests.testdir_with_fail')
         self._testdir_unfinished  = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished')
         self._testdir_unfinished2 = os.path.join(self._testroot, 'scripts_regression_tests.testdir_unfinished2')
-        testdirs = [self._testdir_all_pass, self._testdir_with_fail, self._testdir_unfinished, self._testdir_unfinished2]
-        for testdir in testdirs:
+
+        # live tests
+        self._testdir_teststatus1 = os.path.join(self._testroot, 'scripts_regression_tests.testdir_teststatus1')
+        self._testdir_teststatus2 = os.path.join(self._testroot, 'scripts_regression_tests.testdir_teststatus2')
+
+        self._testdirs = [self._testdir_all_pass, self._testdir_with_fail, self._testdir_unfinished, self._testdir_unfinished2,
+                          self._testdir_teststatus1, self._testdir_teststatus2]
+        basic_tests = self._testdirs[:self._testdirs.index(self._testdir_teststatus1)]
+
+        for testdir in self._testdirs:
             if os.path.exists(testdir):
                 shutil.rmtree(testdir)
             os.makedirs(testdir)
 
         for r in range(10):
-            for testdir in testdirs:
+            for testdir in basic_tests:
                 os.makedirs(os.path.join(testdir, str(r)))
                 make_fake_teststatus(os.path.join(testdir, str(r)), "Test_%d" % r, TEST_PASS_STATUS, RUN_PHASE)
 
@@ -691,6 +702,11 @@ def setUp(self):
         make_fake_teststatus(os.path.join(self._testdir_unfinished,  "5"), "Test_5", TEST_PEND_STATUS, RUN_PHASE)
         make_fake_teststatus(os.path.join(self._testdir_unfinished2, "5"), "Test_5", TEST_PASS_STATUS, SUBMIT_PHASE)
 
+        integration_tests = self._testdirs[len(basic_tests):]
+        for integration_test in integration_tests:
+            os.makedirs(os.path.join(integration_test, "0"))
+            make_fake_teststatus(os.path.join(integration_test, "0"), "Test_0", TEST_PASS_STATUS, CORE_PHASES[0])
+
         # Set up proxy if possible
         self._unset_proxy = setup_proxy()
 
@@ -699,9 +715,11 @@ def setUp(self):
     ###########################################################################
     def tearDown(self):
     ###########################################################################
-        shutil.rmtree(self._testdir_all_pass)
-        shutil.rmtree(self._testdir_with_fail)
-        shutil.rmtree(self._testdir_unfinished)
+        do_teardown = sys.exc_info() == (None, None, None) and not NO_TEARDOWN
+
+        if do_teardown:
+            for testdir in self._testdirs:
+                shutil.rmtree(testdir)
 
         kill_subprocesses()
 
@@ -720,7 +738,7 @@ def simple_test(self, testdir, expected_results, extra_args="", build_name=None)
                                        from_dir=testdir, expected_stat=expected_stat)
 
         lines = [line for line in output.splitlines() if line.startswith("Test '")]
-        self.assertEqual(len(lines), 10)
+        self.assertEqual(len(lines), len(expected_results))
         for idx, line in enumerate(lines):
             testname, status = parse_test_status(line)
             self.assertEqual(status, expected_results[idx])
@@ -869,10 +887,46 @@ def test_wait_for_test_cdash_kill(self):
             xml_contents = open(xml_file, "r").read()
             self.assertTrue(r'<TestList><Test>Test_0</Test><Test>Test_1</Test><Test>Test_2</Test><Test>Test_3</Test><Test>Test_4</Test><Test>Test_5</Test><Test>Test_6</Test><Test>Test_7</Test><Test>Test_8</Test><Test>Test_9</Test></TestList>'
                             in xml_contents)
-            self.assertTrue(r'<Test Status="failed"><Name>Test_5</Name>' in xml_contents)
+            self.assertTrue(r'<Test Status="notrun"><Name>Test_5</Name>' in xml_contents)
 
             # TODO: Any further checking of xml output worth doing?
 
+    ###########################################################################
+    def live_test_impl(self, testdir, expected_results, last_phase, last_status):
+    ###########################################################################
+        run_thread = threading.Thread(target=self.threaded_test, args=(testdir, expected_results))
+        run_thread.daemon = True
+        run_thread.start()
+
+        time.sleep(5)
+
+        self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited")
+
+        for core_phase in CORE_PHASES[1:]:
+            with TestStatus(test_dir=os.path.join(self._testdir_teststatus1, "0")) as ts:
+                ts.set_status(core_phase, last_status if core_phase == last_phase else TEST_PASS_STATUS)
+
+            time.sleep(5)
+
+            if core_phase != last_phase:
+                self.assertTrue(run_thread.isAlive(), msg="wait_for_tests should have waited after passing phase {}".format(core_phase))
+            else:
+                run_thread.join(timeout=10)
+                self.assertFalse(run_thread.isAlive(), msg="wait_for_tests should have finished after phase {}".format(core_phase))
+                break
+
+        self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
+
+    ###########################################################################
+    def test_wait_for_test_test_status_integration_pass(self):
+    ###########################################################################
+        self.live_test_impl(self._testdir_teststatus1, ["PASS"], RUN_PHASE, TEST_PASS_STATUS)
+
+    ###########################################################################
+    def test_wait_for_test_test_status_integration_submit_fail(self):
+    ###########################################################################
+        self.live_test_impl(self._testdir_teststatus1, ["FAIL"], SUBMIT_PHASE, TEST_FAIL_STATUS)
+
 ###############################################################################
 class TestCreateTestCommon(unittest.TestCase):
 ###############################################################################
@@ -888,7 +942,7 @@ def setUp(self):
         self._baseline_area     = os.path.join(TEST_ROOT, "baselines")
         self._testroot          = TEST_ROOT
         self._hasbatch          = MACHINE.has_batch_system() and not NO_BATCH
-        self._do_teardown       = True # Will never do teardown if test failed
+        self._do_teardown       = not NO_TEARDOWN
 
     ###########################################################################
     def tearDown(self):
@@ -1226,6 +1280,14 @@ def setUp(self):
         # Change root to avoid clashing with other jenkins_generic_jobs
         self._jenkins_root = os.path.join(self._testdir, "J")
 
+    ###########################################################################
+    def tearDown(self):
+    ###########################################################################
+        TestCreateTestCommon.tearDown(self)
+
+        if "TESTRUNDIFF_ALTERNATE" in os.environ:
+            del os.environ["TESTRUNDIFF_ALTERNATE"]
+
     ###########################################################################
     def simple_test(self, expect_works, extra_args, build_name=None):
     ###########################################################################
@@ -1236,7 +1298,7 @@ def simple_test(self, expect_works, extra_args, build_name=None):
         if CIME.utils.get_model() == "e3sm" and build_name is not None:
             extra_args += " -p ACME_test --submit-to-cdash --cdash-build-group=Nightly -c %s" % build_name
 
-        run_cmd_assert_result(self, "%s/jenkins_generic_job -r %s %s" % (TOOLS_DIR, self._testdir, extra_args),
+        run_cmd_assert_result(self, "%s/jenkins_generic_job -r %s %s -B %s" % (TOOLS_DIR, self._testdir, extra_args, self._baseline_area),
                               from_dir=self._testdir, expected_stat=(0 if expect_works else CIME.utils.TESTS_FAILED_ERR_CODE))
 
     ###########################################################################
@@ -1248,9 +1310,9 @@ def threaded_test(self, expect_works, extra_args, build_name=None):
             self._thread_error = str(e)
 
     ###########################################################################
-    def assert_num_leftovers(self):
+    def assert_num_leftovers(self, suite):
     ###########################################################################
-        num_tests_in_tiny = len(get_tests.get_test_suite("cime_test_only_pass"))
+        num_tests_in_tiny = len(get_tests.get_test_suite(suite))
 
         jenkins_dirs = glob.glob("%s/*%s*/" % (self._jenkins_root, self._baseline_name.capitalize())) # case dirs
         # scratch_dirs = glob.glob("%s/*%s*/" % (self._testroot, test_id)) # blr/run dirs
@@ -1267,16 +1329,14 @@ def assert_num_leftovers(self):
     ###########################################################################
     def test_jenkins_generic_job(self):
     ###########################################################################
-        # Unfortunately, this test is very long-running
-
         # Generate fresh baselines so that this test is not impacted by
         # unresolved diffs
         self.simple_test(True, "-t cime_test_only_pass -g -b %s" % self._baseline_name)
-        self.assert_num_leftovers()
+        self.assert_num_leftovers("cime_test_only_pass")
 
         build_name = "jenkins_generic_job_pass_%s" % CIME.utils.get_timestamp()
         self.simple_test(True, "-t cime_test_only_pass -b %s" % self._baseline_name, build_name=build_name)
-        self.assert_num_leftovers() # jenkins_generic_job should have automatically cleaned up leftovers from prior run
+        self.assert_num_leftovers("cime_test_only_pass") # jenkins_generic_job should have automatically cleaned up leftovers from prior run
         assert_dashboard_has_build(self, build_name)
 
     ###########################################################################
@@ -1297,6 +1357,44 @@ def test_jenkins_generic_job_kill(self):
         self.assertTrue(self._thread_error is None, msg="Thread had failure: %s" % self._thread_error)
         assert_dashboard_has_build(self, build_name)
 
+    ###########################################################################
+    def test_jenkins_generic_job_realistic_dash(self):
+    ###########################################################################
+        # The actual quality of the cdash results for this test can only
+        # be inspected manually
+
+        # Generate fresh baselines so that this test is not impacted by
+        # unresolved diffs
+        self.simple_test(False, "-t cime_test_all -g -b %s" % self._baseline_name)
+        self.assert_num_leftovers("cime_test_all")
+
+        # Should create a diff
+        os.environ["TESTRUNDIFF_ALTERNATE"] = "True"
+
+        # Should create a nml diff
+        # Modify namelist
+        fake_nl = """
+ &fake_nml
+   fake_item = 'fake'
+   fake = .true.
+/"""
+        baseline_glob = glob.glob(os.path.join(self._baseline_area, self._baseline_name, "TESTRUNPASS*"))
+        self.assertEqual(len(baseline_glob), 1, msg="Expected one match, got:\n%s" % "\n".join(baseline_glob))
+
+        import stat
+        for baseline_dir in baseline_glob:
+            nl_path = os.path.join(baseline_dir, "CaseDocs", "datm_in")
+            self.assertTrue(os.path.isfile(nl_path), msg="Missing file %s" % nl_path)
+
+            os.chmod(nl_path, stat.S_IRUSR | stat.S_IWUSR)
+            with open(nl_path, "a") as nl_file:
+                nl_file.write(fake_nl)
+
+        build_name = "jenkins_generic_job_mixed_%s" % CIME.utils.get_timestamp()
+        self.simple_test(False, "-t cime_test_all -b %s" % self._baseline_name, build_name=build_name)
+        self.assert_num_leftovers("cime_test_all") # jenkins_generic_job should have automatically cleaned up leftovers from prior run
+        assert_dashboard_has_build(self, build_name)
+
 ###############################################################################
 class T_TestRunRestart(TestCreateTestCommon):
 ###############################################################################
@@ -2878,6 +2976,7 @@ def _main_func(description):
     global TEST_MPILIB
     global TEST_ROOT
     global GLOBAL_TIMEOUT
+    global NO_TEARDOWN
     config = CIME.utils.get_cime_config()
 
     help_str = \
@@ -2911,6 +3010,9 @@ def _main_func(description):
     parser.add_argument("--no-cmake", action="store_true",
                         help="Do not run cmake tests")
 
+    parser.add_argument("--no-teardown", action="store_true",
+                        help="Do not delete directories left behind by testing")
+
     parser.add_argument("--machine",
                         help="Select a specific machine setting for cime")
 
@@ -2935,6 +3037,7 @@ def _main_func(description):
     NO_BATCH       = ns.no_batch
     NO_CMAKE       = ns.no_cmake
     GLOBAL_TIMEOUT = ns.timeout
+    NO_TEARDOWN    = ns.no_teardown
 
     if ns.machine is not None:
         MACHINE = Machines(machine=ns.machine)
@@ -3001,7 +3104,7 @@ def _main_func(description):
             print("Detected failures, leaving directory:", TEST_ROOT)
         else:
             print("All pass, removing directory:", TEST_ROOT)
-            if os.path.exists(TEST_ROOT):
+            if os.path.exists(TEST_ROOT) and not NO_TEARDOWN:
                 shutil.rmtree(TEST_ROOT)
 
         raise