Repository: ESMCI/cime

Issue:
Refactor system test scripts to make it easier to add new tests that compare two runs
Most of our system tests do two runs that differ in some way, then compare them to make sure they are identical. There is a desire - from myself and others - to add many more tests along these lines that tweak small things and ensure that the simulation isn't affected (e.g., turn carbon isotopes on in CLM and ensure that this doesn't change the evolution of the bulk carbon pools).

The new python-based system tests are a big step forward from the old shell scripts - thank you to everyone who worked on that! However, to make the test scripts more maintainable, and to make it easier to add new system tests, I'd suggest even more aggressive removal of duplication between these tests. Here I'm using the LII test as an example. 

There are three main things I'd like to see:
1. Introduce a base class for doing two runs
2. Introduce a pre-build stage, so I can move the namelist setup out of the build stage
   
   The main point would be to avoid needing to check if things have already been done and avoid doing them again. This would also make the whole test flow logic more clear.
3. Move this to some common place:

``` python
        self._case.set_value("CONTINUE_RUN",False)
        self._case.set_value("REST_OPTION","none")
        self._case.set_value("HIST_OPTION","$STOP_OPTION")
        self._case.set_value("HIST_N","$STOP_N")
```

I envision something like the following prototype (which is probably buggy, but hopefully conveys what I envision well enough):

``` python
class SystemTestsCompareTwo(SystemTestsCommon):

    def __init__(self, case):
        SystemTestsCommon.__init__(self, case)
        self._test_suffix = 'test'  # will typically be overridden in derived classes

    def run(self):
        self._setup_first_phase()
        success = SystemTestsCommon._run(self)

        if success:
            self._setup_second_phase()
            success = SystemTestsCommon._run(self, self._test_suffix)

        if success:
            success = self._component_compare_test("base", self._test_suffix)

        return success

    def _setup_first_phase(self):
        pass

    def _setup_second_phase(self):
        pass


class LII(SystemTestsCompareTwo):
    def __init__(self, case):
        SystemTestsCompareTwo.__init__(self, case)
        self._test_suffix = 'init_interp_on'

    # This is a new method that does not currently exist in
    # SystemTestsCommon. For many tests it won't do anything, so the default
    # implementation in SystemTestsCommon will just be 'pass'. However, it can
    # be overridden to do something useful in tests where something is needed
    # prior to the build. The advantages of having this in a separate stage
    # rather than as part of the build are:
    #
    # (a) This is just called once, right after test creation, even if the build
    # is called multiple times - so you don't need logic in here ensuring that
    # it's only done once
    #
    # (b) For tests that need this setup but don't need to override the standard
    # build method, the method signature can remain simpler, and you don't need
    # to explicitly call SystemTestsCommon.build(...)
    #
    # (c) Separating things into when they actually need to be done yields
    # better communication about the actual process of setting up the test
    #
    # I haven't looked through the higher-level scripts to determine where this
    # will be called from.
    def initial_setup(self):
        # Note: If copying user_nl files and making some small tweaks to them
        # for the two different runs is a common operation, then this operation
        # could be moved into the SystemTestsCompareTwo class, and called from
        # each derived class that needs it.

        os.makedirs("user_nl_nointerp")
        for filename in glob.glob(r'user_nl_clm*'):
            shutil.copy(filename, os.path.join("user_nl_nointerp",filename))
            with open(os.path.join("user_nl_nointerp",filename), "a") as newfile:
                newfile.write("use_init_interp = .false.")

        os.makedirs("user_nl_interp")
        for filename in glob.glob(r'user_nl_clm*'):
            shutil.copy(filename, os.path.join("user_nl_interp",filename))
            with open(os.path.join("user_nl_interp",filename), "a") as newfile:
                newfile.write("use_init_interp = .false.")


    def _setup_first_phase(self):
        caseroot = self._case.get_value("CASEROOT")
        for filename in glob.glob(r'user_nl_nointerp/*'):
            shutil.copy(filename, os.path.join(caseroot,filename))

    def _setup_second_phase(self):
        caseroot = self._case.get_value("CASEROOT")
        for filename in glob.glob(r'user_nl_interp/*'):
            shutil.copy(filename, os.path.join(caseroot,filename))

```

The intent here is to make the specific test class (LII above) contain only what is unique to that test. Once that's the case, it will become much easier to create and maintain a wide variety of tests.



Discussion:
A lot of what you suggest has already been done.   For example look at the ers.py and err.py

@jedwards4b : I agree that the new python scripts are a big step forward from the shell scripts in this respect! And I did look at ers.py before writing this issue. What I'm suggesting amounts to minor tweaks on top of what is already in place to remove the remaining duplication. I think that this is especially important because the (small) bits of duplication are somewhat cryptic to someone who doesn't understand the whole set of testing scripts, which means that people (myself included) will inevitably copy and paste this without really understanding it... which in my experience leads to bugs and maintainability problems. There is also a greater chance that a critical line will accidentally be left out.

Specifically, I am suggesting moving the run method:

``` python
    def run(self):
        success = self._ers_first_phase()

        if success:
            return self._ers_second_phase()
        else:
            return False
```

as well as this logic (in _ers_second_phase):

``` python
        # Compare restart file
        if success:
            return self._component_compare_test("base", "rest")
        else:
            return False
```

into a shared location, so it doesn't need to be duplicated for each test.

I believe that this further refactoring would have reduced the chances of introducing one of the LII bugs documented in issue #291 , as well as the ERP bug documented in issue #295.

This issue is a revamped version of https://github.com/CESM-Development/cime/issues/146, taking into account the new (and much easier to work with!) python-based system tests. That older issue had some additional ideas; the main ones were:
1. Allow a test to specify two testmods directories. Then the test system does all the work of setting up and running cases with each of those testmods, and comparing them for equality.
2. Allow tests to have a custom comparison script, so that rather than comparing for equality, you can say something like, "if irrigation is turned on in CLM, global latent heat flux should be higher" (to confirm that irrigation is doing something that is at least correct to first (zeroth?) order).

I feel that those additional things would be nice to have, but not essential.


Resolving patch:
diff --git a/utils/python/CIME/SystemTests/lii.py b/utils/python/CIME/SystemTests/lii.py
--- a/utils/python/CIME/SystemTests/lii.py
+++ b/utils/python/CIME/SystemTests/lii.py
@@ -1,75 +1,35 @@
 """
-Implementation of the CIME LII test.  This class inherits from SystemTestsCommon
+Implementation of the CIME LII test.
 
 This is a CLM specific test:
-Verifies that namelist variable 'use_init_interp' works correctly
+Verifies that interpolation of initial conditions onto an identical
+configuration gives identical results:
 (1) do a run with use_init_interp false (suffix base)
 (2) do a run with use_init_interp true (suffix init_interp_on)
 """
 
-import shutil, glob
+from CIME.SystemTests.system_tests_compare_two import SystemTestsCompareTwo
 from CIME.XML.standard_module_setup import *
-from CIME.SystemTests.system_tests_common import SystemTestsCommon
+from CIME.SystemTests.test_utils.user_nl_utils import append_to_user_nl_files
 
 logger = logging.getLogger(__name__)
 
-class LII(SystemTestsCommon):
+class LII(SystemTestsCompareTwo):
 
     def __init__(self, case):
-        """
-        initialize a test object
-        """
-        SystemTestsCommon.__init__(self, case)
+        SystemTestsCompareTwo.__init__(self, case,
+                                       separate_builds = False,
+                                       run_two_suffix = 'interp',
+                                       run_one_description = 'use_init_interp set to false',
+                                       run_two_description = 'use_init_interp set to true')
+
+    def _case_one_setup(self):
+        append_to_user_nl_files(caseroot = self._get_caseroot(),
+                                component = "clm",
+                                contents = "use_init_interp = .false.")
+
+    def _case_two_setup(self):
+        append_to_user_nl_files(caseroot = self._get_caseroot(),
+                                component = "clm",
+                                contents = "use_init_interp = .true.")
 
-    def build_phase(self, sharedlib_only=False, model_only=False):
-
-        # Make copies of the namelist files for each part of the test. Enclose the
-        # copies in conditionals so that we only do this namelist setup the first time
-        # the build script is invoked - otherwise, if the build is rerun, the namelist
-        # files would build up repeated instances of the setting of force_init_intep.
-        #
-        # Note the use of shell wildcards to make sure we apply these mods to
-        # multi-instance versions
-
-        if not os.path.exists("user_nl_nointerp"):
-            os.makedirs("user_nl_nointerp")
-            for filename in glob.glob(r'user_nl_clm*'):
-                shutil.copy(filename, os.path.join("user_nl_nointerp",filename))
-                with open(os.path.join("user_nl_nointerp",filename), "a") as newfile:
-                    newfile.write("use_init_interp = .false.")
-
-        if not os.path.exists("user_nl_interp"):
-            os.makedirs("user_nl_interp")
-            for filename in glob.glob(r'user_nl_clm*'):
-                shutil.copy(filename, os.path.join("user_nl_interp",filename))
-                with open(os.path.join("user_nl_interp",filename), "a") as newfile:
-                    newfile.write("use_init_interp = .true.")
-
-        self.clean_build()
-        self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
-
-    def run_phase(self):
-        '''
-        Do a run with init_interp false, a run with init_interp true and
-        compare
-        '''
-        caseroot = self._case.get_value("CASEROOT")
-
-        self._case.set_value("CONTINUE_RUN",False)
-        self._case.set_value("REST_OPTION","none")
-        self._case.set_value("HIST_OPTION","$STOP_OPTION")
-        self._case.set_value("HIST_N","$STOP_N")
-        self._case.flush()
-        for user_nl_dir in ("nointerp", "interp"):
-            for filename in glob.glob(r'user_nl_%s/*'%user_nl_dir):
-                shutil.copy(filename,
-                            os.path.join(caseroot,os.path.basename(filename)))
-
-            stop_n = self._case.get_value("STOP_N")
-            stop_option = self._case.get_value("STOP_OPTION")
-            logger.info("doing a %d %s initial test with init_interp set to %s, no restarts written"
-                        % (stop_n, stop_option, user_nl_dir == "interp"))
-
-            self.run_indv(suffix=user_nl_dir)
-
-        self._component_compare_test("nointerp", "interp")
diff --git a/utils/python/CIME/SystemTests/pet.py b/utils/python/CIME/SystemTests/pet.py
--- a/utils/python/CIME/SystemTests/pet.py
+++ b/utils/python/CIME/SystemTests/pet.py
@@ -6,63 +6,39 @@
 (2) do another initial run with nthrds=1 for all components (suffix: single_thread)
 """
 
-import shutil
 from CIME.XML.standard_module_setup import *
 from CIME.case_setup import case_setup
-from CIME.SystemTests.system_tests_common import SystemTestsCommon
+from CIME.SystemTests.system_tests_compare_two import SystemTestsCompareTwo
 
 logger = logging.getLogger(__name__)
 
-class PET(SystemTestsCommon):
+class PET(SystemTestsCompareTwo):
+
+    _COMPONENT_LIST = ('ATM','CPL','OCN','WAV','GLC','ICE','ROF','LND')
 
     def __init__(self, case):
         """
         initialize a test object
         """
-        SystemTestsCommon.__init__(self, case)
+        SystemTestsCompareTwo.__init__(self, case,
+                                       separate_builds = False,
+                                       run_two_suffix = 'single_thread',
+                                       run_one_description = 'default threading',
+                                       run_two_description = 'threads set to 1')
 
-    def build_phase(self, sharedlib_only=False, model_only=False):
+    def _case_one_setup(self):
         # first make sure that all components have threaded settings
-        for comp in ['ATM','CPL','OCN','WAV','GLC','ICE','ROF','LND']:
+        for comp in self._COMPONENT_LIST:
             if self._case.get_value("NTHRDS_%s"%comp) <= 1:
                 self._case.set_value("NTHRDS_%s"%comp, 2)
-        self._case.flush()
 
+        # Need to redo case_setup because we may have changed the number of threads
         case_setup(self._case, reset=True)
 
-        self.clean_build()
-        self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
-
-    def _pet_first_phase(self):
-        #Do a run with default threading
-        self._case.set_value("CONTINUE_RUN",False)
-        self._case.set_value("REST_OPTION","none")
-        self._case.set_value("HIST_OPTION","$STOP_OPTION")
-        self._case.set_value("HIST_N","$STOP_N")
-        self._case.flush()
-
-        stop_n = self._case.get_value("STOP_N")
-        stop_option = self._case.get_value("STOP_OPTION")
-        logger.info("doing a %d %s initial test with default threading, no restarts written"
-                    % (stop_n, stop_option))
-
-        self.run_indv()
-
-    def _pet_second_phase(self):
+    def _case_two_setup(self):
         #Do a run with all threads set to 1
-        for comp in ['ATM','CPL','OCN','WAV','GLC','ICE','ROF','LND']:
+        for comp in self._COMPONENT_LIST:
             self._case.set_value("NTHRDS_%s"%comp, 1)
-        self._case.flush()
-        shutil.copy("env_mach_pes.xml", os.path.join("LockedFiles","env_mach_pes.xml"))
-
-        stop_n = self._case.get_value("STOP_N")
-        stop_option = self._case.get_value("STOP_OPTION")
-        logger.info("doing a %d %s initial test with threads set to 1, no restarts written"
-                    % (stop_n, stop_option))
 
-        self.run_indv(suffix="single_thread")
-        self._component_compare_test("base", "single_thread")
-
-    def run_phase(self):
-        self._pet_first_phase()
-        self._pet_second_phase()
+        # Need to redo case_setup because we may have changed the number of threads
+        case_setup(self._case, reset=True)
diff --git a/utils/python/CIME/SystemTests/rep.py b/utils/python/CIME/SystemTests/rep.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/SystemTests/rep.py
@@ -0,0 +1,22 @@
+"""
+Implementation of the CIME REP test
+
+This test verifies that two identical runs give bit-for-bit results
+"""
+
+from CIME.SystemTests.system_tests_compare_two import SystemTestsCompareTwo
+
+class REP(SystemTestsCompareTwo):
+
+    def __init__(self, case):
+        SystemTestsCompareTwo.__init__(self, case,
+                                       separate_builds = False,
+                                       run_two_suffix = 'rep2')
+
+    def _case_one_setup(self):
+        pass
+
+    def _case_two_setup(self):
+        pass
+
+


Test patch:
diff --git a/cime_config/config_tests.xml b/cime_config/config_tests.xml
--- a/cime_config/config_tests.xml
+++ b/cime_config/config_tests.xml
@@ -4,13 +4,15 @@
 
 The following are the test functionality categories:
   1) smoke tests
-  2) restart tests
-  3) threading/pe-count modification tests
-  4) sequencing (layout) modification tests
-  5) multi-instance tests
-  6) archiving (short-term and long-term) tests
-  7) performance tests
-  8) spinup tests
+  2) basic reproducibility tests
+  3) restart tests
+  4) threading/pe-count modification tests
+  5) sequencing (layout) modification tests
+  6) multi-instance tests
+  7) archiving (short-term and long-term) tests
+  8) performance tests
+  9) spinup tests
+  10) other component-specific tests
 
 NOTES:
 - unless otherwise noted everything is run in one executable directory
@@ -29,6 +31,12 @@ SMS    smoke startup test (default length)
 
 SBN    smoke build-namelist test (just run preview_namelist and check_input_data)
 
+======================================================================
+    Basic reproducibility Tests
+======================================================================
+
+REP    reproducibility: do two identical runs give the same results?
+
 ======================================================================
     Restart Tests
 ======================================================================
@@ -143,6 +151,13 @@ SPO    smoke spinup-ocean test
 
 LAR    long term archive test
 
+======================================================================
+    Other component-specific tests
+======================================================================
+
+LII    CLM initial condition interpolation test
+
+
 -->
 
 <config_test>
@@ -323,8 +338,11 @@ LAR    long term archive test
     <DESC>CLM initial condition interpolation test (requires configuration with non-blank finidat)</DESC>
     <INFO_DBUG>1</INFO_DBUG>
     <CCSM_TCOST>0</CCSM_TCOST>
-    <REST_OPTION>none</REST_OPTION>
     <DOUT_S>FALSE</DOUT_S>
+    <CONTINUE_RUN>FALSE</CONTINUE_RUN>
+    <REST_OPTION>never</REST_OPTION>
+    <HIST_OPTION>$STOP_OPTION</HIST_OPTION>
+    <HIST_N>$STOP_N</HIST_N>
   </test>
 
   <test NAME="PEA">
@@ -349,7 +367,10 @@ LAR    long term archive test
     <INFO_DBUG>1</INFO_DBUG>
     <BFBFLAG>TRUE</BFBFLAG>
     <BUILD_THREADED>TRUE</BUILD_THREADED>
-    <REST_OPTION>never</REST_OPTION>
+    <CONTINUE_RUN>FALSE</CONTINUE_RUN>
+    <REST_OPTION>none</REST_OPTION>
+    <HIST_OPTION>$STOP_OPTION</HIST_OPTION>
+    <HIST_N>$STOP_N</HIST_N>
     <CCSM_TCOST>1</CCSM_TCOST>
     <DOUT_S>FALSE</DOUT_S>
   </test>
@@ -397,6 +418,17 @@ LAR    long term archive test
     <DOUT_S>FALSE</DOUT_S>
   </test>
 
+  <test NAME="REP">
+    <DESC>reproducibility test: do two runs give the same answers?</DESC>
+    <CCSM_TCOST>0</CCSM_TCOST>
+    <INFO_DBUG>1</INFO_DBUG>
+    <DOUT_S>FALSE</DOUT_S>
+    <CONTINUE_RUN>FALSE</CONTINUE_RUN>
+    <REST_OPTION>never</REST_OPTION>
+    <HIST_OPTION>$STOP_OPTION</HIST_OPTION>
+    <HIST_N>$STOP_N</HIST_N>
+  </test>
+
   <test NAME="SBN">
     <DESC>smoke build-namelist test (just run preview_namelist and check_input_data)</DESC>
     <INFO_DBUG>1</INFO_DBUG>
diff --git a/utils/python/CIME/SystemTests/system_tests_common.py b/utils/python/CIME/SystemTests/system_tests_common.py
--- a/utils/python/CIME/SystemTests/system_tests_common.py
+++ b/utils/python/CIME/SystemTests/system_tests_common.py
@@ -30,9 +30,23 @@ def __init__(self, case, expected=None):
         self._casebaseid = self._case.get_value("CASEBASEID")
         self._test_status = TestStatus(test_dir=caseroot, test_name=self._casebaseid)
 
+        self._init_environment(caseroot)
+        self._init_locked_files(caseroot, expected)
+        self._init_case_setup()
+
+    def _init_environment(self, caseroot):
+        """
+        Do initializations of environment variables that are needed in __init__
+        """
         # Needed for sh scripts
         os.environ["CASEROOT"] = caseroot
 
+    def _init_locked_files(self, caseroot, expected):
+        """
+        If the file LockedFiles/env_run.orig.xml does not exist, copy the current
+        env_run.xml file. If it does exist, restore values changed in a previous
+        run of the test.
+        """
         if os.path.isfile(os.path.join(caseroot, "LockedFiles", "env_run.orig.xml")):
             self.compare_env_run(expected=expected)
         elif os.path.isfile(os.path.join(caseroot, "env_run.xml")):
@@ -44,6 +58,10 @@ def __init__(self, case, expected=None):
             shutil.copy(os.path.join(caseroot,"env_run.xml"),
                         os.path.join(lockedfiles, "env_run.orig.xml"))
 
+    def _init_case_setup(self):
+        """
+        Do initial case setup needed in __init__
+        """
         if self._case.get_value("IS_FIRST_RUN"):
             self._case.set_initial_test_values()
 
@@ -149,6 +167,12 @@ def run_phase(self):
         """
         self.run_indv()
 
+    def _get_caseroot(self):
+        """
+        Returns the current CASEROOT value
+        """
+        return self._caseroot
+
     def _set_active_case(self, case):
         """
         Use for tests that have multiple cases
diff --git a/utils/python/CIME/SystemTests/system_tests_compare_two.py b/utils/python/CIME/SystemTests/system_tests_compare_two.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/SystemTests/system_tests_compare_two.py
@@ -0,0 +1,362 @@
+"""
+Base class for CIME system tests that involve doing two runs and comparing their
+output.
+
+In the __init__ method for your test, you MUST call
+    SystemTestsCompareTwo.__init__
+See the documentation of that method for details.
+
+Classes that inherit from this are REQUIRED to implement the following methods:
+
+(1) _case_one_setup
+    This method will be called to set up case 1, the "base" case
+
+(2) _case_two_setup
+    This method will be called to set up case 2, the "test" case
+
+In addition, they MAY require the following method:
+
+(1) _common_setup
+    This method will be called to set up both cases. It should contain any setup
+    that's needed in both cases. This is called before _case_one_setup or
+    _case_two_setup.
+
+NOTE: Currently assumes that only one build is needed - i.e., there are no
+build-time settings that need to be changed between the two cases.
+"""
+
+from CIME.XML.standard_module_setup import *
+from CIME.SystemTests.system_tests_common import SystemTestsCommon
+from CIME.case import Case
+
+import shutil, os, glob
+
+logger = logging.getLogger(__name__)
+
+class SystemTestsCompareTwo(SystemTestsCommon):
+
+    def __init__(self,
+                 case,
+                 separate_builds,
+                 run_two_suffix = 'test',
+                 run_one_description = '',
+                 run_two_description = ''):
+        """
+        Initialize a SystemTestsCompareTwo object. Individual test cases that
+        inherit from SystemTestsCompareTwo MUST call this __init__ method.
+
+        Args:
+            case: case object passsed to __init__ method of individual
+                test. This is the main case associated with the test.
+            separate_builds (bool): Whether separate builds are needed for the
+                two cases. If False, case2 uses the case1 executable.
+            run_two_suffix (str, optional): Suffix appended to the case name for
+                the second run. Defaults to 'test'. This can be anything other
+                than 'base'.
+            run_one_description (str, optional): Description printed to log file
+                when starting the first run. Defaults to ''.
+            run_two_description (str, optional): Description printed to log file
+                when starting the second run. Defaults to ''.
+        """
+        SystemTestsCommon.__init__(self, case)
+
+        self._separate_builds = separate_builds
+        expect(not self._separate_builds,
+               "ERROR: separate builds not yet implemented")
+
+        # run_one_suffix is just used as the suffix for the netcdf files
+        # produced by the first case; we may eventually remove this, but for now
+        # it is needed by the various component_*.sh scripts. run_two_suffix is
+        # also used as the suffix for netcdf files, but more importantly is used
+        # to create the case name for the clone case.
+        #
+        # NOTE(wjs, 2016-08-03) It is currently CRITICAL for run_one_suffix to
+        # be 'base', because this is assumed for baseline comparison and
+        # generation. Once that assumption is relaxed, then run_one_suffix can
+        # be set in the call to the constructor just like run_two_suffix
+        # currently is. Or, if these tools are rewritten to work without any
+        # suffix, then run_one_suffix can be removed entirely.
+        self._run_one_suffix = 'base'
+        self._run_two_suffix = run_two_suffix.rstrip()
+        expect(self._run_two_suffix != self._run_one_suffix,
+               "ERROR: Must have different suffixes for run one and run two")
+
+        self._run_one_description = run_one_description
+        self._run_two_description = run_two_description
+
+        # Save case for first run so we can return to it if we switch self._case
+        # to point to self._case2
+        self._case1 = self._case
+        self._caseroot1 = self._get_caseroot()
+
+        self._caseroot2 = self._get_caseroot2()
+        # Initialize self._case2; it will get set to its true value in
+        # _setup_cases_if_not_yet_done
+        self._case2 = None
+
+        self._setup_cases_if_not_yet_done()
+
+    # ========================================================================
+    # Methods that MUST be implemented by specific tests that inherit from this
+    # base class
+    # ========================================================================
+
+    def _case_one_setup(self):
+        """
+        This method will be called to set up case 1, the "base" case.
+
+        This should be written to refer to self._case: this object will point to
+        case1 at the point that this is called.
+        """
+        raise NotImplementedError
+
+    def _case_two_setup(self):
+        """
+        This method will be called to set up case 2, the "test" case
+
+        This should be written to refer to self._case: this object will point to
+        case2 at the point that this is called.
+        """
+        raise NotImplementedError
+
+    # ========================================================================
+    # Methods that MAY be implemented by specific tests that inherit from this
+    # base class, if they have any work to do in these methods
+    # ========================================================================
+
+    def _common_setup(self):
+        """
+        This method will be called to set up both cases. It should contain any setup
+        that's needed in both cases. This is called before _case_one_setup or
+        _case_two_setup.
+
+        This should be written to refer to self._case: It will be called once with
+        self._case pointing to case1, and once with self._case pointing to case2.
+        """
+        pass
+
+    # ========================================================================
+    # Main public methods
+    # ========================================================================
+
+    def build_phase(self, sharedlib_only=False, model_only=False):
+        # TODO(wjs, 2016-08-05) This currently assumes that the two cases use
+        # the same build. Once we relax that assumption, we'll need a
+        # conditional here: If the two cases use the same build (based on
+        # self._separate_builds), then use the below logic; otherwise, do two
+        # builds.
+        self._activate_case1()
+        self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
+
+        # The following is needed when _case_two_setup has a case_setup call
+        # despite sharing the build (e.g., to change NTHRDS)
+        self._case2.set_value("BUILD_COMPLETE",True)
+        self._case2.flush()
+
+    def run_phase(self):
+        """
+        Runs both phases of the two-phase test and compares their results
+        """
+
+        # First run
+        logger.info('Doing first run: ' + self._run_one_description)
+        self._activate_case1()
+        self.run_indv(suffix = self._run_one_suffix)
+
+        # Second run
+        logger.info('Doing second run: ' + self._run_two_description)
+        self._activate_case2()
+        self._force_case2_settings()
+        self.run_indv(suffix = self._run_two_suffix)
+
+        # Compare results
+        # Case1 is the "main" case, and we need to do the comparisons from there
+        self._activate_case1()
+        self._link_to_case2_output()
+
+        self._component_compare_test(self._run_one_suffix, self._run_two_suffix)
+
+    # ========================================================================
+    # Private methods
+    # ========================================================================
+
+    def _get_caseroot2(self):
+        """
+        Determines and returns caseroot for case2
+
+        Assumes that self._case1 is already set to point to the case1 object,
+        and that self._run_two_suffix is already set.
+        """
+        casename1 = self._case1.get_value("CASE")
+        caseroot1 = self._case1.get_value("CASEROOT")
+
+        casename2 = "%s.%s"%(casename1, self._run_two_suffix)
+
+        # Nest the case directory for case2 inside the case directory for case1
+        caseroot2 = os.path.join(caseroot1, casename2)
+
+        return caseroot2
+
+    def _setup_cases_if_not_yet_done(self):
+        """
+        Determines if case2 already exists on disk. If it does, this method
+        creates the self._case2 object pointing to the case directory. If it
+        doesn't exist, then this method creates case2 as a clone of case1, and
+        sets the self._case2 object appropriately.
+
+        This also does the setup for both case1 and case2.
+
+        Assumes that the following variables are already set in self:
+            _caseroot1
+            _caseroot2
+            _case1
+
+        Sets self._case2
+        """
+
+        # Use the existence of the case2 directory to signal whether we have
+        # done the necessary test setup for this test: When we initially create
+        # the case2 directory, we set up both test cases; then, if we find that
+        # the case2 directory already exists, we assume that the setup has
+        # already been done. (In some cases it could be problematic to redo the
+        # test setup when it's not needed - e.g., by appending things to user_nl
+        # files multiple times. This is why we want to make sure to just do the
+        # test setup once.)
+
+        if os.path.exists(self._caseroot2):
+            self._case2 = self._case_from_existing_caseroot(self._caseroot2)
+        else:
+            try:
+                # TODO(wjs, 2016-08-05) For now, we're hard-coding keepexe=True; in
+                # the future, make this based on self._separate_builds.
+                self._case2 = self._case1.create_clone(
+                    newcase = self._caseroot2,
+                    keepexe = True)
+                self._setup_cases()
+            except:
+                # If a problem occurred in setting up the test cases, it's
+                # important to remove the case2 directory: If it's kept around,
+                # that would signal that test setup was done successfully, and
+                # thus doesn't need to be redone - which is not the case. Of
+                # course, we'll likely be left in an inconsistent state in this
+                # case, but if we didn't remove the case2 directory, the next
+                # re-build of the test would think, "okay, setup is done, I can
+                # move on to the build", which would be wrong.
+                shutil.rmtree(self._caseroot2)
+                self._activate_case1()
+                logger.warning("WARNING: Test case setup failed. Case2 has been removed, "
+                               "but the main case may be in an inconsistent state. "
+                               "If you want to rerun this test, you should create "
+                               "a new test rather than trying to rerun this one.")
+                raise
+
+    def _case_from_existing_caseroot(self, caseroot):
+        """
+        Returns a Case object from an existing caseroot directory
+
+        Args:
+            caseroot (str): path to existing caseroot
+        """
+        return Case(case_root=caseroot, read_only=False)
+
+    def _activate_case1(self):
+        """
+        Make case 1 active for upcoming calls
+        """
+        os.chdir(self._caseroot1)
+        self._set_active_case(self._case1)
+
+    def _activate_case2(self):
+        """
+        Make case 2 active for upcoming calls
+        """
+        os.chdir(self._caseroot2)
+        self._set_active_case(self._case2)
+
+    def _setup_cases(self):
+        """
+        Does all test-specific set up for the two test cases.
+        """
+
+        # Set up case 1
+        self._activate_case1()
+        self._common_setup()
+        self._case_one_setup()
+        # Flush the case so that, if errors occur later, then at least case 1 is
+        # in a correct, post-setup state. This is important because the mere
+        # existence of a case 2 directory signals that setup is done. So if the
+        # build fails and the user rebuilds, setup won't be redone - so it's
+        # important to ensure that the results of setup are flushed to disk.
+        #
+        # Note that case 1 will be in its post-setup state even if case 2 setup
+        # fails. Putting the case1 flush after case 2 setup doesn't seem to help
+        # with that (presumably some flush is called automatically), and anyway
+        # wouldn't help with things like appending to user_nl files (which don't
+        # rely on flush). So we just have to live with that possibility (but
+        # note that we print a warning to the log file if that happens, in the
+        # caller of this method).
+        self._case.flush()
+
+        # Set up case 2
+        self._activate_case2()
+        self._common_setup()
+        self._case_two_setup()
+        # Flush the case so that, if errors occur later, then at least case2 is
+        # in a correct, post-setup state
+        self._case.flush()
+
+        # Go back to case 1 to ensure that's where we are for any following code
+        self._activate_case1()
+
+    def _force_case2_settings(self):
+        """
+        Sets some settings in case2 that are normally set automatically.
+
+        This is needed because we aren't running case2 via the normal mechanism
+        (i.e., via the submit script).
+        """
+
+        # RUN_WITH_SUBMIT is normally set when you run the case's submit script.
+        # Trick the scripts into thinking that we are running via the submit
+        # script, like we're supposed to
+        self._case2.set_value("RUN_WITH_SUBMIT",True)
+
+    def _link_to_case2_output(self):
+        """
+        Looks for all files in rundir2 matching the pattern casename2*.nc.run2suffix
+
+        For each file found, makes a link in rundir1 pointing to this file; the
+        link is renamed so that the original occurrence of casename2 is replaced
+        with casename1.
+
+        For example:
+
+        /glade/scratch/sacks/somecase/run/somecase.clm2.h0.nc.run2 ->
+        /glade/scratch/sacks/somecase.run2/run/somecase.run2.clm2.h0.nc.run2
+
+        If the destination link already exists and points to the correct
+        location, it is maintained as is. However, an exception will be raised
+        if the destination link is not exactly as it should be: we avoid
+        overwriting some existing file or link.
+        """
+
+        casename1 = self._case1.get_value("CASE")
+        casename2 = self._case2.get_value("CASE")
+        rundir1 = self._case1.get_value("RUNDIR")
+        rundir2 = self._case2.get_value("RUNDIR")
+        run2suffix = self._run_two_suffix
+
+        pattern = '%s*.nc.%s'%(casename2, run2suffix)
+        case2_files = glob.glob(os.path.join(rundir2, pattern))
+        for one_file in case2_files:
+            file_basename = os.path.basename(one_file)
+            modified_basename = file_basename.replace(casename2, casename1, 1)
+            one_link = os.path.join(rundir1, modified_basename)
+            if (os.path.islink(one_link) and
+                os.readlink(one_link) == one_file):
+                # Link is already set up correctly: do nothing
+                # (os.symlink raises an exception if you try to replace an
+                # existing file)
+                pass
+            else:
+                os.symlink(one_file, one_link)
diff --git a/utils/python/CIME/SystemTests/test_utils/__init__.py b/utils/python/CIME/SystemTests/test_utils/__init__.py
new file mode 100644
diff --git a/utils/python/CIME/SystemTests/test_utils/user_nl_utils.py b/utils/python/CIME/SystemTests/test_utils/user_nl_utils.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/SystemTests/test_utils/user_nl_utils.py
@@ -0,0 +1,52 @@
+"""
+This module contains functions for working with user_nl files in system tests.
+"""
+
+import shutil
+import os
+import glob
+
+def append_to_user_nl_files(caseroot, component, contents):
+    """
+    Append the string given by 'contents' to the end of each user_nl file for
+    the given component (there may be multiple such user_nl files in the case of
+    a multi-instance test).
+
+    Also puts new lines before and after the appended text - so 'contents'
+    does not need to contain a trailing new line (but it's also okay if it
+    does).
+
+    Args:
+        caseroot (str): Full path to the case directory
+
+        component (str): Name of component (e.g., 'clm'). This is used to
+            determine which user_nl files are appended to. For example, for
+            component='clm', this function will operate on all user_nl files
+            matching the pattern 'user_nl_clm*'. (We do a wildcard match to
+            handle multi-instance tests.)
+
+        contents (str): Contents to append to the end of each user_nl file
+    """
+
+    files = _get_list_of_user_nl_files(caseroot, component)
+
+    if len(files) == 0:
+        raise RuntimeError('No user_nl files found for component ' + component)
+
+    for one_file in files:
+        with open(one_file, 'a') as user_nl_file:
+            user_nl_file.write('\n' + contents + '\n')
+
+def _get_list_of_user_nl_files(path, component):
+    """Get a list of all user_nl files in the current path for the component
+    of interest. For a component 'foo', we match all files of the form
+    user_nl_foo* - with a wildcard match at the end in order to match files
+    in a multi-instance case.
+
+    The list of returned files gives their full path.
+    """
+
+    file_pattern = 'user_nl_' + component + '*'
+    file_list = glob.glob(os.path.join(path, file_pattern))
+
+    return file_list
diff --git a/utils/python/CIME/tests/README b/utils/python/CIME/tests/README
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/README
@@ -0,0 +1,6 @@
+The directory structure of this tests directory mirrors the directory structure
+of the parent (production code) directory.
+
+So, for example, unit tests for CIME/foo.py should live in
+CIME/tests/test_foo.py, and unit tests for CIME/SystemTests/bar.py should live
+in CIME/tests/SystemTests/test_bar.py.
diff --git a/utils/python/CIME/tests/SystemTests/__init__.py b/utils/python/CIME/tests/SystemTests/__init__.py
new file mode 100644
diff --git a/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two.py b/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two.py
@@ -0,0 +1,384 @@
+#!/usr/bin/env python
+
+"""
+This module contains unit tests of the core logic in SystemTestsCompareTwo.
+"""
+
+import unittest
+from collections import namedtuple
+import os
+import shutil
+import tempfile
+
+from CIME.SystemTests.system_tests_compare_two import SystemTestsCompareTwo
+import CIME.test_status as test_status
+from CIME.tests.case_fake import CaseFake
+
+# ========================================================================
+# Structure for storing information about calls made to methods
+# ========================================================================
+
+"""
+You can create a Call object to record a single call made to a method:
+
+Call(method, arguments)
+    method (str): name of method
+    arguments (dict): dictionary mapping argument names to values
+
+Example:
+    If you want to record a call to foo(bar = 1, baz = 2):
+        somecall = Call(method = 'foo', arguments = {'bar': 1, 'baz': 2})
+    Or simply:
+        somecall = Call('foo', {'bar': 1, 'baz': 2})
+"""
+Call = namedtuple('Call', ['method', 'arguments'])
+
+# ========================================================================
+# Names of methods for which we want to record calls
+# ========================================================================
+
+# We use constants for these method names because, in some cases, a typo in a
+# hard-coded string could cause a test to always pass, which would be a Bad
+# Thing.
+#
+# For now the names of the constants match the strings they equate to, which
+# match the actual method names. But it's fine if this doesn't remain the case
+# moving forward (which is another reason to use constants rather than
+# hard-coded strings in the tests).
+
+METHOD_component_compare_test = "_component_compare_test"
+METHOD_link_to_case2_output = "_link_to_case2_output"
+METHOD_run_indv = "_run_indv"
+
+# ========================================================================
+# Fake version of SystemTestsCompareTwo that overrides some functionality for
+# the sake of unit testing
+# ========================================================================
+
+# A SystemTestsCompareTwoFake object can be controlled to fail at a given
+# point. See the documentation in its __init__ method for details.
+#
+# It logs what stubbed-out methods have been called in its log attribute; this
+# is a list of Call objects (see above for their definition).
+
+class SystemTestsCompareTwoFake(SystemTestsCompareTwo):
+    def __init__(self,
+                 case1,
+                 run_one_suffix = 'base',
+                 run_two_suffix = 'test',
+                 case2setup_raises_exception = False,
+                 run_one_should_pass = True,
+                 run_two_should_pass = True):
+        """
+        Initialize a SystemTestsCompareTwoFake object
+
+        Args:
+            case1 (CaseFake): existing case
+            run_one_suffix (str, optional): Suffix used for first run. Defaults
+                to 'base'. Currently MUST be 'base'.
+            run_two_suffix (str, optional): Suffix used for the second run. Defaults to 'test'.
+            case2setup_raises_exception (bool, optional): If True, then the call
+                to _case_two_setup will raise an exception. Default is False.
+            run_one_should_pass (bool, optional): Whether the run_indv method should
+                pass for the first run. Default is True, meaning it will pass.
+            run_two_should_pass (bool, optional): Whether the run_indv method should
+                pass for the second run. Default is True, meaning it will pass.
+        """
+
+        self._case2setup_raises_exception = case2setup_raises_exception
+
+        # NOTE(wjs, 2016-08-03) Currently, due to limitations in the test
+        # infrastructure, run_one_suffix MUST be 'base'. However, I'm keeping it
+        # as an explicit argument to the constructor so that it's easy to relax
+        # this requirement later: To relax this assumption, remove the following
+        # assertion and add run_one_suffix as an argument to
+        # SystemTestsCompareTwo.__init__
+        assert(run_one_suffix == 'base')
+
+        SystemTestsCompareTwo.__init__(
+            self,
+            case1,
+            separate_builds = False,
+            run_two_suffix = run_two_suffix)
+
+        # Need to tell test status that case has been built, since this is
+        # checked in the run call
+        with self._test_status:
+            self._test_status.set_status(
+                test_status.MODEL_BUILD_PHASE,
+                test_status.TEST_PASS_STATUS)
+
+        self.run_pass_casenames = []
+        if run_one_should_pass:
+            self.run_pass_casenames.append(self._case1.get_value('CASE'))
+        if run_two_should_pass:
+            self.run_pass_casenames.append(self._case2.get_value('CASE'))
+
+        self.log = []
+
+    # ------------------------------------------------------------------------
+    # Stubs of methods called by SystemTestsCommon.__init__ that interact with
+    # the system or case object in ways we want to avoid here
+    # ------------------------------------------------------------------------
+
+    def _init_environment(self, caseroot):
+        pass
+
+    def _init_locked_files(self, caseroot, expected):
+        pass
+
+    def _init_case_setup(self):
+        pass
+
+    # ------------------------------------------------------------------------
+    # Fake implementations of methods that are typically provided by
+    # SystemTestsCommon
+    # ------------------------------------------------------------------------
+
+    def run_indv(self, suffix="base"):
+        """
+        This fake implementation appends to the log and raises an exception if
+        it's supposed to
+
+        Note that the Call object appended to the log has the current CASE name
+        in addition to the method arguments. (This is mainly to ensure that the
+        proper suffix is used for the proper case, but this extra check can be
+        removed if it's a maintenance problem.)
+        """
+        casename = self._case.get_value('CASE')
+        self.log.append(Call(METHOD_run_indv,
+                             {'suffix': suffix, 'CASE': casename}))
+
+        # Determine whether we should raise an exception
+        #
+        # It's important that this check be based on some attribute of the
+        # self._case object, to ensure that the right case has been activated
+        # for this call to run_indv (e.g., to catch if we forgot to activate
+        # case2 before the second call to run_indv).
+        if casename not in self.run_pass_casenames:
+            raise RuntimeError('casename not in run_pass_casenames')
+
+    def _component_compare_test(self, suffix1, suffix2):
+        # Trying to use the real version of _component_compare_test would pull
+        # too much baggage into these tests. Since the return value from this
+        # method isn't important, it's sufficient for the tests of this class to
+        # just ensure that _component_compare_test was actually called
+        # correctly.
+        #
+        # An alternative would be to extract the main work of
+        # _component_compare_test into a different method that returns a True
+        # (success) / False (failure) result, with _component_compare_test then
+        # updating test_status appropriately. Then we could override that new
+        # method in this Fake class, using the true implementation of
+        # _component_compare_test. Then the test verification would include
+        # verification that TestStatus is set correctly for the COMPARE
+        # phase. But that seems more about testing _component_compare_test than
+        # testing SystemTestsCompareTwo itself, so I don't see much added value
+        # of that.
+
+        self.log.append(Call(METHOD_component_compare_test,
+                             {'suffix1': suffix1, 'suffix2': suffix2}))
+
+    def _check_for_memleak(self):
+        pass
+
+    # ------------------------------------------------------------------------
+    # Fake implementations of methods that are typically provided by
+    # SystemTestsCompareTwo
+    #
+    # Since we're overriding these, their functionality is untested here!
+    # (Though note that _link_to_case2_output is tested elsewhere.)
+    # ------------------------------------------------------------------------
+
+    def _case_from_existing_caseroot(self, caseroot):
+        """
+        Returns a CaseFake object instead of a Case object
+        """
+        return CaseFake(caseroot, create_case_root=False)
+
+    def _link_to_case2_output(self):
+        self.log.append(Call(METHOD_link_to_case2_output, {}))
+
+    # ------------------------------------------------------------------------
+    # Fake implementations of methods that are typically provided by the
+    # individual test
+    #
+    # The values set here are asserted against in some unit tests
+    # ------------------------------------------------------------------------
+
+    def _common_setup(self):
+        self._case.set_value('var_set_in_common_setup', 'common_val')
+
+    def _case_one_setup(self):
+        self._case.set_value('var_set_in_setup', 'case1val')
+
+    def _case_two_setup(self):
+        self._case.set_value('var_set_in_setup', 'case2val')
+        if self._case2setup_raises_exception:
+            raise RuntimeError
+
+# ========================================================================
+# Test class itself
+# ========================================================================
+
+class TestSystemTestsCompareTwo(unittest.TestCase):
+
+    def setUp(self):
+        # create a sandbox in which case directories can be created
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir, ignore_errors=True)
+
+    def test_setup(self):
+        # Ensure that test setup properly sets up case 1 and case 2
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+        case1.set_value('var_preset', 'preset_value')
+
+        # Exercise
+        mytest = SystemTestsCompareTwoFake(case1)
+
+        # Verify
+        # Make sure that pre-existing values in case1 are copied to case2 (via
+        # clone)
+        self.assertEqual('preset_value',
+                         mytest._case2.get_value('var_preset'))
+
+        # Make sure that _common_setup is called for both
+        self.assertEqual('common_val',
+                         mytest._case1.get_value('var_set_in_common_setup'))
+        self.assertEqual('common_val',
+                         mytest._case2.get_value('var_set_in_common_setup'))
+
+        # Make sure that _case_one_setup and _case_two_setup are called
+        # appropriately
+        self.assertEqual('case1val',
+                         mytest._case1.get_value('var_set_in_setup'))
+        self.assertEqual('case2val',
+                         mytest._case2.get_value('var_set_in_setup'))
+
+    def test_setup_case2_exists(self):
+        # If case2 already exists, then setup code should not be called
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+        os.makedirs(os.path.join(case1root, 'case1.test'))
+
+        # Exercise
+        mytest = SystemTestsCompareTwoFake(case1,
+                                           run_two_suffix = 'test')
+
+        # Verify:
+
+        # Make sure that case2 object is set (i.e., that it doesn't remain None)
+        self.assertEqual('case1.test', mytest._case2.get_value('CASE'))
+
+        # Variables set in various setup methods should not be set
+        # (In the real world - i.e., outside of this unit testing fakery - these
+        # values would be set when the Case objects are created.)
+        self.assertIsNone(mytest._case1.get_value('var_set_in_common_setup'))
+        self.assertIsNone(mytest._case2.get_value('var_set_in_common_setup'))
+        self.assertIsNone(mytest._case1.get_value('var_set_in_setup'))
+        self.assertIsNone(mytest._case2.get_value('var_set_in_setup'))
+
+    def test_setup_error(self):
+        # If there is an error in setup, an exception should be raised and the
+        # case2 directory should be removed
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+
+        # Exercise
+        with self.assertRaises(Exception):
+            mytest = SystemTestsCompareTwoFake(case1,
+                                               run_two_suffix = 'test',
+                                               case2setup_raises_exception = True)
+
+        # Verify
+        self.assertFalse(os.path.exists(os.path.join(case1root, 'case1.test')))
+
+    def test_run_phase_passes(self):
+        # Make sure the run phase behaves properly when all runs succeed.
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+        mytest = SystemTestsCompareTwoFake(case1)
+
+        # Exercise
+        mytest.run()
+
+        # Verify
+        self.assertEqual(test_status.TEST_PASS_STATUS,
+                         mytest._test_status.get_status(test_status.RUN_PHASE))
+
+    def test_run_phase_internal_calls(self):
+        # Make sure that the correct calls are made to methods stubbed out by
+        # SystemTestsCompareTwoFake (when runs succeed)
+        #
+        # The point of this is: A number of methods called from the run_phase
+        # method are stubbed out in the Fake test implementation, because their
+        # actions are awkward in these unit tests. But we still want to make
+        # sure that those methods actually got called correctly.
+
+        # Setup
+        run_one_suffix = 'base'
+        run_two_suffix = 'run2'
+        casename = 'mytest'
+        case1root = os.path.join(self.tempdir, casename)
+        case1 = CaseFake(case1root)
+        mytest = SystemTestsCompareTwoFake(case1,
+            run_one_suffix = run_one_suffix,
+            run_two_suffix = run_two_suffix)
+
+        # Exercise
+        mytest.run()
+
+        # Verify
+        expected_calls = [
+            Call(METHOD_run_indv,
+                 {'suffix': run_one_suffix, 'CASE': casename}),
+            Call(METHOD_run_indv,
+                 {'suffix': run_two_suffix, 'CASE': '%s.%s'%(casename, run_two_suffix)}),
+            Call(METHOD_link_to_case2_output, {}),
+            Call(METHOD_component_compare_test,
+                 {'suffix1': run_one_suffix, 'suffix2': run_two_suffix})]
+        self.assertEqual(expected_calls, mytest.log)
+
+    def test_run1_fails(self):
+        # Make sure that a failure in run1 is reported correctly
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+        mytest = SystemTestsCompareTwoFake(case1,
+                                           run_one_should_pass = False)
+
+        # Exercise
+        mytest.run()
+
+        # Verify
+        self.assertEqual(test_status.TEST_FAIL_STATUS,
+                         mytest._test_status.get_status(test_status.RUN_PHASE))
+
+    def test_run2_fails(self):
+        # Make sure that a failure in run2 is reported correctly
+
+        # Setup
+        case1root = os.path.join(self.tempdir, 'case1')
+        case1 = CaseFake(case1root)
+        mytest = SystemTestsCompareTwoFake(case1,
+                                           run_two_should_pass = False)
+
+        # Exercise
+        mytest.run()
+
+        # Verify
+        self.assertEqual(test_status.TEST_FAIL_STATUS,
+                         mytest._test_status.get_status(test_status.RUN_PHASE))
+
diff --git a/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two_link_to_case2_output.py b/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two_link_to_case2_output.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/SystemTests/test_system_tests_compare_two_link_to_case2_output.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+
+"""
+This module contains unit tests of the method
+SystemTestsCompareTwo._link_to_case2_output
+"""
+
+import unittest
+import os
+import shutil
+import tempfile
+from CIME.SystemTests.system_tests_compare_two import SystemTestsCompareTwo
+from CIME.tests.case_fake import CaseFake
+
+# ========================================================================
+# Fake version of SystemTestsCompareTwo that overrides some functionality for
+# the sake of unit testing
+# ========================================================================
+
+class SystemTestsCompareTwoFake(SystemTestsCompareTwo):
+    def __init__(self,
+                 case1,
+                 run_two_suffix = 'test'):
+
+        SystemTestsCompareTwo.__init__(
+            self,
+            case1,
+            separate_builds = False,
+            run_two_suffix = run_two_suffix)
+
+    # ------------------------------------------------------------------------
+    # Stubs of methods called by SystemTestsCommon.__init__ that interact with
+    # the system or case object in ways we want to avoid here
+    # ------------------------------------------------------------------------
+
+    def _init_environment(self, caseroot):
+        pass
+
+    def _init_locked_files(self, caseroot, expected):
+        pass
+
+    def _init_case_setup(self):
+        pass
+
+    # ------------------------------------------------------------------------
+    # Stubs of methods that are typically provided by the individual test
+    # ------------------------------------------------------------------------
+
+    def _case_one_setup(self):
+        pass
+
+    def _case_two_setup(self):
+        pass
+
+# ========================================================================
+# Test class itself
+# ========================================================================
+
+class TestLinkToCase2Output(unittest.TestCase):
+
+    # ========================================================================
+    # Test helper functions
+    # ========================================================================
+
+    def setUp(self):
+        # Create a sandbox in which case directories can be created
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir, ignore_errors=True)
+
+    def setup_test_and_directories(self, casename1, run2_suffix):
+        """
+        Returns test object
+        """
+
+        case1root = os.path.join(self.tempdir, casename1)
+        case1 = CaseFake(case1root)
+        mytest = SystemTestsCompareTwoFake(case1, run_two_suffix = run2_suffix)
+        mytest._case1.make_rundir()
+        mytest._case2.make_rundir()
+
+        return mytest
+
+    def create_file_in_rundir2(self, mytest, core_filename, run2_suffix):
+        """
+        Creates a file in rundir2 named CASE2.CORE_FILENAME.nc.RUN2_SUFFIX
+        (where CASE2 is the casename of case2)
+
+        Returns full path to the file created
+        """
+        filename = '%s.%s.nc.%s'%(
+            mytest._case2.get_value('CASE'),
+            core_filename,
+            run2_suffix)
+        filepath = os.path.join(mytest._case2.get_value('RUNDIR'), filename)
+        open(filepath, 'w').close()
+        return filepath
+
+    # ========================================================================
+    # Begin actual tests
+    # ========================================================================
+
+    def test_basic(self):
+        # Setup
+        casename1 = 'mytest'
+        run2_suffix = 'run2'
+
+        mytest = self.setup_test_and_directories(casename1, run2_suffix)
+        filepath1 = self.create_file_in_rundir2(mytest, 'clm2.h0', run2_suffix)
+        filepath2 = self.create_file_in_rundir2(mytest, 'clm2.h1', run2_suffix)
+
+        # Exercise
+        mytest._link_to_case2_output()
+
+        # Verify
+        expected_link_filename1 = '%s.clm2.h0.nc.%s'%(casename1, run2_suffix)
+        expected_link_filepath1 = os.path.join(mytest._case1.get_value('RUNDIR'),
+                                               expected_link_filename1)
+        self.assertTrue(os.path.islink(expected_link_filepath1))
+        self.assertEqual(filepath1, os.readlink(expected_link_filepath1))
+
+        expected_link_filename2 = '%s.clm2.h1.nc.%s'%(casename1, run2_suffix)
+        expected_link_filepath2 = os.path.join(mytest._case1.get_value('RUNDIR'),
+                                               expected_link_filename2)
+        self.assertTrue(os.path.islink(expected_link_filepath2))
+        self.assertEqual(filepath2, os.readlink(expected_link_filepath2))
+
+    def test_existing_link(self):
+        # Setup
+        casename1 = 'mytest'
+        run2_suffix = 'run2'
+
+        mytest = self.setup_test_and_directories(casename1, run2_suffix)
+        filepath1 = self.create_file_in_rundir2(mytest, 'clm2.h0', run2_suffix)
+
+        # Create initial link via a call to _link_to_case2_output
+        mytest._link_to_case2_output()
+
+        # Exercise
+        # See what happens when we try to recreate that link
+        mytest._link_to_case2_output()
+
+        # (No verification: Test passes if no exception was raised)
diff --git a/utils/python/CIME/tests/SystemTests/test_utils/__init__.py b/utils/python/CIME/tests/SystemTests/test_utils/__init__.py
new file mode 100644
diff --git a/utils/python/CIME/tests/SystemTests/test_utils/test_user_nl_utils.py b/utils/python/CIME/tests/SystemTests/test_utils/test_user_nl_utils.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/SystemTests/test_utils/test_user_nl_utils.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+import unittest
+import os
+import shutil
+import tempfile
+from CIME.SystemTests.test_utils import user_nl_utils
+
+class TestUserNLCopier(unittest.TestCase):
+
+    # ========================================================================
+    # Test helper functions
+    # ========================================================================
+
+    def setUp(self):
+        self._caseroot = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self._caseroot, ignore_errors=True)
+
+    def write_user_nl_file(self, component, contents, suffix=''):
+        """Write contents to a user_nl file in the case directory. Returns the
+        basename (i.e., not the full path) of the file that is created.
+
+        For a component foo, with the default suffix of '', the file name will
+        be user_nl_foo
+
+        If the suffix is '_0001', the file name will be user_nl_foo_0001
+        """
+
+        filename = 'user_nl_' + component + suffix
+
+        with open(os.path.join(self._caseroot, filename), 'w') as user_nl_file:
+            user_nl_file.write(contents)
+
+        return filename
+
+    def assertFileContentsEqual(self, expected, filepath, msg=None):
+        """Asserts that the contents of the file given by 'filepath' are equal to
+        the string given by 'expected'. 'msg' gives an optional message to be
+        printed if the assertion fails."""
+
+        with open(filepath, 'r') as myfile:
+            contents = myfile.read()
+
+        self.assertEqual(expected, contents, msg=msg)
+
+    # ========================================================================
+    # Begin actual tests
+    # ========================================================================
+
+    def test_append(self):
+        # Define some variables
+        component = 'foo'
+        # deliberately exclude new line from file contents, to make sure that's
+        # handled correctly
+        orig_contents = 'bar = 42'
+        contents_to_append = 'baz = 101'
+
+        # Setup
+        filename = self.write_user_nl_file(component, orig_contents)
+
+        # Exercise
+        user_nl_utils.append_to_user_nl_files(caseroot = self._caseroot,
+                                              component = component,
+                                              contents = contents_to_append)
+
+        # Verify
+        expected_contents = orig_contents + '\n' + contents_to_append + '\n'
+        self.assertFileContentsEqual(expected_contents,
+                                     os.path.join(self._caseroot, filename))
+
+    def test_append_multiple_files(self):
+        # Simulates a multi-instance test
+        component = 'foo'
+        orig_contents1 = 'bar = 42'
+        orig_contents2 = 'bar = 17'
+        contents_to_append = 'baz = 101'
+
+        # Setup
+        filename1 = self.write_user_nl_file(component, orig_contents1, suffix='_0001')
+        filename2 = self.write_user_nl_file(component, orig_contents2, suffix='_0002')
+
+        # Exercise
+        user_nl_utils.append_to_user_nl_files(caseroot = self._caseroot,
+                                              component = component,
+                                              contents = contents_to_append)
+
+        # Verify
+        expected_contents1 = orig_contents1 + '\n' + contents_to_append + '\n'
+        expected_contents2 = orig_contents2 + '\n' + contents_to_append + '\n'
+        self.assertFileContentsEqual(expected_contents1,
+                                     os.path.join(self._caseroot, filename1))
+        self.assertFileContentsEqual(expected_contents2,
+                                     os.path.join(self._caseroot, filename2))
+
+
+    def test_append_without_files_raises_exception(self):
+        # This test verifies that you get an exception if you call
+        # append_to_user_nl_files when there are no user_nl files of interest
+
+        # Define some variables
+        component_exists = 'foo'
+        component_for_append = 'bar'
+
+        # Setup
+        # Create file in caseroot for component_exists, but not for component_for_append
+        filename = self.write_user_nl_file(component_exists, 'irrelevant contents')
+
+        # Exercise & verify
+        self.assertRaisesRegexp(RuntimeError, "No user_nl files found",
+                                user_nl_utils.append_to_user_nl_files,
+                                caseroot = self._caseroot,
+                                component = component_for_append,
+                                contents = 'irrelevant contents to append')
+
diff --git a/utils/python/CIME/tests/__init__.py b/utils/python/CIME/tests/__init__.py
new file mode 100644
diff --git a/utils/python/CIME/tests/case_fake.py b/utils/python/CIME/tests/case_fake.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/case_fake.py
@@ -0,0 +1,99 @@
+"""
+This module contains a fake implementation of the Case class that can be used
+for testing the tests.
+"""
+
+import os
+from copy import deepcopy
+
+class CaseFake(object):
+    def __init__(self, case_root, create_case_root=True):
+        """
+        Initialize a new case object for the given case_root directory.
+
+        Args:
+            case_root (str): path to CASEROOT
+            create_case_root (bool): If True, creates the directory given by case_root
+        """
+        self.vars = dict()
+        if create_case_root:
+            os.makedirs(case_root)
+        self.set_value('CASEROOT', case_root)
+        casename = os.path.basename(case_root)
+        self.set_value('CASE', casename)
+        self.set_value('CASEBASEID', casename)
+        self._set_rundir()
+
+    def get_value(self, item):
+        """
+        Get the value of the given item
+
+        Returns None if item isn't set for this case
+
+        Args:
+            item (str): variable of interest
+        """
+        return self.vars.get(item)
+
+    def set_value(self, item, value):
+        """
+        Set the value of the given item to the given value
+
+        Args:
+            item (str): variable of interest
+            value (any type): new value for item
+        """
+        self.vars[item] = value
+
+    def copy(self, newcasename, newcaseroot):
+        """
+        Create and return a copy of self, but with CASE and CASEBASEID set to newcasename,
+        CASEROOT set to newcaseroot, and RUNDIR set appropriately.
+
+        Args:
+            newcasename (str): new value for CASE
+            newcaseroot (str): new value for CASEROOT
+        """
+        newcase = deepcopy(self)
+        newcase.set_value('CASE', newcasename)
+        newcase.set_value('CASEBASEID', newcasename)
+        newcase.set_value('CASEROOT', newcaseroot)
+        newcase._set_rundir()
+
+        return newcase
+
+    def create_clone(self, newcase, keepexe=False):
+        """
+        Create a clone of the current case. Also creates the CASEROOT directory
+        for the clone case (given by newcase).
+
+        Args:
+            newcase (str): full path to the new case. This directory should not
+                already exist; it will be created
+            keepexe (bool, optional): Ignored
+
+        Returns the clone case object
+        """
+        newcaseroot = os.path.abspath(newcase)
+        newcasename = os.path.basename(newcase)
+        os.makedirs(newcaseroot)
+        clone = self.copy(newcasename = newcasename, newcaseroot = newcaseroot)
+
+        return clone
+
+    def flush(self):
+        pass
+
+    def make_rundir(self):
+        """
+        Make directory given by RUNDIR
+        """
+        os.makedirs(self.get_value('RUNDIR'))
+
+    def _set_rundir(self):
+        """
+        Assumes CASEROOT is already set; sets an appropriate RUNDIR (nested
+        inside CASEROOT)
+        """
+        self.set_value('RUNDIR', os.path.join(self.get_value('CASEROOT'), 'run'))
+
diff --git a/utils/python/CIME/tests/test_case_fake.py b/utils/python/CIME/tests/test_case_fake.py
new file mode 100644
--- /dev/null
+++ b/utils/python/CIME/tests/test_case_fake.py
@@ -0,0 +1,39 @@
+#/usr/bin/env python
+
+"""
+This module contains unit tests of CaseFake
+"""
+
+import unittest
+import tempfile
+import os
+import shutil
+
+from CIME.tests.case_fake import CaseFake
+
+class TestCaseFake(unittest.TestCase):
+
+    def setUp(self):
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir, ignore_errors=True)
+
+    def test_create_clone(self):
+        # Setup
+        old_caseroot = os.path.join(self.tempdir, 'oldcase')
+        oldcase = CaseFake(old_caseroot)
+        oldcase.set_value('foo', 'bar')
+
+        # Exercise
+        new_caseroot = os.path.join(self.tempdir, 'newcase')
+        clone = oldcase.create_clone(new_caseroot)
+
+        # Verify
+        self.assertEqual('bar', clone.get_value('foo'))
+        self.assertEqual('newcase', clone.get_value('CASE'))
+        self.assertEqual('newcase', clone.get_value('CASEBASEID'))
+        self.assertEqual(new_caseroot, clone.get_value('CASEROOT'))
+        self.assertEqual(os.path.join(new_caseroot, 'run'),
+                         clone.get_value('RUNDIR'))
+
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -41,13 +41,43 @@ def test_resolve_variable_name(self):
         self.assertTrue(os.path.isfile(machinefile),
                         msg="Path did not resolve to existing file %s" % machinefile)
 
-    def test_CIME_unit_tests(self):
-        # Find and run all the unit tests in the CIME directory tree
+    def test_unittests(self):
+        # Finds all files contained in CIME/tests or its subdirectories that
+        # match the pattern 'test*.py', and runs the unit tests found there
+        # (i.e., tests defined using python's unittest module).
+        #
+        # This is analogous to running:
+        #     python -m unittest discover -s CIME/tests -t .
+        # from cime/utils/python
+        #
+        # Yes, that means we have a bunch of unit tests run from this one unit
+        # test.
+
+        testsuite = unittest.defaultTestLoader.discover(
+            start_dir = os.path.join(LIB_DIR,"CIME","tests"),
+            pattern = 'test*.py',
+            top_level_dir = LIB_DIR)
+
+        testrunner = unittest.TextTestRunner(buffer=False)
+
+        # Disable logging; otherwise log messages written by code under test
+        # clutter the unit test output
+        log_lvl = logging.getLogger().getEffectiveLevel()
+        logging.disable(logging.CRITICAL)
+        try:
+            results = testrunner.run(testsuite)
+        finally:
+            logging.getLogger().setLevel(log_lvl)
+
+        self.assertTrue(results.wasSuccessful())
+
+    def test_CIME_doctests(self):
+        # Find and run all the doctests in the CIME directory tree
         stat, output, _ = run_cmd("PYTHONPATH=%s:$PYTHONPATH python -m doctest *.py 2>&1" % LIB_DIR, from_dir=os.path.join(LIB_DIR,"CIME"))
         self.assertEqual(stat, 0, msg=output)
 
-    def test_CIMEXML_unit_tests(self):
-        # Find and run all the unit tests in the XML directory tree
+    def test_CIMEXML_doctests(self):
+        # Find and run all the doctests in the XML directory tree
         stat, output, _ = run_cmd("PYTHONPATH=%s:$PYTHONPATH python -m doctest *.py 2>&1" % LIB_DIR, from_dir=os.path.join(LIB_DIR,"CIME","XML"))
         self.assertEqual(stat, 0, msg=output)
 
@@ -1635,7 +1665,7 @@ def _main_func():
 
     CIME.utils.handle_standard_logging_options(args)
 
-    unittest.main(verbosity=2)
+    unittest.main(verbosity=2, catchbreak=True)
 
 if (__name__ == "__main__"):
     _main_func()