Repository: ESMCI/cime

Issue:
Feature request: Capture CIME command invocations as an executable script
Currently, various CIME commands are logged to the README along with timestamp and other information.

Although one can technically parse this, it is preferable to have an executable version visible to end-user that can be edited as needed (for instance, to update case name/location).

Moreover, the README is the only place where the short compset names are presently recorded.

cc @rljacob 

Example:
```
2021-03-05 18:13:43: /home/ac.qtang/E3SM/code/20210305/cime/scripts/create_newcase --case 20210305.v2beta3.piControl.naRRMpg2_WC14to60E2r3.chrysalis --case-group v2beta3.piControl --output-root /lcrc/group/e3sm/ac.qtang/E3SM_simulations/20210305.v2beta3.piControl.naRRMpg2_WC14to60E2r3.chrysalis --script-root /lcrc/group/e3sm/ac.qtang/E3SM_simulations/20210305.v2beta3.piControl.naRRMpg2_WC14to60E2r3.chrysalis/case_scripts --handle-preexisting-dirs u --compset WCYCL1850 --res northamericax4v1pg2_WC14to60E2r3 --machine chrysalis --project e3sm --walltime 32:00:00 --pecount M
 ---------------------------------------------------
2021-03-05 18:13:43: Compset longname is 1850SOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP
 ---------------------------------------------------
2021-03-05 18:13:43: Compset specification file is /gpfs/fs1/home/ac.qtang/E3SM/code/20210305/cime_config/allactive/config_compsets.xml
 ---------------------------------------------------
2021-03-05 18:13:43: Pes     specification file is /gpfs/fs1/home/ac.qtang/E3SM/code/20210305/cime_config/allactive/config_pesall.xml
 ---------------------------------------------------
2021-03-05 18:13:43: Forcing is 
 ---------------------------------------------------
2021-03-05 18:13:43: Using None coupler instances
 ---------------------------------------------------
2021-03-05 18:13:43: Component ATM is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK:
 ---------------------------------------------------
2021-03-05 18:13:43: ATM_GRID is ne0np4_northamericax4v1.pg2
 ---------------------------------------------------
2021-03-05 18:13:43: Component LND is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition:
 ---------------------------------------------------
2021-03-05 18:13:43: LND_GRID is ne0np4_northamericax4v1.pg2
 ---------------------------------------------------
2021-03-05 18:13:43: Component ICE is MPAS seaice:
 ---------------------------------------------------
2021-03-05 18:13:43: ICE_GRID is WC14to60E2r3
 ---------------------------------------------------
2021-03-05 18:13:43: Component OCN is mpas ocean default:
 ---------------------------------------------------
2021-03-05 18:13:43: OCN_GRID is WC14to60E2r3
 ---------------------------------------------------
2021-03-05 18:13:43: Component ROF is MOSART:
 ---------------------------------------------------
2021-03-05 18:13:43: ROF_GRID is r0125
 ---------------------------------------------------
2021-03-05 18:13:43: Component GLC is Stub glacier (land ice) component
 ---------------------------------------------------
2021-03-05 18:13:43: GLC_GRID is null
 ---------------------------------------------------
2021-03-05 18:13:43: Component WAV is Stub wave component
 ---------------------------------------------------
2021-03-05 18:13:43: WAV_GRID is null
 ---------------------------------------------------
2021-03-05 18:13:43: Component IAC is Stub iac component
 ---------------------------------------------------
2021-03-05 18:13:43: IAC_GRID is null
 ---------------------------------------------------
2021-03-05 18:13:43: Component ESP is Stub external system processing (ESP) component
 ---------------------------------------------------
2021-03-05 18:13:43: ESP_GRID is None
 ---------------------------------------------------
```



Discussion:
CaseStatus has the commands executed in order but still not in the form of a script that can itself be executed.  README just has the initial create_newcase (which is odd, it should also be in CaseStatus) and some output from create_newcase.

In the CIME-use mode where you constantly xlmchange things in the casedir until you get the config you want, there could be a lot of extra commands in the CaseStatus.  You just need the ones that set up the case the way you finally want it.  But I don't think there's any harm in just re-executing everything as it happened.


@rljacob @sarats 
Were we thinking just a basic shell script? Would a python script be useful?
I think that what is in CaseStatus is recorded from xmlchange commands - so a basic shell script should be straightforward and easy to read. 
Yes basic shell script. 
Ok great. 

Do we want to include all CIME commands ran within the case e.g. case.setup, case.build, xmlchange, etc? or just the xmlchange commands?

If we record all commands, are there any to exclude e.g. preview_*, xmlquery?


I think that you want to capture anything that changes the state of the case so you would exclude preview_run and xmlquery, but you may need to include preview_namelist. 
@sarats @rljacob @jedwards4b 

Here's a sample output of what I have working. This also works when using `create_clone`.

```shell
#!/bin/bash
# Created 2021-05-04 10:51

/data/E3SM/cime/scripts/create_newcase --case /data/cime_testing/scratch/SEQ_Ln9.f19_g16_rx1.A.docker_gnu.20210504_172548_pi2q0s --res f19_g16_rx1 --compset A --test --machine docker --compiler gnu --workflow default --walltime 00:31:40

./case.setup

./case.build --sharedlib-only

./case.build --model-only

./case.submit --skip-preview-namelist --no-batch

./xmlchange STOP_OPTION=nsteps

./case.setup --clean

./case.setup

./check_case

./check_input_data --download

./pelayout --set-ntasks 2

./preview_namelists
```
What are looking at?  This is a file created from the commands you issued in a casedir?
Yea all of the commands are captured from within a casedir except the `create_newcase` and `create_clone`.

Forgot to mention the output from above is placed in a file `replay.sh` in the casedir.

I have one question is the intention that this script should create a case in the way of the original or should just be used to configure an existing case in the same fashion?
Is that what we want?  I thought we would just reformat the CaseStatus a little to accommodate this?
CaseStatus doesn't have the 'create_newcase' and it would be preferable to have an executable script instead of parsing text. 
@sarats there is no reason we couldn't add create_newcase to the CaseStatus file and reformat it a little to make it an sh script. 
👍🏼 That works for me.
That's also fine with me but CaseStatus was recording every command including xmlquery and what we want here is just the state-changing commands.    Should we change that as well?
@rljacob I suggest putting those commands like xmlquery that don't affect outcome in a 
comment (or maybe wrapped in a conditional so if a user wants they can run everything).

An example CaseStatus:
```
2020-12-14 18:46:32: case.setup starting 
 ---------------------------------------------------
2020-12-14 18:46:36: case.setup success 
 ---------------------------------------------------
2020-12-14 18:47:14: xmlchange success <command> ./xmlchange NTASKS=64  </command>
 ---------------------------------------------------
2020-12-14 18:47:17: case.setup starting 
 ---------------------------------------------------
2020-12-14 18:47:20: case.setup success 
 ---------------------------------------------------
2020-12-14 18:47:24: case.build starting 
 ---------------------------------------------------
2020-12-14 18:53:42: case.build success 
```
would become

```
./create_newcase --case Wcyclebi --compset A_WCYCL1850 --res ne4_oQU480 --mach chrysalis
./case.setup
#---------------------------------------------------
#2020-12-14 18:46:32: case.setup starting 
#---------------------------------------------------
#2020-12-14 18:46:36: case.setup success 
#---------------------------------------------------
./xmlchange NTASKS=64  </command>
#---------------------------------------------------
#2020-12-14 18:47:14: xmlchange success
#---------------------------------------------------
./case.setup
#---------------------------------------------------
#2020-12-14 18:47:17: case.setup starting 
#---------------------------------------------------
#2020-12-14 18:47:20: case.setup success 
# ---------------------------------------------------
./case.build
#---------------------------------------------------
#2020-12-14 18:47:24: case.build starting 
# ---------------------------------------------------
#2020-12-14 18:53:42: case.build success 
```
Still seems like CaseStatus as a time-stamped record of what happened is useful as-is and turning it in to a more-portable run-anywhere script while keeping the timestamps would result in a messy script.
> Still seems like CaseStatus as a time-stamped record of what happened is useful as-is and turning it in to a more-portable run-anywhere script while keeping the timestamps would result in a messy script.

I have this same concern, this seems messy. Like we're trying to crowbar two use-cases into one file.
Possibly - also if you run CaseStatus you may not want it to echo the commands back to itself.
This may also be an issue for the replay.sh file.   Can we have a script that reads case status and generates replay.sh or is replay.sh built at the same time as CaseStatus is?
`replay.sh` is built before CaseStatus is appended. The calls to create it exist in the entrypoint scripts e.g. `Toos/case.setup`, `Tools/case.build`, etc. This seemed the simplest solution as each script gets a single function call `append_script(sys.argv, caseroot=caseroot)` to capture the commands.

CaseStatus doesn't record some commands that change state, e.g. check_case, pelayout, preview_namelists. For a script that converts CaseStatus into replay.sh I'll have to add some updats to capture these commands in CaseStatus.
But replay.sh is built as the user issues commands in the casedir like CaseStatus?
Food for thought: Having a separate replay.sh makes it easy to make it reusable and future extensions to port it to other machines/layouts. For instance, making any hard coded paths like case as template variables. 
CaseStatus can be the 'archival record' of what happened.
The way I have it coded right now is each command is capture right away while CaseStatus is modified much further into the code execution.

Resolving patch:
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -5,7 +5,7 @@
 through the Case module.
 """
 from copy import deepcopy
-import glob, os, shutil, math, six
+import glob, os, shutil, math, six, time
 from CIME.XML.standard_module_setup import *
 #pylint: disable=import-error,redefined-builtin
 from six.moves import input
@@ -80,7 +80,7 @@ class Case(object):
     from CIME.case.preview_namelists import create_dirs, create_namelists
     from CIME.case.check_input_data import check_all_input_data, stage_refcase, check_input_data
 
-    def __init__(self, case_root=None, read_only=True):
+    def __init__(self, case_root=None, read_only=True, record=False):
 
         if case_root is None:
             case_root = os.getcwd()
@@ -97,6 +97,9 @@ def __init__(self, case_root=None, read_only=True):
 
         self.read_xml()
 
+        if record:
+            self.record_cmd()
+
         # Hold arbitary values. In create_newcase we may set values
         # for xml files that haven't been created yet. We need a place
         # to store them until we are ready to create the file. At file
@@ -1636,6 +1639,39 @@ def get_latest_cpl_log(self, coupler_log_path=None, cplname="cpl"):
         else:
             return None
 
+    def record_cmd(self, cmd=None, init=False):
+        lines = []
+        caseroot = self.get_value("CASEROOT")
+        cimeroot = self.get_value("CIMEROOT")
+
+        if cmd is None:
+            cmd = list(sys.argv)
+
+        if init:
+            ctime = time.strftime("%Y-%m-%d %H:%M:%S")
+
+            lines.append("#!/bin/bash\n\n")
+            # stop script on error, prevents `create_newcase` from failing
+            # and continuing to execute commands
+            lines.append("set -e\n\n")
+            lines.append("# Created {}\n\n".format(ctime))
+            lines.append("CASEDIR=\"{}\"\n\n".format(caseroot))
+            lines.append("cd \"${CASEDIR}\"\n\n")
+
+            # Ensure program path is absolute
+            cmd[0] = re.sub("^./", "{}/scripts/".format(cimeroot), cmd[0])
+
+        cmd = " ".join(cmd)
+
+        # Replace instances of caseroot with variable
+        cmd = re.sub(caseroot, "\"${CASEDIR}\"", cmd)
+
+        lines_len = len(lines)
+        lines.insert(lines_len-1 if init else lines_len, "{}\n\n".format(cmd))
+
+        with open(os.path.join(caseroot, "replay.sh"), "a") as fd:
+            fd.writelines(lines)
+
     def create(self, casename, srcroot, compset_name, grid_name,
                user_mods_dir=None, machine_name=None,
                project=None, pecount=None, compiler=None, mpilib=None,


Test patch:
diff --git a/scripts/lib/CIME/tests/test_case.py b/scripts/lib/CIME/tests/test_case.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/tests/test_case.py
@@ -0,0 +1,118 @@
+import os
+import sys
+import unittest
+
+from CIME.case import Case
+from CIME import utils as cime_utils
+
+from . import utils
+
+class TestCase_RecordCmd(unittest.TestCase):
+
+    def setUp(self):
+        self.srcroot = os.path.abspath(cime_utils.get_cime_root())
+        self.tempdir = utils.TemporaryDirectory()
+
+        self.mock = utils.Mocker()
+        self.mock.patch(Case, "__init__", ret=None)
+        self.mock.patch(Case, "flush", ret=utils.Mocker())
+        # Case.__init__ = utils.Mocker()
+        # Case.flush = utils.Mocker()
+        Case._force_read_only = False # pylint: disable=protected-access
+
+    def assert_calls_match(self, calls, expected):
+        self.assertTrue(len(calls) == len(expected), calls)
+
+        for x, y in zip(calls, expected):
+            self.assertTrue(x == y, calls)
+
+    def test_init(self):
+        with self.tempdir as tempdir:
+            mock = utils.Mocker()
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker())
+            mock.patch("time.strftime", ret="00:00:00")
+            mock.patch("sys.argv", ret=["/src/create_newcase"], is_property=True)
+
+            with Case(tempdir) as case:
+                case.get_value = utils.Mocker(
+                    side_effect=[tempdir, "/src"]
+                )
+
+                case.record_cmd(init=True)
+
+        self.assertTrue(open_mock.calls[0]["args"] ==
+                        ("{}/replay.sh".format(tempdir), "a"))
+
+        expected = [
+            "#!/bin/bash\n\n",
+            "set -e\n\n",
+            "# Created 00:00:00\n\n",
+            "CASEDIR=\"{}\"\n\n".format(tempdir),
+            "/src/create_newcase\n\n",
+            "cd \"${CASEDIR}\"\n\n",
+        ]
+
+        calls = open_mock.ret.method_calls["writelines"][0]["args"][0]
+
+        self.assert_calls_match(calls, expected)
+
+    def test_sub_relative(self):
+        with self.tempdir as tempdir:
+            mock = utils.Mocker()
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker()
+            )
+            mock.patch("time.strftime", ret="00:00:00")
+            mock.patch("sys.argv", ret=["./create_newcase"], is_property=True)
+
+            with Case(tempdir) as case:
+                case.get_value = utils.Mocker(
+                    side_effect=[tempdir, "/src"]
+                )
+
+                case.record_cmd(init=True)
+
+        expected = [
+            "#!/bin/bash\n\n",
+            "set -e\n\n",
+            "# Created 00:00:00\n\n",
+            "CASEDIR=\"{}\"\n\n".format(tempdir),
+            "/src/scripts/create_newcase\n\n",
+            "cd \"${CASEDIR}\"\n\n",
+        ]
+
+        calls = open_mock.ret.method_calls["writelines"][0]["args"][0]
+
+        self.assert_calls_match(calls, expected)
+
+    def test_cmd_arg(self):
+        with self.tempdir as tempdir:
+            mock = utils.Mocker()
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker()
+            )
+
+            with Case(tempdir) as case:
+                case.get_value = utils.Mocker(
+                    side_effect=[tempdir, "/src"]
+                )
+
+                case.record_cmd(["/some/custom/command", "arg1"])
+
+        expected = [
+            "/some/custom/command arg1\n\n",
+        ]
+
+        calls = open_mock.ret.method_calls["writelines"][0]["args"][0]
+
+        self.assert_calls_match(calls, expected)
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/scripts/lib/CIME/tests/test_provenance.py b/scripts/lib/CIME/tests/test_provenance.py
--- a/scripts/lib/CIME/tests/test_provenance.py
+++ b/scripts/lib/CIME/tests/test_provenance.py
@@ -4,137 +4,122 @@
 
 from CIME import provenance
 
-# TODO replace with actual mock once 2.7 is dropped
-class Mocker:
-    calls = []
-
-    def __init__(self, ret=None, cmd=None):
-        self._orig = []
-        self._ret = ret
-        self._cmd = cmd
-
-    def __getattr__(self, name):
-        return Mocker(self._ret, name)
-
-    def __call__(self, *args, **kwargs):
-        args = list(args)
-        if self._cmd is not None:
-            args.insert(0, self._cmd)
-
-        call_sig = " ".join([
-            x for x in args + [
-                "{}={}".format(y, z) for y, z in kwargs.items()]])
-        self.calls.append(call_sig)
-        return self._ret if self._ret is not None else self
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *args, **kwargs):
-        for m, module, method in self._orig:
-            setattr(sys.modules[module], method, m)
-
-    def patch(self, method, ret=None):
-        x = method.split('.')
-        main = '.'.join(x[:-1])
-        self._orig.append((getattr(sys.modules[main], x[-1]), main, x[-1]))
-        setattr(sys.modules[main], x[-1], Mocker(ret, x[-1]))
+from . import utils
 
 class TestProvenance(unittest.TestCase):
     def test_run_git_cmd_recursively(self):
-        with Mocker() as mock:
-            Mocker.calls =[]
-            mock.patch("CIME.provenance.run_cmd", (0, "data", None))
-            if sys.version_info.major > 2:
-                mock.patch("builtins.open")
-            else:
-                mock.patch("__builtin__.open")
-
+        with utils.Mocker() as mock:
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker()
+            )
+            provenance.run_cmd = utils.Mocker(return_value=(0, "data", None))
             provenance._run_git_cmd_recursively('status', '/srcroot', '/output.txt') # pylint: disable=protected-access
-        expected = [
-            "run_cmd git status from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git status; echo\""
-             " from_dir=/srcroot"),
-            "open /output.txt w",
-            "write data\n\n",
-            "write data\n",
-        ]
 
-        self.assertTrue(len(expected) == len(mock.calls))
+        self.assertTrue(
+            open_mock.calls[0]["args"] == ("/output.txt", "w"),
+            open_mock.calls
+        )
 
-        for x, y in zip(expected, mock.calls):
-            self.assertTrue(x == y, "{} != {}".format(x, y))
+        write = open_mock.ret.method_calls["write"]
 
-    def test_run_git_cmd_recursively_error(self):
-        with Mocker() as mock:
-            Mocker.calls =[]
-            mock.patch("CIME.provenance.run_cmd", (1, "data", "error"))
-            if sys.version_info.major > 2:
-                mock.patch("builtins.open")
-            else:
-                mock.patch("__builtin__.open")
+        self.assertTrue(write[0]["args"][0] == "data\n\n", write)
+        self.assertTrue(write[1]["args"][0] == "data\n", write)
+
+        run_cmd = provenance.run_cmd.calls
+
+        self.assertTrue(run_cmd[0]["args"][0] == "git status")
+        self.assertTrue(run_cmd[0]["kwargs"]["from_dir"] == "/srcroot")
 
+        self.assertTrue(run_cmd[1]["args"][0] == "git submodule foreach"
+                        " --recursive \"git status; echo\"", run_cmd)
+        self.assertTrue(run_cmd[1]["kwargs"]["from_dir"] == "/srcroot")
+
+    def test_run_git_cmd_recursively_error(self):
+        with utils.Mocker() as mock:
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker()
+            )
+            provenance.run_cmd = utils.Mocker(return_value=(1, "data", "error"))
             provenance._run_git_cmd_recursively('status', '/srcroot', '/output.txt') # pylint: disable=protected-access
-        expected = [
-            "run_cmd git status from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git status; echo\""
-             " from_dir=/srcroot"),
-            "open /output.txt w",
-            "write error\n\n",
-            "write error\n",
-        ]
 
-        self.assertTrue(len(expected) == len(mock.calls))
+        write = open_mock.ret.method_calls["write"]
 
-        for x, y in zip(expected, mock.calls):
-            self.assertTrue(x == y, "{} != {}".format(x, y))
+        self.assertTrue(write[0]["args"][0] == "error\n\n", write)
+        self.assertTrue(write[1]["args"][0] == "error\n", write)
 
-    def test_record_git_provenance(self):
-        with Mocker() as mock:
-            Mocker.calls =[]
-            mock.patch("CIME.provenance.safe_copy")
-            mock.patch("CIME.provenance.run_cmd", (0, "data", None))
-            if sys.version_info.major > 2:
-                mock.patch("builtins.open")
-            else:
-                mock.patch("__builtin__.open")
+        run_cmd = provenance.run_cmd.calls
+
+        self.assertTrue(run_cmd[0]["args"][0] == "git status")
+        self.assertTrue(run_cmd[0]["kwargs"]["from_dir"] == "/srcroot")
 
+        self.assertTrue(run_cmd[1]["args"][0] == "git submodule foreach"
+                        " --recursive \"git status; echo\"", run_cmd)
+        self.assertTrue(run_cmd[1]["kwargs"]["from_dir"] == "/srcroot")
+
+    def test_record_git_provenance(self):
+        with utils.Mocker() as mock:
+            open_mock = mock.patch(
+                "builtins.open" if sys.version_info.major > 2 else
+                    "__builtin__.open",
+                ret=utils.Mocker()
+            )
+
+            provenance.safe_copy = utils.Mocker()
+            provenance.run_cmd = utils.Mocker(return_value=(0, "data", None))
             provenance._record_git_provenance("/srcroot", "/output", "5") # pylint: disable=protected-access
 
         expected = [
-            "run_cmd git status from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git status; echo\""
-             " from_dir=/srcroot"),
-            "open /output/GIT_STATUS.5 w",
-            "write data\n\n",
-            "write data\n",
-            "run_cmd git diff from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git diff; echo\""
-             " from_dir=/srcroot"),
-            "open /output/GIT_DIFF.5 w",
-            "write data\n\n",
-            "write data\n",
-            "run_cmd git log --first-parent --pretty=oneline -n 5 from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git log"
-             " --first-parent --pretty=oneline -n 5; echo\" from_dir=/srcroot"),
-            "open /output/GIT_LOG.5 w",
-            "write data\n\n",
-            "write data\n",
-            "run_cmd git remote -v from_dir=/srcroot",
-            ("run_cmd git submodule foreach --recursive \"git remote"
-             " -v; echo\" from_dir=/srcroot"),
-            "open /output/GIT_REMOTE.5 w",
-            "write data\n\n",
-            "write data\n",
-            ("safe_copy /srcroot/.git/config /output/GIT_CONFIG.5"
-             " preserve_meta=False")
+            ("/output/GIT_STATUS.5", "w"),
+            ("/output/GIT_DIFF.5", "w"),
+            ("/output/GIT_LOG.5", "w"),
+            ("/output/GIT_REMOTE.5", "w")
         ]
 
-        self.assertTrue(len(expected) == len(mock.calls),
-                        mock.calls)
+        for i in range(4):
+            self.assertTrue(
+                open_mock.calls[i]["args"] == expected[i],
+                open_mock.calls
+            )
+
+        write = open_mock.ret.method_calls["write"]
+
+        expected = [
+            "data\n\n",
+            "data\n",
+        ]
+
+        for x in range(8):
+            self.assertTrue(write[x]["args"][0] == expected[x%2], write)
+
+        run_cmd = provenance.run_cmd.calls
+
+        expected = [
+            "git status",
+            "git submodule foreach --recursive \"git status; echo\"",
+            "git diff",
+            "git submodule foreach --recursive \"git diff; echo\"",
+            "git log --first-parent --pretty=oneline -n 5",
+            "git submodule foreach --recursive \"git log --first-parent"
+                " --pretty=oneline -n 5; echo\"",
+            "git remote -v",
+            "git submodule foreach --recursive \"git remote -v; echo\"",
+        ]
 
-        for x, y in zip(expected, mock.calls):
-            self.assertTrue(x == y, "{} != {}".format(x, y))
+        for x in range(len(run_cmd)):
+            self.assertTrue(run_cmd[x]["args"][0] == expected[x], run_cmd[x])
+
+        self.assertTrue(
+            provenance.safe_copy.calls[0]["args"][0] == "/srcroot/.git/config",
+            provenance.safe_copy.calls
+        )
+        self.assertTrue(
+            provenance.safe_copy.calls[0]["args"][1] == "/output/GIT_CONFIG.5",
+            provenance.safe_copy.calls
+        )
 
 if __name__ == '__main__':
     sys.path.insert(0, os.path.abspath(os.path.join('.', '..', '..', 'lib')))
diff --git a/scripts/lib/CIME/tests/test_utils.py b/scripts/lib/CIME/tests/test_utils.py
--- a/scripts/lib/CIME/tests/test_utils.py
+++ b/scripts/lib/CIME/tests/test_utils.py
@@ -2,11 +2,11 @@
 
 import sys
 import os
-import shutil
 import unittest
-import tempfile
 from CIME.utils import indent_string, run_and_log_case_status
 
+from . import utils
+
 class TestIndentStr(unittest.TestCase):
     """Test the indent_string function.
 
@@ -36,19 +36,6 @@ def test_indent_string_multiline(self):
 """
         self.assertEqual(expected, result)
 
-# TODO after dropping python 2.7 replace with tempfile.TemporaryDirectory
-class TemporaryDirectory(object):
-    def __init__(self):
-        self._tempdir = None
-
-    def __enter__(self):
-        self._tempdir = tempfile.mkdtemp()
-        return self._tempdir
-
-    def __exit__(self, *args, **kwargs):
-        if os.path.exists(self._tempdir):
-            shutil.rmtree(self._tempdir)
-
 class MockTime(object):
     def __init__(self):
         self._old = None
@@ -56,7 +43,7 @@ def __init__(self):
     def __enter__(self):
         self._old = getattr(sys.modules["time"], "strftime")
         setattr(sys.modules["time"], "strftime", lambda *args: "00:00:00 ")
-        
+
     def __exit__(self, *args, **kwargs):
         setattr(sys.modules["time"], "strftime", self._old)
 
@@ -104,7 +91,7 @@ def test_run_and_log_case_status(self):
             "00:00:00 default success \n",
         ]
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             run_and_log_case_status(self.base_func, "default",
                                     caseroot=tempdir)
 
@@ -116,7 +103,7 @@ def test_run_and_log_case_status_case_submit_on_batch(self):
             "00:00:00 case.submit success \n",
         ]
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             run_and_log_case_status(self.base_func, "case.submit",
                                     caseroot=tempdir, is_batch=True)
 
@@ -128,7 +115,7 @@ def test_run_and_log_case_status_case_submit_no_batch(self):
             "00:00:00 case.submit success \n",
         ]
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             run_and_log_case_status(self.base_func, "case.submit",
                                     caseroot=tempdir, is_batch=False)
 
@@ -141,7 +128,7 @@ def test_run_and_log_case_status_case_submit_error_on_batch(self):
             "Something went wrong\n",
         ]
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             with self.assertRaises(Exception):
                 run_and_log_case_status(self.error_func, "case.submit",
                                         caseroot=tempdir, is_batch=True)
@@ -157,7 +144,7 @@ def test_run_and_log_case_status_custom_msg(self):
         starting_func = lambda *args: "starting extra"
         success_func = lambda *args: "success extra"
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             run_and_log_case_status(self.base_func, "default", 
                                     custom_starting_msg_functor=starting_func,
                                     custom_success_msg_functor=success_func,
@@ -172,7 +159,7 @@ def test_run_and_log_case_status_error(self):
             "Something went wrong\n",
         ]
 
-        with TemporaryDirectory() as tempdir, MockTime():
+        with utils.TemporaryDirectory() as tempdir, MockTime():
             with self.assertRaises(Exception):
                 run_and_log_case_status(self.error_func, "default",
                                         caseroot=tempdir)
diff --git a/scripts/lib/CIME/tests/utils.py b/scripts/lib/CIME/tests/utils.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/tests/utils.py
@@ -0,0 +1,103 @@
+import os
+import tempfile
+import shutil
+import sys
+from collections.abc import Iterable
+
+# TODO after dropping python 2.7 replace with tempfile.TemporaryDirectory
+class TemporaryDirectory(object):
+    def __init__(self):
+        self._tempdir = None
+
+    def __enter__(self):
+        self._tempdir = tempfile.mkdtemp()
+        return self._tempdir
+
+    def __exit__(self, *args, **kwargs):
+        if os.path.exists(self._tempdir):
+            shutil.rmtree(self._tempdir)
+
+# TODO replace with actual mock once 2.7 is dropped
+class Mocker:
+
+    def __init__(self, ret=None, cmd=None, return_value=None, side_effect=None):
+        self._orig = []
+        self._ret = ret or return_value
+        self._cmd = cmd
+        self._calls = []
+
+        if isinstance(side_effect, (list, tuple)):
+            self._side_effect = iter(side_effect)
+        else:
+            self._side_effect = side_effect
+
+        self._method_calls = {}
+
+    @property
+    def calls(self):
+        return self._calls
+
+    @property
+    def method_calls(self):
+        return dict((x, y.calls) for x, y in self._method_calls.items())
+
+    @property
+    def ret(self):
+        return self._ret
+
+    def __getattr__(self, name):
+        if name in self._method_calls:
+            new_method = self._method_calls[name]
+        else:
+            new_method = Mocker(self, cmd=name)
+            self._method_calls[name] = new_method
+
+        return new_method
+
+    def __call__(self, *args, **kwargs):
+        self._calls.append({"args": args, "kwargs": kwargs})
+
+        if (self._side_effect is not None and
+                isinstance(self._side_effect, Iterable)):
+            rv = next(self._side_effect)
+        else:
+            rv = self._ret
+
+        return rv
+
+    def __del__(self):
+        self.revert_mocks()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args, **kwargs):
+        self.revert_mocks()
+
+    def revert_mocks(self):
+        for m, module, method in self._orig:
+            if isinstance(module, str):
+                setattr(sys.modules[module], method, m)
+            else:
+                setattr(module, method, m)
+
+    def patch(self, module, method=None, ret=None, is_property=False):
+        rv = None
+        if isinstance(module, str):
+            x = module.split('.')
+            main = '.'.join(x[:-1])
+            self._orig.append((getattr(sys.modules[main], x[-1]), main, x[-1]))
+            if is_property:
+                setattr(sys.modules[main], x[-1], ret)
+            else:
+                rv = Mocker(ret, cmd=x[-1])
+                setattr(sys.modules[main], x[-1], rv)
+        elif method != None:
+            self._orig.append((getattr(module, method), module, method))
+            rv = Mocker(ret)
+            setattr(module, method, rv)
+        else:
+            raise Exception("Could not patch")
+
+        return rv
+