Repository: ESMCI/cime

Issue:
Include submodule status in provenance
Right now there is no mechanist to find submodules commits from bld folder, only main repo's commit is saved. 

Separately, maybe not for this issue, but it would be good to save status of the clone in bld, not only SHA of the main repo. Right now I have a script that does this to catch possibly all recent changes in it:

```
gitstat=gitstat."${id}"
cd ~/${which}
echo " running stats on clone ${which}"
echo " status is ------------- " >> ${gitstat}
git status >> ${gitstat}
echo " branch is ------------- " >> ${gitstat}
git branch >> ${gitstat}
echo " diffs are ------------- " >> ${gitstat}
git diff >> ${gitstat}
echo " last 10 commits are ------------- " >> ${gitstat}
git log --first-parent  --pretty=oneline  HEAD~10..HEAD  >> ${gitstat}
```



Discussion:
@WesCoomber , I'm assigning you some of my tickets just to give you a few things to choose from.
@oksanaguba Hi Oksana, I am looking into this issue. 

_Is "gitstat" a tool to get SHAs? Could you provide me with more information about this tool and your shell script that you use to find submodules commits from the bld folder?_ (eg. where I can get a copy to recreate the intended behavior, and how you use them.)
Above, gitstat is just a part of a file name; full name is gitstat.${JOB_ID} after the case is submitted and JOB_ID is available.  Above are a few lines of my script that dumps info from git to the file. There ${which} is path to the clone that was used to create a case. It was used in this script like that:
```
which=acme-b3

fol=theta-ne30-nh-oct27

${which}/cime/scripts/create_newcase -case $fol -res ne30_ne30 -mach anvil -compiler intel -compset FC5AV1C-L
cd $fol

./case.setup --reset
./case.setup


./case.build
./xmlchange STOP_N=6 ; ./xmlchange STOP_OPTION=nsteps ; 
#./xmlchange REST_N=7 ; ./xmlchange REST_OPTION=nyears ;


./case.submit

#we only need the latest ID
id=`grep -o case.run:\[0-9\]\* CaseStatus | sed -e 's/.*://' | tail -1`
echo $id


gitstat=gitstat."${id}"
cd ~/${which}
echo " running stats on clone ${which}"
echo " status is ------------- " >> ${gitstat}
git status >> ${gitstat}
echo " branch is ------------- " >> ${gitstat}
git branch >> ${gitstat}
echo " diffs are ------------- " >> ${gitstat}
git diff >> ${gitstat}
echo " last 10 commits are ------------- " >> ${gitstat}
git log --first-parent  --pretty=oneline  HEAD~10..HEAD  >> ${gitstat}

cd ~/$fol
mv ~/${which}/${gitstat} .

```

@jgfouca this needs to be re-opened.  PR #3696 does not provide the info @oksanaguba requested in this issue.
@rljacob @oksanaguba
I have a fix for this and want to verify I'm capturing the pertinent information. I've modeled this based on the script provided by Oksana. I've added three new files that capture the relevant info for the **git clone** and **all submodules**.

- GIT_STATUS
  - Provides a compact status, still includes commit and branch info.
- GIT_LOG
  - Provides last 10 commits.
- GIT_DIFF
  - Provides git diff in patch form.

I have not changed any of the existing git files included in the provenance since they capture slightly different details.
Adding @sarats .   Looks good.  Do you have a sample output to point to?  
@oksanaguba what about submodules and any mods there?
Here's a sample output for the new files.

- [GIT_STATUS](https://gist.github.com/jasonb5/d8a6a773783a634dafff7dd711a7cc00)
- [GIT_LOG](https://gist.github.com/jasonb5/fc59c3f807fca409fe12e18703e0955e)
- [GIT_DIFF](https://gist.github.com/jasonb5/31b23fa217f76d08d78b99ea9503047c)
Sorry I wasn't reading carefully enough.  This also does submodules.

The GIT_DIFF output is great.

I wonder if GIT_LOG could be shortened to 5 log lines each.

I don't understand why GIT_STATUS just says "## HEAD (no branch)" for each submodule.  It should say something like 
```
HEAD detached at 7709c066
nothing to commit, working tree clean
```
for each submodule unless there are mods in the submodules.
I'm fine with changing GIT_LOG to 5, still gives a good snapshot and provides the current commit.

I added the `-s -b` flags to try an shorten GIT_STATUS. Looks like the `-b` flag only prints a commit when on an actual branch. I'll remove those flags.
Here's an updated sample of [GIT_STATUS](https://gist.github.com/jasonb5/30d9b3124620388044e9ac6395aa5a6c), little longer but has the correct information.
Looks good!

Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -44,6 +44,49 @@ def _extract_times(zipfiles, target_file):
     with open(target_file, "w") as fd:
         fd.write(contents)
 
+def _run_git_cmd_recursively(cmd, srcroot, output):
+    """ Runs a git command recursively
+
+    Runs the git command in srcroot then runs it on each submodule.
+    Then output from both commands is written to the output file.
+    """
+    rc1, output1, err1 = run_cmd("git {}".format(cmd), from_dir=srcroot)
+
+    rc2, output2, err2 = run_cmd(
+        "git submodule foreach --recursive \"git {}; echo\"".format(cmd),
+        from_dir=srcroot)
+
+    with open(output, "w") as fd:
+        fd.write((output1 if rc1 == 0 else err1) + "\n\n")
+        fd.write((output2 if rc2 == 0 else err2) + "\n")
+
+def _record_git_provenance(srcroot, exeroot, lid):
+    """ Records git provenance
+
+    Records git status, diff and logs for main repo and all submodules.
+    """
+    # Git Status
+    status_prov = os.path.join(exeroot, "GIT_STATUS.{}".format(lid))
+    _run_git_cmd_recursively("status", srcroot, status_prov)
+
+    # Git Diff
+    diff_prov = os.path.join(exeroot, "GIT_DIFF.{}".format(lid))
+    _run_git_cmd_recursively("diff", srcroot, diff_prov)
+
+    # Git Log
+    log_prov = os.path.join(exeroot, "GIT_LOG.{}".format(lid))
+    cmd = "log --first-parent --pretty=oneline -n 5"
+    _run_git_cmd_recursively(cmd, srcroot, log_prov)
+
+    # Git remote
+    remote_prov = os.path.join(exeroot, "GIT_REMOTE.{}".format(lid))
+    _run_git_cmd_recursively("remote -v", srcroot, remote_prov)
+
+    # Git config
+    config_src = os.path.join(srcroot, ".git", "config")
+    config_prov = os.path.join(exeroot, "GIT_CONFIG.{}".format(lid))
+    safe_copy(config_src, config_prov, preserve_meta=False)
+
 def _save_build_provenance_e3sm(case, lid):
     srcroot = case.get_value("SRCROOT")
     exeroot = case.get_value("EXEROOT")
@@ -69,6 +112,8 @@ def _save_build_provenance_e3sm(case, lid):
     with open(submodule_prov, "w") as fd:
         fd.write(subm_status)
 
+    _record_git_provenance(srcroot, exeroot, lid)
+
     # Save SourceMods
     sourcemods = os.path.join(caseroot, "SourceMods")
     sourcemods_prov = os.path.join(exeroot, "SourceMods.{}.tar.gz".format(lid))
@@ -96,7 +141,10 @@ def _save_build_provenance_e3sm(case, lid):
 
     # For all the just-created post-build provenance files, symlink a generic name
     # to them to indicate that these are the most recent or active.
-    for item in ["GIT_DESCRIBE", "GIT_LOGS_HEAD", "GIT_SUBMODULE_STATUS", "SourceMods", "build_environment", "build_times"]:
+    for item in ["GIT_DESCRIBE", "GIT_LOGS_HEAD", "GIT_SUBMODULE_STATUS",
+                 "GIT_STATUS", "GIT_DIFF", "GIT_LOG", "GIT_CONFIG",
+                 "GIT_REMOTE", "SourceMods", "build_environment",
+                 "build_times"]:
         globstr = "{}/{}.{}*".format(exeroot, item, lid)
         matches = glob.glob(globstr)
         expect(len(matches) < 2, "Multiple matches for glob {} should not have happened".format(globstr))


Test patch:
diff --git a/scripts/lib/CIME/tests/test_provenance.py b/scripts/lib/CIME/tests/test_provenance.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/tests/test_provenance.py
@@ -0,0 +1,141 @@
+import os
+import sys
+import unittest
+
+from CIME import provenance
+
+# TODO replace with actual mock once 2.7 is dropped
+class Mocker:
+    calls = []
+
+    def __init__(self, ret=None, cmd=None):
+        self._orig = []
+        self._ret = ret
+        self._cmd = cmd
+
+    def __getattr__(self, name):
+        return Mocker(self._ret, name)
+
+    def __call__(self, *args, **kwargs):
+        args = list(args)
+        if self._cmd is not None:
+            args.insert(0, self._cmd)
+
+        call_sig = " ".join([
+            x for x in args + [
+                "{}={}".format(y, z) for y, z in kwargs.items()]])
+        self.calls.append(call_sig)
+        return self._ret if self._ret is not None else self
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args, **kwargs):
+        for m, module, method in self._orig:
+            setattr(sys.modules[module], method, m)
+
+    def patch(self, method, ret=None):
+        x = method.split('.')
+        main = '.'.join(x[:-1])
+        self._orig.append((getattr(sys.modules[main], x[-1]), main, x[-1]))
+        setattr(sys.modules[main], x[-1], Mocker(ret, x[-1]))
+
+class TestProvenance(unittest.TestCase):
+    def test_run_git_cmd_recursively(self):
+        with Mocker() as mock:
+            Mocker.calls =[]
+            mock.patch("CIME.provenance.run_cmd", (0, "data", None))
+            if sys.version_info.major > 2:
+                mock.patch("builtins.open")
+            else:
+                mock.patch("__builtin__.open")
+
+            provenance._run_git_cmd_recursively('status', '/srcroot', '/output.txt') # pylint: disable=protected-access
+        expected = [
+            "run_cmd git status from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git status; echo\""
+             " from_dir=/srcroot"),
+            "open /output.txt w",
+            "write data\n\n",
+            "write data\n",
+        ]
+
+        self.assertTrue(len(expected) == len(mock.calls))
+
+        for x, y in zip(expected, mock.calls):
+            self.assertTrue(x == y, "{} != {}".format(x, y))
+
+    def test_run_git_cmd_recursively_error(self):
+        with Mocker() as mock:
+            Mocker.calls =[]
+            mock.patch("CIME.provenance.run_cmd", (1, "data", "error"))
+            if sys.version_info.major > 2:
+                mock.patch("builtins.open")
+            else:
+                mock.patch("__builtin__.open")
+
+            provenance._run_git_cmd_recursively('status', '/srcroot', '/output.txt') # pylint: disable=protected-access
+        expected = [
+            "run_cmd git status from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git status; echo\""
+             " from_dir=/srcroot"),
+            "open /output.txt w",
+            "write error\n\n",
+            "write error\n",
+        ]
+
+        self.assertTrue(len(expected) == len(mock.calls))
+
+        for x, y in zip(expected, mock.calls):
+            self.assertTrue(x == y, "{} != {}".format(x, y))
+
+    def test_record_git_provenance(self):
+        with Mocker() as mock:
+            Mocker.calls =[]
+            mock.patch("CIME.provenance.safe_copy")
+            mock.patch("CIME.provenance.run_cmd", (0, "data", None))
+            if sys.version_info.major > 2:
+                mock.patch("builtins.open")
+            else:
+                mock.patch("__builtin__.open")
+
+            provenance._record_git_provenance("/srcroot", "/output", "5") # pylint: disable=protected-access
+
+        expected = [
+            "run_cmd git status from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git status; echo\""
+             " from_dir=/srcroot"),
+            "open /output/GIT_STATUS.5 w",
+            "write data\n\n",
+            "write data\n",
+            "run_cmd git diff from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git diff; echo\""
+             " from_dir=/srcroot"),
+            "open /output/GIT_DIFF.5 w",
+            "write data\n\n",
+            "write data\n",
+            "run_cmd git log --first-parent --pretty=oneline -n 5 from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git log"
+             " --first-parent --pretty=oneline -n 5; echo\" from_dir=/srcroot"),
+            "open /output/GIT_LOG.5 w",
+            "write data\n\n",
+            "write data\n",
+            "run_cmd git remote -v from_dir=/srcroot",
+            ("run_cmd git submodule foreach --recursive \"git remote"
+             " -v; echo\" from_dir=/srcroot"),
+            "open /output/GIT_REMOTE.5 w",
+            "write data\n\n",
+            "write data\n",
+            ("safe_copy /srcroot/.git/config /output/GIT_CONFIG.5"
+             " preserve_meta=False")
+        ]
+
+        self.assertTrue(len(expected) == len(mock.calls),
+                        mock.calls)
+
+        for x, y in zip(expected, mock.calls):
+            self.assertTrue(x == y, "{} != {}".format(x, y))
+
+if __name__ == '__main__':
+    sys.path.insert(0, os.path.abspath(os.path.join('.', '..', '..', 'lib')))
+    unittest.main()