Repository: ESMCI/cime

Issue:
Include submodule status in provenance
Right now there is no mechanist to find submodules commits from bld folder, only main repo's commit is saved. 

Separately, maybe not for this issue, but it would be good to save status of the clone in bld, not only SHA of the main repo. Right now I have a script that does this to catch possibly all recent changes in it:

```
gitstat=gitstat."${id}"
cd ~/${which}
echo " running stats on clone ${which}"
echo " status is ------------- " >> ${gitstat}
git status >> ${gitstat}
echo " branch is ------------- " >> ${gitstat}
git branch >> ${gitstat}
echo " diffs are ------------- " >> ${gitstat}
git diff >> ${gitstat}
echo " last 10 commits are ------------- " >> ${gitstat}
git log --first-parent  --pretty=oneline  HEAD~10..HEAD  >> ${gitstat}
```



Discussion:
@WesCoomber , I'm assigning you some of my tickets just to give you a few things to choose from.
@oksanaguba Hi Oksana, I am looking into this issue. 

_Is "gitstat" a tool to get SHAs? Could you provide me with more information about this tool and your shell script that you use to find submodules commits from the bld folder?_ (eg. where I can get a copy to recreate the intended behavior, and how you use them.)
Above, gitstat is just a part of a file name; full name is gitstat.${JOB_ID} after the case is submitted and JOB_ID is available.  Above are a few lines of my script that dumps info from git to the file. There ${which} is path to the clone that was used to create a case. It was used in this script like that:
```
which=acme-b3

fol=theta-ne30-nh-oct27

${which}/cime/scripts/create_newcase -case $fol -res ne30_ne30 -mach anvil -compiler intel -compset FC5AV1C-L
cd $fol

./case.setup --reset
./case.setup


./case.build
./xmlchange STOP_N=6 ; ./xmlchange STOP_OPTION=nsteps ; 
#./xmlchange REST_N=7 ; ./xmlchange REST_OPTION=nyears ;


./case.submit

#we only need the latest ID
id=`grep -o case.run:\[0-9\]\* CaseStatus | sed -e 's/.*://' | tail -1`
echo $id


gitstat=gitstat."${id}"
cd ~/${which}
echo " running stats on clone ${which}"
echo " status is ------------- " >> ${gitstat}
git status >> ${gitstat}
echo " branch is ------------- " >> ${gitstat}
git branch >> ${gitstat}
echo " diffs are ------------- " >> ${gitstat}
git diff >> ${gitstat}
echo " last 10 commits are ------------- " >> ${gitstat}
git log --first-parent  --pretty=oneline  HEAD~10..HEAD  >> ${gitstat}

cd ~/$fol
mv ~/${which}/${gitstat} .

```


Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -5,7 +5,7 @@
 """
 
 from CIME.XML.standard_module_setup import *
-from CIME.utils import touch, gzip_existing_file, SharedArea, convert_to_babylonian_time, get_current_commit, indent_string, run_cmd, run_cmd_no_fail, safe_copy
+from CIME.utils import touch, gzip_existing_file, SharedArea, convert_to_babylonian_time, get_current_commit, get_current_submodule_status, indent_string, run_cmd, run_cmd_no_fail, safe_copy
 
 import tarfile, getpass, signal, glob, shutil, sys
 
@@ -49,6 +49,12 @@ def _save_build_provenance_e3sm(case, lid):
     if os.path.exists(headfile):
         safe_copy(headfile, headfile_prov, preserve_meta=False)
 
+    # Save git submodule status
+    submodule_prov = os.path.join(exeroot, "GIT_SUBMODULE_STATUS.{}".format(lid))
+    subm_status = get_current_submodule_status(recursive=True, repo=srcroot)
+    with open(submodule_prov, "w") as fd:
+        fd.write(subm_status)
+
     # Save SourceMods
     sourcemods = os.path.join(caseroot, "SourceMods")
     sourcemods_prov = os.path.join(exeroot, "SourceMods.{}.tar.gz".format(lid))
@@ -67,7 +73,7 @@ def _save_build_provenance_e3sm(case, lid):
 
     # For all the just-created post-build provenance files, symlink a generic name
     # to them to indicate that these are the most recent or active.
-    for item in ["GIT_DESCRIBE", "GIT_LOGS_HEAD", "SourceMods", "build_environment"]:
+    for item in ["GIT_DESCRIBE", "GIT_LOGS_HEAD", "GIT_SUBMODULE_STATUS", "SourceMods", "build_environment"]:
         globstr = "{}/{}.{}*".format(exeroot, item, lid)
         matches = glob.glob(globstr)
         expect(len(matches) < 2, "Multiple matches for glob {} should not have happened".format(globstr))
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -862,6 +862,18 @@ def match_any(item, re_list):
 
     return False
 
+def get_current_submodule_status(recursive=False, repo=None):
+    """
+    Return the sha1s of the current currently checked out commit for each submodule,
+    along with the submodule path and the output of git describe for the SHA-1.
+
+    >>> get_current_submodule_status() is not None
+    True
+    """
+    rc, output, _ = run_cmd("git submodule status {}".format("--recursive" if recursive else ""), from_dir=repo)
+
+    return output if rc == 0 else "unknown"
+
 def safe_copy(src_path, tgt_path, preserve_meta=True):
     """
     A flexbile and safe copy routine. Will try to copy file and metadata, but this


Test patch: