Repository: ESMCI/cime

Issue:
E3SM Provenance: Need to check archival of preview_run.log and GIT_* files
Related issue: https://github.com/ESMCI/cime/issues/3966

There were previously two attempts at recording this information in provenance. However, these files still don't appear in our performance archive. I can see preview_run.log files in a user's run directory but those didn't get archived.
https://github.com/ESMCI/cime/pull/3998
https://github.com/ESMCI/cime/pull/4055

Something to check, the following code snippet from last commit in above PR (https://github.com/ESMCI/cime/pull/4055/commits/b52ce0a07e34e895cd88a8d2545fb5eca7d6c0e0) is _apparently deleting_ the baseline preview_run.log file before making a copy with lid suffix. It should probably make a copy first and then delete. 

https://github.com/ESMCI/cime/blob/master/CIME/provenance.py#L516:L520
```
   if os.path.exists(base_preview_run):
        os.remove(base_preview_run)

    with open(base_preview_run, "w") as fd:
        case.preview_run(lambda x: fd.write("{}\n".format(x)), None)
```

We also need to check why GIT_* files never show up.
https://github.com/ESMCI/cime/blob/master/CIME/provenance.py#L669:675
`        "GIT_STATUS",        "GIT_DIFF",        "GIT_LOG",        "GIT_REMOTE",        "GIT_CONFIG",        "GIT_SUBMODULE_STATUS",
`


Discussion:
FYI @rljacob 
I've confirmed this is an issue with @sarats , these missing files are recorded in the build provenance but are not carried over to the performance archive. There is a fix in progress.
Please also check if the run_e3sm.sh* script is getting properly saved.
@sarats Where is `run_e3sm.sh` stored? From the template script it looks like either `${CASE_ROOT}/case_scripts` for production or `${CASE_ROOT}/tests/${run}/case_scripts` for non production, are these paths static? Do we want to archive both cases or only production?
@golaz should confirm what he prefers. If I recall correctly, the run script used for a particular job is usually stored in the job's run directory.
When I tried the run_e3sm.template.sh, it put a copy of the file with timestamp suffix in ${CASEROOT}/run_script_provenance/.

```
$ ls run_script_provenance/
run_e3sm.template.sh.20220429-124558
```
@sarats Hmm from the script I would have guessed it'd be `${CASE_ROOT}/case_scripts/run_script_provenance`. I'll take a closer look to figure out where it's copied. I really just want to make sure that we shouldn't be looking for it in other places.
FYI, I think the CASE_ROOT variable in the script might be different from CIME CASEROOT. 


```
[ac.sarat@chrlogin1 case_scripts]$ pwd
/home/ac.sarat/E3SMv2/sarat_rune3sm-grp/tests/XS_2x5_ndays/case_scripts
[ac.sarat@chrlogin1 case_scripts]$ ./xmlquery CASEROOT
        CASEROOT: /home/ac.sarat/E3SMv2/sarat_rune3sm-grp/tests/XS_2x5_ndays/case_scripts
[ac.sarat@chrlogin1 case_scripts]$ ls run_script_provenance/
run_e3sm.template.sh.20220429-124558
```


@sarats : archiving script for production runs should be enough.
@sarats I've confirmed that the run script is copied into `run_script_provenance` which is created inside the `CIME` case root.
In the run script, CASE_ROOT is the directory where everything goes:  the CIME CASEROOT, the RUNDIR,  EXEROOT, etc.   In normal use, the CIME CASEROOT is named the same as CASENAME but run_e3sm changes that so that its CASE_ROOT is named after the CASENAME and CIME CASEROOT is a generic "case_scripts".
@rljacob @sarats Looks like we are copying the `run_e3sm.sh*` script, it's under the `CaseDocs` directory in the performance archive.

To find the script we're looking in the  CIME CASEROOT with the glob `run_script_provenance/*`. Is it worth while to update this glob to something more generic like `**/run_e3sm*.sh` in case we ever change where this script gets saved or is the `run_script_provenance/` static directory sufficient?

Resolving patch:
diff --git a/CIME/provenance.py b/CIME/provenance.py
--- a/CIME/provenance.py
+++ b/CIME/provenance.py
@@ -16,6 +16,7 @@
     run_cmd,
     run_cmd_no_fail,
     safe_copy,
+    copy_globs,
 )
 
 import tarfile, getpass, signal, glob, shutil, sys
@@ -411,25 +412,33 @@ def _save_prerun_timing_e3sm(case, lid):
         "Depends.{}.{}".format(mach, compiler),
         "software_environment.txt",
     ]
-    for glob_to_copy in globs_to_copy:
-        for item in glob.glob(os.path.join(caseroot, glob_to_copy)):
-            safe_copy(
-                item,
-                os.path.join(
-                    case_docs, "{}.{}".format(os.path.basename(item).lstrip("."), lid)
-                ),
-                preserve_meta=False,
-            )
+
+    copy_globs([os.path.join(caseroot, x) for x in globs_to_copy], case_docs, lid)
 
     # Copy some items from build provenance
-    blddir_globs_to_copy = ["GIT_LOGS_HEAD", "build_environment.txt", "build_times.txt"]
-    for blddir_glob_to_copy in blddir_globs_to_copy:
-        for item in glob.glob(os.path.join(blddir, blddir_glob_to_copy)):
-            safe_copy(
-                item,
-                os.path.join(full_timing_dir, os.path.basename(item) + "." + lid),
-                preserve_meta=False,
-            )
+    blddir_globs_to_copy = [
+        "GIT_LOGS_HEAD",
+        "GIT_STATUS",
+        "GIT_DIFF",
+        "GIT_LOG",
+        "GIT_REMOTE",
+        "GIT_CONFIG",
+        "GIT_SUBMODULE_STATUS",
+        "build_environment.txt",
+        "build_times.txt",
+    ]
+
+    copy_globs(
+        [os.path.join(blddir, x) for x in blddir_globs_to_copy], full_timing_dir, lid
+    )
+
+    rundir_globs_to_copy = [
+        "preview_run.log",
+    ]
+
+    copy_globs(
+        [os.path.join(rundir, x) for x in rundir_globs_to_copy], full_timing_dir, lid
+    )
 
     # Save state of repo
     from_repo = (
@@ -665,18 +674,6 @@ def _save_postrun_timing_e3sm(case, lid):
     globs_to_copy.append("CaseStatus")
     globs_to_copy.append(os.path.join(rundir, "spio_stats.{}.tar.gz".format(lid)))
     globs_to_copy.append(os.path.join(caseroot, "replay.sh"))
-    # Can't use a single glob, similar files e.g. {filename}.{lid} get picked up.
-    bld_filenames = [
-        "GIT_STATUS",
-        "GIT_DIFF",
-        "GIT_LOG",
-        "GIT_REMOTE",
-        "GIT_CONFIG",
-        "GIT_SUBMODULE_STATUS",
-    ]
-    bld_globs = map(lambda x: f"bld/{x}", bld_filenames)
-    globs_to_copy.extend(bld_globs)
-    globs_to_copy.append("run/preview_run.log")
 
     for glob_to_copy in globs_to_copy:
         for item in glob.glob(os.path.join(caseroot, glob_to_copy)):
diff --git a/CIME/utils.py b/CIME/utils.py
--- a/CIME/utils.py
+++ b/CIME/utils.py
@@ -1312,6 +1312,30 @@ def get_current_submodule_status(recursive=False, repo=None):
     return output if rc == 0 else "unknown"
 
 
+def copy_globs(globs_to_copy, output_directory, lid=None):
+    """
+    Takes a list of globs and copies all files to `output_directory`.
+
+    Hiddens files become unhidden i.e. removing starting dot.
+
+    Output filename is derviced from the basename of the input path and can
+    be appended with the `lid`.
+
+    """
+    for glob_to_copy in globs_to_copy:
+        for item in glob.glob(glob_to_copy):
+            item_basename = os.path.basename(item).lstrip(".")
+
+            if lid is None:
+                filename = item_basename
+            else:
+                filename = f"{item_basename}.{lid}"
+
+            safe_copy(
+                item, os.path.join(output_directory, filename), preserve_meta=False
+            )
+
+
 def safe_copy(src_path, tgt_path, preserve_meta=True):
     """
     A flexbile and safe copy routine. Will try to copy file and metadata, but this


Test patch:
diff --git a/CIME/tests/test_unit_utils.py b/CIME/tests/test_unit_utils.py
--- a/CIME/tests/test_unit_utils.py
+++ b/CIME/tests/test_unit_utils.py
@@ -13,6 +13,7 @@
     import_from_file,
     _line_defines_python_function,
     file_contains_python_function,
+    copy_globs,
 )
 
 
@@ -192,6 +193,29 @@ def _error_func(*args):
 
         self.error_func = _error_func
 
+    @mock.patch("glob.glob")
+    @mock.patch("CIME.utils.safe_copy")
+    def test_copy_globs(self, safe_copy, glob):
+        glob.side_effect = [
+            [],
+            ["/src/run/test.sh", "/src/run/.hidden.sh"],
+            [
+                "/src/bld/test.nc",
+            ],
+        ]
+
+        copy_globs(["CaseDocs/*", "run/*.sh", "bld/*.nc"], "/storage/output", "uid")
+
+        safe_copy.assert_any_call(
+            "/src/run/test.sh", "/storage/output/test.sh.uid", preserve_meta=False
+        )
+        safe_copy.assert_any_call(
+            "/src/run/.hidden.sh", "/storage/output/hidden.sh.uid", preserve_meta=False
+        )
+        safe_copy.assert_any_call(
+            "/src/bld/test.nc", "/storage/output/test.nc.uid", preserve_meta=False
+        )
+
     def assertMatchAllLines(self, tempdir, test_lines):
         with open(os.path.join(tempdir, "CaseStatus")) as fd:
             data = fd.readlines()