Repository: ESMCI/cime

Issue:
CIME errors with git worktree
I believe I have been able to use E3SM just fine in the past from a git worktree.  Here is a simplified example workflow on Chrysalis:
```
git clone git@github.com:E3SM-Project/E3SM.git
cd E3SM
git worktree add ../test
cd ../test
git submodule update --init --recursive
cd cime/scripts/
./create_test --walltime 01:00:00 SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel --wait
```

The error I'm seeing is:
```
No project info available
create_test will do up to 1 tasks simultaneously
create_test will use up to 160 cores simultaneously
Creating test directory /lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c
RUNNING TESTS:
  SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel
Starting CREATE_NEWCASE for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel with 1 procs
Finished CREATE_NEWCASE for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel in 0.975402 seconds (PASS)
Starting XML for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel with 1 procs
Finished XML for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel in 0.189899 seconds (PASS)
Starting SETUP for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel with 1 procs
Finished SETUP for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel in 3.179292 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel with 1 procs
Finished SHAREDLIB_BUILD for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel in 163.753454 seconds (PASS)
Starting MODEL_BUILD for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel with 6 procs
Finished MODEL_BUILD for test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel in 904.674959 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c
    Errors were:
        Building test for SMS in directory /lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c
        Traceback (most recent call last):
          File "/lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c/./case.build", line 167, in <module>
            _main_func(__doc__)
          File "/lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c/./case.build", line 156, in _main_func
            success = test.build(sharedlib_only=sharedlib_only, model_only=model_only, ninja=ninja, dry_run=dry_run, separate_builds=separate_builds)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 95, in build
            self.build_phase(sharedlib_only=(phase_name==SHAREDLIB_BUILD_PHASE),
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 124, in build_phase
            self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 131, in build_indv
            build.case_build(self._caseroot, case=self._case,
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 793, in case_build
            return run_and_log_case_status(functor, cb, caseroot=caseroot)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1810, in run_and_log_case_status
            rv = func()
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 786, in <lambda>
            functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist,
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 751, in _case_build_impl
            post_build(case, logs, build_complete=not (buildlist or sharedlib_only),
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 772, in post_build
            save_build_provenance_sub(case, lid=lid)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/provenance.py", line 199, in save_build_provenance
            _save_build_provenance_e3sm(case, lid)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/provenance.py", line 115, in _save_build_provenance_e3sm
            _record_git_provenance(srcroot, exeroot, lid)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/provenance.py", line 88, in _record_git_provenance
            safe_copy(config_src, config_prov, preserve_meta=False)
          File "/gpfs/fs1/home/ac.xylar/e3sm_work/test/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 937, in safe_copy
            file_util.copy_file(src_path, tgt_path, preserve_mode=preserve_meta, preserve_times=preserve_meta)
          File "/home/ac.xylar/chrysalis/miniconda3/lib/python3.9/distutils/file_util.py", line 104, in copy_file
            raise DistutilsFileError(
        distutils.errors.DistutilsFileError: can't copy '/gpfs/fs1/home/ac.xylar/e3sm_work/test/.git/config': doesn't exist or not a regular file

Waiting for tests to finish
FAIL SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel (phase MODEL_BUILD)
    Case dir: /lcrc/group/e3sm/ac.xylar/scratch/chrys/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.chrysalis_intel.20210712_130312_hpod6c
```

CIME seems to be expecting to find a `.git/config` file, but a git worktree has a `.git` file, rather than a `.git` directory.  The use of `git woktree` is pretty central to my workflow, so this makes E3SM development a bit challenging for me.


Discussion:
This seems to be an issue with save_build_provenance_e3sm and so I have marked it e3sm only.
Thanks @jedwards4b.  Am I right that it's a CIME issue, rather than one for the E3SM repo?
@xylar yes - I think so.
@xylar I also use git worktrees heavily and haven't noticed any issues recently, so also suspect this is e3sm-specific (unless there's a recent issue that I haven't run into).
@billsacks , @xylar , this is an e3sm-specific issue in CIME. We want to, at some point, move more of the e3sm-specific stuff in CIME into E3SM. We haven't done this yet, so our e3sm provenance still lives in CIME.

Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -60,6 +60,38 @@ def _run_git_cmd_recursively(cmd, srcroot, output):
         fd.write((output1 if rc1 == 0 else err1) + "\n\n")
         fd.write((output2 if rc2 == 0 else err2) + "\n")
 
+def _parse_dot_git_path(srcroot):
+    dot_git_pattern = r"^(.*/\.git).*"
+
+    m = re.match(dot_git_pattern, srcroot)
+
+    expect(m is not None, f"Could not parse git path from {srcroot!r}")
+
+    return m.group(1)
+
+def _find_git_root(srcroot):
+    gitroot = f"{srcroot}/.git"
+
+    expect(os.path.exists(gitroot),
+           f"{srcroot!r} is not a git repository, failed to collect provenance")
+
+    # Handle normal git repositories
+    if os.path.isdir(gitroot):
+        return gitroot
+
+    # Handle git worktrees
+    with open(gitroot) as fd:
+        line = fd.readline()
+
+    gitdir_pattern = r"^gitdir:\s?(.*)$"
+
+    m = re.match(gitdir_pattern, line)
+
+    expect(m is not None, f"Could not determine git root in {srcroot!r}")
+
+    # First group is the actual gitroot
+    return m.group(1)
+
 def _record_git_provenance(srcroot, exeroot, lid):
     """ Records git provenance
 
@@ -82,8 +114,11 @@ def _record_git_provenance(srcroot, exeroot, lid):
     remote_prov = os.path.join(exeroot, "GIT_REMOTE.{}".format(lid))
     _run_git_cmd_recursively("remote -v", srcroot, remote_prov)
 
+    gitroot = _find_git_root(srcroot)
+    gitroot = _parse_dot_git_path(gitroot)
+
     # Git config
-    config_src = os.path.join(srcroot, ".git", "config")
+    config_src = os.path.join(gitroot, "config")
     config_prov = os.path.join(exeroot, "GIT_CONFIG.{}".format(lid))
     safe_copy(config_src, config_prov, preserve_meta=False)
 
@@ -98,8 +133,10 @@ def _save_build_provenance_e3sm(case, lid):
     with open(describe_prov, "w") as fd:
         fd.write(desc)
 
+    gitroot = _find_git_root(srcroot)
+
     # Save HEAD
-    headfile = os.path.join(srcroot, ".git", "logs", "HEAD")
+    headfile = os.path.join(gitroot, "logs", "HEAD")
     headfile_prov = os.path.join(exeroot, "GIT_LOGS_HEAD.{}".format(lid))
     if os.path.exists(headfile_prov):
         os.remove(headfile_prov)


Test patch:
diff --git a/scripts/lib/CIME/tests/test_provenance.py b/scripts/lib/CIME/tests/test_provenance.py
--- a/scripts/lib/CIME/tests/test_provenance.py
+++ b/scripts/lib/CIME/tests/test_provenance.py
@@ -2,12 +2,54 @@
 
 import os
 import sys
+import tempfile
 import unittest
 from unittest import mock
 
 from CIME import provenance
+from CIME import utils
 
+# pylint: disable=protected-access
 class TestProvenance(unittest.TestCase):
+    def test_parse_dot_git_path_error(self):
+        with self.assertRaises(utils.CIMEError):
+            provenance._parse_dot_git_path("/src/CIME")
+
+    def test_parse_dot_git_path(self):
+        value = provenance._parse_dot_git_path("/src/CIME/.git/worktrees/test")
+
+        assert value == "/src/CIME/.git"
+
+    def test_find_git_root(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            os.makedirs(os.path.join(tempdir, ".git"))
+
+            value = provenance._find_git_root(tempdir)
+
+            assert value == f"{tempdir}/.git"
+
+    def test_find_git_root_worktree(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            with open(os.path.join(tempdir, ".git"), "w") as fd:
+                fd.write("gitdir: /src/CIME/.git/worktrees/test")
+
+            value = provenance._find_git_root(tempdir)
+
+            assert value == "/src/CIME/.git/worktrees/test"
+
+    def test_find_git_root_worktree_malformed(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            with open(os.path.join(tempdir, ".git"), "w") as fd:
+                fd.write("some value: /src/CIME/.git/worktrees/test")
+
+            with self.assertRaises(utils.CIMEError):
+                provenance._find_git_root(tempdir)
+
+    def test_find_git_root_error(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            with self.assertRaises(utils.CIMEError):
+                provenance._find_git_root(tempdir)
+
     @mock.patch("CIME.provenance.run_cmd")
     def test_run_git_cmd_recursively(self, run_cmd):
         run_cmd.return_value = (0, "data", None)
@@ -52,7 +94,10 @@ def test_record_git_provenance(self, run_cmd, safe_copy):
         run_cmd.return_value = (0, "data", None)
 
         with mock.patch("CIME.provenance.open", mock.mock_open()) as m:
-            provenance._record_git_provenance("/srcroot", "/output", "5") # pylint: disable=protected-access
+            with tempfile.TemporaryDirectory() as tempdir:
+                os.makedirs(os.path.join(tempdir, ".git"))
+
+                provenance._record_git_provenance(tempdir, "/output", "5") # pylint: disable=protected-access
 
         m.assert_any_call("/output/GIT_STATUS.5", "w")
         m.assert_any_call("/output/GIT_DIFF.5", "w")
@@ -64,31 +109,31 @@ def test_record_git_provenance(self, run_cmd, safe_copy):
         write.assert_any_call("data\n\n")
         write.assert_any_call("data\n")
 
-        run_cmd.assert_any_call("git status", from_dir="/srcroot")
+        run_cmd.assert_any_call("git status", from_dir=tempdir)
         run_cmd.assert_any_call(
             "git submodule foreach --recursive \"git status; echo\"",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git diff",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git submodule foreach --recursive \"git diff; echo\"",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git log --first-parent --pretty=oneline -n 5",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git submodule foreach --recursive \"git log --first-parent"
             " --pretty=oneline -n 5; echo\"",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git remote -v",
-            from_dir="/srcroot")
+            from_dir=tempdir)
         run_cmd.assert_any_call(
             "git submodule foreach --recursive \"git remote -v; echo\"",
-            from_dir="/srcroot")
+            from_dir=tempdir)
 
-        safe_copy.assert_any_call("/srcroot/.git/config",
+        safe_copy.assert_any_call(f"{tempdir}/.git/config",
                                   "/output/GIT_CONFIG.5",
                                   preserve_meta=False)