Repository: ESMCI/cime

Issue:
Provenance appears not to be handling relative paths correctly
The [compass](https://github.com/MPAS-Dev/compass) software for testing MPAS-Ocean and MALI contains E3SM as a submodule.  A common workflow is to create a worktree of compass and then to test from the E3SM submodule within the compass worktree.

Today, I encountered an error I have not previously seen with this workflow:
```
$ ./create_test --wait --walltime 01:00:00 ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel
No project info available
create_test will do up to 1 tasks simultaneously
create_test will use up to 160 cores simultaneously
Creating test directory /lcrc/group/e3sm/ac.xylar/scratch/chrys/ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel.20220507_115857_puvq8g
RUNNING TESTS:
  ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel
Starting CREATE_NEWCASE for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel with 1 procs
Finished CREATE_NEWCASE for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel in 0.594735 seconds (PASS)
Starting XML for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel with 1 procs
Finished XML for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel in 0.146541 seconds (PASS)
Starting SETUP for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel with 1 procs
Finished SETUP for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel in 5.213597 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel with 1 procs
Finished SHAREDLIB_BUILD for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel in 186.838065 seconds (PASS)
Starting MODEL_BUILD for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel with 6 procs
Finished MODEL_BUILD for test ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel in 994.970013 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /lcrc/group/e3sm/ac.xylar/scratch/chrys/ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel.20220507_115857_puvq8g
    Errors were:
        Building test for ERS in directory /lcrc/group/e3sm/ac.xylar/scratch/chrys/ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel.20220507_115857_puvq8g
        Traceback (most recent call last):
          File "./case.build", line 256, in <module>
            _main_func(__doc__)
          File "./case.build", line 236, in _main_func
            separate_builds=separate_builds,
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/SystemTests/system_tests_common.py", line 126, in build
            model_only=(phase_name == MODEL_BUILD_PHASE),
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/SystemTests/system_tests_common.py", line 161, in build_phase
            self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/SystemTests/system_tests_common.py", line 176, in build_indv
            separate_builds=self._user_separate_builds,
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/build.py", line 1285, in case_build
            return run_and_log_case_status(functor, cb, caseroot=caseroot)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/utils.py", line 2417, in run_and_log_case_status
            rv = func()
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/build.py", line 1278, in <lambda>
            dry_run,
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/build.py", line 1220, in _case_build_impl
            save_build_provenance=save_build_provenance,
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/build.py", line 1244, in post_build
            save_build_provenance_sub(case, lid=lid)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/provenance.py", line 284, in save_build_provenance
            _save_build_provenance_e3sm(case, lid)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/provenance.py", line 181, in _save_build_provenance_e3sm
            _record_git_provenance(srcroot, exeroot, lid)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/provenance.py", line 151, in _record_git_provenance
            safe_copy(config_src, config_prov, preserve_meta=False)
          File "/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/cime/CIME/utils.py", line 1375, in safe_copy
            preserve_times=preserve_meta,
          File "/usr/lib64/python3.6/distutils/file_util.py", line 105, in copy_file
            "can't copy '%s': doesn't exist or not a regular file" % src)
        distutils.errors.DistutilsFileError: can't copy '../../compass/.git/config': doesn't exist or not a regular file
```

My hunch is that this relative path `../../compass/.git/config` is not being handled correctly because it is relative to the E3SM root rather than the `cime/scripts` subdirectory where I am running `./create_test`.  But this is only a hunch.

Steps to reproduce the problem on Chrysalis:
```
git clone git@github.com:MPAS-Dev/compass.git
cd compass
git worktree add ../test
cd ../test
git submodule update --init --recursive
cd E3SM-Project/cime/scripts
./create_test --wait --walltime 01:00:00 ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel
```



Discussion:
@jgfouca, are there any recent changes to CIME provenance (in E3SM) that might explain this?
@xylar , I'm looking at the git log for this file and seeing only minimal changes in recent months. One significant thing that did happen was the CIME reorganization which moved many of the files around, but, looking at the implementation of _record_git_provenance:

```
def _record_git_provenance(srcroot, exeroot, lid):
    """Records git provenance                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                  
    Records git status, diff and logs for main repo and all submodules.                                                                                                                                                                           
    """
    # Git Status                                                                                                                                                                                                                                  
    status_prov = os.path.join(exeroot, "GIT_STATUS.{}".format(lid))
    _run_git_cmd_recursively("status", srcroot, status_prov)

    # Git Diff                                                                                                                                                                                                                                    
    diff_prov = os.path.join(exeroot, "GIT_DIFF.{}".format(lid))
    _run_git_cmd_recursively("diff", srcroot, diff_prov)

    # Git Log                                                                                                                                                                                                                                     
    log_prov = os.path.join(exeroot, "GIT_LOG.{}".format(lid))
    cmd = "log --first-parent --pretty=oneline -n 5"
    _run_git_cmd_recursively(cmd, srcroot, log_prov)

    # Git remote                                                                                                                                                                                                                                  
    remote_prov = os.path.join(exeroot, "GIT_REMOTE.{}".format(lid))
    _run_git_cmd_recursively("remote -v", srcroot, remote_prov)

    gitroot = _find_git_root(srcroot)
    gitroot = _parse_dot_git_path(gitroot)

    # Git config                                                                                                                                                                                                                                  
    config_src = os.path.join(gitroot, "config")
    config_prov = os.path.join(exeroot, "GIT_CONFIG.{}".format(lid))
    safe_copy(config_src, config_prov, preserve_meta=False)
```

... it doesn't look like anything should be sensitive to the location of the current source file. The error indicates that `gitroot` is incorrect, which means there could be a problem with srcroot (unlikely), _find_get_root, or _parse_dot_git_path. Could you maybe add some print statements around this code:

```
    gitroot = _find_git_root(srcroot)
    gitroot = _parse_dot_git_path(gitroot)
```

to help us see what's going on?
@jgfouca, sure, I can do that.  It will be easiest to wait for LCRC to come back up, since that's where I was testing.
@jgfouca, now that Chrysalis is back, I rad again.  Here's what I'm seeing:
```
srcroot: /gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project
gitroot: ../../compass/.git/worktrees/test_stream_read_input_only/modules/E3SM-Project
gitroot: ../../compass/.git
```
I assume this is obvious but the first `gitroot` is after `_find_git_root()` and the second is after `_parse_dot_git_path()`.  While I don't fully understand the intended outcome of `_parse_dot_git_path()`, I'd hazard a guess that that isn't producing the expected result.
Let me know if I can add more prints to help.  It would also be helpful for me to know if there's a quicker way to reproduce this than rerunning the full test, since that takes about 20 minutes.
Actually, looking at the code, I think what is missing is something to convert a relative path to an absolute path using the original `srcroot`.  At least that seems the most likely problem based on my limited understanding.
``` python
config_src = os.path.join(srcroot, gitroot, "config")
```
would seem to take care of this whether `gitroot` is a relative or absolute path, though it would be more intuitive to so:
``` python
if not os.path.isabs(gitroot):
    gitroot = os.path.abspath(os.path.join(srcroot, gitroot))
config_src = os.path.join(gitroot, "config")
```

Indeed, that seems to fix the problem.  @jgfouca, would you like me to make a PR?
@xylar @jgfouca

From the debug text
```
srcroot: /gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project
gitroot: ../../compass/.git/worktrees/test_stream_read_input_only/modules/E3SM-Project
gitroot: ../../compass/.git
```
the output of
```
gitroot = _find_git_root(srcroot)
gitroot = _parse_dot_git_path(gitroot)
```
should be `/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project` not `/gpfs/fs1/home/ac.xylar/mpas-work/compass`.

The above fix should be added but I think there might be a bigger issue with determining the correct `gitroot`.
@jasonb5, I don't think that's right.  The code is correctly detecting that `/gpfs/fs1/home/ac.xylar/mpas-work/compass/test_stream_read_input_only/E3SM-Project/.git` is a file not a directory.  It needs to go to `/gpfs/fs1/home/ac.xylar/mpas-work/compass/compass/.git` to find a `config` file.  There is no such file within the worktree.
@xylar The E3SM `config` is located in `../../compass/.git/worktrees/test_stream_read_input_only/modules/E3SM-Project`, the provenance should be recording this `config` not the one under `/gpfs/fs1/home/ac.xylar/mpas-work/compass/.git`.
Ah, I see.
Yes, I think you're absolutely right.

Resolving patch:
diff --git a/CIME/provenance.py b/CIME/provenance.py
--- a/CIME/provenance.py
+++ b/CIME/provenance.py
@@ -20,6 +20,7 @@
 )
 
 import tarfile, getpass, signal, glob, shutil, sys
+from contextlib import contextmanager
 
 logger = logging.getLogger(__name__)
 
@@ -85,17 +86,88 @@ def _run_git_cmd_recursively(cmd, srcroot, output):
         fd.write((output2 if rc2 == 0 else err2) + "\n")
 
 
-def _parse_dot_git_path(srcroot):
+def _parse_dot_git_path(gitdir):
+    """Parse `.git` path.
+
+    Take a path e.g. `/storage/cime/.git/worktrees/cime` and parse `.git`
+    directory e.g. `/storage/cime/.git`.
+
+    Args:
+        gitdir (str): Path containing `.git` directory.
+
+    Returns:
+        str: Path ending with `.git`
+    """
     dot_git_pattern = r"^(.*/\.git).*"
 
-    m = re.match(dot_git_pattern, srcroot)
+    m = re.match(dot_git_pattern, gitdir)
+
+    expect(m is not None, f"Could not parse .git from path {gitdir!r}")
+
+    return m.group(1)
+
+
+def _read_gitdir(gitroot):
+    """Read `gitdir` from `.git` file.
+
+    Reads `.git` file in a worktree or submodule and parse `gitdir`.
+
+    Args:
+        gitroot (str): Path ending with `.git` file.
+
+    Returns:
+        str: Path contained in `.git` file.
+    """
+    expect(os.path.isfile(gitroot), f"Expected {gitroot!r} to be a file")
+
+    with open(gitroot) as fd:
+        line = fd.readline()
+
+    gitdir_pattern = r"^gitdir:\s*(.*)$"
+
+    m = re.match(gitdir_pattern, line)
 
-    expect(m is not None, f"Could not parse git path from {srcroot!r}")
+    expect(m is not None, f"Could parse gitdir path from file {gitroot!r}")
 
     return m.group(1)
 
 
+@contextmanager
+def _swap_cwd(new_cwd):
+    old_cwd = os.getcwd()
+
+    os.chdir(new_cwd)
+
+    try:
+        yield
+    finally:
+        os.chdir(old_cwd)
+
+
 def _find_git_root(srcroot):
+    """Finds the `.git` directory.
+
+    NOTICE: It's assumed `srcroot` is an absolute path.
+
+    There are three scenarios to locate the correct `.git` directory:
+
+    NOTICE: In the 3rd case git `.git` directory is not actually `.git`.
+
+    1. In a standard git repository it will be located at `{srcroot}/.git`.
+    2. In a git worktree the `{srcroot}/.git` is a file containing a path,
+       `{gitdir}`, which the `{gitroot}` can be parsed from.
+    3. In a git submodule the `{srcroot}/.git` is a file containing a path,
+       `{gitdir}`, where the `{gitroot}` is `{gitdir}`.
+
+    To aid in finding the correct `{gitroot}` the file `{gitroot}/config`
+    is checked, this file will always be located in the correct directory.
+
+    Args:
+        srcroot (str): Path to the source root.
+
+    Returns:
+        str: Absolute path to `.git` directory.
+    """
     gitroot = f"{srcroot}/.git"
 
     expect(
@@ -103,22 +175,24 @@ def _find_git_root(srcroot):
         f"{srcroot!r} is not a git repository, failed to collect provenance",
     )
 
-    # Handle normal git repositories
+    # Handle 1st scenario
     if os.path.isdir(gitroot):
         return gitroot
 
-    # Handle git worktrees
-    with open(gitroot) as fd:
-        line = fd.readline()
+    # ensure we're in correct directory so abspath works correctly
+    with _swap_cwd(srcroot):
+        gitdir = os.path.abspath(_read_gitdir(gitroot))
 
-    gitdir_pattern = r"^gitdir:\s?(.*)$"
+    # Handle 3rd scenario, gitdir is the `.git` directory
+    config_path = os.path.join(gitdir, "config")
 
-    m = re.match(gitdir_pattern, line)
+    if os.path.exists(config_path):
+        return gitdir
 
-    expect(m is not None, f"Could not determine git root in {srcroot!r}")
+    # Handle 2nd scenario, gitdir should already be absolute path
+    parsed_gitroot = _parse_dot_git_path(gitdir)
 
-    # First group is the actual gitroot
-    return m.group(1)
+    return parsed_gitroot
 
 
 def _record_git_provenance(srcroot, exeroot, lid):
@@ -144,7 +218,6 @@ def _record_git_provenance(srcroot, exeroot, lid):
     _run_git_cmd_recursively("remote -v", srcroot, remote_prov)
 
     gitroot = _find_git_root(srcroot)
-    gitroot = _parse_dot_git_path(gitroot)
 
     # Git config
     config_src = os.path.join(gitroot, "config")


Test patch:
diff --git a/CIME/tests/test_unit_provenance.py b/CIME/tests/test_unit_provenance.py
--- a/CIME/tests/test_unit_provenance.py
+++ b/CIME/tests/test_unit_provenance.py
@@ -20,6 +20,43 @@ def test_parse_dot_git_path(self):
 
         assert value == "/src/CIME/.git"
 
+    def test_read_gitdir(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            dot_git_path = os.path.join(tempdir, ".git")
+
+            with open(dot_git_path, "w") as fd:
+                fd.write("gitdir:     /src/CIME/.git/worktrees/test")
+
+            value = provenance._read_gitdir(dot_git_path)
+
+            assert value == "/src/CIME/.git/worktrees/test"
+
+            with open(dot_git_path, "w") as fd:
+                fd.write("gitdir:/src/CIME/.git/worktrees/test")
+
+            value = provenance._read_gitdir(dot_git_path)
+
+            assert value == "/src/CIME/.git/worktrees/test"
+
+    def test_read_gitdir_not_file(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            dot_git_path = os.path.join(tempdir, ".git")
+
+            os.makedirs(dot_git_path)
+
+            with self.assertRaises(utils.CIMEError):
+                provenance._read_gitdir(dot_git_path)
+
+    def test_read_gitdir_bad_contents(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            dot_git_path = os.path.join(tempdir, ".git")
+
+            with open(dot_git_path, "w") as fd:
+                fd.write("some value: /src/CIME/.git/worktrees/test")
+
+            with self.assertRaises(utils.CIMEError):
+                provenance._read_gitdir(dot_git_path)
+
     def test_find_git_root(self):
         with tempfile.TemporaryDirectory() as tempdir:
             os.makedirs(os.path.join(tempdir, ".git"))
@@ -28,16 +65,55 @@ def test_find_git_root(self):
 
             assert value == f"{tempdir}/.git"
 
+    def test_find_git_root_does_not_exist(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            with self.assertRaises(utils.CIMEError):
+                provenance._find_git_root(tempdir)
+
+    def test_find_git_root_submodule(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            cime_path = os.path.join(tempdir, "cime")
+
+            os.makedirs(cime_path)
+
+            cime_git_dot_file = os.path.join(cime_path, ".git")
+
+            with open(cime_git_dot_file, "w") as fd:
+                fd.write(f"gitdir: {tempdir}/.git/modules/cime")
+
+            temp_dot_git_path = os.path.join(tempdir, ".git", "modules", "cime")
+
+            os.makedirs(temp_dot_git_path)
+
+            temp_config = os.path.join(temp_dot_git_path, "config")
+
+            with open(temp_config, "w") as fd:
+                fd.write("")
+
+            value = provenance._find_git_root(cime_path)
+
+            assert value == f"{tempdir}/.git/modules/cime"
+
+            # relative path
+            with open(cime_git_dot_file, "w") as fd:
+                fd.write(f"gitdir: ../.git/modules/cime")
+
+            value = provenance._find_git_root(cime_path)
+
+            assert value == f"{tempdir}/.git/modules/cime"
+
     def test_find_git_root_worktree(self):
         with tempfile.TemporaryDirectory() as tempdir:
-            with open(os.path.join(tempdir, ".git"), "w") as fd:
+            git_dot_file = os.path.join(tempdir, ".git")
+
+            with open(git_dot_file, "w") as fd:
                 fd.write("gitdir: /src/CIME/.git/worktrees/test")
 
             value = provenance._find_git_root(tempdir)
 
-            assert value == "/src/CIME/.git/worktrees/test"
+            assert value == "/src/CIME/.git"
 
-    def test_find_git_root_worktree_malformed(self):
+    def test_find_git_root_worktree_bad_contents(self):
         with tempfile.TemporaryDirectory() as tempdir:
             with open(os.path.join(tempdir, ".git"), "w") as fd:
                 fd.write("some value: /src/CIME/.git/worktrees/test")
@@ -45,11 +121,6 @@ def test_find_git_root_worktree_malformed(self):
             with self.assertRaises(utils.CIMEError):
                 provenance._find_git_root(tempdir)
 
-    def test_find_git_root_error(self):
-        with tempfile.TemporaryDirectory() as tempdir:
-            with self.assertRaises(utils.CIMEError):
-                provenance._find_git_root(tempdir)
-
     @mock.patch("CIME.provenance.run_cmd")
     def test_run_git_cmd_recursively(self, run_cmd):
         run_cmd.return_value = (0, "data", None)