Repository: LUMC/pytest-workflow (language: python)

Issue:
Collecting tests fails due to git submodules and symlinks
My issue is similar to #139 and I'm on the most recent commit on the development branch ([ef7f312](https://github.com/LUMC/pytest-workflow/commit/ef7f3121f689f9ff1ba8b477ce5e081a23d24a88)). The error appear to be related to copying a symlink within a git submodule:

```
$ git clone --recurse-submodules https://github.com/wholtz/pytest-workflow-submodule-issue.git
Cloning into 'pytest-workflow-submodule-issue'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 9 (delta 0), reused 9 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), done.
Submodule 'ci-functions' (https://code.jgi.doe.gov/user-support-analysis/ci-functions.git) registered for path '.ci-functions'
Cloning into '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions'...
remote: Enumerating objects: 608, done.        
remote: Counting objects: 100% (4/4), done.        
remote: Compressing objects: 100% (4/4), done.        
remote: Total 608 (delta 0), reused 0 (delta 0), pack-reused 604        
Receiving objects: 100% (608/608), 98.69 KiB | 5.80 MiB/s, done.
Resolving deltas: 100% (373/373), done.
Submodule path '.ci-functions': checked out '001b2feb1bcd641589571bb6e13a37e998b36057'
Submodule 'test/bats' (https://github.com/bats-core/bats-core.git) registered for path '.ci-functions/test/bats'
Submodule 'test/test_helper/bats-assert' (https://github.com/bats-core/bats-assert.git) registered for path '.ci-functions/test/test_helper/bats-assert'
Submodule 'test/test_helper/bats-support' (https://github.com/bats-core/bats-support.git) registered for path '.ci-functions/test/test_helper/bats-support'
Cloning into '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats'...
remote: Enumerating objects: 7709, done.        
remote: Counting objects: 100% (560/560), done.        
remote: Compressing objects: 100% (149/149), done.        
remote: Total 7709 (delta 430), reused 494 (delta 390), pack-reused 7149        
Receiving objects: 100% (7709/7709), 1.87 MiB | 5.99 MiB/s, done.
Resolving deltas: 100% (4662/4662), done.
Cloning into '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/test_helper/bats-assert'...
remote: Enumerating objects: 566, done.        
remote: Counting objects: 100% (48/48), done.        
remote: Compressing objects: 100% (23/23), done.        
remote: Total 566 (delta 35), reused 25 (delta 25), pack-reused 518        
Receiving objects: 100% (566/566), 145.47 KiB | 2.14 MiB/s, done.
Resolving deltas: 100% (347/347), done.
Cloning into '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/test_helper/bats-support'...
remote: Enumerating objects: 121, done.        
remote: Counting objects: 100% (16/16), done.        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 121 (delta 4), reused 13 (delta 4), pack-reused 105        
Receiving objects: 100% (121/121), 34.26 KiB | 3.43 MiB/s, done.
Resolving deltas: 100% (43/43), done.
Submodule path '.ci-functions/test/bats': checked out 'dc312135a8de97e149cd0307c45c1e41061c918d'
Submodule path '.ci-functions/test/test_helper/bats-assert': checked out '78fa631d1370562d2cd4a1390989e706158e7bf0'
Submodule path '.ci-functions/test/test_helper/bats-support': checked out '3c8fadc5097c9acfc96d836dced2bb598e48b009'
$ pytest --git-aware
============================================================================================================================== test session starts ===============================================================================================================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/WJHoltz/pytest-workflow-submodule-issue
plugins: workflow-1.7.0.dev0
collecting ... 
collected 0 items / 1 error                                                                                                                                                                                                                                                      

===================================================================================================================================== ERRORS =====================================================================================================================================
_____________________________________________________________________________________________________________________ ERROR collecting tests/test_dummy.yaml _____________________________________________________________________________________________________________________
../Applications/miniconda3/envs/wdl-dev/lib/python3.11/site-packages/pytest_workflow/plugin.py:445: in collect
    workflow = self.queue_workflow()
../Applications/miniconda3/envs/wdl-dev/lib/python3.11/site-packages/pytest_workflow/plugin.py:402: in queue_workflow
    duplicate_tree(root_dir, tempdir,
../Applications/miniconda3/envs/wdl-dev/lib/python3.11/site-packages/pytest_workflow/util.py:183: in duplicate_tree
    copy(src_path, dest_path)
../Applications/miniconda3/envs/wdl-dev/lib/python3.11/shutil.py:436: in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
../Applications/miniconda3/envs/wdl-dev/lib/python3.11/shutil.py:256: in copyfile
    with open(src, 'rb') as fsrc:
E   IsADirectoryError: [Errno 21] Is a directory: '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats/test/fixtures/suite/recursive_with_symlinks/subsuite'
============================================================================================================================ short test summary info =============================================================================================================================
ERROR tests/test_dummy.yaml::dummy - IsADirectoryError: [Errno 21] Is a directory: '/Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats/test/fixtures/suite/recursive_with_symlinks/subsuite'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================================================ 1 error in 1.02s ================================================================================================================================
$ ls -l /Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats/test/fixtures/suite/recursive_with_symlinks/subsuite
lrwxr-xr-x  1 WJHoltz  staff  22 Dec 20 11:11 /Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats/test/fixtures/suite/recursive_with_symlinks/subsuite -> ../recursive/subsuite/
$ ls -l /Users/WJHoltz/pytest-workflow-submodule-issue/.ci-functions/test/bats/test/fixtures/suite/recursive/subsuite/
total 8
-rw-r--r--  1 WJHoltz  staff  40 Dec 20 11:11 test2.bats
$
```

Please let me know if I can provide any other useful information.



Discussion / hints:

Fix patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index 4132e7a1..8d514cf4 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -12,6 +12,7 @@ version 2.0.0-dev
 + Python 3.6 is no longer supported. It has been removed from github actions,
   as such we can no longer guarantee that pytest-workflow works properly
   with python 3.6.
++ Fix an issue where symlinks in git repositories could not be properly copied.
 + Added an optional encoding key for files, stdout and stderr so the file can
   be opened with the proper encoding.
 + Make content tests more efficient by reading each file only once instead of


Test patch:
diff --git a/src/pytest_workflow/util.py b/src/pytest_workflow/util.py
index 94e2ffbf..211e3ab3 100644
--- a/src/pytest_workflow/util.py
+++ b/src/pytest_workflow/util.py
@@ -173,7 +173,9 @@ def duplicate_tree(src: Filepath, dest: Filepath,
         copy: Callable[[Filepath, Filepath], None] = \
             functools.partial(os.symlink, target_is_directory=False)
     else:
-        copy = shutil.copy2  # Preserves metadata, also used by shutil.copytree
+        # shutil.copy2 preserves metadata, also used by shutil.copytree
+        # follow_symlinks False to directly copy links
+        copy = functools.partial(shutil.copy2, follow_symlinks=False)
 
     os.makedirs(dest, exist_ok=False)
     for src_path, dest_path, is_dir in path_iter:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 098f1e67..005d5a72 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -166,17 +166,25 @@ def create_git_repo(path):
     os.mkdir(dir)
     file = dir / "README.md"
     file.write_text("# My new project\n\nHello this project is awesome!\n")
+    subdir = dir / "sub"
+    subdir.mkdir()
+    another_file = subdir / "subtext.md"
+    another_file.write_text("# Subtext\n\nSome other example text.\n")
+    subdir_link = dir / "gosub"
+    subdir_link.symlink_to(subdir.relative_to(dir), target_is_directory=True)
     subprocess.run(["git", "init", "-b", "main"], cwd=dir)
     subprocess.run(["git", "config", "user.name", "A U Thor"], cwd=dir)
     subprocess.run(["git", "config", "user.email", "author@example.com"],
                    cwd=dir)
-    subprocess.run(["git", "add", "README.md"], cwd=dir)
+    subprocess.run(["git", "add", "."], cwd=dir)
     subprocess.run(["git", "commit", "-m", "initial commit"], cwd=dir)
 
 
-def test_git_submodule_check(tmp_path):
-    bird_repo = tmp_path / "bird"
-    nest_repo = tmp_path / "nest"
+@pytest.fixture()
+def git_repo_with_submodules():
+    repo_dir = Path(tempfile.mkdtemp())
+    bird_repo = repo_dir / "bird"
+    nest_repo = repo_dir / "nest"
     create_git_repo(bird_repo)
     create_git_repo(nest_repo)
     # https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586
@@ -185,11 +193,16 @@ def test_git_submodule_check(tmp_path):
                    cwd=nest_repo.absolute())
     subprocess.run(["git", "commit", "-m", "add bird repo as a submodule"],
                    cwd=nest_repo.absolute())
+    yield nest_repo
+    shutil.rmtree(repo_dir)
+
+
+def test_git_submodule_check(git_repo_with_submodules, tmp_path):
     cloned_repo = tmp_path / "cloned"
     subprocess.run(
         # No recursive clone
         ["git", "-c", "protocol.file.allow=always",
-         "clone", nest_repo.absolute(), cloned_repo.absolute()],
+         "clone", git_repo_with_submodules.absolute(), cloned_repo.absolute()],
         cwd=tmp_path
     )
     with pytest.raises(RuntimeError) as error:
@@ -201,3 +214,16 @@ def test_git_submodule_check(tmp_path):
                    cwd=cloned_repo.absolute())
     # Check error does not occur when issue resolved.
     git_check_submodules_cloned(cloned_repo)
+
+
+# https://github.com/LUMC/pytest-workflow/issues/162
+def test_duplicate_git_tree_submodule_symlinks(git_repo_with_submodules):
+    assert (git_repo_with_submodules / ".git").exists()
+    dest = Path(tempfile.mkdtemp()) / "test"
+    duplicate_tree(git_repo_with_submodules, dest, git_aware=True)
+    assert dest.exists()
+    assert not (dest / ".git").exists()
+    link = dest / "bird" / "gosub"
+    assert link.exists()
+    assert link.is_symlink()
+    assert link.resolve() == dest / "bird" / "sub"