Repository: LUMC/pytest-workflow (language: python)

Issue:
Make pytest-workflow git-aware
@DavyCats @Redmar-van-den-Berg , could you please review this? It should reduce the wear and tear on your SSDs significantly :wink:. 

Preliminary testing with Germline-DNA pipeline. Copying everything for all tests: 2.1 GB. Using `--git-aware` option: 460 MB. 
The .git folder contains a lot of cruft!

### Checklist
- [x] Pull request details were added to HISTORY.rst



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 4a1e549f..11d7c966 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -14,11 +14,13 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - 3.6
+          - "3.6"
     steps:
       - uses: actions/checkout@v2.3.4
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
       - name: Install tox
         run: pip install tox
       - name: Lint
@@ -30,6 +32,8 @@ jobs:
       - uses: actions/checkout@v2.3.4
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
       - name: Install tox
         run: pip install tox
       - name: Build docs
@@ -39,15 +43,18 @@ jobs:
     strategy:
       matrix:
         python-version:
-          - 3.6
-          - 3.7
-          - 3.8
-          - 3.9
+          - "3.6"
+          - "3.7"
+          - "3.8"
+          - "3.9"
+          - "3.10"
     needs: lint
     steps:
       - uses: actions/checkout@v2.3.4
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
       - name: Install tox
         run: pip install tox
       - name: Run tests
@@ -58,10 +65,10 @@ jobs:
 
   test-functional:
     runs-on: ubuntu-latest
-    needs: test
+    needs: lint
     strategy:
       matrix:
-        python-version: [3.7]
+        python-version: ["3.7"]
         test-program: [cromwell, snakemake, miniwdl]
     steps:
       - uses: actions/checkout@v2.3.4
@@ -70,6 +77,8 @@ jobs:
       - name: Set up Python ${{ matrix.python-version }}
         if: ${{ matrix.test-program != 'cromwell' }}
         uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
       - name: Install tox
         if: ${{ matrix.test-program != 'cromwell' }}
         run: pip install tox
diff --git a/HISTORY.rst b/HISTORY.rst
index a59b7f4d..dfd74e98 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -7,6 +7,18 @@ Changelog
 .. This document is user facing. Please word the changes in such a way
 .. that users understand how the changes affect the new version.
 
+version 1.6.0
+---------------------------
++ Add a ``--git-aware`` or ``--ga`` option to only copy copy files listed by
+  git ls-files. This omits the ``.git`` folder, all untracked files and
+  everything ignored by ``.gitignore``. This reduces the number of copy
+  operations drastically.
+
+  Pytest-workflow will now emit a warning when copying of a git directory is
+  detected without the ``--git-aware`` option.
+
++ Add support and tests for Python 3.10
+
 version 1.5.0
 ---------------------------
 + Add support for python 3.9
diff --git a/README.rst b/README.rst
index b8de9c2f..a70ead6c 100644
--- a/README.rst
+++ b/README.rst
@@ -32,8 +32,11 @@ pytest-workflow
   :target: https://doi.org/10.5281/zenodo.3757727
   :alt: More information on how to cite pytest-workflow here.
 
-pytest-workflow is a pytest plugin that aims to make pipeline/workflow testing easy
-by using yaml files for the test configuration.
+pytest-workflow is a workflow-system agnostic testing framework that aims 
+to make pipeline/workflow testing easy by using YAML files for the test 
+configuration. Whether you write your pipelines in WDL, snakemake, bash or
+any other workflow framework, pytest-workflow makes testing easy.
+pytest-workflow is build on top of the pytest test framework.
 
 For our complete documentation checkout our
 `readthedocs page <https://pytest-workflow.readthedocs.io/>`_.
@@ -42,7 +45,7 @@ For our complete documentation checkout our
 Installation
 ============
 Pytest-workflow requires Python 3.6 or higher. It is tested on Python 3.6, 3.7,
-3.8 and 3.9. Python 2 is not supported.
+3.8, 3.9 and 3.10. Python 2 is not supported.
 
 - Make sure your virtual environment is activated.
 - Install using pip ``pip install pytest-workflow``
diff --git a/docs/examples.rst b/docs/examples.rst
index 4b823449..166e7e22 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -68,7 +68,7 @@ Cromwell so it can be used as a command, instead of having to use the jar.
         md5sum: 173fd8023240a8016033b33f42db14a2
     stdout:
       contains:
-        - "WorkflowSucceededState"
+        - "workflow finished with status 'Succeeded'"
 
 WDL with miniwdl example
 ------------------------
diff --git a/docs/installation.rst b/docs/installation.rst
index dd939b33..1a38d3bf 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -2,7 +2,7 @@
 Installation
 ============
 
-Pytest-workflow is tested on python 3.6, 3.7, 3.8 and 3.9. Python 2 is not
+Pytest-workflow is tested on python 3.6, 3.7, 3.8, 3.9 and 3.10. Python 2 is not
 supported.
 
 In a virtual environment
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 00000000..62f5f82c
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=51", "wheel"]
+build-backend = "setuptools.build_meta"
diff --git a/setup.py b/setup.py
index 9433b815..8771bba5 100644
--- a/setup.py
+++ b/setup.py
@@ -20,7 +20,7 @@
 
 setup(
     name="pytest-workflow",
-    version="1.5.0",
+    version="1.6.0",
     description="A pytest plugin for configuring workflow/pipeline tests "
                 "using YAML files",
     author="Leiden University Medical Center",
@@ -43,6 +43,7 @@
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
         "Development Status :: 5 - Production/Stable",
         "License :: OSI Approved :: "
         "GNU Affero General Public License v3 or later (AGPLv3+)",
diff --git a/tox.ini b/tox.ini
index 6a232478..6b75d6c2 100644
--- a/tox.ini
+++ b/tox.ini
@@ -15,6 +15,7 @@ deps=flake8
      flake8-import-order
      bandit
      mypy
+     types-pyyaml
 commands =
     flake8 src tests setup.py
     # Skip 'assert used' issue as this is a pytest plugin.'


Test patch:
diff --git a/docs/running_pytest_workflow.rst b/docs/running_pytest_workflow.rst
index dd64bb24..6bee58d7 100644
--- a/docs/running_pytest_workflow.rst
+++ b/docs/running_pytest_workflow.rst
@@ -55,11 +55,18 @@ The temporary directories created are copies of pytest's root directory, the
 directory from which it runs the tests. If you have lots of tests, and if you
 have a large repository, this may take a lot of disk space. To alleviate this
 you can use the ``--symlink`` flag which will create the same directory layout
-but instead symlinks the files instead of copying them. This is *slower* for
-lots of small files, and it carries with it the risk that the tests may alter
-files from your work directory. If there are a lot of large files and files are
-used read-only in tests, then it will use a lot less disk space and be faster
-as well.
+but instead symlinks the files instead of copying them. This carries with it
+the risk that the tests may alter files from your work directory. If there are
+a lot of large files and files are used read-only in tests, then it will use a
+lot less disk space and be faster as well.
+
+.. note::
+
+    When your workflow is version controlled in git please use the
+    ``--git-aware`` option. This omits the ``.git`` folder, all untracked
+    files and everything ignored by ``.gitignore``. This reduces the number of
+    copy operations significantly.
+
 
 Running multiple workflows simultaneously
 -----------------------------------------
diff --git a/src/pytest_workflow/plugin.py b/src/pytest_workflow/plugin.py
index 477ee74c..5e792030 100644
--- a/src/pytest_workflow/plugin.py
+++ b/src/pytest_workflow/plugin.py
@@ -34,7 +34,7 @@
 from .content_tests import ContentTestCollector
 from .file_tests import FileTestCollector
 from .schema import WorkflowTest, workflow_tests_from_schema
-from .util import is_in_dir, link_tree, replace_whitespace
+from .util import duplicate_tree, is_in_dir, replace_whitespace
 from .workflow import Workflow, WorkflowQueue
 
 
@@ -66,6 +66,12 @@ def pytest_addoption(parser: PytestParser):
              "symbolic links. This saves disk space, but should only be used "
              "for tests that do use these files read-only."
     )
+    parser.addoption(
+        "--ga", "--git-aware", action="store_true", dest="git_aware",
+        help="Only copy files that are listed by the 'git ls-files' command. "
+             "This ignores the .git directory, any untracked files and any "
+             "files listed by .gitignore. "
+             "Highly recommended when working in a git project.")
 
     # Why `--tag <tag>` and not simply use `pytest -m <tag>`?
     # `-m` uses a "mark expression". So you have to type a piece of python
@@ -375,12 +381,22 @@ def queue_workflow(self):
                 f"'{tempdir}' already exists. Deleting ...")
             shutil.rmtree(str(tempdir))
 
+        # Warn users of git that they should use the --git-aware option.
+        # The .git directory contains all files ever checked in, and all diffs
+        # in the entire history.
+        root_dir = Path(self.config.rootdir)
+        git_aware = self.config.getoption("git_aware")
+        git_dir = root_dir / ".git"
+        if git_dir.exists() and not git_aware:
+            warnings.warn(
+                f".git dir detected: {str(git_dir)}. pytest-workflow "
+                f"will copy the entire .git directory and all files ignored "
+                f"by git. It is recommended to use the --git-aware option.")
         # Copy the project directory to the temporary directory using pytest's
         # rootdir.
-        if self.config.getoption("symlink"):
-            link_tree(Path(str(self.config.rootdir)), tempdir)
-        else:
-            shutil.copytree(str(self.config.rootdir), str(tempdir))
+        duplicate_tree(root_dir, tempdir,
+                       symlink=self.config.getoption("symlink"),
+                       git_aware=git_aware)
 
         # Create a workflow and make sure it runs in the tempdir
         workflow = Workflow(command=self.workflow_test.command,
diff --git a/src/pytest_workflow/util.py b/src/pytest_workflow/util.py
index ed0a09ea..947eb71a 100644
--- a/src/pytest_workflow/util.py
+++ b/src/pytest_workflow/util.py
@@ -1,8 +1,15 @@
+import functools
 import hashlib
 import os
 import re
+import shutil
+import subprocess  # nosec
+import sys
 import warnings
 from pathlib import Path
+from typing import Callable, Iterator, List, Set, Tuple, Union
+
+Filepath = Union[str, os.PathLike]
 
 
 # This function was created to ensure the same conversion is used throughout
@@ -41,22 +48,128 @@ def is_in_dir(child: Path, parent: Path, strict: bool = False) -> bool:
     return False
 
 
-def link_tree(src: Path, dest: Path) -> None:
+def _run_command(*args):
+    """Run an external command and return the output"""
+    result = subprocess.run(args,  # nosec
+                            stdout=subprocess.PIPE,
+                            # Encoding to output as a string.
+                            encoding=sys.getdefaultencoding(),
+                            check=True)
+    return result.stdout
+
+
+def git_root(path: Filepath) -> str:
+    output = _run_command(
+        "git", "-C", os.fspath(path), "rev-parse", "--show-toplevel")
+    return output.strip()  # Remove trailing newline
+
+
+def git_ls_files(path: Filepath) -> List[str]:
+    output = _run_command("git", "-C", os.fspath(path), "ls-files",
+                          # Make sure submodules are included.
+                          "--recurse-submodules")
+    # Remove trailing newlines and split to output all the paths
+    return output.strip("\n").split("\n")
+
+
+def _duplicate_tree(src: Filepath, dest: Filepath
+                    ) -> Iterator[Tuple[str, str, bool]]:
+    """Traverses src and for each file or directory yields a path to it,
+    its destination, and whether it is a directory."""
+    for entry in os.scandir(src):  # type: os.DirEntry
+        if entry.is_dir():
+            dir_src = entry.path
+            dir_dest = os.path.join(dest, entry.name)
+            yield dir_src, dir_dest, True
+            yield from _duplicate_tree(dir_src, dir_dest)
+        elif entry.is_file() or entry.is_symlink():
+            yield entry.path, os.path.join(dest, entry.name), False
+        else:
+            warnings.warn(f"Unsupported filetype for copying. "
+                          f"Skipping {entry.path}")
+
+
+def _duplicate_git_tree(src: Filepath, dest: Filepath
+                        ) -> Iterator[Tuple[str, str, bool]]:
+    """Traverses src, finds all files registered in git and for each file or
+    directory yields a path to it, its destination and whether it is a
+    directory"""
+    # A set of dirs we have already yielded. '' is the output of
+    # os.path.dirname when the path is in the current directory.
+    yielded_dirs: Set[str] = {''}
+    for path in git_ls_files(src):
+        # git ls-files does not list directories. Yield parent first to prevent
+        # creating files in non-existing directories. Also check if it is
+        # yielded before so each directory is only yielded once.
+        parent = os.path.dirname(path)
+        if parent not in yielded_dirs:
+            # This maybe a nested directory, with non-existing parents itself.
+            # Therefore:
+            # - List parents from deepest to least deep by using os.path.dirname  # noqa: E501
+            # - Reverse the list to yield directories from least deep to deepest  # noqa: E501
+            # This ensures parents are always yielded before children.
+            parents = []
+            while parent not in yielded_dirs:
+                yielded_dirs.add(parent)
+                parents.append(parent)
+                parent = os.path.dirname(parent)
+
+            for parent in reversed(parents):
+                src_parent = os.path.join(src, parent)
+                dest_parent = os.path.join(dest, parent)
+                yield src_parent, dest_parent, True
+
+        # Yield the actual file if the directory has already been yielded.
+        src_path = os.path.join(src, path)
+        dest_path = os.path.join(dest, path)
+        yield src_path, dest_path, False
+
+
+def duplicate_tree(src: Filepath, dest: Filepath,
+                   symlink: bool = False,
+                   git_aware: bool = False):
+    """
+    Duplicates a filetree
+    :param src: The source directory
+    :param dest: The destination directory
+    :param symlink: Create symlinks nstead of copying the files.
+    :param git_aware: Only copy/symlink files registered by git.
+    """
+    if not symlink and not git_aware:
+        shutil.copytree(src, dest)
+        return
+
+    if not os.path.isdir(src):
+        # shutil.copytree also throws a NotADirectoryError
+        raise NotADirectoryError(f"Not a directory: '{src}'")
+
+    if git_aware:
+        path_iter = _duplicate_git_tree(src, dest)
+    else:
+        path_iter = _duplicate_tree(src, dest)
+    if symlink:
+        copy: Callable[[Filepath, Filepath], None] = \
+            functools.partial(os.symlink, target_is_directory=False)
+    else:
+        copy = shutil.copy2  # Preserves metadata, also used by shutil.copytree
+
+    os.makedirs(dest, exist_ok=False)
+    for src_path, dest_path, is_dir in path_iter:
+        if is_dir:
+            os.mkdir(dest_path)
+        else:
+            copy(src_path, dest_path)
+
+
+def link_tree(src: Filepath, dest: Filepath) -> None:
     """
     Copies a tree by mimicking the directory structure and soft-linking the
     files
     :param src: The source directory
     :param dest: The destination directory
     """
-    if src.is_dir():
-        dest.mkdir(parents=True)
-        for path in os.listdir(str(src)):
-            link_tree(Path(src, path), Path(dest, path))
-    elif src.is_file() or src.is_symlink():
-        dest.symlink_to(src, target_is_directory=False)
-    else:  # Only copy files and symlinks, no devices etc.
-        warnings.warn(f"Unsupported filetype. Skipping copying: '{str(src)}' "
-                      f"to '{str(dest)}'.")
+    # THIS FUNCTION IS KEPT FOR BACKWARDS-COMPATIBILITY
+    duplicate_tree(src, dest, symlink=True)
 
 
 # block_size 64k with python is a few percent faster than linux native md5sum.
diff --git a/tests/functional/simple_snakefile_test_cases.yml b/tests/functional/simple_snakefile_test_cases.yml
index 0ca2f666..5b7f969b 100644
--- a/tests/functional/simple_snakefile_test_cases.yml
+++ b/tests/functional/simple_snakefile_test_cases.yml
@@ -3,19 +3,19 @@
 - name: test-config-missing
   command: snakemake -n -r -p -s SimpleSnakefile
   exit_code: 1
-  stdout:
+  stderr:
     contains:
       - "You must set --config N_LINES_TO_READ=<a value>."
 - name: test-config-wrong-type
   command: snakemake -n -r -p -s SimpleSnakefile --config N_LINES_TO_READ=one
   exit_code: 1
-  stdout:
+  stderr:
     contains:
       - "N_LINES_TO_READ must be an integer."
 - name: test-config-invalid-value
   command: snakemake -n -r -p -s SimpleSnakefile --config N_LINES_TO_READ=-1
   exit_code: 1
-  stdout:
+  stderr:
     contains:
       - "N_LINES_TO_READ must at least be 1."
 - name: test-snakemake-run
diff --git a/tests/functional/simple_wdl_test_cases.yml b/tests/functional/simple_wdl_test_cases.yml
index 6dcbc2df..0572a8b2 100644
--- a/tests/functional/simple_wdl_test_cases.yml
+++ b/tests/functional/simple_wdl_test_cases.yml
@@ -37,7 +37,7 @@
     - path: test-output/all_data.gz
   stdout:
     contains:
-      - "WorkflowSucceededState"
+      - "workflow finished with status 'Succeeded'"
 
 
 - name: simple wdl miniwdl
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 54ae7967..aa74946e 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -14,14 +14,16 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with pytest-workflow.  If not, see <https://www.gnu.org/licenses/
 import hashlib
+import os
 import shutil
+import subprocess  # nosec
 import tempfile
 from pathlib import Path
 
 import pytest
 
-from pytest_workflow.util import file_md5sum, is_in_dir, link_tree, \
-    replace_whitespace
+from pytest_workflow.util import duplicate_tree, file_md5sum, git_root, \
+    is_in_dir, link_tree, replace_whitespace
 
 WHITESPACE_TESTS = [
     ("bla\nbla", "bla_bla"),
@@ -77,14 +79,65 @@ def test_link_tree():
 
 
 def test_link_tree_warning():
-    urandom = Path("/dev/urandom")
-    dest = Path("notexist")
+    src_dir = Path(tempfile.mkdtemp())
+    dest_dir = Path(tempfile.mkdtemp(), "test")
+
+    not_a_file = src_dir / "not_a_file"
+    os.mkfifo(not_a_file)
     with pytest.warns(UserWarning,
-                      match="Unsupported filetype. Skipping copying: "
-                            "'/dev/urandom' to 'notexist'."):
-        link_tree(urandom, dest)
+                      match=f"Unsupported filetype for copying. Skipping"
+                            f" {not_a_file}"):
+        link_tree(src_dir, dest_dir)
     # Destination should not be created
-    assert not dest.exists()
+    assert not (dest_dir / "not_a_file").exists()
+    shutil.rmtree(src_dir)
+    shutil.rmtree(dest_dir.parent)
+
+
+@pytest.fixture()
+def git_dir():
+    git_dir = Path(tempfile.mkdtemp())
+    (git_dir / "test").mkdir()
+    test_file = git_dir / "test" / "test.txt"
+    test_file.touch()
+    subprocess.run(["git", "-C", str(git_dir), "init"])  # nosec
+    subprocess.run(["git", "-C", str(git_dir), "add", str(test_file)])  # nosec
+    subprocess.run(["git", "-C", str(git_dir), "commit", "-m",  # nosec
+                    "initial commit"])
+    yield git_dir
+    shutil.rmtree(git_dir)
+
+
+def test_duplicate_git_tree(git_dir):
+    assert (git_dir / ".git").exists()
+    dest = Path(tempfile.mkdtemp()) / "test"
+    duplicate_tree(git_dir, dest, git_aware=True)
+    assert dest.exists()
+    assert not (dest / ".git").exists()
+    assert (dest / "test" / "test.txt").exists()
+
+
+def test_duplicate(git_dir):
+    assert (git_dir / ".git").exists()
+    dest = Path(tempfile.mkdtemp()) / "test"
+    duplicate_tree(git_dir, dest, git_aware=False)
+    assert dest.exists()
+    assert (dest / ".git").exists()
+    assert (dest / "test" / "test.txt").exists()
+
+
+def test_duplicate_notadirerror():
+    fd, file = tempfile.mkstemp()
+    dir = tempfile.mkdtemp()
+    with pytest.raises(NotADirectoryError):
+        duplicate_tree(file, Path(dir, "somedir"), symlink=True)
+    shutil.rmtree(dir)
+    os.close(fd)
+    os.remove(file)
+
+
+def test_git_root(git_dir):
+    assert git_root(git_dir / "test") == str(git_dir)
 
 
 HASH_FILE_DIR = Path(__file__).parent / "hash_files"
diff --git a/tests/test_warnings.py b/tests/test_warnings.py
index 71e0167b..24589f95 100644
--- a/tests/test_warnings.py
+++ b/tests/test_warnings.py
@@ -29,3 +29,13 @@ def test_no_warnings(testdir):
     outcomes = result.parseoutcomes()
     assert outcomes.get('warnings', 0) == 0
     shutil.rmtree(basetemp)
+
+
+def test_git_warning(testdir):
+    basetemp = tempfile.mkdtemp()
+    testdir.mkdir(".git")
+    testdir.makefile(".yml", test_a=MOO_FILE)
+    result = testdir.runpytest("--basetemp", basetemp)
+    outcomes = result.parseoutcomes()
+    assert outcomes.get('warnings') == 1
+    assert ".git dir detected" in result.stdout.str()