Repository: KyleKing/copier-template-tester (language: python)

Issue:
When template repo dir is dirty, it copies last tag, not current commit
**Describe the bug**
<!-- Describe the bug -->


#**To Reproduce**
<!-- How can someone else replicate the issue -->

Steps and/or code snippet(s) to reproduce the behavior:

1. Have a template with git tags.
1. Change something.
2. Commit.
3. Pre-commit hook changes ctt output, pointing to HEAD
4. Fix something
5. Commit again.
6. pre-commit hook changes ctt output to last tag, instead of current HEAD with dirty changes applied.

**Expected behavior**
<!-- What did you expect? -->
CTT should always render current template with current dirty changes applied.

**Additional Information**
<!-- Add any relevant versions -->

- `copier_template_tester` Version: 1.0.2
- OS: \[e.g. macOS/Windows\]: Linux
- OS Version \[e.g. Catalina/10\]: Fedora 37
<!-- Add `pip freeze` or other version information that is relevant -->



Discussion / hints:
Hi @yajo, thanks for trying out CTT and the issues!

I wasn't aware of #3, but I think I was running into a similar issue as this one, where I couldn't figure out how to have the staged files included in the output from copier copy.

One workaround might be for the tool to maintain a shadow copy of the repo, commit and tag the changes there, then use that folder as the source for the copy? Do you think there might be an easier way?
Elaborating on the shadow copy approach with some pseudo code. I would probably use the `user_cache_dir` from https://github.com/platformdirs/platformdirs

```py
# This is the shadow clone
ctt_cache='/Users/kyle/Library/Caches/copier-template-tester'
# And this is the directory the user is actually working with
working_dir='$PWD'

# Run git init in working_dir
subprocess.check_output('git init', cwd=working_dir)

# Then for each directory in `ctt.toml`, ignore them here:
unlinked_dirs = {'.git', *ctt_dirs}
# and copy the matches over:
for match in [_m for _m in _working_dir.glob('*') if _m not in unlinked_dirs]:
    # I will also try symlinking instead of copy, but I'm not confident that it would work
    if match.is_dir():
        shutil.copytree(match, ctt_cache / match.name)
	else:
        shutil.copy(match, ctt_cache / match.name)
```

On each pre-commit, first get the latest tag from the working directory, then commit and tag the shadow directory

```sh
LAST_TAG=$(git tag | tail -n 1)  # In local working repository
git add --all && git commit "(ctt)" --no-verify && git tag $LAST_TAG-ctt  # In user_cache
```

Then `ctt` in `pre-commit` would `copier copy` the files from the `user_cache` to the local destination specified in the config
Have you tried adding `vcs_ref="HEAD"` to the copy call?
I think I have that set here, but it seems that the problem is that `copier` dosn't support using the stash:

https://github.com/KyleKing/copier-template-tester/blob/597181d8c37c9956dd4edc0d4e904b8d982a4cdf/copier_template_tester/main.py#L50-L56

The shadow repo approach proposed above wouldn't strictly need to make tags, but the benefit would be more deterministic file output
> the benefit would be more deterministic file output

That's important indeed. Otherwise, each new `pre-commit run` would produce a different `_commit` in the copier answers file.

Fix patch:
diff --git a/.calcipy_packaging.lock b/.calcipy_packaging.lock
index f90b8fe..6068bc0 100644
--- a/.calcipy_packaging.lock
+++ b/.calcipy_packaging.lock
@@ -959,6 +959,22 @@
         "name": "transitions",
         "version": "0.9.0"
     },
+    "types-pyyaml": {
+        "datetime": "2023-03-27T18:22:27.206635+00:00",
+        "domain": "https://pypi.org/pypi/{name}/json",
+        "latest_datetime": "2023-03-27T18:22:27.206635+00:00",
+        "latest_version": "6.0.12.9",
+        "name": "types-pyyaml",
+        "version": "6.0.12.9"
+    },
+    "types-setuptools": {
+        "datetime": "2023-04-17T21:13:00.597367+00:00",
+        "domain": "https://pypi.org/pypi/{name}/json",
+        "latest_datetime": "2023-04-17T21:13:00.597367+00:00",
+        "latest_version": "67.6.0.8",
+        "name": "types-setuptools",
+        "version": "67.6.0.8"
+    },
     "typing-extensions": {
         "datetime": "2023-02-15T00:17:53.726117+00:00",
         "domain": "https://pypi.org/pypi/{name}/json",
diff --git a/.gitignore b/.gitignore
index eb97d97..d8104a4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -236,6 +236,3 @@ coverage.json
 
 # ----------------------------------------------------------------------------------------------------------------------
 # Custom Rules
-
-# Exclude files that contain absolute paths
-tests/data/copier_demo/.ctt/*/.copier-answers.yml
diff --git a/docs/docs/CHANGELOG.md b/docs/docs/CHANGELOG.md
index ad0615a..185d0f9 100644
--- a/docs/docs/CHANGELOG.md
+++ b/docs/docs/CHANGELOG.md
@@ -1,3 +1,29 @@
+## Unreleased
+
+## 1.2.0 (2023-04-22)
+
+### Feat
+
+- support templated copier answer files
+
+## 1.2.0rc1 (2023-04-22)
+
+### Fix
+
+- write the commit as '-0' each time
+
+## 1.2.0rc0 (2023-04-22)
+
+### Feat
+
+- remove _commit from the answers file
+
+### Refactor
+
+- experiment with a shadow copy
+- use capture_shell
+- split the main logic into three files
+
 ## 1.1.0 (2023-04-22)
 
 ### Feat
diff --git a/docs/docs/DEVELOPER_GUIDE.md b/docs/docs/DEVELOPER_GUIDE.md
index 5e923e5..9015abe 100644
--- a/docs/docs/DEVELOPER_GUIDE.md
+++ b/docs/docs/DEVELOPER_GUIDE.md
@@ -44,11 +44,14 @@ poetry config pypi-token.pypi ...
 ## Current Status
 
 <!-- {cts} COVERAGE -->
-| File                                 |   Statements |   Missing |   Excluded | Coverage   |
-|--------------------------------------|--------------|-----------|------------|------------|
-| `copier_template_tester/__init__.py` |            2 |         0 |          0 | 100.0%     |
-| `copier_template_tester/main.py`     |           65 |         0 |         18 | 100.0%     |
-| **Totals**                           |           67 |         0 |         18 | 100.0%     |
+| File                                            |   Statements |   Missing |   Excluded | Coverage   |
+|-------------------------------------------------|--------------|-----------|------------|------------|
+| `copier_template_tester/__init__.py`            |            2 |         0 |          0 | 100.0%     |
+| `copier_template_tester/_config.py`             |           20 |         2 |          0 | 90.0%      |
+| `copier_template_tester/_pre_commit_support.py` |           19 |         0 |          0 | 100.0%     |
+| `copier_template_tester/_write_output.py`       |           52 |         7 |          2 | 86.5%      |
+| `copier_template_tester/main.py`                |           25 |         0 |         15 | 100.0%     |
+| **Totals**                                      |          118 |         9 |         17 | 92.4%      |
 
 Generated on: 2023-04-22
 <!-- {cte} -->
diff --git a/poetry.lock b/poetry.lock
index e1fbeff..0093674 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -2457,6 +2457,30 @@ six = "*"
 diagrams = ["pygraphviz"]
 test = ["pytest"]
 
+[[package]]
+name = "types-pyyaml"
+version = "6.0.12.9"
+description = "Typing stubs for PyYAML"
+category = "dev"
+optional = false
+python-versions = "*"
+files = [
+    {file = "types-PyYAML-6.0.12.9.tar.gz", hash = "sha256:c51b1bd6d99ddf0aa2884a7a328810ebf70a4262c292195d3f4f9a0005f9eeb6"},
+    {file = "types_PyYAML-6.0.12.9-py3-none-any.whl", hash = "sha256:5aed5aa66bd2d2e158f75dda22b059570ede988559f030cf294871d3b647e3e8"},
+]
+
+[[package]]
+name = "types-setuptools"
+version = "67.6.0.8"
+description = "Typing stubs for setuptools"
+category = "dev"
+optional = false
+python-versions = "*"
+files = [
+    {file = "types-setuptools-67.6.0.8.tar.gz", hash = "sha256:df3e084c3b5af782cb81bd516f40213b572d932fa2f1272aef307d15643736f8"},
+    {file = "types_setuptools-67.6.0.8-py3-none-any.whl", hash = "sha256:0b178cab1d5a7487cd410d1b5448d1a85d9da1bf44aa7be7c0d4f33d65041e5a"},
+]
+
 [[package]]
 name = "typing-extensions"
 version = "4.5.0"
@@ -2749,4 +2773,4 @@ files = [
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.10.5"
-content-hash = "146d0af91407ebcc01284e2ac9dffaac8731e581e1a1a69cbee31cc064bed903"
+content-hash = "ab0fc941cf339be407fc28e76edd4edb4323c7b084e1dd423e37ee6c3de61002"
diff --git a/pyproject.toml b/pyproject.toml
index cd947c6..f86b428 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -3,7 +3,7 @@ build-backend = "poetry.core.masonry.api"
 requires = ["poetry-core"]
 
 [tool.commitizen]
-version = "1.1.0"
+version = "1.2.0"
 version_files = ["copier_template_tester/__init__.py:^__version", "pyproject.toml:^version"]
 
 [tool.poetry]
@@ -24,7 +24,7 @@ maintainers = []
 name = "copier_template_tester"
 readme = "docs/README.md"
 repository = "https://github.com/kyleking/copier-template-tester"
-version = "1.1.0"
+version = "1.2.0"
 
 [tool.poetry.dependencies]
 python = "^3.10.5"
@@ -34,6 +34,8 @@ corallium = ">=0.2.0"
 [tool.poetry.group.dev.dependencies]
 calcipy = {extras = ["doc", "lint", "nox", "stale", "tags", "test", "types"], version = ">=1.2.3"}
 pytest-shell-utilities = ">=1.7.0"
+types-pyyaml = ">=6.0.12.9"
+types-setuptools = ">=67.6.0.8"
 
 [tool.poetry.scripts]
 ctt = "copier_template_tester.main:run_cli"


Test patch:
diff --git a/copier_template_tester/__init__.py b/copier_template_tester/__init__.py
index 66ac97b..ceb1dbe 100644
--- a/copier_template_tester/__init__.py
+++ b/copier_template_tester/__init__.py
@@ -1,6 +1,6 @@
 """copier_template_tester."""
 
-__version__ = '1.1.0'
+__version__ = '1.2.0'
 __pkg_name__ = 'copier_template_tester'
 
 # ====== Above is the recommended code from calcipy_template and may be updated on new releases ======
diff --git a/copier_template_tester/_config.py b/copier_template_tester/_config.py
index a72df85..d6c80f7 100644
--- a/copier_template_tester/_config.py
+++ b/copier_template_tester/_config.py
@@ -21,8 +21,9 @@ def _validate_config(config: dict) -> None:  # type: ignore[type-arg]
 def load_config(base_dir: Path) -> dict:  # type: ignore[type-arg]
     """Read the ctt config from `CWD`."""
     cfg_path = base_dir / 'ctt.toml'
-    if cfg_path.is_file():
-        config: dict = tomllib.loads(cfg_path.read_text())  # type: ignore[type-arg]
-        _validate_config(config)
-        return config
-    raise ValueError(f'No configuration file found. Expected: {cfg_path.absolute()}')  # pragma: no cover # noqa: EM102
+    if not cfg_path.is_file():  # pragma: no cover
+        msg = f'No configuration file found. Expected: {cfg_path.absolute()}'
+        raise ValueError(msg)
+    config: dict = tomllib.loads(cfg_path.read_text())  # type: ignore[type-arg]
+    _validate_config(config)
+    return config
diff --git a/copier_template_tester/_pre_commit_support.py b/copier_template_tester/_pre_commit_support.py
index 4429120..118dfb1 100644
--- a/copier_template_tester/_pre_commit_support.py
+++ b/copier_template_tester/_pre_commit_support.py
@@ -1,12 +1,11 @@
 """Support running `ctt` in `pre-commit`."""
 
-import shlex
-import subprocess  # noqa: S404  # nosec
 import sys
 from pathlib import Path
 
 from beartype import beartype
 from corallium.log import get_logger
+from corallium.shell import capture_shell
 
 logger = get_logger()
 
@@ -15,11 +14,8 @@
 def _ls_untracked_dir(base_dir: Path) -> set[Path]:
     """Use git to list all untracked files."""
     cmd = 'git ls-files --directory --exclude-standard --no-empty-dir --others'
-    process = subprocess.Popen(  # noqa: DUO116  # nosec  # nosemgrep
-        shlex.split(cmd), stdout=subprocess.PIPE, cwd=base_dir,  # noqa: S603
-    )
-    stdout, _stderr = process.communicate()
-    return {base_dir / _d.strip() for _d in stdout.decode().split('\n') if _d}
+    output = capture_shell(cmd=cmd, cwd=base_dir)
+    return {base_dir / _d.strip() for _d in output.split('\n') if _d}
 
 
 @beartype
diff --git a/copier_template_tester/_write_output.py b/copier_template_tester/_write_output.py
index 6cb9e32..00705ce 100644
--- a/copier_template_tester/_write_output.py
+++ b/copier_template_tester/_write_output.py
@@ -1,17 +1,73 @@
 """Template Directory Writer."""
 
+import re
 import shutil
 from pathlib import Path
 
 import copier
+import yaml
 from beartype import beartype
+from corallium.file_helpers import read_lines
 from corallium.log import get_logger
 
 logger = get_logger()
 
+DEFAULT_TEMPLATE_FILE_NAME = 'copier.yaml'
+"""Default answer file name. Alternative is copier.yml."""
+
+
+DEFAULT_ANSWER_FILE_NAME = '.copier-answers.yml'
+"""Default answer file name.
+
+https://github.com/copier-org/copier/blob/7f05baf4f004a4876fb6158e1c532b28290146a4/copier/subproject.py#L39
+
+"""
+
+
+@beartype
+def _read_copier_template(base_dir: Path) -> dict:  # type: ignore[type-arg]
+    """Locate and read the copier configuration file."""
+    copier_path = base_dir / DEFAULT_TEMPLATE_FILE_NAME
+    if not copier_path.is_file():
+        copier_path = copier_path.with_suffix('.yml')
+    if not copier_path.is_file():  # pragma: no cover
+        msg = f"Can't find the copier template file. Expected: {copier_path} (or .yaml)"
+        raise FileNotFoundError(msg)
+
+    return yaml.safe_load(copier_path.read_text())  # type: ignore[no-any-return]
+
+
+@beartype
+def _find_answers_file(*, src_path: Path, dst_path: Path) -> Path:
+    """Locate the copier answers file based on the copier template."""
+    copier_config = _read_copier_template(src_path)
+    answers_filename = copier_config.get('_answers_file') or DEFAULT_ANSWER_FILE_NAME
+    if '{{' in answers_filename:
+        # If the filename is created from the template, just grab the first match
+        search_name = re.sub(r'{{[^}]+}}', '*', answers_filename)
+        matches = [*dst_path.glob(search_name)]
+        if len(matches) != 1:  # pragma: no cover
+            msg = f"Can't find just one copier answers file matching {dst_path / search_name}. Found: {matches}"
+            raise ValueError(msg)
+        return matches[0]
+    return dst_path / answers_filename  # pragma: no cover
+
+
+@beartype
+def _stabilize_commit_id(*, src_path: Path, dst_path: Path) -> None:
+    """Replace part of the _commit for a less variable 'ctt' output."""
+    answers_path = _find_answers_file(src_path=src_path, dst_path=dst_path)
+    lines = (  # noqa: ECE001
+        # Create a stable tag that copier will still utilize
+        f'{line.split("-")[0]}-0' if line.startswith('_commit') else line
+        for line in read_lines(answers_path)
+    )
+    answers_path.write_text('\n'.join(lines))
+
 
 @beartype
 def write_output(  # type: ignore[no-untyped-def]
+    *,
     src_path: Path,
     dst_path: Path,
     data: dict[str, bool | int | float | str | None],
@@ -32,3 +88,9 @@ def write_output(  # type: ignore[no-untyped-def]
     git_path = dst_path / '.git'
     if git_path.is_dir():  # pragma: no cover
         shutil.rmtree(git_path)
+
+    # Reduce variability in the output
+    try:
+        _stabilize_commit_id(src_path=src_path, dst_path=dst_path)
+    except FileNotFoundError as exc:  # pragma: no cover
+        logger.error(str(exc))  # noqa: TRY400
diff --git a/copier_template_tester/main.py b/copier_template_tester/main.py
index 33c1ee7..6eb4af5 100644
--- a/copier_template_tester/main.py
+++ b/copier_template_tester/main.py
@@ -34,7 +34,7 @@ def run(*, base_dir: Path | None = None, check_untracked: bool = False) -> None:
         output_path.mkdir(parents=True, exist_ok=True)
         paths.add(output_path)
         logger.text(f'Creating: {output_path}')
-        write_output(input_path, base_dir / output_path, data=defaults | data)
+        write_output(src_path=input_path, dst_path=base_dir / output_path, data=defaults | data)
 
     if check_untracked:  # pragma: no cover
         check_for_untracked(paths, base_dir)
diff --git a/tests/data/copier_demo/.ctt/no_all/.copier-answers.testing_no_all.yml b/tests/data/copier_demo/.ctt/no_all/.copier-answers.testing_no_all.yml
new file mode 100644
index 0000000..c62f113
--- /dev/null
+++ b/tests/data/copier_demo/.ctt/no_all/.copier-answers.testing_no_all.yml
@@ -0,0 +1,9 @@
+# yamllint disable-file
+# Answer file maintained by Copier for: https://github.com/KyleKing/calcipy_template
+# Check into version control. Edit by re-running copier and changing responses to the questions
+_src_path: ../../copier_demo
+copyright_date: '2023'
+include_all: false
+minimum_python: 3.11.0
+package_name_py: testing_no_all
+project_name: testing-no-all
diff --git a/tests/data/copier_demo/.gitattributes b/tests/data/copier_demo/.gitattributes
deleted file mode 100644
index 77eba7d..0000000
--- a/tests/data/copier_demo/.gitattributes
+++ /dev/null
@@ -1,2 +0,0 @@
-# Ensure whitespace consistency when testing ctt on any OS
-text eol=lf
diff --git a/tests/data/copier_demo/copier.yml b/tests/data/copier_demo/copier.yml
index 12c5bae..030154e 100644
--- a/tests/data/copier_demo/copier.yml
+++ b/tests/data/copier_demo/copier.yml
@@ -2,7 +2,7 @@
 _min_copier_version: 7.0.0
 
 # ====== TEMPLATE ======
-_answers_file: .copier-answers.yml
+_answers_file: .copier-answers.{{ package_name_py }}.yml
 _subdirectory: template_dir
 
 # ====== QUESTIONS ======
diff --git a/tests/helpers.py b/tests/helpers.py
index e54035d..410a47d 100644
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -5,6 +5,7 @@
 from typing import Generator
 
 from beartype import beartype
+from corallium.file_helpers import read_lines
 from corallium.log import get_logger
 from pytestshellutils.shell import Subprocess
 from pytestshellutils.utils.processes import ProcessResult
@@ -71,3 +72,15 @@ def temporary_git_dir(shell: Subprocess, *, source_dir: Path | None = None) -> G
             add_commit(shell, cwd=working_dir)
 
         yield working_dir
+
+
+@beartype
+def reset_path_in_test_answers() -> None:
+    """Replace the absolute path written by copier.."""
+    answers_path = TEST_DATA_DIR / 'copier_demo/.ctt/no_all/.copier-answers.testing_no_all.yml'
+    lines = (
+        '_src_path: ../../copier_demo' if line.startswith('_src_path') else line
+        for line in read_lines(answers_path)
+        if line
+    )
+    answers_path.write_text('\n'.join(lines) + '\n')
diff --git a/tests/test_config.py b/tests/test_config.py
index 34350f4..0dea9a5 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -24,6 +24,10 @@
             does_not_raise(),
         ),
     ],
+    ids=[
+        'Check an incomplete config',
+        'Check a normal config',
+    ],
 )
 @beartype
 def test_validate_config(config: dict, expectation) -> None:
diff --git a/tests/test_main.py b/tests/test_main.py
index 86bd541..1b7663b 100644
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -8,7 +8,7 @@
 from copier_template_tester.main import run
 
 from .configuration import TEST_DATA_DIR
-from .helpers import DEMO_DIR, run_ctt
+from .helpers import DEMO_DIR, reset_path_in_test_answers, run_ctt
 
 logger = get_logger()
 
@@ -23,6 +23,7 @@ def _run_auto(src_path: str, dst_path: Path, **kwargs) -> None:  # noqa: ARG001
     monkeypatch.setattr(copier, 'run_auto', _run_auto)
 
     run(base_dir=DEMO_DIR)
+    reset_path_in_test_answers()
 
 
 @beartype
@@ -32,12 +33,13 @@ def test_main(shell: Subprocess) -> None:
     assert ret.returncode == 0
     ret.stdout.matcher.fnmatch_lines(['*Creating:*copier_demo*no_all*', ''])
     # Check output from copier
-    ret.stderr.matcher.fnmatch_lines_random([  # Order can vary on Windows
-        '*Copying from template*',
-        '* .copier-answers.yml*',
-        '*identical* README.md*',
-        '*identical* script.py*',
-    ])
+    ret.stderr.matcher.fnmatch_lines(['*Copying from template*'])
+    reset_path_in_test_answers()
+    # Check a few of the created files:
+    paths = {pth.relative_to(DEMO_DIR) for pth in (DEMO_DIR / '.ctt').rglob('*.*') if pth.is_file()}
+    assert Path('.ctt/no_all/README.md') in paths
+    assert Path('.ctt/no_all/.copier-answers.testing_no_all.yml') in paths
+    assert Path('.ctt/no_all/.copier-answers.yml') not in paths
 
 
 @beartype
diff --git a/tests/test_pre_commit_support.py b/tests/test_pre_commit_support.py
index dc9d43d..5d1392a 100644
--- a/tests/test_pre_commit_support.py
+++ b/tests/test_pre_commit_support.py
@@ -57,12 +57,18 @@ def test_ctt_with_untracked_files(shell: Subprocess) -> None:
         (copier_dir / untracked_file).write_text('Placeholder\n')
         add_commit(shell, cwd=copier_dir)
         ret = run_ctt(shell, cwd=TEST_DATA_DIR, args=[f'--base-dir={copier_dir}', '--check-untracked'])
+        # Store paths to check later
+        paths = {pth.relative_to(copier_dir) for pth in (copier_dir / '.ctt').rglob('*.*') if pth.is_file()}
 
     assert ret.returncode == 1
     ret.stdout.matcher.fnmatch_lines(['*Creating:*copier_demo*no_all*', ''])
     # Check output from copier
     ret.stderr.matcher.fnmatch_lines_random([  # Order can vary on Windows
         '*Copying from template*',
-        '*conflict* .copier-answers.yml*',
+        '*conflict* .copier-answers.testing_no_all.yml*',
         f'*create* {untracked_file.name}*',
     ])
+    # Check created files:
+    assert Path('.ctt/no_all/.copier-answers.testing_no_all.yml') in paths
+    assert Path('.ctt/no_all/.copier-answers.yml') not in paths
+    assert Path(f'.ctt/no_all/{untracked_file.name}') in paths