Repository: copier-org/copier (python)

Issue:
default option for choices with multiselect
### Describe the problem

When enabling multiselect, the default value is not selected

### Template

```yaml
python_version:
  type: str
  help: What version of python are you targeting?
  default: "3.11"
  choices:
    - "3.8"
    - "3.9"
    - "3.10"
    - "3.11"
    - "3.12"
 
github_runner_python_version:
  type: str
  help: Which Python versions do you want to use on your Github Runner?
  default: "{{ python_version }}"
  multiselect: true
  choices:
    - "3.8"
    - "3.9"
    - "3.10"
    - "3.11"
    - "3.12"
 ```

### To Reproduce

Use the supplied template and copy, see that the version is not selected

### Logs

_No response_

### Expected behavior

`github_runner_python_version` already has the `python_version` selected

### Screenshots/screencasts/logs

_No response_

### Operating system

Linux

### Operating system distribution and version

Debian bookworm

### Copier version

copier 9.2.0

### Python version

Python 3.11.8

### Installation method

pip+pypi

### Additional context

_No response_

Pull request description:
fix: render `default` list items for multi-select choice questions
I've fixed a bug related to rendering templated list items which was discovered when trying to template an item in the default value list of a multi-select choice question (see #1594). The root cause was not directly related to rendering the default value but rather caused by the `Question.render_value()` method passing any non-string value through. To resolve this problem fundamentally, I've extended the `Question.render_value()` method to recurse into a list and render each list item individually. I think we should add support for recursing into a `dict` in a follow-up PR and, thus, add support for rendering even deeply nested value in YAML data (e.g. actual, i.e. non-string, YAML choice items and default values to `type: yaml` questions).

Fixes #1594.

Patch:
diff --git a/copier/main.py b/copier/main.py
index 5e148d0..277f01d 100644
--- a/copier/main.py
+++ b/copier/main.py
@@ -11,7 +11,7 @@ from filecmp import dircmp
 from functools import cached_property, partial
 from itertools import chain
 from pathlib import Path
-from shutil import rmtree
+from shutil import copytree, ignore_patterns, rmtree
 from tempfile import TemporaryDirectory
 from types import TracebackType
 from typing import (
@@ -853,8 +853,12 @@ class Worker:
         subproject_subdir = self.subproject.local_abspath.relative_to(subproject_top)
 
         with TemporaryDirectory(
-            prefix=f"{__name__}.old_copy."
-        ) as old_copy, TemporaryDirectory(prefix=f"{__name__}.new_copy.") as new_copy:
+            prefix=f"{__name__}.old_copy.",
+        ) as old_copy, TemporaryDirectory(
+            prefix=f"{__name__}.new_copy.",
+        ) as new_copy, TemporaryDirectory(
+            prefix=f"{__name__}.dst_copy.",
+        ) as dst_copy:
             # Copy old template into a temporary destination
             with replace(
                 self,
@@ -866,27 +870,23 @@ class Worker:
                 vcs_ref=self.subproject.template.commit,  # type: ignore[union-attr]
             ) as old_worker:
                 old_worker.run_copy()
-            # Extract diff between temporary destination and real destination
-            with local.cwd(old_copy):
-                self._git_initialize_repo()
-                git("remote", "add", "real_dst", "file://" + str(subproject_top))
-                git("fetch", "--depth=1", "real_dst", "HEAD")
-                diff_cmd = git[
-                    "diff-tree", f"--unified={self.context_lines}", "HEAD...FETCH_HEAD"
-                ]
-                try:
-                    diff = diff_cmd("--inter-hunk-context=-1")
-                except ProcessExecutionError:
-                    print(
-                        colors.warn
-                        | "Make sure Git >= 2.24 is installed to improve updates.",
-                        file=sys.stderr,
-                    )
-                    diff = diff_cmd("--inter-hunk-context=0")
             # Run pre-migration tasks
             self._execute_tasks(
                 self.template.migration_tasks("before", self.subproject.template)  # type: ignore[arg-type]
             )
+            # Create a copy of the real destination after applying migrations
+            # but before performing any further update for extracting the diff
+            # between the temporary destination of the old template and the
+            # real destination later.
+            with local.cwd(dst_copy):
+                copytree(
+                    subproject_top,
+                    ".",
+                    symlinks=True,
+                    ignore=ignore_patterns("/.git"),
+                    dirs_exist_ok=True,
+                )
+                self._git_initialize_repo()
             # Clear last answers cache to load possible answers migration, if skip_answered flag is not set
             if self.skip_answered is False:
                 self.answers = AnswersMap()
@@ -913,6 +913,49 @@ class Worker:
                 src_path=self.subproject.template.url,  # type: ignore[union-attr]
             ) as new_worker:
                 new_worker.run_copy()
+            with local.cwd(new_copy):
+                self._git_initialize_repo()
+            # Extract diff between temporary destination and (copy from above)
+            # real destination with some special handling of newly added files
+            # in both the poject and the template.
+            with local.cwd(old_copy):
+                self._git_initialize_repo()
+                git("remote", "add", "dst_copy", "file://" + str(dst_copy))
+                git("fetch", "--depth=1", "dst_copy", "HEAD:dst_copy")
+                git("remote", "add", "new_copy", "file://" + str(new_copy))
+                git("fetch", "--depth=1", "new_copy", "HEAD:new_copy")
+                # Create an empty file in the temporary destination when the
+                # same file was added in *both* the project and the temporary
+                # destination of the new template. With this minor change, the
+                # diff between the temporary destination and the real
+                # destination for such files will use the "update file mode"
+                # instead of the "new file mode" which avoids deleting the file
+                # content previously added in the project.
+                diff_added_cmd = git[
+                    "diff-tree", "-r", "--diff-filter=A", "--name-only"
+                ]
+                for filename in (
+                    set(diff_added_cmd("HEAD...dst_copy").splitlines())
+                ) & set(diff_added_cmd("HEAD...new_copy").splitlines()):
+                    f = Path(filename)
+                    f.parent.mkdir(parents=True, exist_ok=True)
+                    f.touch(Path(dst_copy, filename).stat().st_mode)
+                    git("add", filename)
+                self._git_commit("add new empty files")
+                # Extract diff between temporary destination and real
+                # destination
+                diff_cmd = git[
+                    "diff-tree", f"--unified={self.context_lines}", "HEAD...dst_copy"
+                ]
+                try:
+                    diff = diff_cmd("--inter-hunk-context=-1")
+                except ProcessExecutionError:
+                    print(
+                        colors.warn
+                        | "Make sure Git >= 2.24 is installed to improve updates.",
+                        file=sys.stderr,
+                    )
+                    diff = diff_cmd("--inter-hunk-context=0")
             compared = dircmp(old_copy, new_copy)
             # Try to apply cached diff into final destination
             with local.cwd(subproject_top):
@@ -998,6 +1041,10 @@ class Worker:
         git = get_git()
         git("init", retcode=None)
         git("add", ".")
+        self._git_commit()
+
+    def _git_commit(self, message: str = "dumb commit") -> None:
+        git = get_git()
         git("config", "user.name", "Copier")
         git("config", "user.email", "copier@copier")
         # 1st commit could fail if any pre-commit hook reformats code
@@ -1006,7 +1053,7 @@ class Worker:
             "commit",
             "--allow-empty",
             "-am",
-            "dumb commit 1",
+            f"{message} 1",
             "--no-gpg-sign",
             retcode=None,
         )
@@ -1014,7 +1061,7 @@ class Worker:
             "commit",
             "--allow-empty",
             "-am",
-            "dumb commit 2",
+            f"{message} 2",
             "--no-gpg-sign",
             "--no-verify",
         )
diff --git a/copier/user_data.py b/copier/user_data.py
index 62158e0..ad68f16 100644
--- a/copier/user_data.py
+++ b/copier/user_data.py
@@ -432,7 +432,11 @@ class Question:
             template = self.jinja_env.from_string(value)
         except TypeError:
             # value was not a string
-            return value
+            return (
+                [self.render_value(item) for item in value]
+                if isinstance(value, list)
+                else value
+            )
         try:
             return template.render({**self.answers.combined, **(extra_answers or {})})
         except UndefinedError as error:


Test patch:
diff --git a/tests/test_templated_prompt.py b/tests/test_templated_prompt.py
index 2c14b42..2a1b124 100644
--- a/tests/test_templated_prompt.py
+++ b/tests/test_templated_prompt.py
@@ -362,3 +362,65 @@ def test_templated_prompt_with_conditional_choices(
     for iac in iac_choices:
         tui.expect_exact(iac)
     tui.sendline()
+
+
+def test_multiselect_choices_with_templated_default_value(
+    tmp_path_factory: pytest.TempPathFactory,
+    spawn: Spawn,
+) -> None:
+    src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))
+    build_file_tree(
+        {
+            (src / "copier.yml"): (
+                """\
+                python_version:
+                    type: str
+                    help: What version of python are you targeting?
+                    default: "3.11"
+                    choices:
+                        - "3.8"
+                        - "3.9"
+                        - "3.10"
+                        - "3.11"
+                        - "3.12"
+
+                github_runner_python_version:
+                    type: str
+                    help: Which Python versions do you want to use on your Github Runner?
+                    default: ["{{ python_version }}"]
+                    multiselect: true
+                    choices:
+                        - "3.8"
+                        - "3.9"
+                        - "3.10"
+                        - "3.11"
+                        - "3.12"
+                """
+            ),
+            (src / "{{ _copier_conf.answers_file }}.jinja"): (
+                "{{ _copier_answers|to_nice_yaml }}"
+            ),
+        }
+    )
+
+    tui = spawn(COPIER_PATH + ("copy", str(src), str(dst)), timeout=10)
+    expect_prompt(
+        tui, "python_version", "str", help="What version of python are you targeting?"
+    )
+    tui.sendline()  # select `3.11" (default value)
+    expect_prompt(
+        tui,
+        "github_runner_python_version",
+        "str",
+        help="Which Python versions do you want to use on your Github Runner?",
+    )
+    tui.sendline()  # select "[3.11]" (default value)
+    tui.expect_exact(pexpect.EOF)
+
+    answers_file = dst / ".copier-answers.yml"
+    assert answers_file.exists()
+    assert yaml.safe_load(answers_file.read_text()) == {
+        "_src_path": str(src),
+        "python_version": "3.11",
+        "github_runner_python_version": ["3.11"],
+    }
diff --git a/tests/test_updatediff.py b/tests/test_updatediff.py
index 263ae9f..81a2019 100644
--- a/tests/test_updatediff.py
+++ b/tests/test_updatediff.py
@@ -1076,3 +1076,220 @@ def test_conflicted_files_are_marked_unmerged(
             line.startswith("UU") and normalize_git_path(line[3:]) == filename
             for line in lines
         )
+
+
+def test_update_with_new_file_in_template_and_project(
+    tmp_path_factory: pytest.TempPathFactory,
+) -> None:
+    src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))
+
+    with local.cwd(src):
+        build_file_tree(
+            {
+                "{{ _copier_conf.answers_file }}.jinja": (
+                    "{{ _copier_answers|to_yaml }}"
+                ),
+            }
+        )
+        git_init("v1")
+        git("tag", "v1")
+
+    run_copy(str(src), dst, defaults=True, overwrite=True)
+    assert "_commit: v1" in (dst / ".copier-answers.yml").read_text()
+
+    with local.cwd(dst):
+        git_init("v1")
+        Path(".gitlab-ci.yml").write_text(
+            dedent(
+                """\
+                tests:
+                    stage: test
+                    script:
+                        - ./test.sh
+
+                pages:
+                    stage: deploy
+                    script:
+                        - ./deploy.sh
+                """
+            )
+        )
+        git("add", ".")
+        git("commit", "-m", "v2")
+
+    with local.cwd(src):
+        Path(".gitlab-ci.yml.jinja").write_text(
+            dedent(
+                """\
+                tests:
+                    stage: test
+                    script:
+                        - ./test.sh --slow
+                """
+            )
+        )
+        git("add", ".")
+        git("commit", "-m", "v2")
+        git("tag", "v2")
+
+    run_update(dst_path=dst, defaults=True, overwrite=True, conflict="inline")
+    assert "_commit: v2" in (dst / ".copier-answers.yml").read_text()
+    assert (dst / ".gitlab-ci.yml").read_text() == dedent(
+        """\
+        tests:
+            stage: test
+            script:
+        <<<<<<< before updating
+                - ./test.sh
+
+        pages:
+            stage: deploy
+            script:
+                - ./deploy.sh
+        =======
+                - ./test.sh --slow
+        >>>>>>> after updating
+        """
+    )
+
+
+def test_update_with_new_file_in_template_and_project_via_migration(
+    tmp_path_factory: pytest.TempPathFactory,
+) -> None:
+    """Merge conflicts are yielded when both template and project add same file.
+
+    The project adds content to `.gitlab-ci.yml` on top of what template v1 provides.
+    In a template v2, `.gitlab-ci.yml.jinja` is moved to `.gitlab/ci/main.yml.jinja`
+    and `.gitlab-ci.yml.jinja` now includes the generated `.gitlab/ci/main.yml`. To
+    retain the project's changes/additions to `.gitlab-ci.yml`, a pre-update migration
+    task copies `.gitlab-ci.yml` (containing those changes/additions) to
+    `.gitlab/ci/main.yml` and stages it, then Copier applies template v2's version of
+    that file (which was also moved there, but Git doesn't recognize it as status `R`
+    but as `A`).
+    """
+    src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))
+
+    with local.cwd(src):
+        build_file_tree(
+            {
+                "{{ _copier_conf.answers_file }}.jinja": (
+                    "{{ _copier_answers|to_yaml }}"
+                ),
+                ".gitlab-ci.yml.jinja": (
+                    """\
+                    tests:
+                        stage: test
+                        script:
+                            - ./test.sh
+                    """
+                ),
+            }
+        )
+        git_init("v1")
+        git("tag", "v1")
+
+    run_copy(str(src), dst, defaults=True, overwrite=True)
+    assert "_commit: v1" in (dst / ".copier-answers.yml").read_text()
+    assert (dst / ".gitlab-ci.yml").exists()
+
+    with local.cwd(dst):
+        with Path(".gitlab-ci.yml").open(mode="at") as f:
+            f.write(
+                dedent(
+                    """\
+
+                    pages:
+                        stage: deploy
+                        script:
+                            - ./deploy.sh
+                    """
+                )
+            )
+        git_init("v1")
+
+    with local.cwd(src):
+        old_file = Path(".gitlab-ci.yml.jinja")
+        new_file = Path(".gitlab", "ci", "main.yml.jinja")
+        new_file.parent.mkdir(parents=True)
+        # Move `.gitlab-ci.yml.jinja` to `.gitlab/ci/main.yml.jinja`
+        git("mv", old_file, new_file)
+        # Make a small modification in `.gitlab/ci/main.yml.jinja`
+        new_file.write_text(new_file.read_text().replace("test.sh", "test.sh --slow"))
+        # Include `.gitlab/ci/main.yml.jinja` in `.gitlab-ci.yml.jinja`
+        old_file.write_text(
+            dedent(
+                """\
+                include:
+                    - local: .gitlab/ci/main.yml
+                """
+            )
+        )
+        # Add a pre-migration that copies `.gitlab-ci.yml` to
+        # `.gitlab/ci/main.yml` and stages it, so that the user changes made in
+        # the project are retained after moving the file.
+        build_file_tree(
+            {
+                "copier.yml": (
+                    """\
+                    _migrations:
+                    -   version: v2
+                        before:
+                        -    "{{ _copier_python }} {{ _copier_conf.src_path / 'migrate.py' }}"
+                    """
+                ),
+                "migrate.py": (
+                    """\
+                    from pathlib import Path
+                    from plumbum.cmd import git
+
+                    f = Path(".gitlab", "ci", "main.yml")
+                    f.parent.mkdir(parents=True)
+                    f.write_text(Path(".gitlab-ci.yml").read_text())
+                    git("add", f)
+                    """
+                ),
+            }
+        )
+        git("add", ".")
+        git("commit", "-m", "v2")
+        git("tag", "v2")
+
+    run_update(
+        dst_path=dst, defaults=True, overwrite=True, conflict="inline", unsafe=True
+    )
+    assert "_commit: v2" in (dst / ".copier-answers.yml").read_text()
+    assert (dst / ".gitlab-ci.yml").read_text() == dedent(
+        """\
+        <<<<<<< before updating
+        tests:
+            stage: test
+            script:
+                - ./test.sh
+
+        pages:
+            stage: deploy
+            script:
+                - ./deploy.sh
+        =======
+        include:
+            - local: .gitlab/ci/main.yml
+        >>>>>>> after updating
+        """
+    )
+    assert (dst / ".gitlab" / "ci" / "main.yml").read_text() == dedent(
+        """\
+        tests:
+            stage: test
+            script:
+        <<<<<<< before updating
+                - ./test.sh
+
+        pages:
+            stage: deploy
+            script:
+                - ./deploy.sh
+        =======
+                - ./test.sh --slow
+        >>>>>>> after updating
+        """
+    )