Repository: copier-org/copier
Pull request #2003 (2025-03-03 09:11:28)

fix: auto-detect encoding when reading external data file
I've fixed loading external data files to auto-detect the encoding instead of using the system's default encoding.

Without the fix, the added test fails with this error:

```
self = <yaml.loader.SafeLoader object at 0x7f20921b6e70>, size = 4096

    def update_raw(self, size=4096):
>       data = self.stream.read(size)
E       UnicodeDecodeError: 'cp932' codec can't decode byte 0x98 in position 7: illegal multibyte sequence
```

Follow-up of #1973. Inspired by #1998.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/copier-org/copier/pull/2003?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) Report
Attention: Patch coverage is `95.45455%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 92.59%. Comparing base [(`0229b77`)](https://app.codecov.io/gh/copier-org/copier/commit/0229b779de7cbea4d2d4284b2a8ed6ec69de12ec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) to head [(`3fec4af`)](https://app.codecov.io/gh/copier-org/copier/commit/3fec4af612cb90fbf6b6081604017ab2ce9915b7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org).

| [Files with missing lines](https://app.codecov.io/gh/copier-org/copier/pull/2003?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) | Patch % | Lines |
|---|---|---|
| [tests/test\_answersfile.py](https://app.codecov.io/gh/copier-org/copier/pull/2003?src=pr&el=tree&filepath=tests%2Ftest_answersfile.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org#diff-dGVzdHMvdGVzdF9hbnN3ZXJzZmlsZS5weQ==) | 95.23% | [1 Missing :warning: ](https://app.codecov.io/gh/copier-org/copier/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) |

> :exclamation:  There is a different number of reports uploaded between BASE (0229b77) and HEAD (3fec4af). Click for more details.
> 
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (0229b77) | HEAD (3fec4af) |
>|------|------|------|
>|unittests|15|1|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #2003      +/-   ##
==========================================
- Coverage   97.93%   92.59%   -5.35%     
==========================================
  Files          53       53              
  Lines        5677     5698      +21     
==========================================
- Hits         5560     5276     -284     
- Misses        117      422     +305     
```

| [Flag](https://app.codecov.io/gh/copier-org/copier/pull/2003/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/copier-org/copier/pull/2003/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org) | `92.59% <95.45%> (-5.35%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/copier-org/copier/pull/2003?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=copier-org).

sisp: CI is failing with this error:

```
Error: This request has been automatically failed because it uses a deprecated version of `actions/cache: v3.0.11`. Please update your workflow to use v3/v4 of actions/cache to avoid interruptions. Learn more: https://github.blog/changelog/2024-12-05-notice-of-upcoming-releases-and-breaking-changes-for-github-actions/#actions-cache-v1-v2-and-actions-toolkit-cache-package-closing-down
```

See https://github.com/JRMurr/direnv-nix-action/issues/12.
pawamoy: Nice one 👌

Patch:
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
deleted file mode 100644
index 4941da4..0000000
--- a/.github/dependabot.yml
+++ /dev/null
@@ -1,11 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "daily"
-
-  - package-ecosystem: "pip"
-    directory: "/"
-    schedule:
-      interval: "daily"
diff --git a/copier/settings.py b/copier/settings.py
index 6a9069b..b1d5d36 100644
--- a/copier/settings.py
+++ b/copier/settings.py
@@ -37,7 +37,7 @@ class Settings(BaseModel):
             else:
                 settings_path = user_config_path("copier") / "settings.yml"
         if settings_path.is_file():
-            data = yaml.safe_load(settings_path.read_text())
+            data = yaml.safe_load(settings_path.read_bytes())
             return cls.model_validate(data)
         elif env_path:
             warnings.warn(
diff --git a/copier/user_data.py b/copier/user_data.py
index ea293fe..222ffb4 100644
--- a/copier/user_data.py
+++ b/copier/user_data.py
@@ -522,7 +522,7 @@ def load_answersfile_data(
 ) -> AnyByStrDict:
     """Load answers data from a `$dst_path/$answers_file` file if it exists."""
     try:
-        with Path(dst_path, answers_file).open(encoding="utf-8") as fd:
+        with Path(dst_path, answers_file).open("rb") as fd:
             return yaml.safe_load(fd)
     except (FileNotFoundError, IsADirectoryError):
         warnings.warn(
diff --git a/poetry.lock b/poetry.lock
index 247e16f..5111a58 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1494,18 +1494,17 @@ files = [
 
 [[package]]
 name = "types-pygments"
-version = "2.19.0.20250219"
+version = "2.19.0.20250305"
 description = "Typing stubs for Pygments"
 optional = false
 python-versions = ">=3.9"
 files = [
-    {file = "types_Pygments-2.19.0.20250219-py3-none-any.whl", hash = "sha256:5e3e1f660665b3a2ea946dda794b8d5b05772d789181704b523d646e8a7f4382"},
-    {file = "types_pygments-2.19.0.20250219.tar.gz", hash = "sha256:a4a279338c96f3d4f2eb2c4d7c6c5593c88108b185bb5c664f943f781170cd14"},
+    {file = "types_pygments-2.19.0.20250305-py3-none-any.whl", hash = "sha256:ca88aae5ec426f9b107c0f7adc36dc096d2882d930a49f679eaf4b8b643db35d"},
+    {file = "types_pygments-2.19.0.20250305.tar.gz", hash = "sha256:044c50e80ecd4128c00a7268f20355e16f5c55466d3d49dfda09be920af40b4b"},
 ]
 
 [package.dependencies]
 types-docutils = "*"
-types-setuptools = "*"
 
 [[package]]
 name = "types-pyyaml"
@@ -1518,17 +1517,6 @@ files = [
     {file = "types_pyyaml-6.0.12.20241230.tar.gz", hash = "sha256:7f07622dbd34bb9c8b264fe860a17e0efcad00d50b5f27e93984909d9363498c"},
 ]
 
-[[package]]
-name = "types-setuptools"
-version = "75.2.0.20241025"
-description = "Typing stubs for setuptools"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "types-setuptools-75.2.0.20241025.tar.gz", hash = "sha256:2949913a518d5285ce00a3b7d88961c80a6e72ffb8f3da0a3f5650ea533bd45e"},
-    {file = "types_setuptools-75.2.0.20241025-py3-none-any.whl", hash = "sha256:6721ac0f1a620321e2ccd87a9a747c4a383dc381f78d894ce37f2455b45fcf1c"},
-]
-
 [[package]]
 name = "typing-extensions"
 version = "4.12.2"


Test patch:
diff --git a/tests/test_answersfile.py b/tests/test_answersfile.py
index b5ebaec..619d6fd 100644
--- a/tests/test_answersfile.py
+++ b/tests/test_answersfile.py
@@ -1,10 +1,12 @@
 from __future__ import annotations
 
 import json
+import sys
 from pathlib import Path
 from textwrap import dedent
 
 import pytest
+import yaml
 
 import copier
 from copier.user_data import load_answersfile_data
@@ -252,6 +254,59 @@ def test_external_data_with_umlaut(
     assert answers["ext_umlaut"] == "äöü"
 
 
+@pytest.mark.parametrize(
+    "encoding",
+    ["utf-8", "utf-8-sig", "utf-16-le", "utf-16-be"],
+)
+def test_external_data_with_unicode_characters(
+    tmp_path_factory: pytest.TempPathFactory,
+    monkeypatch: pytest.MonkeyPatch,
+    encoding: str,
+) -> None:
+    def _encode(data: str) -> bytes:
+        if encoding.startswith("utf-16"):
+            data = f"\ufeff{data}"
+        return data.encode(encoding)
+
+    src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))
+
+    build_file_tree(
+        {
+            (src / "copier.yml"): (
+                """\
+                _external_data:
+                    data: data.yml
+                foo: "{{ _external_data.data.foo }}"
+                bar: "{{ _external_data.data.bar }}"
+                """
+            ),
+            (src / "{{ _copier_conf.answers_file }}.jinja"): (
+                "{{ _copier_answers|to_nice_yaml }}"
+            ),
+        }
+    )
+    git_save(src, tag="v1")
+
+    data = {
+        "foo": "\u3053\u3093\u306b\u3061\u306f",  # japanese hiragana
+        "bar": "\U0001f60e",  # smiling face with sunglasses
+    }
+    (dst / "data.yml").write_bytes(_encode(yaml.dump(data, allow_unicode=True)))
+
+    with monkeypatch.context() as m:
+        # Override the factor that determines the default encoding when opening files.
+        if sys.version_info >= (3, 10):
+            m.setattr("io.text_encoding", lambda *_args: "cp932")
+        else:
+            m.setattr("_bootlocale.getpreferredencoding", lambda *_args: "cp932")
+
+        copier.run_copy(str(src), dst, defaults=True, overwrite=True)
+
+    answers = load_answersfile_data(dst)
+    assert answers["foo"] == data["foo"]
+    assert answers["bar"] == data["bar"]
+
+
 def test_undefined_phase_in_external_data(
     tmp_path_factory: pytest.TempPathFactory,
 ) -> None:
diff --git a/tests/test_settings.py b/tests/test_settings.py
index 9cef221..68abec7 100644
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -1,8 +1,10 @@
 from __future__ import annotations
 
+import sys
 from pathlib import Path
 
 import pytest
+import yaml
 
 from copier.errors import MissingSettingsWarning
 from copier.settings import Settings
@@ -72,6 +74,39 @@ def test_settings_defined_but_missing(
         Settings.from_file()
 
 
+@pytest.mark.parametrize(
+    "encoding",
+    ["utf-8", "utf-8-sig", "utf-16-le", "utf-16-be"],
+)
+def test_settings_from_utf_file(
+    settings_path: Path, monkeypatch: pytest.MonkeyPatch, encoding: str
+) -> None:
+    def _encode(data: str) -> bytes:
+        if encoding.startswith("utf-16"):
+            data = f"\ufeff{data}"
+        return data.encode(encoding)
+
+    defaults = {
+        "foo": "\u3053\u3093\u306b\u3061\u306f",  # japanese hiragana
+        "bar": "\U0001f60e",  # smiling face with sunglasses
+    }
+
+    settings_path.write_bytes(
+        _encode(yaml.dump({"defaults": defaults}, allow_unicode=True))
+    )
+
+    with monkeypatch.context() as m:
+        # Override the factor that determines the default encoding when opening files.
+        if sys.version_info >= (3, 10):
+            m.setattr("io.text_encoding", lambda *_args: "cp932")
+        else:
+            m.setattr("_bootlocale.getpreferredencoding", lambda *_args: "cp932")
+
+        settings = Settings.from_file()
+
+    assert settings.defaults == defaults
+
+
 @pytest.mark.parametrize(
     ("repository", "trust", "is_trusted"),
     [