Repository: copier-org/copier (python)

Issue:
--data-file option fails with UTF-8 YAML file on systems with non-UTF-8 default encoding
### Describe the problem

On systems with non-UTF-8 default encodings (like CP932 on Japanese Windows), using the `--data-file` option with UTF-8 encoded YAML files results in encoding errors.

This happens because the current implementation opens files in text mode without specifying an encoding:

```python
def data_file_switch(self, path: cli.ExistingFile) -> None:
    """Update [data][] with provided values.

    Arguments:
        path: The path to the YAML file to load.
    """
    with Path(path).open() as f:
        file_updates: AnyByStrDict = yaml.safe_load(f)
```

When the system's default encoding is not UTF-8, UTF-8 encoded YAML files cannot be read correctly.

To avoid this issue, users must save their YAML file using the system's default encoding rather than the Unicode encoding that is expected for YAML files.

#### Proposed Solution

PyYAML can automatically detect which Unicode encoding is used (UTF-8, UTF-16LE or UTF-16BE).
To use this feature, raw data must be passed to PyYAML like this:

```python
    with Path(path).open("rb") as f:
        file_updates: AnyByStrDict = yaml.safe_load(f)
```

### Template

not a template problem

### To Reproduce

On Japanese Windows 10 (in my case)
1. Create a data file `myparams.yml` and save it in UTF-8.
2. Run `copier copy --data-file .\myparams.yml .\template .\output`

### Logs

```console
Traceback (most recent call last):
  File "M:\Dev\copier\venv\Scripts\\copier", line 6, in <module>
    sys.exit(copier_app_run())
             ^^^^^^^^^^^^^^^^
  File "M:\Dev\copier\venv\Lib\site-packages\plumbum\cli\application.py", line 640, in run
    inst, retcode = subapp.run(argv, exit=False)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\copier\venv\Lib\site-packages\plumbum\cli\application.py", line 631, in run
    f(inst, *a)
  File "M:\Dev\copier\copier-master\copier\cli.py", line 193, in data_file_switch
    file_updates: AnyByStrDict = yaml.safe_load(f)
                                 ^^^^^^^^^^^^^^^^^
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\__init__.py", line 79, in load
    loader = Loader(stream)
             ^^^^^^^^^^^^^^
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\reader.py", line 85, in __init__
    self.determine_encoding()
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\reader.py", line 124, in determine_encoding
    self.update_raw()
  File "M:\Dev\copier\venv\Lib\site-packages\yaml\reader.py", line 178, in update_raw
    data = self.stream.read(size)
           ^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 6: illegal multibyte sequence
```

### Expected behavior

`--data-file` option reads UTF-8 YAML file correctly.

### Screenshots/screencasts/logs

_No response_

### Operating system

Windows

### Operating system distribution and version

Japanese Windows 10

### Copier version

0.0.0

### Python version

CPython 3.11

### Installation method

local build

### Additional context

_No response_

Pull request:
fix: read unicode-encoded file specified with --data-file
Previously, the file specified with --data-file option were read with the system default encoding.
With this change, the binary stream is passed to PyYAML, and PyYAML detects the encoding automatically.
This ensures that UTF-8/UTF-16BE/UTF-16LE YAML files are read correctly, regardless of the system default encoding.

Fixes #1995

Patch:
diff --git a/copier/cli.py b/copier/cli.py
index 47532c1..c3a6380 100644
--- a/copier/cli.py
+++ b/copier/cli.py
@@ -189,7 +189,7 @@ class _Subcommand(cli.Application):  # type: ignore[misc]
         Arguments:
             path: The path to the YAML file to load.
         """
-        with Path(path).open() as f:
+        with Path(path).open("rb") as f:
             file_updates: AnyByStrDict = yaml.safe_load(f)
 
         updates_without_cli_overrides = {


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 8c6331a..d54ca85 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1,7 +1,10 @@
+from __future__ import annotations
+
 import subprocess
 import sys
+from collections.abc import Generator
 from pathlib import Path
-from typing import Callable, Generator
+from typing import Callable
 
 import pytest
 import yaml
@@ -223,6 +226,79 @@ def test_data_cli_takes_precedence_over_data_file(
     assert actual_answers == expected_answers
 
 
+@pytest.mark.parametrize(
+    "data_file_encoding",
+    ["utf-8", "utf-8-sig", "utf-16-le", "utf-16-be"],
+)
+def test_read_utf_data_file(
+    tmp_path_factory: pytest.TempPathFactory,
+    monkeypatch: pytest.MonkeyPatch,
+    data_file_encoding: str,
+) -> None:
+    src, dst, local = map(tmp_path_factory.mktemp, ("src", "dst", "local"))
+
+    def encode_data(data: dict[str, str]) -> bytes:
+        dump = yaml.dump(data, allow_unicode=True)
+        if data_file_encoding.startswith("utf-16"):
+            dump = "\ufeff" + dump
+        return dump.encode(data_file_encoding)
+
+    build_file_tree(
+        {
+            (src / "copier.yml"): (
+                """\
+                text1:
+                    type: str
+                text2:
+                    type: str
+                """
+            ),
+            (src / "{{ _copier_conf.answers_file }}.jinja"): (
+                """\
+                # Changes here will be overwritten by Copier
+                {{ _copier_answers|to_nice_yaml }}
+                """
+            ),
+            (local / "data.yml"): encode_data(
+                {
+                    "text1": "\u3053\u3093\u306b\u3061\u306f",  # japanese hiragana
+                    "text2": "\U0001f60e",  # smiling face with sunglasses
+                }
+            ),
+        }
+    )
+
+    with monkeypatch.context() as m:
+        # Override the factor that determine the default encoding when opening files.
+        # data.yml should be read correctly regardless of this value.
+        if sys.version_info >= (3, 10):
+            m.setattr("io.text_encoding", lambda *_args: "cp932")
+        else:
+            m.setattr("_bootlocale.getpreferredencoding", lambda *_args: "cp932")
+
+        run_result = CopierApp.run(
+            [
+                "copier",
+                "copy",
+                f"--data-file={local / 'data.yml'}",
+                str(src),
+                str(dst),
+            ],
+            exit=False,
+        )
+
+    assert run_result[1] == 0
+    expected_answers = {
+        "_src_path": str(src),
+        "text1": "\u3053\u3093\u306b\u3061\u306f",
+        "text2": "\U0001f60e",
+    }
+    with Path(dst / ".copier-answers.yml").open(encoding="utf-8") as f:
+        content = f.read()
+        actual_answers = yaml.safe_load(content)
+    assert actual_answers == expected_answers
+
+
 @pytest.mark.parametrize(
     "config_folder",
     map(