Repository: pypa/setuptools

Issue:
[BUG] StaticModule not choosing the Python file encoding
### setuptools version

setuptools==65.4.0

### Python version

3.7, 3.8, 3.9, 3.10, 3.11

### OS

Windows mainly, but reproducible on Linux

### Additional environment information

_No response_

### Description

The [StaticModule](https://github.com/pypa/setuptools/blob/badd7064e202635b8a94d5603b31a37ca7d58d6c/setuptools/config/expand.py#L61) opens Python file without looking at `-*-` encoding, and without specifying the encoding:

```python
    with open(spec.origin) as strm:
```

On Linux as most Python files are UTF-8 and as Python defaults to UTF-8 to open files, it goes right.

But on Windows, Python will default to CP1252 at file opening, possibly leading to a UnicodeDecodeError.

When the module don't have an import on an external dependency the `fallback to evaluate module` hides the bug.

### Expected behavior

I expect `StaticModule` to first look at the encoding marker (`# -*- coding: encoding -*-`) of the file before trying to decode it, and defaulting to UTF-8 if no maker is found.

### How to Reproduce

I first found the bug on a Windows CI runner on github so it was reading UTF-8 file as CP1252. I'm on Debian so I'll repro it the other way around: writing a latin1 file and I let setuptools fail to decode it.

```bash
mkdir repro
cd repro
echo $'# -*- coding: latin1 -*-\nimport regex\n__version__ = "0.1"\nprint("\xe9")' > repro.py 
cat <<EOF >pyproject.toml 
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my_package"
dynamic = ["version"]
dependencies = [
    "regex",
]

[tool.setuptool]
py-modules = ['repro.py']

[tool.setuptools.dynamic.version]
attr = "repro.__version__"
EOF
pip install .
```



### Output

<details>
<summary>
output
</summary>

```text
Processing /home/mdk/clones/setuptools/repro
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [58 lines of output]
      /tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
        warnings.warn(msg, _BetaConfiguration)
      Traceback (most recent call last):
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 191, in read_attr
          return getattr(StaticModule(module_name, spec), attr_name)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 66, in __init__
          src = strm.read()
        File "/usr/lib/python3.10/codecs.py", line 322, in decode
          (result, consumed) = self._buffer_decode(data, self.errors, final)
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 65: invalid continuation byte
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 1, in <module>
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 159, in setup
          dist.parse_config_files()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 62, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 140, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 195, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 242, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 286, in _expand_all_dynamic
          version=self._obtain_version(dist, package_dir),
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 332, in _obtain_version
          return _expand.version(self._obtain(dist, "version", package_dir))
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 321, in _obtain
          return self._expand_directive(
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 315, in _expand_directive
          return _expand.read_attr(directive["attr"], package_dir, root_dir)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 194, in read_attr
          module = _load_spec(spec, module_name)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 214, in _load_spec
          spec.loader.exec_module(module)  # type: ignore
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/home/mdk/clones/setuptools/repro/repro.py", line 2, in <module>
          import regex
      ModuleNotFoundError: No module named 'regex'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
```

</details>


Discussion:
Ohh, looks like ast.parse handle it if given bytes!

Resolving patch:
diff --git a/setuptools/config/expand.py b/setuptools/config/expand.py
--- a/setuptools/config/expand.py
+++ b/setuptools/config/expand.py
@@ -21,6 +21,7 @@
 import importlib
 import io
 import os
+import pathlib
 import sys
 import warnings
 from glob import iglob
@@ -62,9 +63,7 @@ class StaticModule:
     """Proxy to a module object that avoids executing arbitrary code."""
 
     def __init__(self, name: str, spec: ModuleSpec):
-        with open(spec.origin) as strm:  # type: ignore
-            src = strm.read()
-        module = ast.parse(src)
+        module = ast.parse(pathlib.Path(spec.origin).read_bytes())
         vars(self).update(locals())
         del self.self
 


Test patch:
diff --git a/setuptools/tests/config/test_expand.py b/setuptools/tests/config/test_expand.py
--- a/setuptools/tests/config/test_expand.py
+++ b/setuptools/tests/config/test_expand.py
@@ -60,6 +60,20 @@ def test_read_files(tmp_path, monkeypatch):
 
 
 class TestReadAttr:
+    @pytest.mark.parametrize(
+        "example",
+        [
+            # No cookie means UTF-8:
+            b"__version__ = '\xc3\xa9'\nraise SystemExit(1)\n",
+            # If a cookie is present, honor it:
+            b"# -*- coding: utf-8 -*-\n__version__ = '\xc3\xa9'\nraise SystemExit(1)\n",
+            b"# -*- coding: latin1 -*-\n__version__ = '\xe9'\nraise SystemExit(1)\n",
+        ]
+    )
+    def test_read_attr_encoding_cookie(self, example, tmp_path):
+        (tmp_path / "mod.py").write_bytes(example)
+        assert expand.read_attr('mod.__version__', root_dir=tmp_path) == 'é'
+
     def test_read_attr(self, tmp_path, monkeypatch):
         files = {
             "pkg/__init__.py": "",