Repository: pypa/setuptools

Issue:
[BUG] UnicodeDecodeError with unicode setup.py on windows with non english primary language
### setuptools version

69.02

### Python version

3.11

### OS

Windows 11

### Additional environment information

```
Package    Version
---------- -------
pip        23.3.1
setuptools 69.0.2
wheel      0.42.0
```

I installed Korean language feature on windows and set it primary additionally.

### Description

I encountered this when using poetry install.

I suspect it's because uctools' setup.py author field contains unicode character.

I don't know if this is the right place for the issue, if not transfer it.

### Expected behavior

Maybe uctools should be installed.

### How to Reproduce

Run the following with admin powershell.

```
pip wheel --no-cache-dir --use-pep517 "uctools (==1.3.0)"
```

### Output

```console
➜ jeiea  pip wheel --no-cache-dir --use-pep517 "uctools (==1.3.0)"
Collecting uctools==1.3.0
  Downloading uctools-1.3.0.tar.gz (4.6 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Traceback (most recent call last):
        File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Python311\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\jeiea\AppData\Local\Temp\pip-build-env-1da1ty1i\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\jeiea\AppData\Local\Temp\pip-build-env-1da1ty1i\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "C:\Users\jeiea\AppData\Local\Temp\pip-build-env-1da1ty1i\overlay\Lib\site-packages\setuptools\build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "C:\Users\jeiea\AppData\Local\Temp\pip-build-env-1da1ty1i\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 7, in <module>
      UnicodeDecodeError: 'cp949' codec can't decode byte 0xe2 in position 2953: illegal multibyte sequence
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
```



Discussion:
Hi @jeiea that is very odd indeed...
The part that I don't understand is that setuptools specifically uses `tokenize.open` to avoid any encoding errors for `setup.py`. In theory `tokenize.open` should automatically detect and handle encoding.

I think it might be worthy for you to do an experiment:

1. Download the `sdist` (`.tar.gz`) from the PyPI for the specific version you want and extract it in a folder.
2. Try to read the contents for the `setup.py` using `tokenize.open` and then `exec` them... Does the same error still happens?
    - if there is a different error you might have to pass the `local` and `global` dict to exec (probably with `__name__` set to `__main__`) and/or set `sys.argv = ["python", "setup.py", "egg_info"]`.
    - it is probably better to create a virtual environment for that to avoid missing with your global Python... Also a good idea to install `setuptools`, `wheel` and any other build dependencies the package might require.

This is just an experiment removing `setuptools.build_meta` out of the equation to see if the problem is within that module. Probably it is very hard to replicate the exact environment that you have in your machine.

Please let us know how it goes.
1. Downloaded [this](https://pypi.org/project/uctools/#files)
2. Wrote test.py
```python
from tokenize import open

with open('setup.py') as f:
  s = f.read()
  print(s)
  exec(s)
```

With the above there was no error(regardless of virtual environment). Following is the full history.

```
➜ uctools-1.3.0  poetry init

This command will guide you through creating your pyproject.toml config.

Package name [uctools-1.3.0]:
Version [0.1.0]:
Description []:
Author [jeiea <>, n to skip]:  n
License []:
Compatible Python versions [^3.11]:

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "uctools-1-3-0"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "uctools_1"}]

[tool.poetry.dependencies]
python = "^3.11"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes]
➜ uctools-1.3.0  poetry shell
Creating virtualenv uctools-1-3-0-Q6_KiIg6-py3.11 in C:\Users\jeiea\AppData\Local\pypoetry\Cache\virtualenvs
Spawning shell within C:\Users\jeiea\AppData\Local\pypoetry\Cache\virtualenvs\uctools-1-3-0-Q6_KiIg6-py3.11
PowerShell 7.2.17
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

   A new PowerShell stable release is available: v7.4.0 
   Upgrade now, or check out the release page at:       
     https://aka.ms/PowerShell-Release?tag=v7.4.0       

Loading personal and system profiles took 503ms.
(uctools-1-3-0-py3.11) ➜ uctools-1.3.0  python test.py
from setuptools import setup

setup(
    name='uctools',
    version="1.3.0",
    description='Unicode tools.',
    long_description=open('README.rst').read(),
    url='https://github.com/luismsgomes/uctools',
    author='Luís Gomes',
    author_email='luismsgomes@gmail.com',
    license='MIT',
    # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Topic :: Text Processing :: General',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3',
    ],
    python_requires='>=3',
    keywords='text unicode character',
    package_dir={'': 'src'},
    py_modules=['ucinfo', 'ucenum', 'ucnorm'],
    entry_points={
        'console_scripts': [
            'ucinfo=ucinfo:_main',
            'ucenum=ucenum:_main',
            'ucnorm=ucnorm:_main',
        ],
    },
)

usage: test.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: test.py --help [cmd1 cmd2 ...]
   or: test.py --help-commands
   or: test.py cmd --help

error: no commands supplied
```
Amazing, I found something.

```python
import tokenize

with getattr(tokenize, 'open', open)('setup.py') as f:
  s = f.read()
  print(s)
  exec(s)
```

```
(uctools-1-3-0-py3.11) ➜ uctools-1.3.0  ✗ python test.py
from setuptools import setup

setup(
    name='uctools',
    version="1.3.0",
    description='Unicode tools.',
    long_description=open('README.rst').read(),
    url='https://github.com/luismsgomes/uctools',
    author='Luís Gomes',
    author_email='luismsgomes@gmail.com',
    license='MIT',
    # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Topic :: Text Processing :: General',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3',
    ],
    python_requires='>=3',
    keywords='text unicode character',
    package_dir={'': 'src'},
    py_modules=['ucinfo', 'ucenum', 'ucnorm'],
    entry_points={
        'console_scripts': [
            'ucinfo=ucinfo:_main',
            'ucenum=ucenum:_main',
            'ucnorm=ucnorm:_main',
        ],
    },
)

Traceback (most recent call last):
  File "D:\jeiea\uctools-1.3.0\test.py", line 6, in <module>
    exec(s)
  File "<string>", line 7, in <module>
UnicodeDecodeError: 'cp949' codec can't decode byte 
```
Does it mean that `getattr(tokenize, 'open', open)`is working differently than `tokenize.open`? In any currently  supported version of Python they should be equivalent isn't it? They should only differ for very old Python versions that are not used anymore, isn't it the case?
> Does it mean that getattr(tokenize, 'open', open)is working differently than tokenize.open?

Yes

>  They should only differ for very old Python versions that are not used anymore, isn't it the case?

I'm sorry but I'm not python expert.
Yeah, so according to https://docs.python.org/3/library/tokenize.html#tokenize.open, `tokenize.open` exists in any Python >= 3.2... I don't understand why the behaviour would be different... But anyway, I guess we can remove the `getattr` part and write `with tokenize.open(...)`  directly in `setuptools.build_meta`. Would you like to provide a PR for that?

The bad news is that I will only be able to cut a release once I am back in January :P.

Resolving patch:
diff --git a/setuptools/build_meta.py b/setuptools/build_meta.py
--- a/setuptools/build_meta.py
+++ b/setuptools/build_meta.py
@@ -130,7 +130,7 @@ def _open_setup_script(setup_script):
         # Supply a default setup.py
         return io.StringIO("from setuptools import setup; setup()")
 
-    return getattr(tokenize, 'open', open)(setup_script)
+    return tokenize.open(setup_script)
 
 
 @contextlib.contextmanager


Test patch: