Repository: pypa/setuptools

Issue:
[BUG] `pip install -e .` on Windows creates broken .pth files
### setuptools version

67.8.0

### Python version

3.8.10

### OS

Windows 10

### Additional environment information

Tried with newer Python versions, same problem.

### Description

I tried to install my PEP571 package in editable mode and it works, but since my installation path is not ASCII, the .pth file is created as UTF-8 and Python just ignores it.

### Expected behavior

Using `pip install -e` should create non-UTF-8 .pth files.

### How to Reproduce

1. Start with an empty folder

2. Setup the venv and the project:
```
> git clone https://support.powertools-tech.com/PowertoolsTech/tkmilan.git -b setup/pyproject tkmilan_ç_utf8
Cloning into 'tkmilan_ç_utf8'...
remote: Enumerating objects: 2520, done.
remote: Counting objects: 100% (1011/1011), done.
remote: Compressing objects: 100% (624/624), done.
remote: Total 2520 (delta 557), reused 484 (delta 236), pack-reused 1509
Receiving objects: 100% (2520/2520), 995.17 KiB | 21.63 MiB/s, done.
Resolving deltas: 100% (1445/1445), done.

> py -3.8 -m venv venv

> venv\Scripts\activate.bat

> pip install -U pip setuptools wheel
Requirement already satisfied: pip in %TEMP%\setuptools_test\venv\lib\site-packages (21.1.1)
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in %TEMP%\setuptools_test\venv\lib\site-packages (56.0.0)
Collecting setuptools
  Using cached setuptools-67.8.0-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-23.1.2 setuptools-67.8.0 wheel-0.40.0

>pip install -e tkmilan_ç_utf8
Obtaining file:///%TEMP%/setuptools_test/tkmilan_%C3%A7_utf8
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: tkmilan
  Building editable for tkmilan (pyproject.toml) ... done
  Created wheel for tkmilan: filename=tkmilan-0.31.dev5+g89be321-0.editable-py3-none-any.whl size=7606 sha256=af651bdbad009100b19fe77bf9e920808d738d260e908894f46d95b1e22d3977
  Stored in directory: %TEMP%\pip-ephem-wheel-cache-hzsqdlvi\wheels\0b\a3\d4\32ec164451dad8b10051ddc54df87ea215cc03fcb18d742dfa
Successfully built tkmilan
Installing collected packages: tkmilan
Successfully installed tkmilan-0.31.dev5+g89be321
```

3. Try and run the showcase, notice it fails:
```
> tkmilan-showcase
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "%TEMP%\setuptools_test\venv\Scripts\tkmilan-showcase.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'tkmilan'
```

4. Edit `venv\Lib\site-packages\__editable__.tkmilan-0.31.dev5+g89be321.pth` on Notepad and save as ANSI format.

5. Try and run the showcase, notice it works:
```
> tkmilan-showcase
```


### Output

See above


Discussion:
Hi @simaoafonso-pwt, could you please post here what would be the contents of the `.pth` file?
Any chance this is a bug with Python itself? Python3 is supposed to have support for UTF-8 isn't it?
> Hi @simaoafonso-pwt, could you please post here what would be the contents of the .pth file?

It's a single line the correct path (`%TEMP%/setuptools_test/tkmilan_ç_utf8/src`), but as UTF-8 text. As mentioned, if I open the file in Notepad and save it as ANSI text, python picks it up correctly.

> Any chance this is a bug with Python itself? Python3 is supposed to have support for UTF-8 isn't it?

I think paths on Windows are a special case, they are not UTF-8.
I guess Python itself reads these .pth files using the default locale. So the producer of .pth files should write it using the default locale too, which I think it's `setuptools`, correct me if I'm wrong.
Does anything get added to `sys.path` at all, or is it all ignored?
It's not added to `sys.path`. I tried adding the correct path manually and the package is found.

As UTF-8:

```python
> python
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Program Files\\Python38\\python38.zip', 'C:\\Program Files\\Python38\\DLLs', 'C:\\Program Files\\Python38\\lib', 'C:\\Program Files\\Python38', '%CD%', '%CD%\\lib\\site-packages']
>>> import tkmilan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tkmilan'
>>> sys.path.append(r"%HOME%\tkmilan\src")
>>> import tkmilan
>>>
```

As ANSI:

```python
>python
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', 'C:\\Program Files\\Python38\\python38.zip', 'C:\\Program Files\\Python38\\DLLs', 'C:\\Program Files\\Python38\\lib', 'C:\\Program Files\\Python38', '%CD%', '%CD%\\lib\\site-packages', '%HOME%\\tkmilan\\src']
```
@simaoafonso-pwt, I am trying to create a reproducer with a windows container to ask the Python maintainers for advice about this matter.

When I try to change the encoding to ASCII I am no longer able to use the `ç` character in the terminal (but I see that you are able to do that in your example). What is the encoding your system is using?

In my machine I have `$OutputEncoding` set to `US-ASCII`, [System.Text.Encoding]::Default set to `iso-8859-1` and `chcp` configured to `437` (I got these values using PowerShell).
When I run a windows container (`python:3.10-windowsservercore-1809`), I get `$OutputEncoding` => `US-ASCII`, `[System.Text.Encoding]::Default` => `UTF-8` and `chcp` => 0.

Could you please tell me what are the values in your system?


Not sure if this answers your question, I'm not familiar with all these PowerShell commands.

```
PS %HOME%> Get-SystemLanguage
pt-PT

PS %HOME%> [System.Text.Encoding]::Default

IsSingleByte      : True
BodyName          : iso-8859-1
EncodingName      : Western European (Windows)
HeaderName        : Windows-1252
WebName           : Windows-1252
WindowsCodePage   : 1252
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.InternalEncoderBestFitFallback
DecoderFallback   : System.Text.InternalDecoderBestFitFallback
IsReadOnly        : True
CodePage          : 1252

PS %HOME%> echo $OutputEncoding

IsSingleByte      : True
BodyName          : us-ascii
EncodingName      : US-ASCII
HeaderName        : us-ascii
WebName           : us-ascii
WindowsCodePage   : 1252
IsBrowserDisplay  : False
IsBrowserSave     : False
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 20127
PS %HOME%> chcp
Active code page: 850
```

![2023-05-30T15:16:53+01:00](https://github.com/pypa/setuptools/assets/49308688/dbe225ec-0e10-48c5-9b63-197b422f3a05)

Thank you very much Simão.

I did not manage to reproduce this with a Windows container (for some reason they seem to always be UTF-8, and I did not find out how to change that), however I managed to run this in a Windows machine I have access to, with the following steps:

1. Open the run prompt with `Windows+R`
2. Run `intl.cpl`
3. Go to the `Administrative` tab, then `Change system locale`
4. Make sure the `UTF-8` option is not used and that the appropriate encoding is used (compatible with `iso8859-1`)
5. Using PowerShell check if the `[System.Text.Encoding]::Default` is compatible with `iso8859-1`/ Windows 1252
   (if not it might be necessary to change the system locale again and/or restart)
6. Create virtual environment using PowerShell:
   ```powershell
   # rm -R -Force $env:TEMP\test
   mkdir $env:TEMP\test
   cd $env:TEMP\test
   python3 -m venv .venv
   ```
7. In the same PowerShell terminal, simulate an UTF-8 `.pth` file:
   ```powershell
   new-item create_files.py
   @"
   from pathlib import Path
   proj = Path("tkmilan_ç_utf8")
   (proj / "src/mypkg").mkdir(parents=True, exist_ok=True)
   (proj / "src/mypkg/__init__.py").write_text("print('hello world')", encoding="utf-8")
   Path(".venv/Lib/site-packages/custom.pth").write_text(str((proj / "src").resolve()), encoding="utf-8")
   "@ | add-content -Encoding utf8 -Path create_files.py
   .venv\Scripts\python create_files.py
   ```
8. Verify that the UTF-8 entry in the `custom.pth` is not added to `sys.path` when Python runs:
   ```powershell
   .venv\Scripts\python -c 'import sys; print(sys.path)'
   ```
9. Modify the `custom.pth` file to the iso8859-1 encoding:
   ```powershell
   rm create_files.py
   new-item create_files.py
   @"
   from pathlib import Path
   proj = Path("tkmilan_ç_utf8")
   Path(".venv/Lib/site-packages/custom.pth").write_text(str((proj / "src").resolve()), encoding='iso8859-1')
   "@ | add-content -Encoding utf8 -Path create_files.py
   .venv\Scripts\python
   ```
10. Verify that the ISO 8859 entry is `custom.pth` is added to `sys.path` when Python runs:
    ```powershell
    .venv\Scripts\python -c 'import sys; print(sys.path)'
    ```
@jaraco do you have any advice here?

In the `editable_wheel` command, we write the `.pth` file to the wheel using `writestr` (which accepts a `bytes` object. To create this `bytes` object we currently use `utf-8`:

https://github.com/pypa/setuptools/blob/1aed0ff9ba8091f127cd3638bd26661fa0ff8371/setuptools/command/editable_wheel.py#L404-L405

We could change the `utf-8` to `locale.getpreferredencoding()`, however I am in doubt about this approach because of the following:
1. [`locale.getpreferredencoding()`](https://docs.python.org/3/library/locale.html#locale.getpreferredencoding) has side effects and is not thread safe (it can be invoked with `do_setlocale=False`, but then, the docs seem to indicate it may fail) -- Not sure if side-effects and thread-safety is something we have to worry about in setuptools.
2. Python3 uses `UTF-8` by default for source files... `.pth` files are kind of source files, they can even contain Python code, it would seem to me that they should be treated as `UTF-8`.
   There is an open issue in the CPython repository about it https://github.com/python/cpython/issues/77102, but I don't think there is a clear position from the core maintainers if the issue is considered a pending bug, a wontfix, or not a bug at all...
3. I am not sure what will happen in other versions of Python/Windows... What if we change to `locale.getpreferredencoding()`, any chance we end up having the opposite problem later?
   Additionally, if we produce a `.pth` file with ISO 8859 encoding and later the user sets the `PYTHONUTF8` environment variable or use `python -X utf8`, then the `.pth` file will not work.

Resolving patch:
diff --git a/setuptools/command/editable_wheel.py b/setuptools/command/editable_wheel.py
--- a/setuptools/command/editable_wheel.py
+++ b/setuptools/command/editable_wheel.py
@@ -11,6 +11,7 @@
 """
 
 import logging
+import io
 import os
 import shutil
 import sys
@@ -401,7 +402,7 @@ def __init__(self, dist: Distribution, name: str, path_entries: List[Path]):
 
     def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]):
         entries = "\n".join((str(p.resolve()) for p in self.path_entries))
-        contents = bytes(f"{entries}\n", "utf-8")
+        contents = _encode_pth(f"{entries}\n")
         wheel.writestr(f"__editable__.{self.name}.pth", contents)
 
     def __enter__(self):
@@ -509,7 +510,7 @@ def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]
         content = bytes(_finder_template(name, roots, namespaces_), "utf-8")
         wheel.writestr(f"{finder}.py", content)
 
-        content = bytes(f"import {finder}; {finder}.install()", "utf-8")
+        content = _encode_pth(f"import {finder}; {finder}.install()")
         wheel.writestr(f"__editable__.{self.name}.pth", content)
 
     def __enter__(self):
@@ -525,6 +526,24 @@ def __exit__(self, _exc_type, _exc_value, _traceback):
         InformationOnly.emit("Editable installation.", msg)
 
 
+def _encode_pth(content: str) -> bytes:
+    """.pth files are always read with 'locale' encoding, the recommendation
+    from the cpython core developers is to write them as ``open(path, "w")``
+    and ignore warnings (see python/cpython#77102, pypa/setuptools#3937).
+    This function tries to simulate this behaviour without having to create an
+    actual file, in a way that supports a range of active Python versions.
+    (There seems to be some variety in the way different version of Python handle
+    ``encoding=None``, not all of them use ``locale.getpreferredencoding(False)``).
+    """
+    encoding = "locale" if sys.version_info >= (3, 10) else None
+    with io.BytesIO() as buffer:
+        wrapper = io.TextIOWrapper(buffer, encoding)
+        wrapper.write(content)
+        wrapper.flush()
+        buffer.seek(0)
+        return buffer.read()
+
+
 def _can_symlink_files(base_dir: Path) -> bool:
     with TemporaryDirectory(dir=str(base_dir.resolve())) as tmp:
         path1, path2 = Path(tmp, "file1.txt"), Path(tmp, "file2.txt")


Test patch:
diff --git a/setuptools/tests/test_editable_install.py b/setuptools/tests/test_editable_install.py
--- a/setuptools/tests/test_editable_install.py
+++ b/setuptools/tests/test_editable_install.py
@@ -22,6 +22,7 @@
 from setuptools.command.editable_wheel import (
     _DebuggingTips,
     _LinkTree,
+    _encode_pth,
     _find_virtual_namespaces,
     _find_namespaces,
     _find_package_roots,
@@ -1025,6 +1026,13 @@ def test_debugging_tips(tmpdir_cwd, monkeypatch):
         cmd.run()
 
 
+@pytest.mark.filterwarnings("error")
+def test_encode_pth():
+    """Ensure _encode_pth function does not produce encoding warnings"""
+    content = _encode_pth("tkmilan_ç_utf8")  # no warnings (would be turned into errors)
+    assert isinstance(content, bytes)
+
+
 def install_project(name, venv, tmp_path, files, *opts):
     project = tmp_path / name
     project.mkdir()