Repository: python/mypy
Pull request: Fixes #16669 
This Pull Request Fixes #16669 . This pull request makes a one line change to mypy/pyinfo.py to force the stdout of the python process to use utf-8.

One can replicate this error in Windows using Python3.8 just by calling the mypy/pyinfo.py module using a slightly modified code of the  `get_search_dirs` function where the python executable doesn't match the value of sys.executable. The only modification made to this code from `get_search_dirs` is the adding of a non-ascii-path to the env parameter to subprocess. 
```python
    python_executable: str = r"<your_path_to_python_executable>\python.exe"
    non_ascii_path = r"<any_non_ascii_path>"
    env = {**dict(os.environ), "PYTHONSAFEPATH": "1"}
    env.update({"PYTHONPATH": non_ascii_path}) 
 # all code below is the same as get_search_dirs
    try:
        sys_path, site_packages = ast.literal_eval(
            subprocess.check_output(
                [python_executable, pyinfo.__file__, "getsearchdirs"],
                env=env,
                stderr=subprocess.PIPE,
            ).decode(encoding="utf-8")
        )
    except subprocess.CalledProcessError as err:
        print(err.stderr)
        print(err.stdout)
        raise
    except OSError as err:
        reason = os.strerror(err.errno)
        raise CompileError(
             [f"mypy: Invalid python executable '{python_executable}': {reason}"]
        ) from err
```

Running this code with out the fix returns this error in windows
![image](https://github.com/python/mypy/assets/46231621/5a727d72-c6a4-4edb-ba1a-9a01922df3a1)

Running pyinfo.py after forcing stdout to use utf-8 returns the expected value for sys_path and site_packages

<!--
Checklist:
- Read the [Contributing Guidelines](https://github.com/python/mypy/blob/master/CONTRIBUTING.md)
- Add tests for all changed behaviour.
- If you can't add a test, please explain why and how you verified your changes work.
- Make sure CI passes.
- Please do not force push to the PR once it has been reviewed.
-->


Problem statement:
Crash when get_search_dirs called with `sys.path` containing non-ascii characters on Windows
**Crash Report**

On non-English Windows machines, if `sys.path` contains non-ASCII characters, the `get_search_dirs` function in `modulefinder.py` will raise a `UnicodeDecodeError`. 

It seems that this issue arises because the binary string returned by `subprocess.check_output` is encoded with the encoding of the current default active code page, rather than the default UTF-8 encoding.

**Traceback**

```
Traceback (most recent call last):
  File "mypy\dmypy_server.py", line 236, in serve
  File "mypy\dmypy_server.py", line 285, in run_command
  File "mypy\dmypy_server.py", line 353, in cmd_run
  File "mypy\dmypy_server.py", line 424, in check
  File "mypy\dmypy_server.py", line 463, in initialize_fine_grained
  File "mypy\build.py", line 189, in build
  File "mypy\build.py", line 222, in _build
  File "mypy\modulefinder.py", line 832, in compute_search_paths
  File "mypy\modulefinder.py", line 752, in get_search_dirs
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 547: invalid start byte
```

**To Reproduce**

1. Create a Python package under a path that includes non-ASCII characters.
2. Perform an editable installation using either `pip -e` or using `poetry install` to update `sys.path`.
3. Execute `mypy`, causing the exception.

**Your Environment**

- Mypy version used: mypy 1.7.1
- Python version used: cp3.8
- Operating system and version: Windows 10

Hints from issue discussion:

Commit message: forced windosws to encode stdout to utf-8

Patch to review:
diff --git a/mypy/pyinfo.py b/mypy/pyinfo.py
index f262ac8b2132..4f14ffa033d4 100644
--- a/mypy/pyinfo.py
+++ b/mypy/pyinfo.py
@@ -71,6 +71,7 @@ def getsearchdirs() -> tuple[list[str], list[str]]:
 
 
 if __name__ == "__main__":
+    sys.stdout.reconfigure(encoding="utf-8") #type: ignore [attr-defined]
     if sys.argv[-1] == "getsearchdirs":
         print(repr(getsearchdirs()))
     else:


Code review comments:
diff_hunk: @@ -71,6 +71,7 @@ def getsearchdirs() -> tuple[list[str], list[str]]:
 
 
 if __name__ == "__main__":
+    sys.stdout.reconfigure(encoding="utf-8") #type: ignore [attr-defined]
path: mypy/pyinfo.py
text: @user1:
```suggestion
    sys.stdout.reconfigure(encoding="utf-8") # type: ignore[attr-defined]
```