Repository: python-poetry/poetry

Issue:
utils.env._full_python_path output encoding on Windows
<!--
    Hi there! Thank you for submitting a bug report!

    Before you submit your issue, please review and follow the instructions at
    https://python-poetry.org/docs/contributing/#reporting-bugs. Fully fill out the template below; insufficient
    information or bad reproduction instructions will impair the ability of others to help you.
-->

<!-- All the below information must be provided for others to understand and help with your issue. -->
- **Poetry version**: 1.3.1
- **Python version**: 3.9.13
- **OS version and name**: Windows 11 22H2
- **pyproject.toml**: [gist](https://gist.github.com/Winand/45ded1d76be2e3ed85208c06cada30ca)

<!-- All the below steps should be completed before submitting your issue. Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) stable Poetry version, installed using a recommended method.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [ ] I have consulted the [FAQ](https://python-poetry.org/docs/faq/) and [blog](https://python-poetry.org/blog/) for any relevant entries or release notes.
- [ ] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option) and have included the output below.

## Issue
I want to use poetry along with pyenv-win.
When I set `prefer-active-python = true` and call `poetry install` I get "Command cannot be found" error:
```
PS F:\python\projects\tutorial-hypermodern-python-winand> poetry install
El sistema no puede encontrar la ruta especificada.

Command 'C:\Users\Àíäðåé\.pyenv\pyenv-win\versions\3.8.10\python.exe -c "import sys; print('.'.join([str(s) for s in sys.version_info[:3]]))"' returned non-zero exit status 1.
```
Username in the output has broken encoding (Àíäðåé instead of Андрей). I've found out that poetry tries to get current interpreter using `utils.env._full_python_path` function which returns broken path in my case.

I've found two possible solutions:
* Add `universal_newlines=True` / `text=True` to `subprocess.check_output` call. That fixes only paths which can be encoded with local encoding - cp1251.
* Encode output to UTF-8 in child process:
```python
subprocess.check_output(
    list_to_shell_command(
        # [python, "-c", '"import sys; print(sys.executable)"']
        [python, "-c", '"import sys; sys.stdout.buffer.write(sys.executable.encode())"']
    ),
    shell=True,
).strip()
```



Discussion:
Path encoding is also a problem for `poetry install`.
poetry thinks that pyenv's 3.9.13 base environment is a virtual environment and installs everything there. (actually 3.8.10 is selected in pyenv)
```
PS F:\python\projects\tutorial-hypermodern-python-winand> poetry env info

Virtualenv
Python:         3.9.13
Implementation: CPython
Path:           C:\Users\Андрей\.pyenv\pyenv-win\versions\3.9.13
Executable:     C:\Users\Андрей\.pyenv\pyenv-win\versions\3.9.13\python.exe
Valid:          True

System
Platform:   win32
OS:         nt
Python:     3.8.10
Path:       C:\Users\Àíäðåé\.pyenv\pyenv-win\versions\3.9.13
Executable: python
```

If I patch `utils.env.GET_BASE_PREFIX`:
```
GET_BASE_PREFIX = """\
import sys

if hasattr(sys, "real_prefix"):
    sys.stdout.buffer.write(sys.real_prefix.encode())
elif hasattr(sys, "base_prefix"):
    sys.stdout.buffer.write(sys.base_prefix.encode())
else:
    sys.stdout.buffer.write(sys.prefix.encode())
"""
```

`poetry env info` still does not show 3.8.10 but `poetry install` creates a new virtual environment in .venv folder as expected (`prefer-active-python`, `in-project` are set)

```
PS F:\python\projects\tutorial-hypermodern-python-winand> poetry env info   

Virtualenv
Python:         3.9.13
Implementation: CPython
Path:           NA
Executable:     NA

System
Platform:   win32
OS:         nt
Python:     3.9.13
Path:       C:\Users\Андрей\.pyenv\pyenv-win\versions\3.9.13
Executable: C:\Users\Андрей\.pyenv\pyenv-win\versions\3.9.13\python.exe
```

Resolving patch:
diff --git a/src/poetry/utils/env/base_env.py b/src/poetry/utils/env/base_env.py
--- a/src/poetry/utils/env/base_env.py
+++ b/src/poetry/utils/env/base_env.py
@@ -14,8 +14,6 @@
 
 from virtualenv.seed.wheels.embed import get_embed_wheel
 
-from poetry.utils._compat import decode
-from poetry.utils._compat import encode
 from poetry.utils.env.exceptions import EnvCommandError
 from poetry.utils.env.site_packages import SitePackages
 from poetry.utils.helpers import get_real_windows_path
@@ -343,13 +341,14 @@ def _run(self, cmd: list[str], **kwargs: Any) -> str:
 
         try:
             if input_:
-                output = subprocess.run(
+                output: str = subprocess.run(
                     cmd,
                     stdout=subprocess.PIPE,
                     stderr=stderr,
-                    input=encode(input_),
+                    input=input_,
                     check=True,
                     env=env,
+                    text=True,
                     **kwargs,
                 ).stdout
             elif call:
@@ -357,11 +356,13 @@ def _run(self, cmd: list[str], **kwargs: Any) -> str:
                 subprocess.check_call(cmd, stderr=stderr, env=env, **kwargs)
                 output = ""
             else:
-                output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)
+                output = subprocess.check_output(
+                    cmd, stderr=stderr, env=env, text=True, **kwargs
+                )
         except CalledProcessError as e:
             raise EnvCommandError(e, input=input_)
 
-        return decode(output)
+        return output
 
     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
         command = self.get_command_from_bin(bin) + list(args)
diff --git a/src/poetry/utils/env/env_manager.py b/src/poetry/utils/env/env_manager.py
--- a/src/poetry/utils/env/env_manager.py
+++ b/src/poetry/utils/env/env_manager.py
@@ -23,7 +23,6 @@
 
 from poetry.toml.file import TOMLFile
 from poetry.utils._compat import WINDOWS
-from poetry.utils._compat import decode
 from poetry.utils._compat import encode
 from poetry.utils.env.exceptions import EnvCommandError
 from poetry.utils.env.exceptions import IncorrectEnvError
@@ -67,11 +66,9 @@ def _full_python_path(python: str) -> Path | None:
             return None
 
         try:
-            executable = decode(
-                subprocess.check_output(
-                    [path_python, "-c", "import sys; print(sys.executable)"],
-                ).strip()
-            )
+            executable = subprocess.check_output(
+                [path_python, "-c", "import sys; print(sys.executable)"], text=True
+            ).strip()
             return Path(executable)
 
         except CalledProcessError:
@@ -115,11 +112,9 @@ def get_python_version(
             executable = EnvManager._detect_active_python(io)
 
             if executable:
-                python_patch = decode(
-                    subprocess.check_output(
-                        [executable, "-c", GET_PYTHON_VERSION_ONELINER],
-                    ).strip()
-                )
+                python_patch = subprocess.check_output(
+                    [executable, "-c", GET_PYTHON_VERSION_ONELINER], text=True
+                ).strip()
 
                 version = ".".join(str(v) for v in python_patch.split(".")[:precision])
 
@@ -150,10 +145,8 @@ def activate(self, python: str) -> Env:
             raise PythonVersionNotFound(python)
 
         try:
-            python_version_string = decode(
-                subprocess.check_output(
-                    [python_path, "-c", GET_PYTHON_VERSION_ONELINER],
-                )
+            python_version_string = subprocess.check_output(
+                [python_path, "-c", GET_PYTHON_VERSION_ONELINER], text=True
             )
         except CalledProcessError as e:
             raise EnvCommandError(e)
@@ -334,10 +327,8 @@ def remove(self, python: str) -> Env:
         if python_path.is_file():
             # Validate env name if provided env is a full path to python
             try:
-                env_dir = decode(
-                    subprocess.check_output(
-                        [python, "-c", GET_ENV_PATH_ONELINER],
-                    )
+                env_dir = subprocess.check_output(
+                    [python, "-c", GET_ENV_PATH_ONELINER], text=True
                 ).strip("\n")
                 env_name = Path(env_dir).name
                 if not self.check_env_is_for_current_project(env_name, base_env_name):
@@ -393,10 +384,8 @@ def remove(self, python: str) -> Env:
             pass
 
         try:
-            python_version_string = decode(
-                subprocess.check_output(
-                    [python, "-c", GET_PYTHON_VERSION_ONELINER],
-                )
+            python_version_string = subprocess.check_output(
+                [python, "-c", GET_PYTHON_VERSION_ONELINER], text=True
             )
         except CalledProcessError as e:
             raise EnvCommandError(e)
@@ -485,11 +474,9 @@ def create_venv(
         python_patch = ".".join([str(v) for v in sys.version_info[:3]])
         python_minor = ".".join([str(v) for v in sys.version_info[:2]])
         if executable:
-            python_patch = decode(
-                subprocess.check_output(
-                    [executable, "-c", GET_PYTHON_VERSION_ONELINER],
-                ).strip()
-            )
+            python_patch = subprocess.check_output(
+                [executable, "-c", GET_PYTHON_VERSION_ONELINER], text=True
+            ).strip()
             python_minor = ".".join(python_patch.split(".")[:2])
 
         supported_python = self._poetry.package.python_constraint
@@ -533,12 +520,11 @@ def create_venv(
                     continue
 
                 try:
-                    python_patch = decode(
-                        subprocess.check_output(
-                            [python, "-c", GET_PYTHON_VERSION_ONELINER],
-                            stderr=subprocess.STDOUT,
-                        ).strip()
-                    )
+                    python_patch = subprocess.check_output(
+                        [python, "-c", GET_PYTHON_VERSION_ONELINER],
+                        stderr=subprocess.STDOUT,
+                        text=True,
+                    ).strip()
                 except CalledProcessError:
                     continue
 
diff --git a/src/poetry/vcs/git/system.py b/src/poetry/vcs/git/system.py
--- a/src/poetry/vcs/git/system.py
+++ b/src/poetry/vcs/git/system.py
@@ -53,15 +53,12 @@ def run(*args: Any, **kwargs: Any) -> str:
         git_command = find_git_command()
         env = os.environ.copy()
         env["GIT_TERMINAL_PROMPT"] = "0"
-        return (
-            subprocess.check_output(
-                git_command + list(args),
-                stderr=subprocess.STDOUT,
-                env=env,
-            )
-            .decode()
-            .strip()
-        )
+        return subprocess.check_output(
+            git_command + list(args),
+            stderr=subprocess.STDOUT,
+            env=env,
+            text=True,
+        ).strip()
 
     @staticmethod
     def _check_parameter(parameter: str) -> None:


Test patch:
diff --git a/tests/utils/test_env.py b/tests/utils/test_env.py
--- a/tests/utils/test_env.py
+++ b/tests/utils/test_env.py
@@ -968,7 +968,7 @@ def test_env_has_symlinks_on_nix(tmp_path: Path, tmp_venv: VirtualEnv) -> None:
 def test_run_with_input(tmp_path: Path, tmp_venv: VirtualEnv) -> None:
     result = tmp_venv.run("python", "-", input_=MINIMAL_SCRIPT)
 
-    assert result == "Minimal Output" + os.linesep
+    assert result == "Minimal Output\n"
 
 
 def test_run_with_input_non_zero_return(tmp_path: Path, tmp_venv: VirtualEnv) -> None: