Repository: pyenv-win/pyenv-win (language: python)

Issue:
bug: virtualenv is broken if created using a versioned shim (pip command does not work)
**Describe the bug**
When a virtualenv is created ising a shim of a particular Python version, e.g. `python3.9` or `python3.10` instead of just `python` the virtualenv is broken and `pip` command does not work.

**To Reproduce**
Steps to reproduce the behavior:
1. `pyenv set global 3.9.13`
2. `python3.9.bat -m venv broken` (or `pyenv exec python3.9 -m venv broken`)
3. `.\broken\Scripts\pip.exe` (produces no output)

**Expected behavior**
Pip command should work, as it does if the virtualenv was created using just `python`:
1. `pyenv set global 3.9.13`
2. `python -m venv working` (or `pyenv exec python -m venv working`)
3. `.\working\Scripts\pip.exe` (works fine)

**Screenshots**
N/A.

**Desktop (please complete the following information):**
 - OS: Windows11 Pro 22H2
 - Tools used: Windows Command Prompt
 - PyEnv: 3.1.1
 - Python: 3.9.13 (installed using `pyenv install`)

**Additional context**
* I have tried also with Python 3.9.12 and 3.10.8 and got the same behaviour.
* This might be related to #458.

bug: virtualenv is broken if created using a versioned shim (pip command does not work)
**Describe the bug**
When a virtualenv is created ising a shim of a particular Python version, e.g. `python3.9` or `python3.10` instead of just `python` the virtualenv is broken and `pip` command does not work.

**To Reproduce**
Steps to reproduce the behavior:
1. `pyenv set global 3.9.13`
2. `python3.9.bat -m venv broken` (or `pyenv exec python3.9 -m venv broken`)
3. `.\broken\Scripts\pip.exe` (produces no output)

**Expected behavior**
Pip command should work, as it does if the virtualenv was created using just `python`:
1. `pyenv set global 3.9.13`
2. `python -m venv working` (or `pyenv exec python -m venv working`)
3. `.\working\Scripts\pip.exe` (works fine)

**Screenshots**
N/A.

**Desktop (please complete the following information):**
 - OS: Windows11 Pro 22H2
 - Tools used: Windows Command Prompt
 - PyEnv: 3.1.1
 - Python: 3.9.13 (installed using `pyenv install`)

**Additional context**
* I have tried also with Python 3.9.12 and 3.10.8 and got the same behaviour.
* This might be related to #458.



Discussion / hints:
It looks like `python3.9.exe` that the venv module puts in the virtual environment is not like the `python.exe` it places there. The latter will look for the `python.dll` in the based on the `pyenv.cfg` file, while the former requires it to be in the same directory.

`python3.9.bat -m venv broken --symlinks` and having my Windows 10 PC in Developer mode (to allow symlinks) makes it work for me.
If I rename `python3.9.exe` and make a copy of `python.exe` that I name `python3.9.exe` it also works.

So it seems this is a venv module issue that few people complain about???
@Darsstar we have to work on this, https://github.com/pyenv-win/pyenv-win/pull/463 which contains process of creating shortcut links (similar to sym links) for exe files
Wait, what? I just tried to communicate that this is CPython's fault.

I guess we could add code that patches the venv module after every `pyenv install`. Ddo we want to do something like that instead of creating a CPython PR that fixes it and tell people who want it with current versions to install virtualenv and configure it to always use the venv backend with --symlinks?

Does it suck that it doesn't work? yes!
Is there a good way forward for us to fix `venv` and make it work on all python versions? I don't see one.
To put it differently. When using the Python `venv` module without `--symlinks` only `python.exe` and `python_d.exe` can be used to create a virtual environment with a working `pip.exe`.

So if we replace `python3<minor>.exe` and `python3.<minor>.exe` with `~\.pyenv\pyenv-win\versions\<version>\Lib\venv\scripts\nt\python.exe` during `pyenv install` it might start working?
It looks like `python3.9.exe` that the venv module puts in the virtual environment is not like the `python.exe` it places there. The latter will look for the `python.dll` in the based on the `pyenv.cfg` file, while the former requires it to be in the same directory.

`python3.9.bat -m venv broken --symlinks` and having my Windows 10 PC in Developer mode (to allow symlinks) makes it work for me.
If I rename `python3.9.exe` and make a copy of `python.exe` that I name `python3.9.exe` it also works.

So it seems this is a venv module issue that few people complain about???
@Darsstar we have to work on this, https://github.com/pyenv-win/pyenv-win/pull/463 which contains process of creating shortcut links (similar to sym links) for exe files
Wait, what? I just tried to communicate that this is CPython's fault.

I guess we could add code that patches the venv module after every `pyenv install`. Ddo we want to do something like that instead of creating a CPython PR that fixes it and tell people who want it with current versions to install virtualenv and configure it to always use the venv backend with --symlinks?

Does it suck that it doesn't work? yes!
Is there a good way forward for us to fix `venv` and make it work on all python versions? I don't see one.
To put it differently. When using the Python `venv` module without `--symlinks` only `python.exe` and `python_d.exe` can be used to create a virtual environment with a working `pip.exe`.

So if we replace `python3<minor>.exe` and `python3.<minor>.exe` with `~\.pyenv\pyenv-win\versions\<version>\Lib\venv\scripts\nt\python.exe` during `pyenv install` it might start working?

Fix patch:
diff --git a/pyenv-win/libexec/pyenv-install.vbs b/pyenv-win/libexec/pyenv-install.vbs
index 0cc66dd1..7e25c7eb 100644
--- a/pyenv-win/libexec/pyenv-install.vbs
+++ b/pyenv-win/libexec/pyenv-install.vbs
@@ -150,10 +150,11 @@ Function deepExtract(params, web)
     ' Windows try to execute pythonX.Y file (considers Y as en extension)
     ' It requires explicit .bat extension to work (pythonX.Y.bat)
     ' That's why we also use the pattern pythonXY
-    Dim version, pythonExe, pythonwExe, major, minor, majorMinor, majorDotMinor
+    Dim version, pythonExe, pythonwExe, venvlauncherExe, major, minor, majorMinor, majorDotMinor
     version = params(LV_Code)
     pythonExe = installPath &"\python.exe"
     pythonwExe = installPath &"\pythonw.exe"
+    venvlauncherExe = installPath &"\Lib\venv\scripts\nt\python.exe"
     major = Split(version,".")(0)
     minor = Split(version, ".")(1)
     majorMinor = major & minor
@@ -164,6 +165,14 @@ Function deepExtract(params, web)
     objfs.CopyFile pythonwExe, installPath &"\pythonw"& major &".exe"
     objfs.CopyFile pythonwExe, installPath &"\pythonw"& majorMinor &".exe"
     objfs.CopyFile pythonwExe, installPath &"\pythonw"& majorDotMinor &".exe"
+    If objfs.FileExists(venvlauncherExe) Then
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\python"& major &".exe"
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\python"& majorMinor &".exe"
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\python"& majorDotMinor &".exe"
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\pythonw"& major &".exe"
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\pythonw"& majorMinor &".exe"
+        objfs.CopyFile venvlauncherExe, installPath &"\Lib\venv\scripts\nt\pythonw"& majorDotMinor &".exe"
+    End If
 End Function
 
 Function unzip(installFile, installPath, zipRootDir)


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 59f13f94..200267a0 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -58,11 +58,11 @@ def pyenv_file(shell, bin_path, shell_ext):
 @pytest.fixture(scope='session', autouse=True, params=['AMD64', 'X86'],
                 ids=['PYENV_FORCE_ARCH=AMD64', 'PYENV_FORCE_ARCH=X86'])
 def arch(request):
-    arch = request.param
+    value = request.param
     if request.session.testsfailed:
-        pytest.skip(f'Skipping PYENV_FORCE_ARCH={arch} since at lease one test failed for PYENV_FORCE_ARCH=AMD64')
-    with(TemporaryEnvironment({'PYENV_FORCE_ARCH': arch})):
-        yield arch
+        pytest.skip(f'Skipping PYENV_FORCE_ARCH={value} since at lease one test failed for PYENV_FORCE_ARCH=AMD64')
+    with(TemporaryEnvironment({'PYENV_FORCE_ARCH': value})):
+        yield value
 
 
 @pytest.fixture()
@@ -105,10 +105,22 @@ def run(run_args, pyenv_path, bin_path, shims_path):
             environ[key] = str(pyenv_path)
             environ['PATH'] = environ['PATH'].replace(str(Path(os.environ[key])), environ[key])
 
-    def run(*args, env={}):
-        env = {**environ, **env}
+    def remove_python_paths(path):
+        path = Path(path)
+        if path.joinpath("python.exe").exists():
+            return False
+        if path.parent.name.lower() == "scripts" and path.parent.joinpath("python.exe").exists():
+            return False
+        return True
+
+    environ["PATH"] = os.pathsep.join(filter(remove_python_paths, environ["PATH"].split(os.pathsep)))
+    environ.pop("VIRTUAL_ENV", None)
+
+    def run(*args, **kwargs):
+        env = {**environ, **kwargs.pop("env", {})}
+        env.pop("PYTHONPATH", None)
         args = run_args + list(args)
-        return do_run(*args, env=env)
+        return do_run(*args, env=env, **kwargs)
 
     with TemporaryEnvironment({'PATH': environ['PATH']}):
         return run
diff --git a/tests/test_pyenv_feature_install.py b/tests/test_pyenv_feature_install.py
index 6e091aa9..52f2d887 100644
--- a/tests/test_pyenv_feature_install.py
+++ b/tests/test_pyenv_feature_install.py
@@ -1,4 +1,7 @@
-import subprocess
+import pytest
+
+import os
+from test_pyenv_helpers import Native
 
 
 def test_check_pyenv_install_list(pyenv):
@@ -32,3 +35,16 @@ def test_check_pyenv_install_list(pyenv):
 def test_check_pyenv_installation():
     # TODO: tracking the logs of installation and checking the folder
     pass
+
+
+@pytest.mark.parametrize("version, python", (("3.9.13", "python39"), ("3.10.11", "python310"), ("3.11.3", "python311")))
+def test_patched_venv_module(version, python, arch, pyenv, run, tmp_path):
+    if arch != os.environ["PROCESSOR_ARCHITECTURE"]:
+        pytest.skip()
+    pyenv.install(Native(version), check=True)
+    pyenv.rehash(check=True)
+    pyenv("global", Native(version), check=True)
+    pyenv.exec(python, "-m", "venv", str(tmp_path / "venv"), check=True)
+    stdout, stderr = run(str(tmp_path / "venv" / "Scripts" / "pip.exe"), "--version")
+    assert stderr == "", stdout
+
diff --git a/tests/test_pyenv_helpers.py b/tests/test_pyenv_helpers.py
index e383b5e0..25695484 100644
--- a/tests/test_pyenv_helpers.py
+++ b/tests/test_pyenv_helpers.py
@@ -52,12 +52,15 @@ def pyenv_setup(settings):
     if isinstance(local_ver, list):
         local_ver = '\n'.join(local_ver)
     src_path = Path(__file__).resolve().parents[1].joinpath('pyenv-win')
-    dirs = [r'bin', r'libexec\libs', r'shims', r'versions']
+    dirs = [r'bin', r'bin\WiX', r'libexec\libs', r'shims', r'versions']
     for d in dirs:
         os.makedirs(Path(pyenv_path, d))
     _, _, libexec_files = next(os.walk(src_path.joinpath('libexec')))
     for f in libexec_files:
         shutil.copy(Path(src_path, 'libexec', f), Path(pyenv_path, 'libexec', f))
+    _, _, libexec_files = next(os.walk(src_path.joinpath(r'bin\WiX')))
+    for f in libexec_files:
+        shutil.copy(Path(src_path, r'bin\WiX', f), Path(pyenv_path, r'bin\WiX', f))
     files = [r'.versions_cache.xml',
              r'..\.version',
              r'bin\pyenv.bat',
@@ -114,9 +117,9 @@ def local_python_versions(path):
         return f.read().strip()
 
 
-def do_run(*args, env={}):
+def do_run(*args, **kwargs):
     args = list(args)
-    result = subprocess.run(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+    result = subprocess.run(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE,  **kwargs)
     stderr = str(result.stderr, "utf-8").strip()
     # \x0c: generated by cls in cmd AutoRun
     stdout = str(result.stdout, "utf-8").rpartition('\x0c')[2].strip("\r\n")
@@ -137,6 +140,8 @@ def __new__(cls, content):
         ver = content
         if os.environ['PYENV_FORCE_ARCH'] == 'X86':
             content = content + '-win32'
+        if os.environ['PYENV_FORCE_ARCH'] == 'ARM64':
+            content = content + '-arm64'
         self = super().__new__(cls, content)
         self.version = ver
         return self