Repository: pypa/pip

Issue:
Can not install editable pip into venv on Windows
### Description

It is not possible to install an editable version of pip into a virtual environment on Windows using Python 3.12 and pip 24.0, I have tried this multiple ways on multiple machines now

### Expected behavior

It is possible to install an editable version of pip

### pip version

24.0

### Python version

3.12.3

### OS

Windows 10

### How to Reproduce

1. Install Python 3.12.3
2. Open terminal powershell in normal user mode
3. `cd C:\temp` (or similiar)
4. `git clone https://github.com/pypa/pip`
5. `cd pip`
6. ` py -3.12 -m venv .venv`
7. `.\.venv\Scripts\activate`
8. `python -m pip install -e .`

### Output

```shell
(.venv) PS C:\temp\pip> python -m pip install -e .
Obtaining file:///C:/temp/pip
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pip
  Building editable for pip (pyproject.toml) ... done
  Created wheel for pip: filename=pip-24.1.dev0-0.editable-py3-none-any.whl size=9576 sha256=fb94e098d7a3b8a37a51ca7d80e534734358e3ba327d8de1723e8c91d43d2354
  Stored in directory: C:\Users\damia\AppData\Local\Temp\pip-ephem-wheel-cache-xomgbjc4\wheels\ae\13\10\f08f9034efc6531b45d1b33b8e18d1fcd461a050eda59dfdf1
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
  Rolling back uninstall of pip
  Moving to c:\temp\pip\.venv\lib\site-packages\pip-24.0.dist-info\
   from C:\temp\pip\.venv\Lib\site-packages\~ip-24.0.dist-info
  Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip
  Moving to c:\temp\pip\.venv\scripts\pip.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.12.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.12.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.exe
ERROR: Exception:
Traceback (most recent call last):
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 452, in run
                ^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\req_install.py", line 856, in install
    install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 725, in install_wheel
    _install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 642, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 451, in make_multiple
    filenames.extend(self.make(specification, options))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 423, in make
    return super().make(specification, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make
    self._make_script(entry, filenames, options=options)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 339, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 260, in _write_script
    launcher = self._get_launcher('t')
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 419, in _get_launcher
    raise ValueError(msg)
ValueError: Unable to find resource t64.exe in package pip._vendor.distlib
```
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).
Can not install editable pip into venv on Windows
### Description

It is not possible to install an editable version of pip into a virtual environment on Windows using Python 3.12 and pip 24.0, I have tried this multiple ways on multiple machines now

### Expected behavior

It is possible to install an editable version of pip

### pip version

24.0

### Python version

3.12.3

### OS

Windows 10

### How to Reproduce

1. Install Python 3.12.3
2. Open terminal powershell in normal user mode
3. `cd C:\temp` (or similiar)
4. `git clone https://github.com/pypa/pip`
5. `cd pip`
6. ` py -3.12 -m venv .venv`
7. `.\.venv\Scripts\activate`
8. `python -m pip install -e .`

### Output

```shell
(.venv) PS C:\temp\pip> python -m pip install -e .
Obtaining file:///C:/temp/pip
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pip
  Building editable for pip (pyproject.toml) ... done
  Created wheel for pip: filename=pip-24.1.dev0-0.editable-py3-none-any.whl size=9576 sha256=fb94e098d7a3b8a37a51ca7d80e534734358e3ba327d8de1723e8c91d43d2354
  Stored in directory: C:\Users\damia\AppData\Local\Temp\pip-ephem-wheel-cache-xomgbjc4\wheels\ae\13\10\f08f9034efc6531b45d1b33b8e18d1fcd461a050eda59dfdf1
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
  Rolling back uninstall of pip
  Moving to c:\temp\pip\.venv\lib\site-packages\pip-24.0.dist-info\
   from C:\temp\pip\.venv\Lib\site-packages\~ip-24.0.dist-info
  Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip
  Moving to c:\temp\pip\.venv\scripts\pip.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.12.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.12.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.exe
ERROR: Exception:
Traceback (most recent call last):
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 452, in run
                ^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\req_install.py", line 856, in install
    install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 725, in install_wheel
    _install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 642, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 451, in make_multiple
    filenames.extend(self.make(specification, options))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 423, in make
    return super().make(specification, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make
    self._make_script(entry, filenames, options=options)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 339, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 260, in _write_script
    launcher = self._get_launcher('t')
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 419, in _get_launcher
    raise ValueError(msg)
ValueError: Unable to find resource t64.exe in package pip._vendor.distlib
```
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
My workaround is to use `uv`:

```
(.venv) PS C:\temp\pip> python -m uv pip install -e .
   Built file:///C:/temp/pip                                                                                                                                    Built 1 editable in 1.86s
Resolved 1 package in 1ms
Installed 1 package in 4ms
 - pip==24.0
 - pip==24.1.dev0
 + pip==24.1.dev0 (from file:///C:/temp/pip)
```
This might be a duplicate of https://github.com/pypa/pip/issues/11349, but I find this 100% reproducible, it seems like it might be an edge case with pip specifically, and if not for uv I would struggle to test pip on Windows.
I can confirm this.

I note the lines

```
Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip
```

in the "rolling back" section of the output. I wonder if the problem here is that pip renames itself to `~ip` as part of the process of uninstalling the old version of pip, but then when it comes to writing the new wrappers, distlib looks for `t64.exe` relative to `__file__`, which was calculated *before* the rename.

I don't know how we'd fix this, to be honest. It seems pretty fundamental that moving the package that you're using while it's running would make it impossible for it to find its own resources. The best I can think of is for distlib to load all of its wrapper executables into memory *at import time* - or for us to patch distlib to do that, as our use case is almost unique, and I wouldn't be surprised if distlib wasn't willing to do this.

I'll see if I can work up a patch for this.
Adding a note here so I don't forget later.

The reproducer is tricky. The version of pip that's being tested is the one that was installed in non-editable mode as part of creating the venv. So when I patch pip, in order to test, I need to:

1. Install the patched version *in non-editable mode*. Installing in editable mode deosn't work, as editable mode just puts a `.pth` file in site-packages, and the pip directory never gets renamed.
2. Try to install *any version of pip* in editable mode. The version of pip you use here is irrelevant, as all that matters is that it triggers the rename of the in-use pip.

Writing a test for this is going to be fun! I suspect I'll be fighting our test infrastructure all the way 🙁
OK. I have a test that demonstrates the issue all the way back to Python 3.8. I also have a fix, but it relies on the `importlib.resources.files` API, which wasn't available in 3.8, so I need to work out how to add 3.8 support (without relying on `importlib_resources` as that would involve a new vendored dependency just for a weird edge case in a very old Python version...)

I think I know *how* to fix 3.8, but that will have to wait for tomorrow.
My workaround is to use `uv`:

```
(.venv) PS C:\temp\pip> python -m uv pip install -e .
   Built file:///C:/temp/pip                                                                                                                                    Built 1 editable in 1.86s
Resolved 1 package in 1ms
Installed 1 package in 4ms
 - pip==24.0
 - pip==24.1.dev0
 + pip==24.1.dev0 (from file:///C:/temp/pip)
```
This might be a duplicate of https://github.com/pypa/pip/issues/11349, but I find this 100% reproducible, it seems like it might be an edge case with pip specifically, and if not for uv I would struggle to test pip on Windows.
I can confirm this.

I note the lines

```
Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip
```

in the "rolling back" section of the output. I wonder if the problem here is that pip renames itself to `~ip` as part of the process of uninstalling the old version of pip, but then when it comes to writing the new wrappers, distlib looks for `t64.exe` relative to `__file__`, which was calculated *before* the rename.

I don't know how we'd fix this, to be honest. It seems pretty fundamental that moving the package that you're using while it's running would make it impossible for it to find its own resources. The best I can think of is for distlib to load all of its wrapper executables into memory *at import time* - or for us to patch distlib to do that, as our use case is almost unique, and I wouldn't be surprised if distlib wasn't willing to do this.

I'll see if I can work up a patch for this.
Adding a note here so I don't forget later.

The reproducer is tricky. The version of pip that's being tested is the one that was installed in non-editable mode as part of creating the venv. So when I patch pip, in order to test, I need to:

1. Install the patched version *in non-editable mode*. Installing in editable mode deosn't work, as editable mode just puts a `.pth` file in site-packages, and the pip directory never gets renamed.
2. Try to install *any version of pip* in editable mode. The version of pip you use here is irrelevant, as all that matters is that it triggers the rename of the in-use pip.

Writing a test for this is going to be fun! I suspect I'll be fighting our test infrastructure all the way 🙁
OK. I have a test that demonstrates the issue all the way back to Python 3.8. I also have a fix, but it relies on the `importlib.resources.files` API, which wasn't available in 3.8, so I need to work out how to add 3.8 support (without relying on `importlib_resources` as that would involve a new vendored dependency just for a weird edge case in a very old Python version...)

I think I know *how* to fix 3.8, but that will have to wait for tomorrow.

Resolving patch:
diff --git a/src/pip/_vendor/distlib/scripts.py b/src/pip/_vendor/distlib/scripts.py
--- a/src/pip/_vendor/distlib/scripts.py
+++ b/src/pip/_vendor/distlib/scripts.py
@@ -49,6 +49,24 @@
     sys.exit(%(func)s())
 '''
 
+# Pre-fetch the contents of all executable wrapper stubs.
+# This is to address https://github.com/pypa/pip/issues/12666.
+# When updating pip, we rename the old pip in place before installing the
+# new version. If we try to fetch a wrapper *after* that rename, the finder
+# machinery will be confused as the package is no longer available at the
+# location where it was imported from. So we load everything into memory in
+# advance.
+
+# Issue 31: don't hardcode an absolute package name, but
+# determine it relative to the current package
+distlib_package = __name__.rsplit('.', 1)[0]
+
+WRAPPERS = {
+    r.name: r.bytes
+    for r in finder(distlib_package).iterator("")
+    if r.name.endswith(".exe")
+}
+
 
 def enquote_executable(executable):
     if ' ' in executable:
@@ -409,15 +427,11 @@ def _get_launcher(self, kind):
                 bits = '32'
             platform_suffix = '-arm' if get_platform() == 'win-arm64' else ''
             name = '%s%s%s.exe' % (kind, bits, platform_suffix)
-            # Issue 31: don't hardcode an absolute package name, but
-            # determine it relative to the current package
-            distlib_package = __name__.rsplit('.', 1)[0]
-            resource = finder(distlib_package).find(name)
-            if not resource:
+            if name not in WRAPPERS:
                 msg = ('Unable to find resource %s in package %s' %
                        (name, distlib_package))
                 raise ValueError(msg)
-            return resource.bytes
+            return WRAPPERS[name]
 
     # Public API follows
 


Test patch:
diff --git a/tests/functional/test_self_update.py b/tests/functional/test_self_update.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/test_self_update.py
@@ -0,0 +1,22 @@
+# Check that pip can update itself correctly
+
+from typing import Any
+
+
+def test_self_update_editable(script: Any, pip_src: Any) -> None:
+    # Test that if we have an environment with pip installed in non-editable
+    # mode, that pip can safely update itself to an editable install.
+    # See https://github.com/pypa/pip/issues/12666 for details.
+
+    # Step 1. Install pip as non-editable. This is expected to succeed as
+    # the existing pip in the environment is installed in editable mode, so
+    # it only places a .pth file in the environment.
+    proc = script.pip("install", pip_src)
+    assert proc.returncode == 0
+    # Step 2. Using the pip we just installed, install pip *again*, but
+    # in editable mode. This could fail, as we'll need to uninstall the running
+    # pip in order to install the new copy, and uninstalling pip while it's
+    # running could fail. This test is specifically to ensure that doesn't
+    # happen...
+    proc = script.pip("install", "-e", pip_src)
+    assert proc.returncode == 0