Repository: pypa/pip

Issue:
upgrade from 22.2 to 22.2.1. fails with ImportError for name '_distutils'
### Description

When trying to upgrade from 22.2 to 22.2.1, the install fails because of an ImportError.
The setuptools version is 63.2.0.

```
Traceback (most recent call last):
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py)
```


### Expected behavior

Pip should be upgraded from 22.2 to 22.2.1

### pip version

22.2

### Python version

3.9.13

### OS

macOS 10.15.7

### How to Reproduce

With version 22.2 user-installed:

```
python3 -m pip install --user --upgrade pip
```


### Output

```shell
% python3 -m pip install --user --upgrade pip    
Requirement already satisfied: pip in ./Library/Python/3.9/lib/python/site-packages (22.2)
Collecting pip
  Using cached pip-22.2.1-py3-none-any.whl (2.0 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2
    Uninstalling pip-22.2:
      Successfully uninstalled pip-22.2
  Rolling back uninstall of pip
  Moving to /Users/mm/Library/Python/3.9/bin/pip
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip
  Moving to /Users/mm/Library/Python/3.9/bin/pip3
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip3
  Moving to /Users/mm/Library/Python/3.9/bin/pip3.9
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip3.9
  Moving to /Users/mm/Library/Python/3.9/lib/python/site-packages/pip-22.2.dist-info/
   from /Users/mm/Library/Python/3.9/lib/python/site-packages/~ip-22.2.dist-info
  Moving to /Users/mm/Library/Python/3.9/lib/python/site-packages/pip/
   from /Users/mm/Library/Python/3.9/lib/python/site-packages/~ip
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py)
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
Does installing anything else (not pip) work?
Yes. A test install of numpy, pandas, seaborn works.
Interesting. I wonder if for all our effort to make the distutils import lazy, we actually made it _too_ lazy that it’s not imported in time, before the file is moved out of place.
Same problem on Windows:
```
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\cli\req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\commands\install.py", line 461, in run
    installed = install_given_reqs(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\req\req_install.py", line 752, in install
    scheme = get_scheme(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\locations\__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\locations\__init__.py)
```
Can someone provide a reproducer for this ?
@sbidoul:

1. Spin up an instance of Ubuntu 20.04 on your favorite VPS provider, or see below.

2. The VPS provider I chose does an immediate update/upgrade, I don't know if this is necessary, but to keep things consistent:

        apt update
        apt upgrade
        reboot

3. Run:

        pip3 install --upgrade --user pip==22.2
        ~/.local/bin/pip3 install --upgrade --user pip

Here's a VPS you can test with, just skip straight to step 3 on it:

IP: 144.202.6.36
username: pip

Please give me an SSH pubkey for the account above, or email me for a password.
@gurnec thanks, I could reproduce now with your procedure.
@sbidoul You may have already figured this out, but upgrading pip from 22.1.2 to 22.2.1 works fine, it's only 22.2 → 22.2.1 which generates an `ImportError`.

Resolving patch:
diff --git a/src/pip/_internal/locations/__init__.py b/src/pip/_internal/locations/__init__.py
--- a/src/pip/_internal/locations/__init__.py
+++ b/src/pip/_internal/locations/__init__.py
@@ -60,6 +60,12 @@ def _should_use_sysconfig() -> bool:
 
 _USE_SYSCONFIG = _should_use_sysconfig()
 
+if not _USE_SYSCONFIG:
+    # Import distutils lazily to avoid deprecation warnings,
+    # but import it soon enough that it is in memory and available during
+    # a pip reinstall.
+    from . import _distutils
+
 # Be noisy about incompatibilities if this platforms "should" be using
 # sysconfig, but is explicitly opting out and using distutils instead.
 if _USE_SYSCONFIG_DEFAULT and not _USE_SYSCONFIG:
@@ -241,8 +247,6 @@ def get_scheme(
     if _USE_SYSCONFIG:
         return new
 
-    from . import _distutils
-
     old = _distutils.get_scheme(
         dist_name,
         user=user,
@@ -407,8 +411,6 @@ def get_bin_prefix() -> str:
     if _USE_SYSCONFIG:
         return new
 
-    from . import _distutils
-
     old = _distutils.get_bin_prefix()
     if _warn_if_mismatch(pathlib.Path(old), pathlib.Path(new), key="bin_prefix"):
         _log_context()
@@ -442,8 +444,6 @@ def get_purelib() -> str:
     if _USE_SYSCONFIG:
         return new
 
-    from . import _distutils
-
     old = _distutils.get_purelib()
     if _looks_like_deb_system_dist_packages(old):
         return old
@@ -488,8 +488,6 @@ def get_prefixed_libs(prefix: str) -> List[str]:
     if _USE_SYSCONFIG:
         return _deduplicated(new_pure, new_plat)
 
-    from . import _distutils
-
     old_pure, old_plat = _distutils.get_prefixed_libs(prefix)
     old_lib_paths = _deduplicated(old_pure, old_plat)
 


Test patch: