Repository: pypa/pip

Issue:
Unable to upgrade setuptools with adopted distutils
**Environment**

* pip version: 20.2.2
* Python version: 3.8.5
* OS: macOS 10.15.6

<!-- Feel free to add more information about your environment here -->

**Description**

In pypa/setuptools#417, Setuptools is seeking to adopt distutils. With the current release (49.6), Setuptools has a vendored copy of distutils, but that copy is not activated except when `SETUPTOOLS_USE_DISTUTILS=local` in the environment.

In pypa/setuptools#2259, the team identified the need to make the distutils replacement more aggressive so that packages that `import distutils` before `import setuptools` would get the version from Setuptools (and not some unfortunate mix). This behavior was released in Setuptools 49.3, but still dormant unless `SETUPTOOLS_USE_DISTUTILS=local`.

In pypa/setuptools#2255, we're seeking to make that behavior the default (opt-out with `SETUPTOOLS_USE_DISTUTILS=stdlib`), but the test suite caught an undesirable behavior when attempting to upgrade setuptools--it seems that there's a race condition in pip when the distutils that setuptools is supplying disappears during an upgrade.

**Expected behavior**

`pip` should be able to upgrade setuptools even when setuptools supplies distutils.

**How to Reproduce**

1. Install setuptools 49.3 or later.
2. Install any other version of setuptools with `SETUPTOOLS_USE_DISTUTILS=local` set.

**Output**

```
draft $ virtualenv env --no-setuptools
created virtual environment CPython3.8.5.final.0-64 in 271ms
  creator CPython3Posix(dest=/Users/jaraco/draft/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, via=copy, app_data_dir=/Users/jaraco/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
draft $ env/bin/pip install 'setuptools<49.6'
Collecting setuptools<49.6
  Using cached setuptools-49.5.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
Successfully installed setuptools-49.5.0
draft $ env SETUPTOOLS_USE_DISTUTILS=local env/bin/pip install setuptools==49.6
Collecting setuptools==49.6
  Using cached setuptools-49.6.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.5.0
    Uninstalling setuptools-49.5.0:
      Successfully uninstalled setuptools-49.5.0
  Rolling back uninstall of setuptools
  Moving to /Users/jaraco/draft/env/bin/easy_install
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0v5mjgxu/easy_install
  Moving to /Users/jaraco/draft/env/bin/easy_install-3.8
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0v5mjgxu/easy_install-3.8
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/__pycache__/easy_install.cpython-38.pyc
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-jg6byxv1/easy_install.cpython-38.pyc
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/_distutils_hack/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~distutils_hack
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/distutils-precedence.pth
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-te4egp43/distutils-precedence.pth
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/easy_install.py
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-te4egp43/easy_install.py
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/pkg_resources/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~kg_resources
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools-49.5.0.dist-info/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools-49.5.0.dist-info
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 412, in run
    installed = install_given_reqs(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 82, in install_given_reqs
    requirement.install(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 778, in install
    scheme = get_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 185, in get_scheme
    scheme = distutils_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 381, in finalize_options
    self.set_undefined_options('build',
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'
```



Discussion:
cc @pganssle , @ncoghlan , @pfmoore 

I see the last implicated code in pip is in [locations.distutils_scheme](https://github.com/pypa/pip/blob/4312b940b4c8426fb428809556236b724aace80c/src/pip/_internal/locations.py#L95-L156).

I'm pretty sure what's happening here is pip is getting distutils from the installed version of setuptools, and when that version of setuptools is removed, the command discovery no longer works.

More importantly, the stdlib version of distutils is intended to be deprecated, so pip will need to find an alternative.

Here are some ideas:

- A short-term solution could be for pip to vendor setuptools and use `Distribution` from setuptools instead of from distutils.
- An even shorter-term solution might be to require users to set `SETUPTOOLS_USE_DISTUTILS=stdlib` when upgrading setuptools with pip (yuck) or have pip do that implicitly (requires new bugfix release(s) of pip).
- Setuptools could re-write distutils to eagerly import its commands and avoid resolving them late.
- Pip could re-write locations not to rely on distutils, possibly factoring this behavior out into a shared library (packaging?).
> * A short-term solution could be for pip to vendor setuptools and use `Distribution` from setuptools instead of from distutils.

This feels like it could be a long-term solution, it's hardly a band-aid if `pip` is relying on this behavior that will now live in `setuptools`, though you may be right that factoring out the relevant code into some shared library might be a more surgical approach.

> * Setuptools could re-write distutils to eagerly import its commands and avoid resolving them late.

I think our short-term solution should be setuptools-based if possible, because we don't have any good way to declare an incompatibility with older versions of `pip`, and if we want to avoid breaking a whole bunch of existing packages / workflows, we'd have to delay the `distutils` adoption until we could be reasonably sure that `pip`'s fix is in.

I don't totally understand the potential impact here, but I'm assuming it's just performance at the time you do `import distutils`? It might be worth attempting to profile import times and seeing how much of an impact we're actually having here.
Another possible temporary mitigation:

- The distutils import hack could detect when pip is importing distutils and in that case disable the local copy.
We've done a lot of work to decouple pip from setuptools (as a build backend). It seems like a step backward to just re-couple ourselves to get distutils functionality. Particularly as we'd end up vendoring setuptools, which vendors other stuff that we already vendor (like packaging) increasing our already-large footprint further. So -1 on vendoring setuptools from me.

What's the "official" migration strategy for removing distutils from the stdlib? Are any tools that use distutils expected to add a new dependency on setuptools? I've not seen much discussion of this. I note that packaging depends on `distutils.util.get_platform()` - will we now have packaging depend on setuptools as well? That seems like another potential issue - particularly considering the discussion at https://discuss.python.org/t/split-tags-into-a-separate-package/4965/8.

My feeling is that pip should continue to use distutils on the basis that it's in the stdlib, until it's officially removed from there. I'd be fine with moving to a non-distutils solution, and I would strongly prefer that as the solution over vendoring setuptools. But I don't feel we need to do anything urgently - until distutils is formally removed from the stdlib, I'd expect to be able to use it. If there's an issue with upgrading setuptools, that's really a matter for how the migration out of the stdlib is handled, and one for setuptools to handle smoothly in the first instance.

That's not to say I'm not willing to support the process, but we shouldn't take on significant maintenance burden to do so.
> My feeling is that pip should continue to use distutils on the basis that it's in the stdlib, until it's officially removed from there. I'd be fine with moving to a non-distutils solution, and I would strongly prefer that as the solution over vendoring setuptools. But I don't feel we need to do anything urgently - until distutils is formally removed from the stdlib, I'd expect to be able to use it. If there's an issue with upgrading setuptools, that's really a matter for how the migration out of the stdlib is handled, and one for setuptools to handle smoothly in the first instance.

I mostly disagree with this idea — people should be attempting to abandon `distutils` *now*, as soon as possible, even though it is not officially deprecated or removed yet. I would say in some ways this is more important for `pip` than other libraries because of how long people tend to continue using `pip`.

I also expect that the idea that `distutils` is "in the standard library" will get fuzzier once the `setuptools` adoption takes place. It's already fuzzy because `setuptools` monkey-patches `distutils`, but once the `setuptools` adoption is the default, if you have `setuptools` installed, `import distutils` will no longer come from the standard library.

Given that there's a lot of uncertainty about the way forward for dropping `distutils`. I'm thinking maybe it would be a good idea for there to be a PEP for the `distutils` deprecation and removal plan, possibly *before* `setuptools` adoption becomes the default (as much as it pains me to delay it more).
> people should be attempting to abandon `distutils` _now_, as soon as possible, even though it is not officially deprecated or removed yet.

I agree 100% with this conceptually, but the reality is there are systems everywhere depending on distutils, and pip gets blamed when it tries to “fix” that (#5193 comes to mind).
> I agree 100% with this conceptually, but the reality is there are systems everywhere depending on distutils, and pip gets blamed when it tries to “fix” that (#5193 comes to mind).

I'm not sure I see how this is relevant. That bug seems to be "debian's weird patching and removal of `distutils` from the standard library breaks stuff", which happens whether or not we drop `distutils` *anyway*. It seems like this (and another issue that came up in `setuptools`) is an extreme version of Debian relying on implementation details. I think we should give Debian maintainers an early head's up to start backing out their changes or getting them implemented upstream in a platform-independent manner or something.

This migration is going to be complicated and we should try to disentangle as much of the deep relationships between `distutils` and other packaging tools like `pip` as early in the process as possible, to unblock later changes and minimize the impact of future version incompatibilities ("can't run `pip<20.1 with `setuptools>50`" is not great when those are both in active use).
> people should be attempting to abandon distutils now, as soon as possible, even though it is not officially deprecated or removed yet

To clarify, what I meant was that we should handle moving off distutils based on the idea that "distutils is going to be dropped from the stdlib". So we have to deal with that, and while it may be a long way off we don't want to leave it till the last minute, conversely we shouldn't treat it as an emergency that needs an immediate band-aid solution. We should focus on getting the solution *right*, because the timescale for pip is about distutils no longer being in the stdlib, not about the intermediate stage where setuptools is trying to expose distutils while it is also still in the stdlib (sorry if I mischaracterise what's going on here, I haven't really been following the migration).

And my view is that the right long-term solution is to replace the functionality with something that doesn't need yet another large vendored library just to provide one function. So we should be looking for either a stdlib-based solution, or something that we add to a tool support library like packaging, and definitely *not* pull in a full build backend just to get distribution layout information.

And with all that in mind, we have to prioritise the resource we have and dealing with the consequences of distutils disappearing is way down the list right now.
Would it be a good idea to make migrating off `distutils` (and probbly also `pkg_resources`) a [fundable project](https://github.com/psf/fundable-packaging-improvements)? Not sure who would be interested in funding it, but this is very well-scoped and largely only requires rigorous (and tedious) work, making it a suitable funded project.
> Would it be a good idea to make migrating off distutils (and probably also pkg_resources) a fundable project?

Quite likely, in particular as we get closer to any sort of hard deadline. While there's still a relatively large amount of time, I still have some hope that someone might feel inclined to do it as volunteer work. (That's one reason I want to avoid putting undue time pressure on the work - nothing drains enthusiasm more than cries of "we need this yesterday" 🙁)
Gentle reminder that in less than a year, Python 3.12 alpha will be released without distutils.
Hmm, I think this is done, actually. Since 21.3, pip running on Python 3.10 or later will use sysconfig by default; the distutils code path is still present to support 3.9 and lower (and as a stopgap measure for Python redistributors not yet able to implement a working sysconfig). I don’t think there are any more things left for pip to do.
The reported issue applies to all supported versions of Python. Since the plan is for pip to remain reliant on distutils from the stdlib for older Pythons, then I think pip should also be responsible for ensuring that distutils is used, so that Setuptools doesn't need to try to detect if pip is the caller and keep track of which Python versions pip need special treatment. In particular, I'd like to get rid of [this special exemption](https://github.com/pypa/setuptools/blob/b2ba0e39918d58bd6c6b15093ddcdc42605040ce/_distutils_hack/__init__.py#L132-L140).
I don't think it's done either. pip [unconditionally imports distutils](https://github.com/pypa/pip/blob/7b6c5c9e7ced238375c11c33284dc26e43847812/src/pip/_internal/locations/_distutils.py#L9).

I started drafting a PR to have pip set `SETUPTOOLS_USE_DISTUTILS=stdlib`, but I realized pip can't do that early enough for it to have effect. I'm still trying to figure out a technique by which setuptools doesn't have to infer the presence of pip in order to hide distutils.
> pip unconditionally imports distutils.

Ah right, I forgot to add a conditional import. Thanks for spotting it. I’ll add some mechanism so the module is only imported if the distutils backend is used.

> Since the plan is for pip to remain reliant on distutils from the stdlib for older Pythons, then I think pip should also be responsible for ensuring that distutils is used, so that Setuptools doesn't need to try to detect if pip is the caller and keep track of which Python versions pip need special treatment.

Does setting the environment variable all that’s needed? I can incorporate this to the said conditional import mechanism to make this work. How would you recommend to reliably test that the stdlib copy is used in CI?
> Does setting the environment variable all that’s needed? I can incorporate this to the said conditional import mechanism to make this work. How would you recommend to reliably test that the stdlib copy is used in CI?

I don't think setting an environment will work. Because Setuptools uses a .pth file, the environment variable will already have been evaluated before any runtime has a chance to set it. It's only honored during `import site`.

I'm now wondering if pip should invoke something like:

```
with contextlib.suppress(ImportError):
    __import__('_distutils_hack').remove_shim()
```

(undoing the effect of the .pth file)

I'm not loving that approach, but it is in some ways the most direct. Do you have any other ideas?

> How would you recommend to reliably test that the stdlib copy is used in CI?

Probably `'setuptools' not in distutils.__spec__.origin` (or `'_distutils'`) would work.


> ```
> with contextlib.suppress(ImportError):
>     __import__('_distutils_hack').remove_shim()
> ```

I'm happy to just do this on pip's end.
FWIW, all of pip's imports of distutils now live within a single module -- https://github.com/pypa/pip/blob/9351efd78b0c0f7814b01188f6e877c5411cf104/src/pip/_internal/locations/_distutils.py

We could just add this snippet on top of that module, which should make things work as expected?
And, this is now only affecting Python <3.10, since starting with 3.10, pip's using importlib.metadata as the source of truth for loading metadata and sysconfig for the various paths of the packages.

Resolving patch:
diff --git a/src/pip/_internal/locations/_distutils.py b/src/pip/_internal/locations/_distutils.py
--- a/src/pip/_internal/locations/_distutils.py
+++ b/src/pip/_internal/locations/_distutils.py
@@ -3,6 +3,17 @@
 # The following comment should be removed at some point in the future.
 # mypy: strict-optional=False
 
+# If pip's going to use distutils, it should not be using the copy that setuptools
+# might have injected into the environment. This is done by removing the injected
+# shim, if it's injected.
+#
+# See https://github.com/pypa/pip/issues/8761 for the original discussion and
+# rationale for why this is done within pip.
+try:
+    __import__("_distutils_hack").remove_shim()
+except ImportError:
+    pass
+
 import logging
 import os
 import sys


Test patch: