Repository: pypa/setuptools

Issue:
AssertionError no _distutils when running get-pip with Setuptools and USE_DISTUTILS=local
Somehow related: Since `setuptools` v60 at least on Debian running `get-pip.py` over an existing `pip` + `setuptools` fails, when the Debian `python3-distutils` package is installed, e.g. as dependency of the Python development headers package:
```
/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/tmp/DietPi-Software/./get-pip.py", line 27081, in <module>
    main()
  File "/tmp/DietPi-Software/./get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/tmp/DietPi-Software/./get-pip.py", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/tmp/DietPi-Software/./get-pip.py", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/usr/local/lib/python3.9/dist-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py", line 73, in do_override
    ensure_local_distutils()
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/lib/python3.9/distutils/core.py
```
This `_distutils_hack` comes from `get-pip.py`. Looks like removing any use and detangling `distutils` from `pip` in favour of `setuptools` would solve both issues.

_Originally posted by @MichaIng in https://github.com/pypa/pip/issues/10742#issuecomment-1003741034_


Discussion:
Setuptools 60 intentionally introduced its local copy of distutils as default. As an escape hatch, you should be able to set the environment variable SETUPTOOLS_USE_DISTUTILS=stdlib while running get-pip to bypass that behavior.

I know the distutils hack has an exclusion for when running under pip. That was, however, to account for pip's use of distutils. In this case, get-pip is specifically importing setuptools. Importing setuptools causes the more aggressive form of requiring distutils, where if the local copy of distutils isn't found, it fails with the reported error.

So the question is - what is it about this environment that triggers this behavior? In particular, how is it that [this code](https://github.com/pypa/setuptools/blob/80045ce11c74d1b50892d15feef82415e6badfe9/_distutils_hack/__init__.py#L51-L61) doesn't ensure that any existing distutils is unloaded, then loaded from Setuptools?

Can someone create a set of docker steps that will replicate the issue?
I was able to replicate the error with this Dockerfile:

```dockerfile
FROM ubuntu:focal
RUN apt update
RUN apt upgrade -y
RUN apt install -y python3-pip python3-distutils wget
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN python3 -m pip install -U setuptools
CMD python3 get-pip.py
```
I did a bit of debugging and it seems that the `spec_for_pip` had gotten invoked, causing the DistutilsMetaFinder to disable the distutils hook:

```
draft $ docker run -it @$(docker build -q .) python3 -m pdb get-pip.py
> /get-pip.py(23)<module>()
-> import sys
(Pdb) b _distutils_hack/__init__.py:57
Breakpoint 1 at /usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:57
(Pdb) c
/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
> /usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py(57)ensure_local_distutils()
-> importlib.import_module('distutils')
(Pdb) 'distutils' in sys.modules
False
(Pdb) sys.meta_path[0]
<_distutils_hack.DistutilsMetaFinder object at 0x7fad398a9370>
(Pdb) mf = sys.meta_path[0]
(Pdb) mf.pip_imported_during_build()
False
(Pdb) mf.spec_for_distutils()
(Pdb) mf.spec_for_distutils
<function DistutilsMetaFinder.spec_for_pip.<locals>.<lambda> at 0x7fad39199550>
```

The hack makes the assumption that if pip is imported and it wasn't imported as part of a `setup.py` invocation, then it's a pip invocation and needs to avoid making the local version of distutils available.

But setuptools makes the assertion that the hack must be available (if SETUPTOOLS_USE_DISTUTILS=local).

The question is - what is it about get-pip that (a) imports pip and then (b) imports setuptools?
I see it's [these two lines](https://github.com/pypa/get-pip/blob/6043201e89b1193f7bf050979436eff6a7337247/templates/default.py#L119-L120) where `import pip` and then `import setuptools` happens.

I no longer think this issue is specific to Debian or distutils.
Indeed, the same issue occurs on my mac. All it takes is an invocation of get-pip on an existing install of `Setuptools>=60` (or `Setuptools>=50` with SETUPTOOLS_USE_DISTUTILS='local').

<details>

```
draft $ python -m venv venv
draft $ venv/bin/pip install -U setuptools
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (58.1.0)
Collecting setuptools
  Using cached setuptools-60.2.0-py3-none-any.whl (953 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
Successfully installed setuptools-60.2.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/jaraco/draft/venv/bin/python -m pip install --upgrade pip' command.
draft $ venv/bin/python get-pip.py
/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Users/jaraco/draft/get-pip.py", line 27081, in <module>
    main()
  File "/Users/jaraco/draft/get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/Users/jaraco/draft/get-pip.py", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/Users/jaraco/draft/get-pip.py", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 73, in do_override
    ensure_local_distutils()
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/core.py
draft $
```

</details>
In the aforementioned commit, I've devised another exception that bypasses the issue (and solves the problem at least for me in this environment). I'm a little concerned that the approach is unsound, because it's essentially bypassing the _other_ exception that was created for pip due to pypa/pip#8761.

I suspect a better approach would be for get-pip not to "import setuptools" to determine if it's present, but merely to "find" it or check metadata for it.
On further consideration, the conflict with the other exception may not be a real concern, because that exception was to support the pip _upgrading_ setuptools case. Since get-pip is specifically not upgrading Setuptools, the concern may be moot. So my temptation is to apply the drafted fix while get-pip explores the reported issue.
it looks like the release job failed  https://github.com/pypa/setuptools/actions/runs/1650359780

so while the tag is available on github its not published to pypi currently
Thanks for letting me know. I've identified and fixed the issue and a new release should be going out.
In https://github.com/pypa/pip/issues/10775#issuecomment-1009375819, I summarize why I think the fix for this issue was inadequate:

> ...get-pip's import of setuptools happens before get-pip's import of pip, but the hook for the hack is on the import of pip.

I believe instead, the distutils hack should (a) detect get-pip, (b) detect "import setuptools", and in that case, satisfy "import setuptools" without actually importing anything and bypassing the distutils replacement.

Resolving patch:
diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py
--- a/_distutils_hack/__init__.py
+++ b/_distutils_hack/__init__.py
@@ -136,11 +136,36 @@ def spec_for_pip(self):
         """
         if self.pip_imported_during_build():
             return
-        if self.is_get_pip():
-            return
         clear_distutils()
         self.spec_for_distutils = lambda: None
 
+    def spec_for_setuptools(self):
+        """
+        get-pip imports setuptools solely for the purpose of
+        determining if it's installed. In this case, provide
+        a stubbed spec to represent setuptools being present
+        without invoking any behavior.
+
+        Workaround for pypa/get-pip#137.
+        """
+        if not self.is_get_pip():
+            return
+
+        import importlib
+
+        class StubbedLoader(importlib.abc.Loader):
+
+            def create_module(self, spec):
+                import types
+                return types.ModuleType('setuptools')
+
+            def exec_module(self, module):
+                pass
+
+        return importlib.util.spec_from_loader(
+            'setuptools', StubbedLoader(),
+        )
+
     @classmethod
     def pip_imported_during_build(cls):
         """


Test patch: