Repository: pypa/setuptools

Issue:
[BUG] DistutilsMetaFinder breaks distutils imports when old setuptools is higher on path
### setuptools version

setuptools==60.0.4

### Python version

python3.8

### OS

RHEL8, FreeBSD12/13, Ubuntu 20.04

### Additional environment information

Any environment where an old setuptools (eg OS package) is higher in `sys.path`, but a newer setuptools is installed anywhere on `sys.path` (with its distutils shimming `.pth` file active)

### Description

Any import of `distutils` will bomb when a new setuptools is present, but an older setuptools is higher on the path (since the new setuptools' path shim blindly tries to import from `setuptools._distutils`, which doesn't exist in old versions of setuptools). This has probably been the case ever since the distutils shim was created, but now that it's active by default, it's a bigger problem.

The place we hit this was running `pip install` from a venv with `--system-site-packages`- pip's build isolation installs latest `setuptools`, but then shuffles `sys.path` around such that the ancient OS-packaged `setuptools` is higher on the path. The first thing that tries to import `distutils` invokes the metapath finder, which has no validation or recovery and just lets the `ModuleNotFoundError` fly.

At first blush, it seems like the metapath shim's `spec_for_distutils` should try to at least verify the co-location of `setuptools` with the shim impl (ie, that they're under the same path prefix) and/or that the `_distutils` subpackage is available via `find_spec` before it actually imports it. Mixing and matching the shim version with random setuptools versions seems like a recipe for trouble down the road...

I could probably throw together a PR for this if there's a consensus that this approach is sane.

### Expected behavior

The presence of a new-but-not-loadable setuptools on sys.path doesn't break old setuptools/distutils.


### How to Reproduce

(NB: this is a much simpler repro than the OS installed setuptools that's actually failing, but illustrates the problem more easily on any OS)
```
python3 -m venv /tmp/bang && source /tmp/bang/bin/activate  # give us a clean venv to work in
python -m pip install 'setuptools>60'  # install new setuptools to venv site-packages
python -m pip install 'setuptools<45' -t .  # install old setuptools to cwd- higher than site-packages on path for most systems
python -c 'import distutils'
```

### Output

```console
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/mdavis/setuptools/__init__.py", line 6, in <module>
    import distutils.core
  File "/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools._distutils'

```


### Code of Conduct

- [X] I agree to follow the PSF Code of Conduct


Discussion:

Resolving patch:
diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py
--- a/_distutils_hack/__init__.py
+++ b/_distutils_hack/__init__.py
@@ -86,17 +86,23 @@ def spec_for_distutils(self):
         import importlib.abc
         import importlib.util
 
-        # In cases of path manipulation during sitecustomize,
-        # Setuptools might actually not be present even though
-        # the hook has been loaded. Allow the caller to fall
-        # back to stdlib behavior. See #2980.
-        if not importlib.util.find_spec('setuptools'):
+        try:
+            mod = importlib.import_module('setuptools._distutils')
+        except Exception:
+            # There are a couple of cases where setuptools._distutils
+            # may not be present:
+            # - An older Setuptools without a local distutils is
+            #   taking precedence. Ref #2957.
+            # - Path manipulation during sitecustomize removes
+            #   setuptools from the path but only after the hook
+            #   has been loaded. Ref #2980.
+            # In either case, fall back to stdlib behavior.
             return
 
         class DistutilsLoader(importlib.abc.Loader):
 
             def create_module(self, spec):
-                return importlib.import_module('setuptools._distutils')
+                return mod
 
             def exec_module(self, module):
                 pass


Test patch: