Repository: pypa/setuptools

Issue:
[Forward compatibility][BUG] pkg_resources.extern.VendorImporter lacks find_spec() method
### setuptools version

54.1.2 and main

### Python version

3.10.0a7

### OS

All

### Additional environment information

When running tests for setuptools_scm.

### Description

We get:

```python-traceback
<frozen importlib._bootstrap>:933: in _find_spec
    ???
E   AttributeError: 'VendorImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:
/usr/lib/python3.10/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/lib/python3.10/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/lib/python3.10/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
/usr/lib/python3.10/site-packages/_pytest/doctest.py:129: in pytest_collect_file
    elif _is_doctest(config, path, parent):
/usr/lib/python3.10/site-packages/_pytest/doctest.py:147: in _is_doctest
    if path.check(fnmatch=glob):
/usr/lib/python3.10/site-packages/py/_path/local.py:387: in check
    return super(LocalPath, self).check(**kw)
/usr/lib/python3.10/site-packages/py/_path/common.py:241: in check
    return self.Checkers(self)._evaluate(kw)
/usr/lib/python3.10/site-packages/py/_path/common.py:108: in _evaluate
    if py.code.getrawcode(meth).co_argcount > 1:
/usr/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py:152: in __makeattr
    result = importobj(modpath, attrname)
/usr/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py:72: in importobj
    module = __import__(modpath, None, None, ['__doc__'])
/usr/lib/python3.10/site-packages/py/_code/code.py:7: in <module>
    reprlib = py.builtin._tryimport('repr', 'reprlib')
/usr/lib/python3.10/site-packages/py/_builtin.py:144: in _tryimport
    __import__(name)
<frozen importlib._bootstrap>:1021: in _find_and_load
    ???
<frozen importlib._bootstrap>:996: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _find_spec
    ???
<frozen importlib._bootstrap>:908: in _find_spec_legacy
    ???
E   ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
```

The traceback is not very helpful, but the VendorImporter is from pkg_resourecs.

Python added a warning: https://bugs.python.org/issue42134 but other packages treat it as error.

### Expected behavior

No ImportWarning.

### How to Reproduce

1. Get Python 3.10.0a7 and tox (e.g. on Fedora via `$ sudo dnf --enablerepo=updates-testing install python3.10 tox`)
2. git clone https://github.com/pypa/setuptools_scm.git and cd in
3. Run `tox -e py310-test`

### Output

```console
py310-test develop-inst-nodeps: .../setuptools_scm
py310-test installed: attrs==20.3.0,iniconfig==1.1.1,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.3,-e git+https://github.com/pypa/setuptools_scm.git@abb67b15985f380d8cf4451b9f2ef3dd11cb8a91#egg=setuptools_scm,toml==0.10.2
py310-test run-test-pre: PYTHONHASHSEED='1426720794'
py310-test run-test: commands[0] | pytest
============================= test session starts ==============================
platform linux -- Python 3.10.0a7, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py310-test/.pytest_cache
setuptools version 53.0.0 from '.../setuptools_scm/.tox/py310-test/lib/python3.10/site-packages/setuptools/__init__.py'
setuptools_scm version 6.0.1 from '.../setuptools_scm/src/setuptools_scm/__init__.py'
rootdir: .../setuptools_scm, configfile: tox.ini, testpaths: testing
collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
<frozen importlib._bootstrap>:933: in _find_spec
    ???
E   AttributeError: 'VendorImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:
.tox/py310-test/lib/python3.10/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
.tox/py310-test/lib/python3.10/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
.tox/py310-test/lib/python3.10/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
.tox/py310-test/lib/python3.10/site-packages/_pytest/doctest.py:129: in pytest_collect_file
    elif _is_doctest(config, path, parent):
.tox/py310-test/lib/python3.10/site-packages/_pytest/doctest.py:147: in _is_doctest
    if path.check(fnmatch=glob):
.tox/py310-test/lib/python3.10/site-packages/py/_path/local.py:387: in check
    return super(LocalPath, self).check(**kw)
.tox/py310-test/lib/python3.10/site-packages/py/_path/common.py:241: in check
    return self.Checkers(self)._evaluate(kw)
.tox/py310-test/lib/python3.10/site-packages/py/_path/common.py:108: in _evaluate
    if py.code.getrawcode(meth).co_argcount > 1:
.tox/py310-test/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py:152: in __makeattr
    result = importobj(modpath, attrname)
.tox/py310-test/lib/python3.10/site-packages/py/_vendored_packages/apipkg/__init__.py:72: in importobj
    module = __import__(modpath, None, None, ['__doc__'])
.tox/py310-test/lib/python3.10/site-packages/py/_code/code.py:7: in <module>
    reprlib = py.builtin._tryimport('repr', 'reprlib')
.tox/py310-test/lib/python3.10/site-packages/py/_builtin.py:144: in _tryimport
    __import__(name)
<frozen importlib._bootstrap>:1021: in _find_and_load
    ???
<frozen importlib._bootstrap>:996: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _find_spec
    ???
<frozen importlib._bootstrap>:908: in _find_spec_legacy
    ???
E   ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
=========================== short test summary info ============================
ERROR  - ImportWarning: VendorImporter.find_spec() not found; falling back to...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.16s ===============================
ERROR: InvocationError for command .../setuptools_scm/.tox/py310-test/bin/pytest (exited with code 2)
___________________________________ summary ____________________________________
ERROR:   py310-test: commands failed
```


### Code of Conduct

I agree to follow the PSF Code of Conduct


Discussion:
https://github.com/pypa/setuptools/blob/ee6a5ffd1b3ec686339e00713542a530dbb65bc8/pkg_resources/extern/__init__.py#L23-L33

https://github.com/pypa/setuptools/blob/ee6a5ffd1b3ec686339e00713542a530dbb65bc8/setuptools/extern/__init__.py#L23-L33
@hroncok pro tip: you can mark code blocks that have tracebacks with `python-traceback` and they'll be more colorful.
Ref: https://bugs.python.org/issue43540
In that bug, @vstinner asks @brettcannon how to fix this issue — I think the answer would be more useful right here on GitHub rather than on bpo so I tag you both in case you want to comment here :)
@hroncok @jaraco a more minimalist reproducer is
```python-traceback
$ python3.10 -W error -c 'import pkg_resources'                 
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 939, in _find_spec
AttributeError: 'VendorImporter' object has no attribute 'find_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/pkg_resources/__init__.py", line 74, in <module>
    from pkg_resources.extern import appdirs
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 941, in _find_spec
  File "<frozen importlib._bootstrap>", line 914, in _find_spec_legacy
ImportWarning: VendorImporter.find_spec() not found; falling back to find_module()
```
```console
$ python3.10 -m pip show setuptools                                     
Name: setuptools
Version: 52.0.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email: distutils-sig@python.org
License: UNKNOWN
Location: ~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages
Requires: 
Required-by:
```
_(I used `pyenv install 3.10-dev` which just got the bleeding edge CPython from the main repository, I didn't even need to change the setuptools version — the one bundled with Python 3.10 causes this same problem; And yeah... it's under Gentoo Linux but the bug is clearly unrelated to the OS)_

Resolving patch:
diff --git a/pkg_resources/extern/__init__.py b/pkg_resources/extern/__init__.py
--- a/pkg_resources/extern/__init__.py
+++ b/pkg_resources/extern/__init__.py
@@ -1,3 +1,4 @@
+import importlib.util
 import sys
 
 
@@ -20,17 +21,10 @@ def search_path(self):
         yield self.vendor_pkg + '.'
         yield ''
 
-    def find_module(self, fullname, path=None):
-        """
-        Return self when fullname starts with root_name and the
-        target module is one vendored through this importer.
-        """
+    def _module_matches_namespace(self, fullname):
+        """Figure out if the target module is vendored."""
         root, base, target = fullname.partition(self.root_name + '.')
-        if root:
-            return
-        if not any(map(target.startswith, self.vendored_names)):
-            return
-        return self
+        return not root and any(map(target.startswith, self.vendored_names))
 
     def load_module(self, fullname):
         """
@@ -60,6 +54,13 @@ def create_module(self, spec):
     def exec_module(self, module):
         pass
 
+    def find_spec(self, fullname, path=None, target=None):
+        """Return a module spec for vendored names."""
+        return (
+            importlib.util.spec_from_loader(fullname, self)
+            if self._module_matches_namespace(fullname) else None
+        )
+
     def install(self):
         """
         Install this importer into sys.meta_path if not already present.
diff --git a/setuptools/extern/__init__.py b/setuptools/extern/__init__.py
--- a/setuptools/extern/__init__.py
+++ b/setuptools/extern/__init__.py
@@ -1,3 +1,4 @@
+import importlib.util
 import sys
 
 
@@ -20,17 +21,10 @@ def search_path(self):
         yield self.vendor_pkg + '.'
         yield ''
 
-    def find_module(self, fullname, path=None):
-        """
-        Return self when fullname starts with root_name and the
-        target module is one vendored through this importer.
-        """
+    def _module_matches_namespace(self, fullname):
+        """Figure out if the target module is vendored."""
         root, base, target = fullname.partition(self.root_name + '.')
-        if root:
-            return
-        if not any(map(target.startswith, self.vendored_names)):
-            return
-        return self
+        return not root and any(map(target.startswith, self.vendored_names))
 
     def load_module(self, fullname):
         """
@@ -60,6 +54,13 @@ def create_module(self, spec):
     def exec_module(self, module):
         pass
 
+    def find_spec(self, fullname, path=None, target=None):
+        """Return a module spec for vendored names."""
+        return (
+            importlib.util.spec_from_loader(fullname, self)
+            if self._module_matches_namespace(fullname) else None
+        )
+
     def install(self):
         """
         Install this importer into sys.meta_path if not already present.


Test patch: