Repository: pypa/setuptools

Issue:
ImportWarning when importing a PEP 420 namespace package
The `pkg_resources` code is raising an `ImportWarning` when it attempts to import a [PEP 420](https://www.python.org/dev/peps/pep-0420/) implicit namespace package:

```
ImportError: Failed to import test module: test.test_methods
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/bignose/Projects/debian/dput/trunk/test/test_methods.py", line 26, in <module>
    import testscenarios
  File "/usr/lib/python3/dist-packages/testscenarios/__init__.py", line 57, in <module>
    from testscenarios.scenarios import (
  File "/usr/lib/python3/dist-packages/testscenarios/scenarios.py", line 34, in <module>
    from testtools import iterate_tests
  File "/usr/lib/python3/dist-packages/testtools/__init__.py", line 124, in <module>
    from pbr.version import VersionInfo
  File "/usr/lib/python3/dist-packages/pbr/version.py", line 25, in <module>
    import pkg_resources
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3037, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3021, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in _initialize_master_working_set
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in <genexpr>
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2595, in activate
    declare_namespace(pkg)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2163, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2083, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /usr/lib/python3/dist-packages/keyrings: missing __init__
```

(To reproduce this, enable the `-W error` option to `python3`. I did this to try to catch warnings in *my* code, assuming that libraries would not cause them :-/)

The “Not importing directory […] missing __init__” contradicts the explicit [specification from PEP 420](https://www.python.org/dev/peps/pep-0420/#specification):

> Namespace packages [as specified in this PEP] cannot contain an __init__.py .



Discussion:
I am not very familiar with namespace packages, so I could be wrong about where the fault lies. I am sure, though, that there *is* a fault, and it needs to be resolved to avoid the `ImportWarning`.

I'm seeing this as well. It is causing loads of noise in my logs when Python warnings are enabled. Here is a minimal bash script to demonstrate the error:

```bash
#!/bin/bash

rm -rf venv
python3 -m venv venv
venv/bin/pip install sphinxcontrib-websupport
venv/bin/python -Werror -c 'import pkg_resources'
```

Actual output:

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2958, in <module>
    @_call_aside
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2944, in _call_aside
    f(*args, **kwargs)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2985, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2521, in activate
    declare_namespace(pkg)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2103, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2037, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /home/jon/warning-test/venv/lib64/python3.5/site-packages/sphinxcontrib: missing __init__
```
As noted above, the lack of __init__.py is fine according to PEP420. The error itself comes from the builtin imp module in python2.7:

```
/usr/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/home/kleptog/venv/lib/python2.7/site-packages/zc': missing __init__.py
  file, filename, etc = imp.find_module(subname, path)
```

So the real fix for this is for python itself to stop complaining (given it just works), or barring that suppress the warning in the package that calls it, which is setuptools.
One plan that addresses this issue is to replace the vast majority of uses of pkg_resources with stdlib functionality (and backports), such as with [importlib_resources](https://pypi.org/project/importlib_resources/), [importlib_metadata](https://pypi.org/project/importlib_metadata/) and [entrypoints](https://pypi.org/project/entrypoints).

> a minimal bash script to demonstrate the error

I tried replicating the minimal script on Python 3.7, but it fails first on another warning:

```
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
    collections.MutableMapping.register(ParseResults)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 52, in __getattr__
    DeprecationWarning, stacklevel=2)
DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
```

I've reported that in #1480.

I tried replicating the issue on Python 3.6 but couldn't:

```
~ $ python3.6 -m venv venv
~ $ venv/bin/pip install sphinxcontrib-websupport
Collecting sphinxcontrib-websupport
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
~ $ venv/bin/python -c 'import pkg_resources'; echo no problem
no problem
```

I should note that if you have [rwt](https://pypi.org/project/rwt) (and you should), it's easier and cleaner to test with this one liner (and doesn't leave you with ./venv):

```
~ $ python3.6 -m rwt sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
Collecting sphinxcontrib-websupport
  Using cached https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
no problem
```

Let's see if we can ascertain why the issue exists in your environment but not in mine.
I upgraded my setuptools and pip and still no issue:

```
~ $ python3.6 -m pip install -q -U setuptools pip
~ $ python3.6 -m rwt -q sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
no problem
```
I tested on Python 3.5.2 and 3.6.4 and 3.6.6, but none of those exhibit the reported failure. Maybe the issue has gone away?
The issue still occurs for me, with Python 3.6.6.

Using a slightly enhanced version of @jdufresne's test script:

```bash
#! /bin/bash

VENV_DIR="$(realpath ./venv)"

rm -rf "$VENV_DIR"
python3 -m venv "$VENV_DIR"
"$VENV_DIR"/bin/python3 --version
"$VENV_DIR"/bin/python3 -m pip install sphinxcontrib-websupport
"$VENV_DIR"/bin/python3 -W error -c 'import pkg_resources'
```

The output I get:

```
$ ./jdufresne.test 
Python 3.6.6
Collecting sphinxcontrib-websupport
  Using cached https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in _initialize_master_working_set
    for dist in working_set
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in <genexpr>
    for dist in working_set
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2639, in activate
    declare_namespace(pkg)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2176, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2096, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory /home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__
```
 
@bignose-debian Would you consider creating a Dockerfile that replicates the issue? With that, I could troubleshoot in the environment where the issue occurs.
@jaraco wrote:
> Would you consider creating a Dockerfile that replicates the issue?

Yes, though I haven't a clue how to do that (nor how to know whether the result would correctly replicate the behaviour).
I'll take a stab at it.
I was able to replicate thus:

```
draft $ cat > Dockerfile
from python:3.6
RUN python -m venv venv
RUN venv/bin/python --version
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkg_resources'
draft $ docker build .
Sending build context to Docker daemon  13.04MB
Step 1/5 : from python:3.6
3.6: Pulling from library/python
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
28bdf9e584cc: Pull complete
dec1a1f0462b: Pull complete
a4670d125615: Pull complete
547b45a875f5: Pull complete
102a0247b454: Pull complete
Digest: sha256:0c3b6384565b8524a82096bee143174a1f07a28cfdc090ca63c040251ab10604
Status: Downloaded newer image for python:3.6
 ---> 4f13b7f2138e
Step 2/5 : RUN python -m venv venv
 ---> Running in 31b6974b096e
Removing intermediate container 31b6974b096e
 ---> 48aa5b2ac9ae
Step 3/5 : RUN venv/bin/python --version
 ---> Running in 46a4c6c2e3a2
Python 3.6.6
Removing intermediate container 46a4c6c2e3a2
 ---> 5d232b495a62
Step 4/5 : RUN venv/bin/python -m pip install sphinxcontrib-websupport
 ---> Running in 134e7c778b19
Collecting sphinxcontrib-websupport
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 134e7c778b19
 ---> 86e837b01253
Step 5/5 : CMD venv/bin/python -W error -c 'import pkg_resources'
 ---> Running in 0ce64945506a
Removing intermediate container 0ce64945506a
 ---> d90cede4387c
Successfully built d90cede4387c
draft $ docker run d90cede4387c
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in _initialize_master_working_set
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in <genexpr>
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2639, in activate
    declare_namespace(pkg)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2176, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2096, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory /venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__
```

Not sure if this will be of any help but the warning seems to occur on any namespace packages present in sys.modules that have a "namespace_packages.txt" file in their metadata.
So I've confirmed the .pth file is present and that causes `sphinxcontrib` to exist in `sys.modules` before any import of pkg_resources, which is good.
Modules that have `namespace_packages.txt` in their metadata are those that use `setuptools` to declare `namespace_packages`, a deprecated but common technique (and difficult to migrate away from).
> I tried replicating the issue on Python 3.6 but couldn't

Glad I pasted my exact commands, because I started replicating it on my local workstation and couldn't figure out what I'd done wrong. I see now I neglected to include the important `-W error` when I tried to replicate the issue above.
I learned that removing the `.pth` file works around the issue. That explains why the issue didn't replicate when under `rwt` (because the .pth files aren't honored in that mode).

```
draft $ rm venv/lib/python*/site-packages/sphinxcontrib_websupport*.pth
draft $ venv/bin/python -W error -c 'import pkg_resources'; echo no problem
no problem
```
Digging deeper, I see why the absence of the `.pth` file bypasses the issue. It's because in `pkg_resources.Distribution.activate`, the package is not in `sys.modules`, so the `declare_namespace` never gets called.
Removing setuptools from the equation, I can replicate the failure with this Dockerfile:

```
from python:3.6
RUN python -m venv venv
RUN venv/bin/python -m pip install -U pip setuptools
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkgutil; importer = pkgutil.get_importer("venv/lib/python3.6/site-packages/"); importer.find_module("sphinxcontrib")'
```

emits

```
draft $ docker run -it 07e6f2229b27
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__
```

Resolving patch:
diff --git a/pkg_resources/__init__.py b/pkg_resources/__init__.py
--- a/pkg_resources/__init__.py
+++ b/pkg_resources/__init__.py
@@ -2094,7 +2094,12 @@ def _handle_ns(packageName, path_item):
     importer = get_importer(path_item)
     if importer is None:
         return None
-    loader = importer.find_module(packageName)
+
+    # capture warnings due to #1111
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        loader = importer.find_module(packageName)
+
     if loader is None:
         return None
     module = sys.modules.get(packageName)


Test patch: