Repository: pypa/setuptools

Issue:
python3.6 can't import installed namespace package if another package in the same namespace is put locally and pkg_resources are imported
Recently I've met an issue with python3.6 and ``pkg_resources``-style namespace packages when for one namespace one package is installed into virtualenv (a) and one is located in current directory (b) - after importing ``pkg_resources`` or ``setuptools`` package a becomes inaccessible. But no such behaviour is observed with python2.7. To be more clear I have an example:
```
# package a:
a $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkga
│       └── __init__.py
├── setup.cfg
└── setup.py

a $ cat pkgns/__init__.py 
__import__('pkg_resources').declare_namespace(__name__)

a $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkga',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# package b:
b $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkgb
│       └── __init__.py
├── setup.cfg
├── setup.py
└── show-bug.py

b $ cat pkgns/__init__.py 
import pkg_resources

pkg_resources.declare_namespace(__name__)

b $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkgb',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# this file will show a bug, pkg_resources import is necessary to see unexpected behaviour
b $ cat show-bug.py 
import pkg_resources

import pkgns.pkga

# create virtualenv with python2.7 in env27 and python3.6 in env36, install pkgns.pkga to both
# and see problem
b $ ./env36/bin/python show-bug.py 
Traceback (most recent call last):
  File "show-bug.py", line 3, in <module>
    import pkgns.pkga
ModuleNotFoundError: No module named 'pkgns.pkga'

b $ ./env27/bin/python show-bug.py 
b $ echo $?
0
```
As you can see ``pkgns.pkga`` installed in virtualenv was successfully imported under python2.7 and failed under python3.6. For me this is an issue because our code is structured similarly and some our packages contain helper functions to use in ``setup.py`` but this bug prevents me from using them. I had a workaround by using older setuptools to build helper packages but this is not a very reliable solution.

I investigated an issue further:
```
# with python2.7
b $ ./env27/bin/python 
Python 2.7.14 (default, Feb 20 2018, 17:13:09) 
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
['.../b/env27/lib/python2.7/site-packages/pkgns']
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['.../b/pkgns', '.../b/env27/lib/python2.7/site-packages/pkgns']

# and with python3.6
b $ ./env36/bin/python 
Python 3.6.4 (default, Feb 20 2018, 20:29:59) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
_NamespacePath(['.../b/env36/lib/python3.6/site-packages/pkgns'])
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['./pkgns']
```
So ``pkg_resources`` during import changed namespace package ``__path__`` attribute in one case correctly and in second not - old path was lost. Also one may notice that under python3.6 original path is not list but ``_NamespacePath`` object and this gives us a key to understanding source of problem. Let's look at ``nspkg.pth`` file:
```
b $ cat env36/lib/python3.6/site-packages/pkgns.pkga-1.0-py2.7-nspkg.pth 
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('pkgns',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('pkgns', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('pkgns', [os.path.dirname(p)])));m = m or sys.modules.setdefault('pkgns', types.ModuleType('pkgns'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
```
If python version is greater than 3.5 ``pkgns`` module entry is created using ``importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec(...))`` function which gives real pep420 namespace package, ``pkg_resources`` on import adjusts namespace packages paths but handles only lists as original path in ``_rebuild_mod_path()`` in ``handle_ns()`` function (commit https://github.com/pypa/setuptools/commit/7c0c39ef1f60571709e5a8e6680f2147c38cd4ff to fix #885):
```
2141     if not isinstance(orig_path, list):
2142         # Is this behavior useful when module.__path__ is not a list?
2143         return
2144 
2145     orig_path.sort(key=position_in_sys_path)
2146     module.__path__[:] = [_normalize_cached(p) for p in orig_path]
```
So what happens briefly: ``pkg_resources`` tries to fix namespace packages paths on import, and inside ``_handle_ns()`` loads module (``pkgns.pkgb`` in my example), after this ``pkgns``' ``__path__`` attribute points to package in current directory, ``orig_path`` points to old ``_NamespacePath`` object and ``_rebuild_mod_path`` does not handle this combination leaving new path not fixed.

Here are versions of software I used: setuptools-39.0.1, pip-9.0.3, python-2.7.14, python-3.6.4 and python-3.6.5.

I can see at least 2 ways to fix the issue - assign ``module.__path__`` to ``orig_path`` when the latter is not a list or to convert it to list and proceed with sorting and normalizing, but I cannot currently predict consequences of both decisions.
python3.6 can't import installed namespace package if another package in the same namespace is put locally and pkg_resources are imported
Recently I've met an issue with python3.6 and ``pkg_resources``-style namespace packages when for one namespace one package is installed into virtualenv (a) and one is located in current directory (b) - after importing ``pkg_resources`` or ``setuptools`` package a becomes inaccessible. But no such behaviour is observed with python2.7. To be more clear I have an example:
```
# package a:
a $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkga
│       └── __init__.py
├── setup.cfg
└── setup.py

a $ cat pkgns/__init__.py 
__import__('pkg_resources').declare_namespace(__name__)

a $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkga',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# package b:
b $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkgb
│       └── __init__.py
├── setup.cfg
├── setup.py
└── show-bug.py

b $ cat pkgns/__init__.py 
import pkg_resources

pkg_resources.declare_namespace(__name__)

b $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkgb',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# this file will show a bug, pkg_resources import is necessary to see unexpected behaviour
b $ cat show-bug.py 
import pkg_resources

import pkgns.pkga

# create virtualenv with python2.7 in env27 and python3.6 in env36, install pkgns.pkga to both
# and see problem
b $ ./env36/bin/python show-bug.py 
Traceback (most recent call last):
  File "show-bug.py", line 3, in <module>
    import pkgns.pkga
ModuleNotFoundError: No module named 'pkgns.pkga'

b $ ./env27/bin/python show-bug.py 
b $ echo $?
0
```
As you can see ``pkgns.pkga`` installed in virtualenv was successfully imported under python2.7 and failed under python3.6. For me this is an issue because our code is structured similarly and some our packages contain helper functions to use in ``setup.py`` but this bug prevents me from using them. I had a workaround by using older setuptools to build helper packages but this is not a very reliable solution.

I investigated an issue further:
```
# with python2.7
b $ ./env27/bin/python 
Python 2.7.14 (default, Feb 20 2018, 17:13:09) 
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
['.../b/env27/lib/python2.7/site-packages/pkgns']
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['.../b/pkgns', '.../b/env27/lib/python2.7/site-packages/pkgns']

# and with python3.6
b $ ./env36/bin/python 
Python 3.6.4 (default, Feb 20 2018, 20:29:59) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
_NamespacePath(['.../b/env36/lib/python3.6/site-packages/pkgns'])
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['./pkgns']
```
So ``pkg_resources`` during import changed namespace package ``__path__`` attribute in one case correctly and in second not - old path was lost. Also one may notice that under python3.6 original path is not list but ``_NamespacePath`` object and this gives us a key to understanding source of problem. Let's look at ``nspkg.pth`` file:
```
b $ cat env36/lib/python3.6/site-packages/pkgns.pkga-1.0-py2.7-nspkg.pth 
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('pkgns',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('pkgns', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('pkgns', [os.path.dirname(p)])));m = m or sys.modules.setdefault('pkgns', types.ModuleType('pkgns'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
```
If python version is greater than 3.5 ``pkgns`` module entry is created using ``importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec(...))`` function which gives real pep420 namespace package, ``pkg_resources`` on import adjusts namespace packages paths but handles only lists as original path in ``_rebuild_mod_path()`` in ``handle_ns()`` function (commit https://github.com/pypa/setuptools/commit/7c0c39ef1f60571709e5a8e6680f2147c38cd4ff to fix #885):
```
2141     if not isinstance(orig_path, list):
2142         # Is this behavior useful when module.__path__ is not a list?
2143         return
2144 
2145     orig_path.sort(key=position_in_sys_path)
2146     module.__path__[:] = [_normalize_cached(p) for p in orig_path]
```
So what happens briefly: ``pkg_resources`` tries to fix namespace packages paths on import, and inside ``_handle_ns()`` loads module (``pkgns.pkgb`` in my example), after this ``pkgns``' ``__path__`` attribute points to package in current directory, ``orig_path`` points to old ``_NamespacePath`` object and ``_rebuild_mod_path`` does not handle this combination leaving new path not fixed.

Here are versions of software I used: setuptools-39.0.1, pip-9.0.3, python-2.7.14, python-3.6.4 and python-3.6.5.

I can see at least 2 ways to fix the issue - assign ``module.__path__`` to ``orig_path`` when the latter is not a list or to convert it to list and proceed with sorting and normalizing, but I cannot currently predict consequences of both decisions.


Discussion:

Resolving patch:
diff --git a/pkg_resources/__init__.py b/pkg_resources/__init__.py
--- a/pkg_resources/__init__.py
+++ b/pkg_resources/__init__.py
@@ -2135,12 +2135,13 @@ def position_in_sys_path(path):
         parts = path_parts[:-module_parts]
         return safe_sys_path_index(_normalize_cached(os.sep.join(parts)))
 
-    if not isinstance(orig_path, list):
-        # Is this behavior useful when module.__path__ is not a list?
-        return
+    new_path = sorted(orig_path, key=position_in_sys_path)
+    new_path = [_normalize_cached(p) for p in new_path]
 
-    orig_path.sort(key=position_in_sys_path)
-    module.__path__[:] = [_normalize_cached(p) for p in orig_path]
+    if isinstance(module.__path__, list):
+        module.__path__[:] = new_path
+    else:
+        module.__path__ = new_path
 
 
 def declare_namespace(packageName):


Test patch:
diff --git a/setuptools/tests/test_namespaces.py b/setuptools/tests/test_namespaces.py
--- a/setuptools/tests/test_namespaces.py
+++ b/setuptools/tests/test_namespaces.py
@@ -12,10 +12,10 @@
 
 class TestNamespaces:
 
-    @pytest.mark.xfail(sys.version_info < (3, 5),
-        reason="Requires importlib.util.module_from_spec")
-    @pytest.mark.skipif(bool(os.environ.get("APPVEYOR")),
-        reason="https://github.com/pypa/setuptools/issues/851")
+    @pytest.mark.xfail(
+        sys.version_info < (3, 5),
+        reason="Requires importlib.util.module_from_spec",
+    )
     def test_mixed_site_and_non_site(self, tmpdir):
         """
         Installing two packages sharing the same namespace, one installed
@@ -55,8 +55,6 @@ def test_mixed_site_and_non_site(self, tmpdir):
         with test.test.paths_on_pythonpath(map(str, targets)):
             subprocess.check_call(try_import)
 
-    @pytest.mark.skipif(bool(os.environ.get("APPVEYOR")),
-        reason="https://github.com/pypa/setuptools/issues/851")
     def test_pkg_resources_import(self, tmpdir):
         """
         Ensure that a namespace package doesn't break on import
@@ -81,8 +79,6 @@ def test_pkg_resources_import(self, tmpdir):
         with test.test.paths_on_pythonpath([str(target)]):
             subprocess.check_call(try_import)
 
-    @pytest.mark.skipif(bool(os.environ.get("APPVEYOR")),
-        reason="https://github.com/pypa/setuptools/issues/851")
     def test_namespace_package_installed_and_cwd(self, tmpdir):
         """
         Installing a namespace packages but also having it in the current
@@ -109,3 +105,32 @@ def test_namespace_package_installed_and_cwd(self, tmpdir):
         ]
         with test.test.paths_on_pythonpath([str(target)]):
             subprocess.check_call(pkg_resources_imp, cwd=str(pkg_A))
+
+    def test_packages_in_the_same_namespace_installed_and_cwd(self, tmpdir):
+        """
+        Installing one namespace package and also have another in the same
+        namespace in the current working directory, both of them must be
+        importable.
+        """
+        pkg_A = namespaces.build_namespace_package(tmpdir, 'myns.pkgA')
+        pkg_B = namespaces.build_namespace_package(tmpdir, 'myns.pkgB')
+        target = tmpdir / 'packages'
+        # use pip to install to the target directory
+        install_cmd = [
+            sys.executable,
+            '-m',
+            'pip.__main__',
+            'install',
+            str(pkg_A),
+            '-t', str(target),
+        ]
+        subprocess.check_call(install_cmd)
+        namespaces.make_site_dir(target)
+
+        # ensure that all packages import and pkg_resources imports
+        pkg_resources_imp = [
+            sys.executable,
+            '-c', 'import pkg_resources; import myns.pkgA; import myns.pkgB',
+        ]
+        with test.test.paths_on_pythonpath([str(target)]):
+            subprocess.check_call(pkg_resources_imp, cwd=str(pkg_B))