Repository: pypa/setuptools

Issue:
ModuleNotFound for editable install of namespace package with two dots
### setuptools version

68.1.2, same with main

### Python version

3.11.5

### OS

MacOS

### Additional environment information

Also happening on GitHub-actions on a Linux env.
See also my comment here: https://github.com/plone/meta/issues/172#issuecomment-1705329010

Locally, on Python 3.10 all goes well, on 3.11 not.

### Description

I maintain various setuptools/pkg_resources style namespace packages with two dots, for example: ``plone.app.uuid``.  Until the beginning of August, an editable install worked fine.  Now it fails, because setuptools 68.1.0+ is used:
`ModuleNotFoundError: No module named 'plone.app.uuid'`

Packages with just one dot seem fine, for example `plone.session`.

### Expected behavior

After `pip install -e .` of the `plone.app.uuid` package, I should be able to do `import plone.app.uuid`.

### How to Reproduce

I tried to create a stripped-down copy of `plone.app.uuid`, but there an import works fine.  I suspect it only goes wrong if another `plone.app.*` package is installed next to it.  So use the original repo, but use a branch I created for this issue with minor changes (mainly a much smaller `tox.ini`):

```
git clone -b maurits-setuptools-namespace git@github.com:plone/plone.app.uuid.git
cd plone.app.uuid
tox -e test
```

This can take a few minutes, because it needs lots of packages.

### Output

```console
$ git clone -b maurits-setuptools-namespace git@github.com:plone/plone.app.uuid.git
Cloning into 'plone.app.uuid'...
remote: Enumerating objects: 698, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 698 (delta 0), reused 0 (delta 0), pack-reused 696
Receiving objects: 100% (698/698), 134.94 KiB | 746.00 KiB/s, done.
Resolving deltas: 100% (353/353), done.
$ cd plone.app.uuid
$ tox -e test
test: install_deps> python -I -m pip install zope.testrunner -c https://dist.plone.org/release/6.0-dev/constraints.txt
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /Users/maurits/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /Users/maurits/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /Users/maurits/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_editable> python -I -m pip install wheel
.pkg: build_editable> python /Users/maurits/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
test: install_package_deps> python -I -m pip install Products.CMFCore Products.ZCatalog plone.app.testing plone.dexterity plone.indexer plone.testing plone.uuid setuptools zope.interface zope.publisher -c/Users/maurits/foo/plone.app.uuid/.tox/test/constraints.txt
test: install_package> python -I -m pip install --force-reinstall --no-deps /Users/maurits/foo/plone.app.uuid/.tox/.tmp/package/1/plone.app.uuid-2.2.3.dev0-0.editable-py3-none-any.whl
test: commands[0]> zope-testrunner --all --test-path=/Users/maurits/foo/plone.app.uuid -s plone.app.uuid
Traceback (most recent call last):
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/bin/zope-testrunner", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/__init__.py", line 31, in run
    failed = run_internal(defaults, args, script_parts=script_parts, cwd=cwd,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/__init__.py", line 55, in run_internal
    runner.run()
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/runner.py", line 179, in run
    feature.global_setup()
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 505, in global_setup
    tests = find_tests(self.runner.options, self.runner.found_suites)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 178, in find_tests
    for suite in found_suites:
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 199, in find_suites
    for fpath, package in find_test_files(options):
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 265, in find_test_files
    for f, package in find_test_files_(options):
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 293, in find_test_files_
    for (p, package) in test_dirs(options, {}):
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 348, in test_dirs
    p = import_name(p)
        ^^^^^^^^^^^^^^
  File "/Users/maurits/foo/plone.app.uuid/.tox/test/lib/python3.11/site-packages/zope/testrunner/find.py", line 423, in import_name
    __import__(name)
ModuleNotFoundError: No module named 'plone.app.uuid'
test: exit 1 (6.56 seconds) /Users/maurits/foo/plone.app.uuid> zope-testrunner --all --test-path=/Users/maurits/foo/plone.app.uuid -s plone.app.uuid pid=38144
.pkg: _exit> python /Users/maurits/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  test: FAIL code 1 (76.97=setup[70.41]+cmd[6.56] seconds)
  evaluation failed :( (77.12 seconds)
```

The editable install generated this file from a template in setuptools:

```
$ cd .tox/test
$ cat lib/python3.11/site-packages/__editable___plone_app_uuid_2_2_3_dev0_finder.py 
import sys
from importlib.machinery import ModuleSpec, PathFinder
from importlib.machinery import all_suffixes as module_suffixes
from importlib.util import spec_from_file_location
from itertools import chain
from pathlib import Path

MAPPING = {'plone': '/Users/maurits/foo/plone.app.uuid/plone'}
NAMESPACES = {}
PATH_PLACEHOLDER = '__editable__.plone.app.uuid-2.2.3.dev0.finder' + ".__path_hook__"
...
```

When I edit `MAPPING` to the following, and directly call the test command, instead of running tox again, it works:

```
MAPPING = {
    'plone': '/Users/maurits/foo/plone.app.uuid/plone',
    'plone.app': '/Users/maurits/foo/plone.app.uuid/plone/app',
}
```

I don't know if that makes any sense, or if the `NAMESPACES` should be filled.

Workaround: force using an older `setuptools`, by adding this in `pyproject.toml`:

```
[build-system]
# See https://snarky.ca/what-the-heck-is-pyproject-toml/
requires = ["setuptools<68.1.0", "wheel"]
build-backend = "setuptools.build_meta"
```

I suspect this is a regression caused by PR #3995, so cc @aganders3.
It may need a fix similar to PR #4020.



Discussion:
Thank you very much @mauritsvanrees for opening the issue.

This is my take on the minimal reproducer:

```bash
> docker run --rm -it python:3.11-bullseye /bin/bash

mkdir -p /tmp/pkg1/plone/app/uuid
cd /tmp/pkg1    
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/uuid/__init__.py
cat <<EOF > setup.py
from setuptools import setup, find_packages

setup(
    name="plone.app.uuid",
    version="42",
    packages=find_packages(),
    namespace_packages=[
        "plone",
        "plone.app",
    ]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==68.1.2", "wheel"]
build-backend = "setuptools.build_meta"
EOF

mkdir -p /tmp/pkg2/plone/app/other
cd /tmp/pkg2
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/other/__init__.py

cat <<EOF > setup.py
from setuptools import setup, find_packages

setup(
    name="plone.app.other",
    version="42",
    packages=find_packages(),
    namespace_packages=[
        "plone",
        "plone.app",
    ]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==68.1.2", "wheel"]
build-backend = "setuptools.build_meta"
EOF


cd /tmp
python3.11 -m venv .venv
.venv/bin/python -m pip install -U 'pip==23.2'
.venv/bin/python -m pip install -e pkg1
.venv/bin/python -m pip install -e pkg2
.venv/bin/python -c 'from plone.app import uuid, other; print(uuid, other)'
```

I think this is related to the fact that it is very hard to support namespace packages with PEP 660.

I have the impression that the `-nspkg.pth`[^0] was already broken before `68.1.0` but it was masked by the specific implementation of the meta path finder[^1].


I think a fix similar to #4020 would not be possible[^1]. Instead maybe the best is to fix the `-nspkg.pth` file that is generated[^2].

[^0]: `pkg_resources.declare_namespace` requires a special `-nspkg.pth` file to be produced.

[^1]: Changing the finder again would not be a good option. Personally, I don't know how to solve this problem without using the logic that was previously implemented in the Finder, but that was problematic from the point of view of case sensitivity.
I also would not like to add the churn of handling file system case sensitivity on `setuptools` itself as initially suggested in #3995 (it not only hinders maintenance but also decrease performance), so I still believe the best is to delegate to the stdlib's import machinery.

[^2]: I suspected the problem is that it is getting rewritten every time a new package using the same namespace is installed. My guess is that each package should generate a new one, with a different name. The `-nspkg.pth` file only get rewritten if its name contain `.`.

Resolving patch:
diff --git a/setuptools/command/editable_wheel.py b/setuptools/command/editable_wheel.py
--- a/setuptools/command/editable_wheel.py
+++ b/setuptools/command/editable_wheel.py
@@ -341,7 +341,7 @@ def _create_wheel_file(self, bdist_wheel):
         with unpacked_wheel as unpacked, build_lib as lib, build_tmp as tmp:
             unpacked_dist_info = Path(unpacked, Path(self.dist_info_dir).name)
             shutil.copytree(self.dist_info_dir, unpacked_dist_info)
-            self._install_namespaces(unpacked, dist_info.name)
+            self._install_namespaces(unpacked, dist_name)
             files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
             strategy = self._select_strategy(dist_name, tag, lib)
             with strategy, WheelFile(wheel_path, "w") as wheel_obj:
@@ -505,9 +505,19 @@ def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]
             )
         )
 
+        legacy_namespaces = {
+            pkg: find_package_path(pkg, roots, self.dist.src_root or "")
+            for pkg in self.dist.namespace_packages or []
+        }
+
+        mapping = {**roots, **legacy_namespaces}
+        # ^-- We need to explicitly add the legacy_namespaces to the mapping to be
+        #     able to import their modules even if another package sharing the same
+        #     namespace is installed in a conventional (non-editable) way.
+
         name = f"__editable__.{self.name}.finder"
         finder = _normalization.safe_identifier(name)
-        content = bytes(_finder_template(name, roots, namespaces_), "utf-8")
+        content = bytes(_finder_template(name, mapping, namespaces_), "utf-8")
         wheel.writestr(f"{finder}.py", content)
 
         content = _encode_pth(f"import {finder}; {finder}.install()")
@@ -752,9 +762,9 @@ def __init__(self, distribution, installation_dir, editable_name, src_root):
         self.outputs = []
         self.dry_run = False
 
-    def _get_target(self):
+    def _get_nspkg_file(self):
         """Installation target."""
-        return os.path.join(self.installation_dir, self.editable_name)
+        return os.path.join(self.installation_dir, self.editable_name + self.nspkg_ext)
 
     def _get_root(self):
         """Where the modules/packages should be loaded from."""
@@ -777,6 +787,8 @@ def _get_root(self):
 class _EditableFinder:  # MetaPathFinder
     @classmethod
     def find_spec(cls, fullname, path=None, target=None):
+        extra_path = []
+
         # Top-level packages and modules (we know these exist in the FS)
         if fullname in MAPPING:
             pkg_path = MAPPING[fullname]
@@ -787,7 +799,7 @@ def find_spec(cls, fullname, path=None, target=None):
         # to the importlib.machinery implementation.
         parent, _, child = fullname.rpartition(".")
         if parent and parent in MAPPING:
-            return PathFinder.find_spec(fullname, path=[MAPPING[parent]])
+            return PathFinder.find_spec(fullname, path=[MAPPING[parent], *extra_path])
 
         # Other levels of nesting should be handled automatically by importlib
         # using the parent path.
diff --git a/setuptools/namespaces.py b/setuptools/namespaces.py
--- a/setuptools/namespaces.py
+++ b/setuptools/namespaces.py
@@ -13,8 +13,7 @@ def install_namespaces(self):
         nsp = self._get_all_ns_packages()
         if not nsp:
             return
-        filename, ext = os.path.splitext(self._get_target())
-        filename += self.nspkg_ext
+        filename = self._get_nspkg_file()
         self.outputs.append(filename)
         log.info("Installing %s", filename)
         lines = map(self._gen_nspkg_line, nsp)
@@ -28,13 +27,16 @@ def install_namespaces(self):
             f.writelines(lines)
 
     def uninstall_namespaces(self):
-        filename, ext = os.path.splitext(self._get_target())
-        filename += self.nspkg_ext
+        filename = self._get_nspkg_file()
         if not os.path.exists(filename):
             return
         log.info("Removing %s", filename)
         os.remove(filename)
 
+    def _get_nspkg_file(self):
+        filename, _ = os.path.splitext(self._get_target())
+        return filename + self.nspkg_ext
+
     def _get_target(self):
         return self.target
 
@@ -75,7 +77,7 @@ def _gen_nspkg_line(self, pkg):
     def _get_all_ns_packages(self):
         """Return sorted list of all package namespaces"""
         pkgs = self.distribution.namespace_packages or []
-        return sorted(flatten(map(self._pkg_names, pkgs)))
+        return sorted(set(flatten(map(self._pkg_names, pkgs))))
 
     @staticmethod
     def _pkg_names(pkg):


Test patch:
diff --git a/setuptools/tests/namespaces.py b/setuptools/tests/namespaces.py
--- a/setuptools/tests/namespaces.py
+++ b/setuptools/tests/namespaces.py
@@ -1,29 +1,54 @@
+import ast
+import json
 import textwrap
+from pathlib import Path
 
 
-def build_namespace_package(tmpdir, name):
+def iter_namespace_pkgs(namespace):
+    parts = namespace.split(".")
+    for i in range(len(parts)):
+        yield ".".join(parts[: i + 1])
+
+
+def build_namespace_package(tmpdir, name, version="1.0", impl="pkg_resources"):
     src_dir = tmpdir / name
     src_dir.mkdir()
     setup_py = src_dir / 'setup.py'
-    namespace, sep, rest = name.partition('.')
+    namespace, _, rest = name.rpartition('.')
+    namespaces = list(iter_namespace_pkgs(namespace))
+    setup_args = {
+        "name": name,
+        "version": version,
+        "packages": namespaces,
+    }
+
+    if impl == "pkg_resources":
+        tmpl = '__import__("pkg_resources").declare_namespace(__name__)'
+        setup_args["namespace_packages"] = namespaces
+    elif impl == "pkgutil":
+        tmpl = '__path__ = __import__("pkgutil").extend_path(__path__, __name__)'
+    else:
+        raise ValueError(f"Cannot recognise {impl=} when creating namespaces")
+
+    args = json.dumps(setup_args, indent=4)
+    assert ast.literal_eval(args)  # ensure it is valid Python
+
     script = textwrap.dedent(
-        """
+        """\
         import setuptools
-        setuptools.setup(
-            name={name!r},
-            version="1.0",
-            namespace_packages=[{namespace!r}],
-            packages=[{namespace!r}],
-        )
+        args = {args}
+        setuptools.setup(**args)
         """
-    ).format(**locals())
+    ).format(args=args)
     setup_py.write_text(script, encoding='utf-8')
-    ns_pkg_dir = src_dir / namespace
-    ns_pkg_dir.mkdir()
-    pkg_init = ns_pkg_dir / '__init__.py'
-    tmpl = '__import__("pkg_resources").declare_namespace({namespace!r})'
-    decl = tmpl.format(**locals())
-    pkg_init.write_text(decl, encoding='utf-8')
+
+    ns_pkg_dir = Path(src_dir, namespace.replace(".", "/"))
+    ns_pkg_dir.mkdir(parents=True)
+
+    for ns in namespaces:
+        pkg_init = src_dir / ns.replace(".", "/") / '__init__.py'
+        pkg_init.write_text(tmpl, encoding='utf-8')
+
     pkg_mod = ns_pkg_dir / (rest + '.py')
     some_functionality = 'name = {rest!r}'.format(**locals())
     pkg_mod.write_text(some_functionality, encoding='utf-8')
@@ -34,7 +59,7 @@ def build_pep420_namespace_package(tmpdir, name):
     src_dir = tmpdir / name
     src_dir.mkdir()
     pyproject = src_dir / "pyproject.toml"
-    namespace, sep, rest = name.rpartition(".")
+    namespace, _, rest = name.rpartition(".")
     script = f"""\
         [build-system]
         requires = ["setuptools"]
@@ -45,7 +70,7 @@ def build_pep420_namespace_package(tmpdir, name):
         version = "3.14159"
         """
     pyproject.write_text(textwrap.dedent(script), encoding='utf-8')
-    ns_pkg_dir = src_dir / namespace.replace(".", "/")
+    ns_pkg_dir = Path(src_dir, namespace.replace(".", "/"))
     ns_pkg_dir.mkdir(parents=True)
     pkg_mod = ns_pkg_dir / (rest + ".py")
     some_functionality = f"name = {rest!r}"
diff --git a/setuptools/tests/test_editable_install.py b/setuptools/tests/test_editable_install.py
--- a/setuptools/tests/test_editable_install.py
+++ b/setuptools/tests/test_editable_install.py
@@ -11,6 +11,8 @@
 from unittest.mock import Mock
 from uuid import uuid4
 
+from distutils.core import run_setup
+
 import jaraco.envs
 import jaraco.path
 import pytest
@@ -31,6 +33,7 @@
 )
 from setuptools.dist import Distribution
 from setuptools.extension import Extension
+from setuptools.warnings import SetuptoolsDeprecationWarning
 
 
 @pytest.fixture(params=["strict", "lenient"])
@@ -230,22 +233,59 @@ def test_editable_with_single_module(tmp_path, venv, editable_opts):
 
 
 class TestLegacyNamespaces:
-    """Ported from test_develop"""
+    # legacy => pkg_resources.declare_namespace(...) + setup(namespace_packages=...)
 
-    def test_namespace_package_importable(self, venv, tmp_path, editable_opts):
+    def test_nspkg_file_is_unique(self, tmp_path, monkeypatch):
+        deprecation = pytest.warns(
+            SetuptoolsDeprecationWarning, match=".*namespace_packages parameter.*"
+        )
+        installation_dir = tmp_path / ".installation_dir"
+        installation_dir.mkdir()
+        examples = (
+            "myns.pkgA",
+            "myns.pkgB",
+            "myns.n.pkgA",
+            "myns.n.pkgB",
+        )
+
+        for name in examples:
+            pkg = namespaces.build_namespace_package(tmp_path, name, version="42")
+            with deprecation, monkeypatch.context() as ctx:
+                ctx.chdir(pkg)
+                dist = run_setup("setup.py", stop_after="config")
+                cmd = editable_wheel(dist)
+                cmd.finalize_options()
+                editable_name = cmd.get_finalized_command("dist_info").name
+                cmd._install_namespaces(installation_dir, editable_name)
+
+        files = list(installation_dir.glob("*-nspkg.pth"))
+        assert len(files) == len(examples)
+
+    @pytest.mark.parametrize(
+        "impl",
+        (
+            "pkg_resources",
+            #  "pkgutil",  => does not work
+        ),
+    )
+    @pytest.mark.parametrize("ns", ("myns.n",))
+    def test_namespace_package_importable(
+        self, venv, tmp_path, ns, impl, editable_opts
+    ):
         """
         Installing two packages sharing the same namespace, one installed
         naturally using pip or `--single-version-externally-managed`
         and the other installed in editable mode should leave the namespace
         intact and both packages reachable by import.
+        (Ported from test_develop).
         """
         build_system = """\
         [build-system]
         requires = ["setuptools"]
         build-backend = "setuptools.build_meta"
         """
-        pkg_A = namespaces.build_namespace_package(tmp_path, 'myns.pkgA')
-        pkg_B = namespaces.build_namespace_package(tmp_path, 'myns.pkgB')
+        pkg_A = namespaces.build_namespace_package(tmp_path, f"{ns}.pkgA", impl=impl)
+        pkg_B = namespaces.build_namespace_package(tmp_path, f"{ns}.pkgB", impl=impl)
         (pkg_A / "pyproject.toml").write_text(build_system, encoding="utf-8")
         (pkg_B / "pyproject.toml").write_text(build_system, encoding="utf-8")
         # use pip to install to the target directory
@@ -253,7 +293,7 @@ def test_namespace_package_importable(self, venv, tmp_path, editable_opts):
         opts.append("--no-build-isolation")  # force current version of setuptools
         venv.run(["python", "-m", "pip", "install", str(pkg_A), *opts])
         venv.run(["python", "-m", "pip", "install", "-e", str(pkg_B), *opts])
-        venv.run(["python", "-c", "import myns.pkgA; import myns.pkgB"])
+        venv.run(["python", "-c", f"import {ns}.pkgA; import {ns}.pkgB"])
         # additionally ensure that pkg_resources import works
         venv.run(["python", "-c", "import pkg_resources"])