Repository: pypa/setuptools

Issue:
[BUG] package-dir with subpackages (namespaces) does not work with editable install
### setuptools version

setuptools==69.1.0

### Python version

Python 3.12.2

### OS

Artix Linux

### Additional environment information

I'm running this in a fresh conda environment.

### Description

I have the following slightly odd directory structure:

```
pyproject.toml
src
- my_package
- - my_module.py
src2
- my_package2
- - my_module2.py
```

And this clause in my pyproject.toml:

```
[tool.setuptools.package-dir]
"different_name" = "src/my_package"
"different_name.subpkg" = "src2/my_package2"
```

After running `pip install .` (ie *non*-editable), I'm able to import modules like this:

```
from different_name import my_module
from different_name.subpkg import my_module2
```

... but with `pip install -e .` (*editable*), the second import fails with:

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'
```

### Expected behavior

I expect the import behavior to be the same whether the package is installed editable or not.

### How to Reproduce

1. `git clone https://github.com/DavidRV00/package-dir-editable` and cd inside.
2. Create a virtual environment (I use `conda create -n package-dir-editable python=3`) and activate it.
3. `pip install -e .` inside the project dir.
4. `python -c "from different_name.subpkg import my_module2"`

### Output

```console
❯ pip install -e .

Obtaining file:///home/david/projects/package-dir-editable
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: package-dir-editable
  Building editable for package-dir-editable (pyproject.toml) ... done
  Created wheel for package-dir-editable: filename=package_dir_editable-0.0.1-0.editable-py3-none-any.whl size=3548 sha256=7754770e4c45aeed608f47517905691f06c62b31638ed8614815d60564a29f72
  Stored in directory: /tmp/pip-ephem-wheel-cache-6sz16frl/wheels/9f/30/48/30cb3e3e3b5268cb75fe782edf9c59670e56859d2cc847d60f
Successfully built package-dir-editable
Installing collected packages: package-dir-editable
Successfully installed package-dir-editable-0.0.1

❯ python -c "from different_name import my_module" # works fine

❯ python -c "from different_name.subpkg import my_module2"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'
```



Discussion:
Hi @DavidRV00 , ~what is the configuration in your `packages` field?~

Sorry, I just saw that you provided a github repository example.

I believe that to make it work you need to set the `tool.setuptools.packages` field with a list of all packages and subpackages. Because your package structure deviates from the 2 conventional styles we support natively  (vanilla src- and flat-layout, without directory name remapping), the auto discovery is limited.

Can you please remove all the cache (by `rm`-ing the `build`, `dist` and any `.egg-info` folders), try doing the change I pointed above in `pyproject.toml`, and let me know if it works?
Hi @abravalheri ,

I want to avoid explicitly listing the packages, because auto-discovery seems to work great, at least for the non-editable install; seems like it's really close to being able to handle it. But if the issue is just that what I'm doing isn't fully supported, then I think we can close this as not-a-bug. Maybe I can make it work with packages.find.
Hi @DavidRV00 , with a customisation like that, auto-discovery is not supposed to work.

It is probably just working by accident...
Alright, thanks for letting me know. I'll see if I can find some other way to make what I'm trying to do work.
Hi @DavidRV00, thank you very much for the feedback.

I will close this issue for now, because the customisation of `package-dir` does require explicitly listing `packages` for everything to work fine. (The only situations where `packages` can be safely auto-discovered are vanilla [`src-layout`](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout) and vanilla [`flat-layout`](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#flat-layout), i.e. no directory re-mapping).
Hello again @abravalheri ,

commenting to re-open, because now I've tried out using an explicit list of packages, but I get the same result as before with editable installs.

This is what I've added to pyproject.toml:
```
[tool.setuptools]
packages = [
	"different_name",
	"different_name.subpkg",
]
```

Again `pip install .` works fine, but trying to import the subpkg after `pip install -e .` fails.

Here, again, is the log output from `pip install -e .`:
```
(package-dir-editable) package-dir-editable ❯ pip install -e .
Obtaining file:///home/david/projects/package-dir-editable
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: package-dir-editable
  Building editable for package-dir-editable (pyproject.toml) ... done
  Created wheel for package-dir-editable: filename=package_dir_editable-0.0.1-0.editable-py3-none-any.whl size=3548 sha256=6c6c2bbedd1e41a8e0eb18574447f13af5fcb81b75673d07fe3eff32d8cbfd42
  Stored in directory: /tmp/pip-ephem-wheel-cache-9mokx92b/wheels/9f/30/48/30cb3e3e3b5268cb75fe782edf9c59670e56859d2cc847d60f
Successfully built package-dir-editable
Installing collected packages: package-dir-editable
Successfully installed package-dir-editable-0.0.1
```

## Reproducible example
1. Either pull and switch branches on the github repo above to branch `explicit-packages`, or just run `git clone -b explicit-packages https://github.com/DavidRV00/package-dir-editable` to get it directly.
2. cd inside, then `pip install -e .`
3. `python -c "from different_name.subpkg import my_module2"` (you should see this fail)

Hopefully this is still just a case of me doing something unsupported, but I'm not sure what it could be.

Thanks again for any help!
Hi @DavidRV00, thank you for the reproducer.

This seems to be a tricky one.

If I do:

```python
>>> import __editable___package_dir_editable_0_0_1_finder as ed
>>> ed._EditableNamespaceFinder.find_spec("different_name.subpkg")
ModuleSpec(name='different_name.subpkg', loader=None, submodule_search_locations=['/tmp/package-dir-editable/src2/my_package2'])
>>> ed._EditableFinder.find_spec("different_name.subpkg.my_module2")
ModuleSpec(name='different_name.subpkg.my_module2', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fcd914fd490>, origin='/tmp/package-dir-editable/src2/my_package2/my_module2.py')

###

>>> import sys
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>, <class '__editable___package_dir_editable_0_0_1_finder._EditableFinder'>]
>>> sys.path_hooks
[<class 'zipimport.zipimporter'>, <function FileFinder.path_hook.<locals>.path_hook_for_FileFinder at 0x7fcd917245e0>, <bound method _EditableNamespaceFinder._path_hook of <class '__editable___package_dir_editable_0_0_1_finder._EditableNamespaceFinder'>>]
>>> sys.path
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/tmp/package-dir-editable/.venv/lib/python3.12/site-packages', '__editable__.package_dir_editable-0.0.1.finder.__path_hook__']
>>> from different_name.subpkg import my_module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'different_name.subpkg'
```

I can see that the custom finders can successfully locate the packages themselves... But somehow they don't seem to work effectively togheter with the import machinery.

This might be a very tricky problem to solve (and take time research, debugging).
Related to https://github.com/python/cpython/issues/92054.

If anyone in the community would like to provide a PR, that would be much appreciated.
Thanks for the quick reply, and for looking into it so quick!

FWIW, for a little context, the reason I want this kind of directory structure is that I'm trying to put together a system for writing hybrid Python/C packages using just setuptools, CMake, and SWIG. CMake tends to generate code in parallel directory trees that mirror the src tree, so I end up with directories that look like `src/a/b/c... , build/src/a/b/c...`. The challenge is to find a nice way to import the SWIG-generated python interfaces under the `build` dir, homogenously with the regular python packages under the `src` dir, hence the remapping.
Maybe as a workaround while this issue is not solved, try adding `__init__.py` to the directories?

Namespaces are notably more complex to handle in custom finders (and that is related to the advice I asked in https://github.com/python/cpython/issues/92054).
Great, using `__init__.py`'s does make it work. That'll be a fine workaround for now, thanks.
Another note if it's useful: Using `pip install -e . --config-settings editable_mode=strict` also makes it work.

Resolving patch:
diff --git a/setuptools/command/editable_wheel.py b/setuptools/command/editable_wheel.py
--- a/setuptools/command/editable_wheel.py
+++ b/setuptools/command/editable_wheel.py
@@ -505,7 +505,7 @@ def __init__(self, dist: Distribution, name: str):
         self.dist = dist
         self.name = name
 
-    def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]):
+    def template_vars(self) -> Tuple[str, str, Dict[str, str], Dict[str, List[str]]]:
         src_root = self.dist.src_root or os.curdir
         top_level = chain(_find_packages(self.dist), _find_top_level_modules(self.dist))
         package_dir = self.dist.package_dir or {}
@@ -519,7 +519,7 @@ def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]
         )
 
         legacy_namespaces = {
-            pkg: find_package_path(pkg, roots, self.dist.src_root or "")
+            cast(str, pkg): find_package_path(pkg, roots, self.dist.src_root or "")
             for pkg in self.dist.namespace_packages or []
         }
 
@@ -530,11 +530,20 @@ def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]
 
         name = f"__editable__.{self.name}.finder"
         finder = _normalization.safe_identifier(name)
+        return finder, name, mapping, namespaces_
+
+    def get_implementation(self) -> Iterator[Tuple[str, bytes]]:
+        finder, name, mapping, namespaces_ = self.template_vars()
+
         content = bytes(_finder_template(name, mapping, namespaces_), "utf-8")
-        wheel.writestr(f"{finder}.py", content)
+        yield (f"{finder}.py", content)
 
         content = _encode_pth(f"import {finder}; {finder}.install()")
-        wheel.writestr(f"__editable__.{self.name}.pth", content)
+        yield (f"__editable__.{self.name}.pth", content)
+
+    def __call__(self, wheel: "WheelFile", files: List[str], mapping: Dict[str, str]):
+        for file, content in self.get_implementation():
+            wheel.writestr(file, content)
 
     def __enter__(self):
         msg = "Editable install will be performed using a meta path finder.\n"
@@ -784,6 +793,7 @@ def _get_root(self):
 
 
 _FINDER_TEMPLATE = """\
+from __future__ import annotations
 import sys
 from importlib.machinery import ModuleSpec, PathFinder
 from importlib.machinery import all_suffixes as module_suffixes
@@ -791,16 +801,14 @@ def _get_root(self):
 from itertools import chain
 from pathlib import Path
 
-MAPPING = {mapping!r}
-NAMESPACES = {namespaces!r}
+MAPPING: dict[str, str] = {mapping!r}
+NAMESPACES: dict[str, list[str]] = {namespaces!r}
 PATH_PLACEHOLDER = {name!r} + ".__path_hook__"
 
 
 class _EditableFinder:  # MetaPathFinder
     @classmethod
-    def find_spec(cls, fullname, path=None, target=None):
-        extra_path = []
-
+    def find_spec(cls, fullname: str, _path=None, _target=None) -> ModuleSpec | None:
         # Top-level packages and modules (we know these exist in the FS)
         if fullname in MAPPING:
             pkg_path = MAPPING[fullname]
@@ -811,35 +819,42 @@ def find_spec(cls, fullname, path=None, target=None):
         # to the importlib.machinery implementation.
         parent, _, child = fullname.rpartition(".")
         if parent and parent in MAPPING:
-            return PathFinder.find_spec(fullname, path=[MAPPING[parent], *extra_path])
+            return PathFinder.find_spec(fullname, path=[MAPPING[parent]])
 
         # Other levels of nesting should be handled automatically by importlib
         # using the parent path.
         return None
 
     @classmethod
-    def _find_spec(cls, fullname, candidate_path):
+    def _find_spec(cls, fullname: str, candidate_path: Path) -> ModuleSpec | None:
         init = candidate_path / "__init__.py"
         candidates = (candidate_path.with_suffix(x) for x in module_suffixes())
         for candidate in chain([init], candidates):
             if candidate.exists():
                 return spec_from_file_location(fullname, candidate)
+        return None
 
 
 class _EditableNamespaceFinder:  # PathEntryFinder
     @classmethod
-    def _path_hook(cls, path):
+    def _path_hook(cls, path) -> type[_EditableNamespaceFinder]:
         if path == PATH_PLACEHOLDER:
             return cls
         raise ImportError
 
     @classmethod
-    def _paths(cls, fullname):
-        # Ensure __path__ is not empty for the spec to be considered a namespace.
-        return NAMESPACES[fullname] or MAPPING.get(fullname) or [PATH_PLACEHOLDER]
+    def _paths(cls, fullname: str) -> list[str]:
+        paths = NAMESPACES[fullname]
+        if not paths and fullname in MAPPING:
+            paths = [MAPPING[fullname]]
+        # Always add placeholder, for 2 reasons:
+        # 1. __path__ cannot be empty for the spec to be considered namespace.
+        # 2. In the case of nested namespaces, we need to force
+        #    import machinery to query _EditableNamespaceFinder again.
+        return [*paths, PATH_PLACEHOLDER]
 
     @classmethod
-    def find_spec(cls, fullname, target=None):
+    def find_spec(cls, fullname: str, _target=None) -> ModuleSpec | None:
         if fullname in NAMESPACES:
             spec = ModuleSpec(fullname, None, is_package=True)
             spec.submodule_search_locations = cls._paths(fullname)
@@ -847,7 +862,7 @@ def find_spec(cls, fullname, target=None):
         return None
 
     @classmethod
-    def find_module(cls, fullname):
+    def find_module(cls, _fullname) -> None:
         return None
 
 


Test patch:
diff --git a/setuptools/tests/test_editable_install.py b/setuptools/tests/test_editable_install.py
--- a/setuptools/tests/test_editable_install.py
+++ b/setuptools/tests/test_editable_install.py
@@ -24,6 +24,7 @@
 from setuptools.command.editable_wheel import (
     _DebuggingTips,
     _LinkTree,
+    _TopLevelFinder,
     _encode_pth,
     _find_virtual_namespaces,
     _find_namespaces,
@@ -530,6 +531,49 @@ def test_combine_namespaces(self, tmp_path):
             assert pkgA.a == 13
             assert mod2.b == 37
 
+    def test_combine_namespaces_nested(self, tmp_path):
+        """
+        Users may attempt to combine namespace packages in a nested way via
+        ``package_dir`` as shown in pypa/setuptools#4248.
+        """
+
+        files = {
+            "src": {"my_package": {"my_module.py": "a = 13"}},
+            "src2": {"my_package2": {"my_module2.py": "b = 37"}},
+        }
+
+        stack = jaraco.path.DirectoryStack()
+        with stack.context(tmp_path):
+            jaraco.path.build(files)
+            attrs = {
+                "script_name": "%PEP 517%",
+                "package_dir": {
+                    "different_name": "src/my_package",
+                    "different_name.subpkg": "src2/my_package2",
+                },
+                "packages": ["different_name", "different_name.subpkg"],
+            }
+            dist = Distribution(attrs)
+            finder = _TopLevelFinder(dist, str(uuid4()))
+            code = next(v for k, v in finder.get_implementation() if k.endswith(".py"))
+
+        with contexts.save_paths(), contexts.save_sys_modules():
+            for mod in attrs["packages"]:
+                sys.modules.pop(mod, None)
+
+            self.install_finder(code)
+            mod1 = import_module("different_name.my_module")
+            mod2 = import_module("different_name.subpkg.my_module2")
+
+            expected = str((tmp_path / "src/my_package/my_module.py").resolve())
+            assert str(Path(mod1.__file__).resolve()) == expected
+
+            expected = str((tmp_path / "src2/my_package2/my_module2.py").resolve())
+            assert str(Path(mod2.__file__).resolve()) == expected
+
+            assert mod1.a == 13
+            assert mod2.b == 37
+
     def test_dynamic_path_computation(self, tmp_path):
         # Follows the example in PEP 420
         files = {