Repository: pypa/setuptools

Issue:
[BUG] setuptools 64.0.0 breaks editable install
### setuptools version

setuptools==64.0.0

### Python version

Python 3.10.6

### OS

Ubuntu jammy

### Additional environment information

_No response_

### Description

After creating a simple project and installing it as editable with `pip install -e`, I'm unable to import modules from the package.

### Expected behavior

We should be able to import a package installed as editable.

### How to Reproduce

*1. Create this simple project structure*

`pyproject.toml`
```
[build-system]
requires = [
  "setuptools==64.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]
```

`setup.py`
```
from setuptools import setup

setup()
```

`foo/__init__.py`
```
__version__ = "1.0.0"
```

*2. Create and activate a venv:*
```
python -mvenv venv
. venv/bin/activate
```

*3. Install the project as editable:*
```
python install -e .
```

Change to another directory and try to import the package:
```
python -c "import foo"
```




### Output

```console
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'foo'
```



Discussion:
Hi @ajasmin thank you very much for reporting this.

I went through your reproduction steps, but I have a doubt: have you activated your virtual environment? (It does not seem to be part of the steps).

Instead of relying on the activation, I tried to reproduce the same experiment, but using the path to the Python executable directly. This is what I run in my terminal:

```bash
cd /tmp
rm -rf foo
mkdir -p foo/foo
cd foo
cat <<EOF > pyproject.toml
[build-system]
requires = [
  "setuptools==64.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]
EOF

cat <<EOF > setup.py
from setuptools import setup

setup()
EOF

echo '__version__ = "1.0.0"' > foo/__init__.py

python3.8 -mvenv venv

./venv/bin/python -m pip install -e .

cd /tmp

/tmp/foo/venv/bin/python -c "import foo"
```

... I cannot see any error with these steps.

Could you please verify if the same problem persists if you use:

a) the activation script before running Python
b) the explicit path for the Python executable

?

If you still see the error, maybe we will need to create a reproducer using something like a container...
> I went through your reproduction steps, but I have a doubt: have you activated your virtual environment? (It does not seem to be part of the steps).

I activated the venv in my tests. It was missing from the description.

> If you still see the error, maybe we will need to create a reproducer using something like a container...

Thanks. I will try to do so
> I activated the venv in my tests it was missing from the description.

Umm... Could you also try using the explicit path instead of the activation script? (Sometimes the activation script doesn't work very well in my setup, so I got used to just call everything with the explicit path to ensure I am using the correct environment).
Sorry @ajasmin, my bad! I don't trust the activation script in my own machine so I biased against it.

There was an error when I tried to replicate the error: when I `cd` into the `tmp` folder, the root directory of the project was also named `foo` and that created another accidental namespace package...

I was able to replicate your error by renaming the root directory to a name different than the package.

---

What is happening here is not related to the editable installation, you should be able to have the same error even if you omit `-e` and use v63 of setuptools, for example by running:

```bash
cd /tmp
rm -rf proj
mkdir -p proj/foo
cd proj
cat <<EOF > pyproject.toml
[build-system]
requires = [
  "setuptools==63.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]
EOF

cat <<EOF > setup.py
from setuptools import setup

setup()
EOF

echo '__version__ = "1.0.0"' > foo/__init__.py

python3.10 -mvenv venv

/tmp/proj/venv/bin/python -m pip install .

cd /tmp

/tmp/proj/venv/bin/python -c "import foo"
```

The reason why the experiment is failing is related to the line:

```
packages.find.include = ["foo.*"]
```

`include` simply runs a glob pattern matcher with the names of the packages. In your case the package name is `foo` which does not match `foo.*` because it  does not contain a `.`.

If you replace:

```diff
- packages.find.include = ["foo.*"]
+ packages.find.include = ["foo*"]
```

things should work.

---

This behaviour is explained in our [docs](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html):

> *include* and *exclude* accept strings representing [glob](https://docs.python.org/3/library/glob.html#module-glob) patterns. These patterns should match the full name of the Python module (as if it was written in an import statement).
>
> For example if you have `util` pattern, it will match `util/__init__.py` but not `util/files/__init__.py`.
>
> The fact that the parent package is matched by the pattern will not dictate if the submodule will be included or excluded from the distribution. You will need to explicitly add a wildcard (e.g. `util*`) if you want the pattern to also match submodules.





Please let me know if this works for you!

Resolving patch:
diff --git a/setuptools/command/editable_wheel.py b/setuptools/command/editable_wheel.py
--- a/setuptools/command/editable_wheel.py
+++ b/setuptools/command/editable_wheel.py
@@ -585,7 +585,14 @@ def _absolute_root(path: _Path) -> str:
 def _find_virtual_namespaces(pkg_roots: Dict[str, str]) -> Iterator[str]:
     """By carefully designing ``package_dir``, it is possible to implement the logical
     structure of PEP 420 in a package without the corresponding directories.
-    This function will try to find this kind of namespaces.
+
+    Moreover a parent package can be purposefully/accidentally skipped in the discovery
+    phase (e.g. ``find_packages(include=["mypkg.*"])``, when ``mypkg.foo`` is included
+    by ``mypkg`` itself is not).
+    We consider this case to also be a virtual namespace (ignoring the original
+    directory) to emulate a non-editable installation.
+
+    This function will try to find these kinds of namespaces.
     """
     for pkg in pkg_roots:
         if "." not in pkg:
@@ -594,7 +601,8 @@ def _find_virtual_namespaces(pkg_roots: Dict[str, str]) -> Iterator[str]:
         for i in range(len(parts) - 1, 0, -1):
             partial_name = ".".join(parts[:i])
             path = Path(find_package_path(partial_name, pkg_roots, ""))
-            if not path.exists():
+            if not path.exists() or partial_name not in pkg_roots:
+                # partial_name not in pkg_roots ==> purposefully/accidentally skipped
                 yield partial_name
 
 


Test patch:
diff --git a/setuptools/tests/test_editable_install.py b/setuptools/tests/test_editable_install.py
--- a/setuptools/tests/test_editable_install.py
+++ b/setuptools/tests/test_editable_install.py
@@ -269,6 +269,54 @@ def test_namespace_created_via_package_dir(self, venv, tmp_path, editable_opts):
         venv.run(["python", "-m", "pip", "install", "-e", str(pkg_C), *opts])
         venv.run(["python", "-c", "from myns.n import pkgA, pkgB, pkgC"])
 
+    def test_namespace_accidental_config_in_lenient_mode(self, venv, tmp_path):
+        """Sometimes users might specify an ``include`` pattern that ignores parent
+        packages. In a normal installation this would ignore all modules inside the
+        parent packages, and make them namespaces (reported in issue #3504),
+        so the editable mode should preserve this behaviour.
+        """
+        files = {
+            "pkgA": {
+                "pyproject.toml": dedent("""\
+                    [build-system]
+                    requires = ["setuptools", "wheel"]
+                    build-backend = "setuptools.build_meta"
+
+                    [project]
+                    name = "pkgA"
+                    version = "3.14159"
+
+                    [tool.setuptools]
+                    packages.find.include = ["mypkg.*"]
+                    """),
+                "mypkg": {
+                    "__init__.py": "",
+                    "other.py": "b = 1",
+                    "n": {
+                        "__init__.py": "",
+                        "pkgA.py": "a = 1",
+                    },
+                 },
+                "MANIFEST.in": EXAMPLE["MANIFEST.in"],
+            },
+        }
+        jaraco.path.build(files, prefix=tmp_path)
+        pkg_A = tmp_path / "pkgA"
+
+        # use pip to install to the target directory
+        opts = ["--no-build-isolation"]  # force current version of setuptools
+        venv.run(["python", "-m", "pip", "-v", "install", "-e", str(pkg_A), *opts])
+        out = venv.run(["python", "-c", "from mypkg.n import pkgA; print(pkgA.a)"])
+        assert str(out, "utf-8").strip() == "1"
+        cmd = """\
+        try:
+            import mypkg.other
+        except ImportError:
+            print("mypkg.other not defined")
+        """
+        out = venv.run(["python", "-c", dedent(cmd)])
+        assert "mypkg.other not defined" in str(out, "utf-8")
+
 
 # Moved here from test_develop:
 @pytest.mark.xfail(
@@ -490,7 +538,7 @@ def test_pkg_roots(tmp_path):
     assert ns == {"f", "f.g"}
 
     ns = set(_find_virtual_namespaces(roots))
-    assert ns == {"a.b.c.x", "a.b.c.x.y", "m", "m.n", "m.n.o", "m.n.o.p"}
+    assert ns == {"a.b", "a.b.c.x", "a.b.c.x.y", "m", "m.n", "m.n.o", "m.n.o.p"}
 
 
 class TestOverallBehaviour: