Repository: python/mypy

Issue:
MYPYPATH doesn't respect PEP420 namespace packages
Take this directory tree:

```
/path/to/src
└───namespace
    └───package
        │   __init__.py
        │
        └───__pycache__
                __init__.cpython-35.pyc
```

And this MYPYPATH:

```
> $env:MYPYPATH
/path/to/src
```

then running `mypy -p namespace.package` gives:

```
> mypy -p namespace.package
Can't find package 'namespace.package'
```

However, mypy works fine when you add an empty `__init__.py` into `namepace` (turning it into a regular package).



Discussion:
I know... I'm not sure what to do about this. The [algorithm from PEP 420](https://www.python.org/dev/peps/pep-0420/#specification) is quite complex (namespace packages are only considered if no "classic" package or module is found). The workaround of adding a dummy `__init__.py` is simple enough (you could also have a dummy `__init__.pyi` so that Python is not affected by this file).

I also expect that usually you'd want to create a separate set of stubs rather than type-check the original source code (though obviously this may vary per package).

I'd like to bump this. In future, there will be python programmers who never heard of pre PEP420 era of `__init__.py` and that's good. What's not good is mypy forcing them to create dummy files. 
It might get the ball rolling if somebody explained why this is such a big problem for mypy.
I just bumped into Jukka Lehtosalo and he said there shouldn't be any major issue blocking this. But given the easy workaround that exists for this, it'll likely require an outsider to write a PR.
> What's not good is mypy forcing them to create dummy files.

This issue is more complex than you think.
I don't think it's trivial, I've read your previous comment - so I don't even think I'd be able to write a PR. I just think it's an issue worth looking into eventually
I'm working on a project with a directory structure like the following:

<pre>
./a
./a/c
./a/c/file1.py
./b
./b/c
./b/c/file2.py
</pre>

Note that both, a and b, contain a subfolder called c. Adding an `__init__`  everywhere defines some modules multiple times (the actual project contains dozens folders) and things get hairy. Any progress on this issue would be much appreciated.
> Adding an `__init__` everywhere defines some modules multiple times [...]

I presume you mean in each `c` folder but not in `a` or `b` right? So `c` is a namespace package and you have both `a` and `b` in `PYTHONPATH` or `sys.path`. (And when running mypy, you pass it both `a` and `b`, which will have the same effect -- or perhaps you pass it their parent directory or the current directory.)

With that setup, I agree you have a predicament. We will have to support namespace packages in mypy. At the very least it should be possible to do something like setting `MYPYPATH=a:b` and then run `mypy -m c` and have it find both parts of `c`.
Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without `__init__.py` are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty `__init__.py` turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add `__init__.py` (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of `__init__.py` becomes the special case, used only when you need some package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one.
Or perhaps this reflects your particular culture.

On Nov 25, 2017 21:48, "Carl Meyer" <notifications@github.com> wrote:

> Just a philosophical note here that might influence the way mypy handles
> this: in my experience in a Py3-only world, packages without __init__.py
> are not a special edge case for namespace packages, they are the default
> case. It's pretty unreasonable to expect a new Py3 developer to remember to
> drop empty __init__.py turds everywhere when their code demonstrably runs
> perfectly fine without them. It's hard to even make a sensible case in code
> review for why they should add __init__.py (other than "it's the way
> we've always done things in the old world, get off my lawn" or "the
> typechecker is buggy and won't find your code without it.") The presence of
> __init__.py becomes the special case, used only when you need some
> package-level code or imports.
>
> This reversal may not have been an intended consequence of PEP 420, but
> AFAICS it's an inevitable one.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/python/mypy/issues/1645#issuecomment-346986174>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ACwrMunwfsZJ03cqKrWM6ONIipLvzuh_ks5s6Pu5gaJpZM4IuKil>
> .
>

> Or perhaps this reflects your particular culture.

Certainly possible. I'd be curious to hear from another Py3-only culture
which (convincing to new Python 3 developers) arguments are used in
favor of continuing the tradition of empty `__init__.py` files. I'm not
aware of a strong practical argument that can be made for them, but if
I'm missing a good one I may start using it ;)
This seems to be hitting quite a few people, so raising priority to high.
This seems to be one the most important Python features that mypy doesn't properly support. Some potential users have told me that they were unable to use mypy because of this.
Unfortunately the work-around of adding dummy` __init__.py` files does not work when you are using tools that expect namespace packages to not have` __init__.py` files. This limitation is preventing me from getting full value from mypy.
@jfgossage As a dirty trick you can try adding dummy `__init__.pyi` files (note `i` at the end), this might work.
I am working with a structure like this:

```
project
   src
      wiseoldbird  (Python package)
          utilities (Python package)
              module1.py
              module2.py
  tests
        wiseoldbird (Python package)
             utilities (Python package)
                   test_module1.py
                   test_module2.py

```
Unfortunately if I try to use `__init__.pyi `files in all the package directories I get the following mypy error:

`tests/wiseoldbird/__init__.pyi`: error: Duplicate module named 'wiseoldbird'

As shown with #5575 I hit the same problem.  I think mypy needs to somehow handle namespace packages and be prepared for an all Python 3 world.

Resolving patch:
diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py
--- a/mypy/modulefinder.py
+++ b/mypy/modulefinder.py
@@ -170,6 +170,7 @@ def _find_module(self, id: str) -> Optional[str]:
         # Now just look for 'baz.pyi', 'baz/__init__.py', etc., inside those directories.
         seplast = os.sep + components[-1]  # so e.g. '/baz'
         sepinit = os.sep + '__init__'
+        near_misses = []  # Collect near misses for namespace mode (see below).
         for base_dir, verify in candidate_base_dirs:
             base_path = base_dir + seplast  # so e.g. '/usr/lib/python3.4/foo/bar/baz'
             # Prefer package over module, i.e. baz/__init__.py* over baz.py*.
@@ -178,10 +179,12 @@ def _find_module(self, id: str) -> Optional[str]:
                 path_stubs = base_path + '-stubs' + sepinit + extension
                 if fscache.isfile_case(path):
                     if verify and not verify_module(fscache, id, path):
+                        near_misses.append(path)
                         continue
                     return path
                 elif fscache.isfile_case(path_stubs):
                     if verify and not verify_module(fscache, id, path_stubs):
+                        near_misses.append(path_stubs)
                         continue
                     return path_stubs
             # No package, look for module.
@@ -189,8 +192,36 @@ def _find_module(self, id: str) -> Optional[str]:
                 path = base_path + extension
                 if fscache.isfile_case(path):
                     if verify and not verify_module(fscache, id, path):
+                        near_misses.append(path)
                         continue
                     return path
+
+        # In namespace mode, re-check those entries that had 'verify'.
+        # Assume search path entries xxx, yyy and zzz, and we're
+        # looking for foo.bar.baz.  Suppose near_misses has:
+        #
+        # - xxx/foo/bar/baz.py
+        # - yyy/foo/bar/baz/__init__.py
+        # - zzz/foo/bar/baz.pyi
+        #
+        # If any of the foo directories has __init__.py[i], it wins.
+        # Else, we look for foo/bar/__init__.py[i], etc.  If there are
+        # none, the first hit wins.  Note that this does not take into
+        # account whether the lowest-level module is a file (baz.py),
+        # a package (baz/__init__.py), or a stub file (baz.pyi) -- for
+        # these the first one encountered along the search path wins.
+        #
+        # The helper function highest_init_level() returns an int that
+        # indicates the highest level at which a __init__.py[i] file
+        # is found; if no __init__ was found it returns 0, if we find
+        # only foo/bar/__init__.py it returns 1, and if we have
+        # foo/__init__.py it returns 2 (regardless of what's in
+        # foo/bar).  It doesn't look higher than that.
+        if self.options and self.options.namespace_packages and near_misses:
+            levels = [highest_init_level(fscache, id, path) for path in near_misses]
+            index = levels.index(max(levels))
+            return near_misses[index]
+
         return None
 
     def find_modules_recursive(self, module: str) -> List[BuildSource]:
@@ -236,6 +267,19 @@ def verify_module(fscache: FileSystemCache, id: str, path: str) -> bool:
     return True
 
 
+def highest_init_level(fscache: FileSystemCache, id: str, path: str) -> int:
+    """Compute the highest level where an __init__ file is found."""
+    if path.endswith(('__init__.py', '__init__.pyi')):
+        path = os.path.dirname(path)
+    level = 0
+    for i in range(id.count('.')):
+        path = os.path.dirname(path)
+        if any(fscache.isfile_case(os.path.join(path, '__init__{}'.format(extension)))
+               for extension in PYTHON_EXTENSIONS):
+            level = i + 1
+    return level
+
+
 def mypy_path() -> List[str]:
     path_env = os.getenv('MYPYPATH')
     if not path_env:


Test patch:
diff --git a/test-data/unit/check-modules.test b/test-data/unit/check-modules.test
--- a/test-data/unit/check-modules.test
+++ b/test-data/unit/check-modules.test
@@ -1,4 +1,5 @@
 -- Type checker test cases dealing with modules and imports.
+-- Towards the end there are tests for PEP 420 (namespace packages, i.e. __init__.py-less packages).
 
 [case testAccessImportedDefinitions]
 import m
@@ -2525,3 +2526,97 @@ def __radd__(self) -> int: ...
 
 [case testFunctionWithInPlaceDunderName]
 def __iadd__(self) -> int: ...
+
+-- Tests for PEP 420 namespace packages.
+
+[case testClassicPackage]
+from foo.bar import x
+[file foo/__init__.py]
+# empty
+[file foo/bar.py]
+x = 0
+
+[case testClassicNotPackage]
+from foo.bar import x
+[file foo/bar.py]
+x = 0
+[out]
+main:1: error: Cannot find module named 'foo.bar'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+
+[case testNamespacePackage]
+# flags: --namespace-packages
+from foo.bar import x
+reveal_type(x)  # E: Revealed type is 'builtins.int'
+[file foo/bar.py]
+x = 0
+
+[case testNamespacePackageWithMypyPath]
+# flags: --namespace-packages --config-file tmp/mypy.ini
+from foo.bax import x
+from foo.bay import y
+from foo.baz import z
+reveal_type(x)  # E: Revealed type is 'builtins.int'
+reveal_type(y)  # E: Revealed type is 'builtins.int'
+reveal_type(z)  # E: Revealed type is 'builtins.int'
+[file xx/foo/bax.py]
+x = 0
+[file yy/foo/bay.py]
+y = 0
+[file foo/baz.py]
+z = 0
+[file mypy.ini]
+[[mypy]
+mypy_path = tmp/xx, tmp/yy
+
+[case testClassicPackageIgnoresEarlierNamespacePackage]
+# flags: --namespace-packages --config-file tmp/mypy.ini
+from foo.bar import y
+reveal_type(y)  # E: Revealed type is 'builtins.int'
+[file xx/foo/bar.py]
+x = ''
+[file yy/foo/bar.py]
+y = 0
+[file yy/foo/__init__.py]
+[file mypy.ini]
+[[mypy]
+mypy_path = tmp/xx, tmp/yy
+
+[case testNamespacePackagePickFirstOnMypyPath]
+# flags: --namespace-packages --config-file tmp/mypy.ini
+from foo.bar import x
+reveal_type(x)  # E: Revealed type is 'builtins.int'
+[file xx/foo/bar.py]
+x = 0
+[file yy/foo/bar.py]
+x = ''
+[file mypy.ini]
+[[mypy]
+mypy_path = tmp/xx, tmp/yy
+
+[case testNamespacePackageInsideClassicPackage]
+# flags: --namespace-packages --config-file tmp/mypy.ini
+from foo.bar.baz import x
+reveal_type(x)  # E: Revealed type is 'builtins.int'
+[file xx/foo/bar/baz.py]
+x = ''
+[file yy/foo/bar/baz.py]
+x = 0
+[file yy/foo/__init__.py]
+[file mypy.ini]
+[[mypy]
+mypy_path = tmp/xx, tmp/yy
+
+[case testClassicPackageInsideNamespacePackage]
+# flags: --namespace-packages --config-file tmp/mypy.ini
+from foo.bar.baz.boo import x
+reveal_type(x)  # E: Revealed type is 'builtins.int'
+[file xx/foo/bar/baz/boo.py]
+x = ''
+[file xx/foo/bar/baz/__init__.py]
+[file yy/foo/bar/baz/boo.py]
+x = 0
+[file yy/foo/bar/__init__.py]
+[file mypy.ini]
+[[mypy]
+mypy_path = tmp/xx, tmp/yy