Repository: python/mypy

Issue:
Importing types from namespace packages emits false positives without `from ... import ...`
Importing type hints from namespace packages only works when using the `from ... import ...` form.

Using the `--namespace-packages` flag works properly when the `typedpkg_namespace-alpha` is not installed but `mypy` is executed from the directory containing the package contents (as expected) but only for imports using the `from ... import ...` style. `import ...` does not work as expected. Likewise the same is true without the `--namespace-packages` flag if `typedpkg_namespace-alpha` is installed.

Specifically, the type hints are correctly imported but a false positive error message is still emitted of `Cannot find module named 'typedpkg_namespace'`.

# Repro steps
- Virtual env state
```
$ pip list
Package         Version
--------------- --------------------------------------------------
mypy            0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions 0.4.1
pip             18.1
setuptools      40.4.3
typed-ast       1.1.0
```
- Repro file structure
```
$ find . \( -path ./.mypy_cache -o -path ./venv \) -prune -o -print
.
./typedpkg_namespace
./typedpkg_namespace/alpha
./typedpkg_namespace/alpha/__init__.py
./typedpkg_namespace/alpha/py.typed
./typedpkg_namespace/alpha/alpha_module.py
./not_from_imp.py
./from_imp.py
./setup.py
```
- Repro files
    - `setup.py`
    ```
    setup(
        name='typedpkg_namespace.alpha',
        version='1.0.0',
        packages=find_namespace_packages(),
        namespace_packages=['typedpkg_namespace'],
        zip_safe=False,
        package_data={'typedpkg_namespace.alpha': ['py.typed']}
    )
    ```
    - `typedpkg_namespace/alpha/alpha_module.py`
    ```
    def alpha_func(a: bool) -> bool:
        return not a
    ```
    - `from_imp.py`
    ```
    from typedpkg_namespace.alpha.alpha_module import alpha_func
    alpha_func(False)
    alpha_func(2)
    ```
    - `not_from_imp.py`
    ```
    import typedpkg_namespace.alpha.alpha_module as am; alpha_func = am.alpha_func
    alpha_func(False)
    alpha_func(2)
    ```
    - `typedpkg_namespace/alpha/__init__.py` and `typedpkg_namespace/alpha/py.typed` are empty
    
# Repro Example
- using `--namespace-packages`
```
$ mypy from_imp.py  --namespace-packages
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py --namespace-packages
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"
```
- using installed package
```
$ pip install .
Looking in indexes: http://artifactory.factset.com/artifactory/api/pypi/python/simple/
Processing /home/user/cphilip/side_projects/test_namespace/typedpkg_namespace-alpha
Installing collected packages: typedpkg-namespace.alpha
  Running setup.py install for typedpkg-namespace.alpha ... done
Successfully installed typedpkg-namespace.alpha-1.0.0

$ pip list
Package                  Version
------------------------ --------------------------------------------------
mypy                     0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions          0.4.1
pip                      18.1
setuptools               40.4.3
typed-ast                1.1.0
typedpkg-namespace.alpha 1.0.0

$ mypy from_imp.py
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"
```


Discussion:
Yup, I can repro this. There must be a corner case that I don't handle correctly yet.

Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -572,12 +572,20 @@ def correct_rel_imp(imp: Union[ImportFrom, ImportAll]) -> str:
                     pri = import_priority(imp, PRI_MED)
                     ancestor_pri = import_priority(imp, PRI_LOW)
                     for id, _ in imp.ids:
+                        # We append the target (e.g. foo.bar.baz)
+                        # before the ancestors (e.g. foo and foo.bar)
+                        # so that, if FindModuleCache finds the target
+                        # module in a package marked with py.typed
+                        # underneath a namespace package installed in
+                        # site-packages, (gasp), that cache's
+                        # knowledge of the ancestors can be primed
+                        # when it is asked to find the target.
+                        res.append((pri, id, imp.line))
                         ancestor_parts = id.split(".")[:-1]
                         ancestors = []
                         for part in ancestor_parts:
                             ancestors.append(part)
                             res.append((ancestor_pri, ".".join(ancestors), imp.line))
-                        res.append((pri, id, imp.line))
                 elif isinstance(imp, ImportFrom):
                     cur_id = correct_rel_imp(imp)
                     pos = len(res)
@@ -1400,6 +1408,8 @@ def __init__(self,
                 self.ignore_all = True
         self.path = path
         self.xpath = path or '<string>'
+        if path and source is None and self.manager.fscache.isdir(path):
+            source = ''
         self.source = source
         if path and source is None and self.manager.cache_enabled:
             self.meta = find_cache_meta(self.id, path, manager)
diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py
--- a/mypy/modulefinder.py
+++ b/mypy/modulefinder.py
@@ -74,11 +74,13 @@ def __init__(self,
         self.dirs = {}  # type: Dict[Tuple[str, Tuple[str, ...]], PackageDirs]
         # Cache find_module: id -> result
         self.results = {}  # type: Dict[str, Optional[str]]
+        self.ns_ancestors = {}  # type: Dict[str, str]
         self.options = options
 
     def clear(self) -> None:
         self.results.clear()
         self.dirs.clear()
+        self.ns_ancestors.clear()
 
     def find_lib_path_dirs(self, dir_chain: str, lib_path: Tuple[str, ...]) -> PackageDirs:
         # Cache some repeated work within distinct find_module calls: finding which
@@ -115,6 +117,14 @@ def _find_module_non_stub_helper(self, components: List[str],
                 return os.path.join(pkg_dir, *components[:-1]), index == 0
         return None
 
+    def _update_ns_ancestors(self, components: List[str], match: Tuple[str, bool]) -> None:
+        path, verify = match
+        for i in range(1, len(components)):
+            pkg_id = '.'.join(components[:-i])
+            if pkg_id not in self.ns_ancestors:
+                self.ns_ancestors[pkg_id] = path
+            path = os.path.dirname(path)
+
     def _find_module(self, id: str) -> Optional[str]:
         fscache = self.fscache
 
@@ -155,6 +165,7 @@ def _find_module(self, id: str) -> Optional[str]:
             non_stub_match = self._find_module_non_stub_helper(components, pkg_dir)
             if non_stub_match:
                 third_party_inline_dirs.append(non_stub_match)
+                self._update_ns_ancestors(components, non_stub_match)
         if self.options and self.options.use_builtins_fixtures:
             # Everything should be in fixtures.
             third_party_inline_dirs.clear()
@@ -187,6 +198,8 @@ def _find_module(self, id: str) -> Optional[str]:
                         near_misses.append(path_stubs)
                         continue
                     return path_stubs
+                elif self.options and self.options.namespace_packages and fscache.isdir(base_path):
+                    near_misses.append(base_path)
             # No package, look for module.
             for extension in PYTHON_EXTENSIONS:
                 path = base_path + extension
@@ -222,7 +235,11 @@ def _find_module(self, id: str) -> Optional[str]:
             index = levels.index(max(levels))
             return near_misses[index]
 
-        return None
+        # Finally, we may be asked to produce an ancestor for an
+        # installed package with a py.typed marker that is a
+        # subpackage of a namespace package.  We only fess up to these
+        # if we would otherwise return "not found".
+        return self.ns_ancestors.get(id)
 
     def find_modules_recursive(self, module: str) -> List[BuildSource]:
         module_path = self.find_module(module)


Test patch:
diff --git a/test-data/unit/check-modules.test b/test-data/unit/check-modules.test
--- a/test-data/unit/check-modules.test
+++ b/test-data/unit/check-modules.test
@@ -2280,9 +2280,9 @@ def __getattr__(attr: str) -> Any: ...
 # empty (i.e. complete subpackage)
 [builtins fixtures/module.pyi]
 [out]
-main:1: error: Cannot find module named 'a.b.c'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'a.b.c.d'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'a.b.c'
 
 [case testModuleGetattrInit8a]
 import a.b.c  # Error
@@ -2620,3 +2620,10 @@ x = 0
 [file mypy.ini]
 [[mypy]
 mypy_path = tmp/xx, tmp/yy
+
+[case testNamespacePackagePlainImport]
+# flags: --namespace-packages
+import foo.bar.baz
+reveal_type(foo.bar.baz.x)  # E: Revealed type is 'builtins.int'
+[file foo/bar/baz.py]
+x = 0
diff --git a/test-data/unit/cmdline.test b/test-data/unit/cmdline.test
--- a/test-data/unit/cmdline.test
+++ b/test-data/unit/cmdline.test
@@ -486,9 +486,9 @@ import a.b
 [file a.b.py]
 whatever
 [out]
-main.py:1: error: Cannot find module named 'a'
-main.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main.py:1: error: Cannot find module named 'a.b'
+main.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main.py:1: error: Cannot find module named 'a'
 
 [case testPythonVersionTooOld10]
 # cmd: mypy -c pass
diff --git a/test-data/unit/fine-grained-modules.test b/test-data/unit/fine-grained-modules.test
--- a/test-data/unit/fine-grained-modules.test
+++ b/test-data/unit/fine-grained-modules.test
@@ -189,9 +189,9 @@ p.a.f(1)
 [file p/a.py.2]
 def f(x: str) -> None: pass
 [out]
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 ==
 main:2: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 
@@ -215,9 +215,9 @@ p.a.f(1)
 [file p/a.py.3]
 def f(x: str) -> None: pass
 [out]
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 ==
 main:1: error: Cannot find module named 'p.a'
 main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
@@ -232,13 +232,13 @@ p.a.f(1)
 def f(x: str) -> None: pass
 [file p/__init__.py.3]
 [out]
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
-==
-main:1: error: Cannot find module named 'p'
 main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
+==
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 ==
 main:2: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 
@@ -266,13 +266,13 @@ p.a.f(1)
 def f(x: str) -> None: pass
 [file p/__init__.py.3]
 [out]
-main:4: error: Cannot find module named 'p'
-main:4: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:4: error: Cannot find module named 'p.a'
-==
-main:4: error: Cannot find module named 'p'
 main:4: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:4: error: Cannot find module named 'p'
+==
 main:4: error: Cannot find module named 'p.a'
+main:4: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:4: error: Cannot find module named 'p'
 ==
 main:5: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 
@@ -839,9 +839,9 @@ def f(x: str) -> None: pass
 [out]
 main:2: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 ==
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 
 [case testDeletePackage2]
 import p
@@ -873,9 +873,9 @@ main:2: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 main:1: error: Cannot find module named 'p.a'
 main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 ==
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 
 [case testDeletePackage4]
 import p.a
@@ -888,13 +888,13 @@ def f(x: str) -> None: pass
 [out]
 main:2: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 ==
-main:1: error: Cannot find module named 'p'
-main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:1: error: Cannot find module named 'p.a'
-==
-main:1: error: Cannot find module named 'p'
 main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
+==
 main:1: error: Cannot find module named 'p.a'
+main:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:1: error: Cannot find module named 'p'
 
 [case testDeletePackage5]
 # cmd1: mypy main p/a.py p/__init__.py
@@ -911,13 +911,13 @@ def f(x: str) -> None: pass
 [out]
 main:6: error: Argument 1 to "f" has incompatible type "int"; expected "str"
 ==
-main:5: error: Cannot find module named 'p'
-main:5: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:5: error: Cannot find module named 'p.a'
-==
-main:5: error: Cannot find module named 'p'
 main:5: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:5: error: Cannot find module named 'p'
+==
 main:5: error: Cannot find module named 'p.a'
+main:5: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:5: error: Cannot find module named 'p'
 
 
 [case testDeletePackage6]
diff --git a/test-data/unit/semanal-errors.test b/test-data/unit/semanal-errors.test
--- a/test-data/unit/semanal-errors.test
+++ b/test-data/unit/semanal-errors.test
@@ -279,11 +279,11 @@ main:4: error: Name 'n' is not defined
 import typing
 import m.n
 [out]
-main:2: error: Cannot find module named 'm'
-main:2: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:2: error: Cannot find module named 'm.n'
+main:2: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:2: error: Cannot find module named 'm'
 
-[case testMissingPackage]
+[case testMissingPackage2]
 import typing
 from m.n import x
 from a.b import *
@@ -318,9 +318,9 @@ m.n.x
 [file m/n.py]
 x = 1
 [out]
-main:2: error: Cannot find module named 'm'
-main:2: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
 main:2: error: Cannot find module named 'm.n'
+main:2: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
+main:2: error: Cannot find module named 'm'
 
 [case testBreakOutsideLoop]
 break