Repository: python/mypy

Issue:
Incremental bug involving aliases and/or nested modules or classes and/or import cycles
I'm encountering a complex bug in incremental. One of the ways it appears is spurious errors complaining about some types in the requests stubs, e.g.

```
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/__init__.pyi:1: note: In module imported here,
(somewhere)/main.py:1: note: ... from here:
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi: note: At top level:
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi:59: error: Invalid type "requests.sessions.Request"
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi:61: error: Invalid type "requests.sessions.Request"
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi:68: error: Invalid type "requests.sessions.RequestsCookieJar"
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi: note: In function "request":
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi:75: error: Invalid type "requests.sessions.RequestsCookieJar"
/Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi:75: error: Invalid type "requests.sessions.Request"
```

There's no need to panic but I'd like to have an issue I can point to.



Discussion:
The issue appears elusive. I wonder if it's related to #2126 (the second bullet)? I may have run mypy with strict optional and that may have corrupted the cache.

(Issue #2126 looking less likely now that I've seen the exact same set of errors from two different repos.)

This is almost certainly related to import cycles; the error "Invalid type" almost exclusively stems from those.

I've got a repro!

```
bash-3.2$ my -2 -i -m requests.sessions
bash-3.2$ touch typeshed/third_party/2.7/requests/sessions.pyi 
bash-3.2$ my -2 -i -m requests.sessions
typeshed/third_party/2.7/requests/sessions.pyi:59: error: Invalid type "requests.sessions.Request"
typeshed/third_party/2.7/requests/sessions.pyi:61: error: Invalid type "requests.sessions.Request"
typeshed/third_party/2.7/requests/sessions.pyi:68: error: Invalid type "requests.sessions.RequestsCookieJar"
typeshed/third_party/2.7/requests/sessions.pyi: note: In function "request":
typeshed/third_party/2.7/requests/sessions.pyi:75: error: Invalid type "requests.sessions.RequestsCookieJar"
typeshed/third_party/2.7/requests/sessions.pyi:75: error: Invalid type "requests.sessions.Request"
```

When run with `-v`, the second run contains the log line

```
LOG:  Metadata abandoned for requests.sessions: file /Users/guido/src/mypy/typeshed/third_party/2.7/requests/sessions.pyi is modified
```

(which is correct, since that file was touched). This is a clue that something not quite right has ended up in the cache. I'll investigate...

Shorter repro, involving a package with just three minimal files:

```
==> requests/__init__.pyi <==
from . import sessions

==> requests/models.pyi <==
class Request:
    pass

==> requests/sessions.pyi <==
from . import models
Request = models.Request
a = ... # type: Request
```

Then run

```
rm -rf .mypy_cache
mypy -i -m requests.sessions
touch requests/sessions.pyi
mypy -i -m requests.sessions
```

This produces one error message on the second run:

```
requests/sessions.pyi:3: error: Invalid type "requests.sessions.Request"
```

Some more of the story:
- With a cold cache, when SemanticAnalyzer.visit_import_from() processes the line "from . import models" in sessions.py, it finds self.modules['requests'] existing, and makes the models variable a MODULE_REF to requests.models.
- But with a warm cache, at that point self.modules doesn't have the key 'requests' yet, and it skips this code, instead calling self.add_unknown_symbol(). This then cascades into the error message.
- The reason appears that with a cold cache, `requests/__init__.pyi` gets parsed and modules['requests'] gets updated early (right after requests/sessions.pyi is parsed). But with a warm cache modules['requests'] gets updated too late, after requests.sessions is already fully processed.

I've got a possible fix at https://github.com/python/mypy/tree/fix2133. Not sure whether I like it that much. The solution inserts None entries into the modules dict just so that SemanticAnalyzer.visit_import_from() can tell whether a module is supposed to exist or not.

An alternative fix would be to add the parent package to the submodule's dependencies with a low priority, rather than the current approach of maintaining a separate list of ancestors which are taken into account sometimes but not other times. I haven't tried to code that up yet to see how it would go.

@Michael0x2a Any thoughts about this? It's taken me all day Friday to get this far...

@gvanrossum --ah, sorry. I meant to respond sooner, but got side-tracked by grading :( 

I'll try and take a look tonight/tomorrow morning.

The alt fix also misses tests; I am still meditating on which fix I like better.

Note that there would also be a cop-out way to fix this, by making some subtle changes to requests/sessions.pyi. But that would just keep the bug in incremental until the next time someone has a package with a similar structure.

The cost of increased circularity is real. The alt fix puts requests and 7 submodules in a single SCC (requests.cookies requests.utils requests.auth requests.models requests.api requests.adapters requests.sessions requests) where there were previously 8 independent singletons.

There's yet another fix I'd like to try. It's more similar to the first version, making subtle changes to SemanticAnalyzer.visit_import_from() in case a module is deemed to exist but hasn't been processed yet. But instead of pre-populating the  modules dict with None values, it should use a separate data structure to indicate whether the module exists or not (e.g. derived from graph.keys()).


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -774,7 +774,7 @@ def is_meta_fresh(meta: Optional[CacheMeta], id: str, path: str, manager: BuildM
     st = manager.get_stat(path)  # TODO: Errors
     if st.st_mtime != meta.mtime or st.st_size != meta.size:
         manager.log('Metadata abandoned for {}: file {} is modified'.format(id, path))
-        return None
+        return False
 
     # It's a match on (id, path, mtime, size).
     # Check data_json; assume if its mtime matches it's good.
@@ -1650,7 +1650,11 @@ def process_graph(graph: Graph, manager: BuildManager) -> None:
             process_stale_scc(graph, scc)
 
     sccs_left = len(fresh_scc_queue)
-    manager.log("{} fresh SCCs left in queue (and will remain unprocessed)".format(sccs_left))
+    if sccs_left:
+        manager.log("{} fresh SCCs left in queue (and will remain unprocessed)".format(sccs_left))
+        manager.trace(str(fresh_scc_queue))
+    else:
+        manager.log("No fresh SCCs left in queue")
 
 
 def order_ascc(graph: Graph, ascc: AbstractSet[str], pri_max: int = PRI_ALL) -> List[str]:
diff --git a/mypy/semanal.py b/mypy/semanal.py
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -913,11 +913,13 @@ def add_submodules_to_parent_modules(self, id: str, module_public: bool) -> None
         """
         while '.' in id:
             parent, child = id.rsplit('.', 1)
-            modules_loaded = parent in self.modules and id in self.modules
-            if modules_loaded and child not in self.modules[parent].names:
-                sym = SymbolTableNode(MODULE_REF, self.modules[id], parent,
-                        module_public=module_public)
-                self.modules[parent].names[child] = sym
+            parent_mod = self.modules.get(parent)
+            if parent_mod and child not in parent_mod.names:
+                child_mod = self.modules.get(id)
+                if child_mod:
+                    sym = SymbolTableNode(MODULE_REF, child_mod, parent,
+                                          module_public=module_public)
+                    parent_mod.names[child] = sym
             id = parent
 
     def add_module_symbol(self, id: str, as_id: str, module_public: bool,
@@ -931,48 +933,47 @@ def add_module_symbol(self, id: str, as_id: str, module_public: bool,
 
     def visit_import_from(self, imp: ImportFrom) -> None:
         import_id = self.correct_relative_import(imp)
-        if import_id in self.modules:
-            module = self.modules[import_id]
-            self.add_submodules_to_parent_modules(import_id, True)
-            for id, as_id in imp.names:
-                node = module.names.get(id)
-
-                # If the module does not contain a symbol with the name 'id',
-                # try checking if it's a module instead.
-                if id not in module.names or node.kind == UNBOUND_IMPORTED:
-                    possible_module_id = import_id + '.' + id
-                    mod = self.modules.get(possible_module_id)
-                    if mod is not None:
-                        node = SymbolTableNode(MODULE_REF, mod, import_id)
-                        self.add_submodules_to_parent_modules(possible_module_id, True)
-
-                if node and node.kind != UNBOUND_IMPORTED:
-                    node = self.normalize_type_alias(node, imp)
-                    if not node:
-                        return
-                    imported_id = as_id or id
-                    existing_symbol = self.globals.get(imported_id)
-                    if existing_symbol:
-                        # Import can redefine a variable. They get special treatment.
-                        if self.process_import_over_existing_name(
-                                imported_id, existing_symbol, node, imp):
-                            continue
-                    # 'from m import x as x' exports x in a stub file.
-                    module_public = not self.is_stub_file or as_id is not None
-                    symbol = SymbolTableNode(node.kind, node.node,
-                                             self.cur_mod_id,
-                                             node.type_override,
-                                             module_public=module_public)
-                    self.add_symbol(imported_id, symbol, imp)
-                else:
-                    message = "Module '{}' has no attribute '{}'".format(import_id, id)
-                    extra = self.undefined_name_extra_info('{}.{}'.format(import_id, id))
-                    if extra:
-                        message += " {}".format(extra)
-                    self.fail(message, imp)
-        else:
-            # Missing module.
-            for id, as_id in imp.names:
+        self.add_submodules_to_parent_modules(import_id, True)
+        module = self.modules.get(import_id)
+        for id, as_id in imp.names:
+            node = module.names.get(id) if module else None
+
+            # If the module does not contain a symbol with the name 'id',
+            # try checking if it's a module instead.
+            if not node or node.kind == UNBOUND_IMPORTED:
+                possible_module_id = import_id + '.' + id
+                mod = self.modules.get(possible_module_id)
+                if mod is not None:
+                    node = SymbolTableNode(MODULE_REF, mod, import_id)
+                    self.add_submodules_to_parent_modules(possible_module_id, True)
+
+            if node and node.kind != UNBOUND_IMPORTED:
+                node = self.normalize_type_alias(node, imp)
+                if not node:
+                    return
+                imported_id = as_id or id
+                existing_symbol = self.globals.get(imported_id)
+                if existing_symbol:
+                    # Import can redefine a variable. They get special treatment.
+                    if self.process_import_over_existing_name(
+                            imported_id, existing_symbol, node, imp):
+                        continue
+                # 'from m import x as x' exports x in a stub file.
+                module_public = not self.is_stub_file or as_id is not None
+                symbol = SymbolTableNode(node.kind, node.node,
+                                         self.cur_mod_id,
+                                         node.type_override,
+                                         module_public=module_public)
+                self.add_symbol(imported_id, symbol, imp)
+            elif module:
+                # Missing attribute.
+                message = "Module '{}' has no attribute '{}'".format(import_id, id)
+                extra = self.undefined_name_extra_info('{}.{}'.format(import_id, id))
+                if extra:
+                    message += " {}".format(extra)
+                self.fail(message, imp)
+            else:
+                # Missing module.
                 self.add_unknown_symbol(as_id or id, imp, is_import=True)
 
     def process_import_over_existing_name(self,


Test patch:
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -1615,3 +1615,22 @@ from funcs import callee
 from classes import Outer
 def caller(a: Outer.Inner) -> int:
     callee(a)
+
+[case testIncrementalLoadsParentAfterChild]
+# cmd: mypy -m r.s
+
+[file r/__init__.py]
+from . import s
+
+[file r/m.py]
+class R: pass
+
+[file r/s.py]
+from . import m
+R = m.R
+a = None  # type: R
+
+[file r/s.py.next]
+from . import m
+R = m.R
+a = None  # type: R