Repository: python/mypy

Issue:
Incremental mode crashes on `#type: ignore` comment on module when it was previously imported
Suppose you have the following two files:

**a.py**

``` python
import b  # type: ignore
```

**b.py**

``` python
# This file can be empty or non-empty
```

If you delete `.mypy_cache` then run `mypy -i b.py a.py` twice, you obtain the following error the second time you run mypy:

```
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/michaellee/Work/mypy/mypy/__main__.py", line 5, in <module>
    main(None)
  File "/Users/michaellee/Work/mypy/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/michaellee/Work/mypy/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/Users/michaellee/Work/mypy/mypy/build.py", line 178, in build
    dispatch(sources, manager)
  File "/Users/michaellee/Work/mypy/mypy/build.py", line 1339, in dispatch
    process_graph(graph, manager)
  File "/Users/michaellee/Work/mypy/mypy/build.py", line 1480, in process_graph
    process_fresh_scc(graph, scc)
  File "/Users/michaellee/Work/mypy/mypy/build.py", line 1539, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "/Users/michaellee/Work/mypy/mypy/build.py", line 1202, in fix_cross_refs
    fixup_module_pass_one(self.tree, self.manager.modules)
  File "/Users/michaellee/Work/mypy/mypy/fixup.py", line 17, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "/Users/michaellee/Work/mypy/mypy/fixup.py", line 75, in visit_symbol_table
    stnode = lookup_qualified_stnode(self.modules, cross_ref)
  File "/Users/michaellee/Work/mypy/mypy/fixup.py", line 237, in lookup_qualified_stnode
    assert '.' in head, "Cannot find %s" % (name,)
AssertionError: Cannot find b
```

However, just running `mypy -i a.py` twice or swapping the order of the files and running `mypy -a a.py b.py` appears to both work fine -- it appears that incremental mode does not behave correctly when it first parses a module normally then encounters it a second time when it's annotated with `# type: ignore`.

Fix crash with # type: ignore in incremental mode
This commit fixes the bug described in https://github.com/python/mypy/issues/1904.

More precisely, the bug manifests when...
1. You import a module normally
2. From another file, you import the same module, but mark it with the `# type: ignore` comment
3. You run mypy twice in this setup in incremental mode, making sure mypy encounters the `#type: ignore` comment after the regular import.

The bug appears to stem from how the semantic analyzer seemed to always mark import symbols as being of kind `MODULE_REF` and did not preserve whether the module was previously marked with the `# type: ignore` comment or not.

This caused a contradiction since how mypy decides to order dependencies differs whether you're starting with a cold cache or a warm cache.

When you start with a cold cache, mypy determines what each module's dependencies are based roughly on the order mypy encounters each file while parsing and whether the import is marked as ignorable or not.

However, when you start with a warm cache, mypy will use the previously computed list of dependencies, but will also attempt to typecheck each symbol marked as being of kind `MODULE_REF`. This presents a problem, since if the module was marked as being ignorable, it wouldn't be in the list of dependencies, but since it was marked as `MODULE_REF`, mypy assumes it is...

This pull request fixes this problem by modifying the semantic analyzer so that it determines whether it needs to mark import symbols of being kind `MODULE_REF` or `UNBOUND_IMPORTED` to keep the behavior consistent during both runs.



Discussion:
Could be caused by #1911?

I'm holding off reviewing this pending a fix for #1911.


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -416,23 +416,17 @@ def parse_file(self, id: str, path: str, source: str) -> MypyFile:
         tree = parse(source, path, self.errors, options=self.options)
         tree._fullname = id
 
-        # We don't want to warn about 'type: ignore' comments on
-        # imports, but we're about to modify tree.imports, so grab
-        # these first.
-        import_lines = set(node.line for node in tree.imports)
-
-        # Skip imports that have been ignored (so that we can ignore a C extension module without
-        # stub, for example), except for 'from x import *', because we wouldn't be able to
-        # determine which names should be defined unless we process the module. We can still
-        # ignore errors such as redefinitions when using the latter form.
-        imports = [node for node in tree.imports
-                   if node.line not in tree.ignored_lines or isinstance(node, ImportAll)]
-        tree.imports = imports
-
         if self.errors.num_messages() != num_errs:
             self.log("Bailing due to parse errors")
             self.errors.raise_error()
 
+        # We don't want to warn about unused 'type: ignore' comments on imports.
+        # That way, if we want to conditionally import a module that is valid
+        # only on one particular platform, we can silence the import so we don't
+        # get spurious error messages when mypy tries checking it. For example,
+        # we might want to conditionally import a Python 2 only module while
+        # checking the file as Python 3.
+        import_lines = set(node.line for node in tree.imports)
         self.errors.set_file_ignored_lines(path, tree.ignored_lines)
         self.errors.mark_file_ignored_lines_used(path, import_lines)
         return tree


Test patch:
diff --git a/test-data/unit/check-ignore.test b/test-data/unit/check-ignore.test
--- a/test-data/unit/check-ignore.test
+++ b/test-data/unit/check-ignore.test
@@ -36,7 +36,23 @@ x   # E: Name 'x' is not defined
 import m # type: ignore
 from m import a # type: ignore
 [file m.py]
-+ # A parse error, but we shouldn't even parse m.py
++
+[out]
+main:1: note: In module imported here:
+tmp/m.py:1: error: Parse error before end of line
+
+[case testIgnoreAppliesOnlyToMissing]
+import a # type: ignore
+import b # type: ignore
+reveal_type(a.foo) # E: Revealed type is 'Any'
+reveal_type(b.foo) # E: Revealed type is 'builtins.int'
+a.bar()
+b.bar() # E: "module" has no attribute "bar"
+
+[file b.py]
+foo = 3
+
+[builtins fixtures/module_all.py]
 [out]
 
 [case testIgnoreImportStarFromBadModule]
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -185,3 +185,65 @@ from parent import b
 
 [stale parent.a]
 [out]
+
+[case testIncrementalWithTypeIgnoreOnDirectImport]
+import a, b
+
+[file a.py]
+import b  # type: ignore
+
+[file b.py]
+import c
+
+[file c.py]
+
+[stale]
+[out]
+
+[case testIncrementalWithTypeIgnoreOnImportFrom]
+import a, b
+
+[file a.py]
+from b import something # type: ignore
+
+[file b.py]
+import c
+something = 3
+
+[file c.py]
+
+[stale]
+[out]
+
+[case testIncrementalWithPartialTypeIgnore]
+import a  # type: ignore
+import a.b
+
+[file a/__init__.py]
+
+[file a/b.py]
+
+[stale]
+[out]
+
+[case testIncrementalAnyIsDifferentFromIgnore]
+import b
+
+[file b.py]
+from typing import Any
+import a.b
+
+[file b.py.next]
+from typing import Any
+
+a = 3  # type: Any
+import a.b
+
+[file a/__init__.py]
+
+[file a/b.py]
+
+[stale b]
+[out]
+main:1: note: In module imported here:
+tmp/b.py:4: error: Name 'a' already defined