Repository: python/mypy

Issue:
incremental: Crash after deleting a function.
I ran into this crash with 0.501 today on the Zulip codebase when running mypy with this version of the code https://github.com/zulip/zulip/commit/09f66b5c6dfc5c8bf5d52fb51259189ade0f336e (which deleted the below function).  

I've attached the cache state tarball in case it's helpful: [mypy.tar.gz](https://github.com/python/mypy/files/866606/mypy.tar.gz)


```
Traceback (most recent call last):
  File "/srv/zulip-py3-venv/bin/mypy", line 6, in <module>
    main(__file__)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 42, in main
    res = type_check_only(sources, bin_dir, options)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 86, in type_check_only
    options=options)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 183, in build
    dispatch(sources, manager)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1531, in dispatch
    process_graph(graph, manager)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1761, in process_graph
    process_fresh_scc(graph, prev_scc)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1830, in process_fresh_scc
    graph[id].fix_cross_refs()
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1311, in fix_cross_refs
    fixup_module_pass_one(self.tree, self.manager.modules)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/fixup.py", line 21, in fixup_module_pass_one
    node_fixer.visit_symbol_table(tree.names)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/fixup.py", line 79, in visit_symbol_table
    stnode = lookup_qualified_stnode(self.modules, cross_ref)
  File "/srv/zulip-venv-cache/015153c2b7c837e69f696afeaa7df4d23e89fed1/zulip-py3-venv/lib/python3.5/site-packages/mypy/fixup.py", line 255, in lookup_qualified_stnode
    assert key in names, "Cannot find %s for %s" % (key, name)
AssertionError: Cannot find do_set_realm_description for zerver.lib.actions.do_set_realm_description
```




Discussion:
FWIW, I couldn't actually reproduce this with (inside the Zulip directory):

```
rm -rf var/mypy_cache
git checkout 09f66b5c6dfc5c8bf5d52fb51259189ade0f336e^
tools/run-mypy
git checkout 09f66b5c6dfc5c8bf5d52fb51259189ade0f336e
tools/run-mypy
```
Yeah the missing info is all the mtimes of the files in Tim's working directory when it failed. :-(
(FWIW if this occurs again on an open source package, as of mypy 0.521 the mtimes in the `.mypy_cache` directory are no longer a problem, as we now use a hash of the source file.)
I think @hackerkid saw this earlier today in the Zulip project; but we may be a bit behind 0.521 (haven't updated since PyCon).
I got the error after I switched from a branch [invite-link](https://github.com/hackerkid/zulip/tree/mypy-debug-invite-link) which had the model `MultiUserInvitation` to the branch [get-user-2g](https://github.com/hackerkid/zulip/tree/mypy-debug-get-user-2g) which didn't had the model. But interestingly I was not able to reproduce the error when I switched from [invite-link](https://github.com/hackerkid/zulip/tree/mypy-debug-invite-link) to [master](mypy-debug-master). I tried the switching multiple times and was able to obtain the same behavior. All the branches were rebased on top of upstream/master. 

During each switch I did
* `./tools/provision`
* `./tools/run-mypy `

 **I have attached the tar of my Zulip directory including the cache [here](https://drive.google.com/open?id=0B5Ux_wsMN5N-c0tTRlA5cmRheTg).** 

**Update**: fixed the get-user-2g-link. Thanks **ethanhs** 
(the correct [get-user-2g link](https://github.com/hackerkid/zulip/tree/mypy-debug-get-user-2g))

@hackerkid did this happen consistently? Could you try updating the version of Mypy that you use? It appears your version is old.

I downloaded the tarball but could not repro the issue with mypy 0.521, so I think it's not worth looking into further. Let us know whether you still see this with master.
Hmm, I've definitely seen this since 0.521.  I just upgraded to the latest mypy master in case that helps; will let you know when I next see it (it's about once a week, usually after a series of switches between branches including one that deletes things).
If and when you see it, tarring up your entire zulip tree (.git and all) like @hackerkid did would be very helpful.
We've seen a similar crash internally, where the assert referenced a class that was deleted by the change. Sadly it was also impossible to repro. Which means that at least the work-around of deleting the cache shouldn't be too painful.
An internal user claims this can be reproduced as follows:

0. Start with a clean slate
1. Run mypy
2. Make an edit
3. Run it again, with `-i`
4. Revert the edit (e.g. `git stash`)
5. Run it again, with `-i` -- BOOM

The edit in step 2 could be a whitespace edit, as long as it's a type-checked file.
@gvanrossum I now have a simple repro for this crash inspired by #4043. Here is the recipe:

Initial setup:
```python
# a.py
from b import x

# b.py
from c import x

# c.py
x = 1
```
with any previous mypy caches removed. Steps to reproduce:
* run ``mypy a.py``
* delete (or comment out) definition of ``x`` in ``c.py``
* run ``mypy -i a.py`` (it will show an obvious error)
* add random whitespace to ``a.py``
* run ``mypy -i a.py`` again -- ``AssertionError: Cannot find x for c.x``
Thanks! Adding an unused variable definition to a.py in the 4th step will work too. I've also confirmed that this crash was introduced by the same PR (#2014) that caused #4043. I haven't played with fixes yet.

Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -21,6 +21,7 @@
 import sys
 import time
 from os.path import dirname, basename
+import errno
 
 from typing import (AbstractSet, Dict, Iterable, Iterator, List, cast, Any,
                     NamedTuple, Optional, Set, Tuple, Union, Callable)
@@ -469,7 +470,7 @@ class BuildManager:
       all_types:       Map {Expression: Type} collected from all modules
       options:         Build options
       missing_modules: Set of modules that could not be imported encountered so far
-      stale_modules:   Set of modules that needed to be rechecked
+      stale_modules:   Set of modules that needed to be rechecked (only used by tests)
       version_id:      The current mypy version (based on commit id when possible)
       plugin:          Active mypy plugin(s)
       errors:          Used for reporting all errors
@@ -1165,6 +1166,25 @@ def write_cache(id: str, path: str, tree: MypyFile,
     return interface_hash
 
 
+def delete_cache(id: str, path: str, manager: BuildManager) -> None:
+    """Delete cache files for a module.
+
+    The cache files for a module are deleted when mypy finds errors there.
+    This avoids inconsistent states with cache files from different mypy runs,
+    see #4043 for an example.
+    """
+    path = os.path.abspath(path)
+    meta_json, data_json = get_cache_names(id, path, manager)
+    manager.log('Deleting {} {} {} {}'.format(id, path, meta_json, data_json))
+
+    for filename in [data_json, meta_json]:
+        try:
+            os.remove(filename)
+        except OSError as e:
+            if e.errno != errno.ENOENT:
+                manager.log("Error deleting cache file {}: {}".format(filename, e.strerror))
+
+
 """Dependency manager.
 
 Design
@@ -1534,11 +1554,12 @@ def mark_as_rechecked(self) -> None:
         """Marks this module as having been fully re-analyzed by the type-checker."""
         self.manager.rechecked_modules.add(self.id)
 
-    def mark_interface_stale(self) -> None:
+    def mark_interface_stale(self, *, on_errors: bool = False) -> None:
         """Marks this module as having a stale public interface, and discards the cache data."""
         self.meta = None
         self.externally_same = False
-        self.manager.stale_modules.add(self.id)
+        if not on_errors:
+            self.manager.stale_modules.add(self.id)
 
     def check_blockers(self) -> None:
         """Raise CompileError if a blocking error is detected."""
@@ -1813,6 +1834,8 @@ def write_cache(self) -> None:
         else:
             is_errors = self.manager.errors.is_errors()
         if is_errors:
+            delete_cache(self.id, self.path, self.manager)
+            self.mark_interface_stale(on_errors=True)
             return
         dep_prios = [self.priorities.get(dep, PRI_HIGH) for dep in self.dependencies]
         new_interface_hash = write_cache(


Test patch:
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -27,7 +27,8 @@
 -- Any files that we expect to be rechecked should be annotated in the [rechecked]
 -- annotation, and any files expect to be stale (aka have a modified interface)
 -- should be annotated in the [stale] annotation. Note that a file that ends up
--- producing an error does not create a new cache file and so is not considered stale.
+-- producing an error has its caches deleted and is marked stale automatically.
+-- Such files don't need to be included in [stale ...] list.
 --
 -- The test suite will automatically assume that __main__ is stale and rechecked in
 -- all cases so we can avoid constantly having to annotate it. The list of
@@ -200,7 +201,7 @@ def foo() -> int:
         return "foo"
     return inner2()
 
-[rechecked mod2]
+[rechecked mod1, mod2]
 [stale]
 [out2]
 tmp/mod2.py:4: error: Incompatible return value type (got "str", expected "int")
@@ -2800,6 +2801,87 @@ b.x.y
 tmp/c.py:2: error: Revealed type is '<stale cache: consider running mypy without --quick>'
 tmp/c.py:5: error: "<stale cache: consider running mypy without --quick>" has no attribute "y"
 
+[case testCacheDeletedAfterErrorsFound]
+import a
+[file a.py]
+from b import x
+[file b.py]
+from c import x
+[file c.py]
+x = 1
+[file c.py.2]
+1 + 1
+[file a.py.3]
+from b import x
+1 + 1
+[out]
+[out2]
+tmp/b.py:1: error: Module 'c' has no attribute 'x'
+tmp/a.py:1: error: Module 'b' has no attribute 'x'
+[out3]
+tmp/b.py:1: error: Module 'c' has no attribute 'x'
+tmp/a.py:1: error: Module 'b' has no attribute 'x'
+
+[case testCacheDeletedAfterErrorsFound2]
+import a
+[file a.py]
+from b import x
+[file b.py]
+from c import C
+x: C
+[file c.py]
+class C: pass
+[file c.py.2]
+def C(): pass
+[file a.py.3]
+from b import x
+1 + 1
+[out]
+[out2]
+tmp/b.py:2: error: Invalid type "c.C"
+[out3]
+tmp/b.py:2: error: Invalid type "c.C"
+
+[case testCacheDeletedAfterErrorsFound3]
+import a
+[file a.py]
+import b
+b.f()
+[file b.py]
+def f() -> None: pass
+[file b.py.2]
+def f(x) -> None: pass
+[out]
+[out2]
+tmp/a.py:2: error: Too few arguments for "f"
+[out3]
+tmp/a.py:2: error: Too few arguments for "f"
+
+[case testCacheDeletedAfterErrorsFound4]
+import a
+[file a.py]
+from b import x
+[file b.py]
+from c import x
+[file c.py]
+from d import x
+[file d.py]
+x = 1
+[file d.py.2]
+1 + 1
+[file a.py.3]
+from b import x
+1 + 1
+[out]
+[out2]
+tmp/c.py:1: error: Module 'd' has no attribute 'x'
+tmp/b.py:1: error: Module 'c' has no attribute 'x'
+tmp/a.py:1: error: Module 'b' has no attribute 'x'
+[out3]
+tmp/c.py:1: error: Module 'd' has no attribute 'x'
+tmp/b.py:1: error: Module 'c' has no attribute 'x'
+tmp/a.py:1: error: Module 'b' has no attribute 'x'
+
 [case testNoCrashOnDoubleImportAliasQuick]
 # cmd: mypy -m e
 # cmd2: mypy -m c