Repository: python/mypy

Issue:
"KeyError: 'configuration'" error in incremental mode
I'm not sure whether the name "configuration" is specific to our code (we do have a module called `configuration`) or is it mypy-internal thing.

CPython 3.5.1, mypy a284c48, command: `python -m mypy --silent-imports --incremental --fast-parser BUNCH OF MODULES AND PACKAGES`, result:

```
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/bin/mypy", line 6, in <module>
    main(__file__)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 180, in build
    dispatch(sources, manager)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1335, in dispatch
    process_graph(graph, manager)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1424, in process_graph
    stale_deps = {id for id in deps if not graph[id].is_fresh()}
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1424, in <setcomp>
    stale_deps = {id for id in deps if not graph[id].is_fresh()}
KeyError: 'configuration'
```

I've backed up `.mypy_cache`, I'll see whether I can reproduce this reliably and provide a minimal test case.



Discussion:
Can you try again with the latest master from the repo, after wiping the cache? Michael has fixed a few things in this area, specifically rev c6f58b3 (the most recent HEAD).

So, two things:
- "configuration" there is a name of one of our modules
- wiping the cache will help even with the version we're using (at least temporarily), I'm trying to figure out what in the cache generates this failure

Yeah, wiping the cache _always_ helps for incremental-related crashes. :-)
Please do check out the very latest, Michael made another fix.

Ping?

I'm sorry, didin't have time to reproduce/test/fiddle with it and I won't have a chance to do it in the near future, let me just close this issue and reopen in case I can repoduce it with more recent mypy.

Thanks for keeping our tracker clean!

--Guido (mobile)

I've gotten another (internal) report of an identical crash, so I'm going to reopen this. 

The internal report also gave me a full dump of the `.mypy_cache` from the crash.

The key thing of interest here seems to be that the module whose name appears in the KeyError message (a different one for our internal report) is not listed on the command line and hence does not get analyzed (because of `--silent-imports`).

Another observation: in the dump, a certain module (a _sibling_ of the KeyError module) was suppressed. But in the crashing run, that module was added to the command line. Hm....

OK, here's a repro.

```
rm -rf .mypy_cache a
mkdir a
cat >a/__init__.py <<EOF
from .b import B
from .c import C
EOF
cat >a/b.py <<EOF
class B: pass
EOF
cat >a/c.py <<EOF
class C: pass
EOF
mypy --silent-imports -i a/__init__.py
echo '#' >>a/c.py
mypy --silent-imports -i a/__init__.py a/c.py
```

The last line crashes with

```
KeyError: 'a.b'
```

Note that a/b.py is not checked, and it's not updated, but on the second run its name (`'a.b'`) somehow ends up in `deps`.

I also have a fix, just before the offending line in build.py (`stale_deps = ...`) insert

```
deps &= set(graph)
```

But I am not satisfied, because I don't understand the cause of the crash, I can just prevent it.

@Michael0x2a what do you think? Here's the test case:

```
[case testKeyErrorCrash]
# cmd: mypy -m a
# cmd2: mypy -m a a.c
# flags: --silent-imports
[file a/__init__.py]
from .b import B
from .c import C
[file a/b.py]
class B: pass
[file a/c.py]
class C: pass
[file a/c.py.next]
class C: pass
pass
```

**UPDATE the second:** The above test case reproduces the KeyError.


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -1536,6 +1536,7 @@ def load_graph(sources: List[BuildSource], manager: BuildManager) -> Graph:
     for id, g in graph.items():
         if g.has_new_submodules():
             g.parse_file()
+            g.fix_suppressed_dependencies(graph)
             g.mark_interface_stale()
     return graph
 


Test patch:
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -1567,3 +1567,26 @@ class Outer:
 [out2]
 main:1: note: In module imported here:
 tmp/foo.py:3: error: Argument 1 to "accept_int" has incompatible type "str"; expected "int"
+
+[case testIncrementalPartialSubmoduleUpdate]
+# cmd: mypy -m a
+# cmd2: mypy -m a a.c
+# flags: --silent-imports
+
+[file a/__init__.py]
+from .b import B
+from .c import C
+
+[file a/b.py]
+class B: pass
+
+[file a/c.py]
+class C: pass
+
+[file a/c.py.next]
+class C: pass
+pass
+
+[rechecked a, a.c]
+[stale a, a.c]
+[out]