Repository: python/mypy

Issue:
Internal error: "TypeError: 'NoneType' object is not iterable" in incremental mode
Python 3.5.2, mypy 2e2a94e30724f061a835eb2f72a89115250a37ee

Steps to reproduce (structure extracted from a larger application, I could probably simplify it further but this should be good enough):

```
mkdir -p testdir/package/subpackage
touch \
    testdir/package/__init__.py \
    testdir/package/subpackage/__init__.py \
    testdir/package/subpackage/mod2.py

echo "from package.subpackage.mod1 import Class

def handle(c: Class) -> None:
    c.some_attribute" > testdir/main.py

echo "import six

class Class(object): pass" > testdir/package/subpackage/mod1.py

rm -rf .mypy_cache

python -m mypy --silent-imports --incremental testdir/main.py testdir/package/subpackage/mod2.py
python -m mypy --show-traceback --silent-imports --incremental testdir/main.py testdir/package/subpackage/mod1.py
```

Result:

```
testdir/main.py:4: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/__main__.py", line 5, in <module>
    main(None)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/build.py", line 180, in build
    dispatch(sources, manager)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/build.py", line 1335, in dispatch
    process_graph(graph, manager)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/build.py", line 1478, in process_graph
    process_stale_scc(graph, scc)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/build.py", line 1555, in process_stale_scc
    graph[id].type_check()
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/build.py", line 1313, in type_check
    manager.type_checker.visit_file(self.tree, self.xpath)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 152, in visit_file
    self.accept(d)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 201, in accept
    typ = node.accept(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/nodes.py", line 477, in accept
    return visitor.visit_func_def(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 408, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 471, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 591, in check_func_def
    self.accept(item.body)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 201, in accept
    typ = node.accept(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/nodes.py", line 727, in accept
    return visitor.visit_block(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 999, in visit_block
    self.accept(s)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 201, in accept
    typ = node.accept(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/nodes.py", line 741, in accept
    return visitor.visit_expression_stmt(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 1437, in visit_expression_stmt
    self.accept(s.expr)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 201, in accept
    typ = node.accept(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/nodes.py", line 1157, in accept
    return visitor.visit_member_expr(self)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checker.py", line 1978, in visit_member_expr
    return self.expr_checker.visit_member_expr(e)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checkexpr.py", line 861, in visit_member_expr
    result = self.analyze_ordinary_member_access(e, False)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checkexpr.py", line 874, in analyze_ordinary_member_access
    self.named_type, self.not_ready_callback, self.msg)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checkmember.py", line 77, in analyze_member_access
    report_type=report_type)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checkmember.py", line 174, in analyze_member_var_access
    v = lookup_member_var_or_accessor(info, name, is_lvalue)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/checkmember.py", line 267, in lookup_member_var_or_accessor
    node = info.get(name)
  File "/Users/me/ve/35/lib/python3.5/site-packages/mypy/nodes.py", line 1853, in get
    for cls in self.mro:
TypeError: 'NoneType' object is not iterable
testdir/main.py:4: note: use --pdb to drop into pdb
```



Discussion:
Thanks for the detailed bug report!  I was able to repro following your steps.  We'll look into it!


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -1359,7 +1359,7 @@ def load_graph(sources: List[BuildSource], manager: BuildManager) -> Graph:
     # Collect dependencies.  We go breadth-first.
     while new:
         st = new.popleft()
-        for dep in st.ancestors + st.dependencies:
+        for dep in st.ancestors + st.dependencies + st.suppressed:
             if dep not in graph:
                 try:
                     if dep in st.ancestors:
@@ -1380,6 +1380,11 @@ def load_graph(sources: List[BuildSource], manager: BuildManager) -> Graph:
                     new.append(newst)
             if dep in st.ancestors and dep in graph:
                 graph[dep].child_modules.add(st.id)
+            if dep in graph and dep in st.suppressed:
+                # Previously suppressed file is now visible
+                if dep in st.suppressed:
+                    st.suppressed.remove(dep)
+                    st.dependencies.append(dep)
     for id, g in graph.items():
         if g.has_new_submodules():
             g.parse_file()


Test patch:
diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py
--- a/mypy/test/testcheck.py
+++ b/mypy/test/testcheck.py
@@ -106,10 +106,10 @@ def clear_cache(self) -> None:
 
     def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
         find_module_clear_caches()
-        program_text = '\n'.join(testcase.input)
-        module_name, program_name, program_text = self.parse_module(program_text)
+        original_program_text = '\n'.join(testcase.input)
+        module_data = self.parse_module(original_program_text, incremental)
 
-        options = self.parse_options(program_text)
+        options = self.parse_options(original_program_text)
         options.use_builtins_fixtures = True
         options.python_version = testcase_pyversion(testcase.file, testcase.name)
         set_show_tb(True)  # Show traceback on crash.
@@ -120,8 +120,11 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
             if incremental == 1:
                 # In run 1, copy program text to program file.
                 output = []
-                with open(program_name, 'w') as f:
-                    f.write(program_text)
+                for module_name, program_path, program_text in module_data:
+                    if module_name == '__main__':
+                        with open(program_path, 'w') as f:
+                            f.write(program_text)
+                        break
             elif incremental == 2:
                 # In run 2, copy *.py.next files to *.py files.
                 for dn, dirs, files in os.walk(os.curdir):
@@ -137,11 +140,13 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
                             new_time = os.stat(target).st_mtime + 1
                             os.utime(target, times=(new_time, new_time))
 
-            # Always set to none so we're forced to reread program_name
-            program_text = None
-        source = BuildSource(program_name, module_name, program_text)
+        sources = []
+        for module_name, program_path, program_text in module_data:
+            # Always set to none so we're forced to reread the module in incremental mode
+            program_text = None if incremental else program_text
+            sources.append(BuildSource(program_path, module_name, program_text))
         try:
-            res = build.build(sources=[source],
+            res = build.build(sources=sources,
                               options=options,
                               alt_lib_path=test_temp_dir)
             a = res.errors
@@ -159,14 +164,15 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
                 testcase.file, testcase.line))
 
         if incremental and res:
-            self.verify_cache(module_name, program_name, a, res.manager)
+            if not options.silent_imports:
+                self.verify_cache(module_data, a, res.manager)
             if testcase.expected_stale_modules is not None and incremental == 2:
                 assert_string_arrays_equal(
                     list(sorted(testcase.expected_stale_modules)),
                     list(sorted(res.manager.stale_modules.difference({"__main__"}))),
                     'Set of stale modules does not match expected set')
 
-    def verify_cache(self, module_name: str, program_name: str, a: List[str],
+    def verify_cache(self, module_data: List[Tuple[str, str, str]], a: List[str],
                      manager: build.BuildManager) -> None:
         # There should be valid cache metadata for each module except
         # those in error_paths; for those there should not be.
@@ -176,7 +182,7 @@ def verify_cache(self, module_name: str, program_name: str, a: List[str],
         # However build.process_graphs() will ignore A's cache data.
         error_paths = self.find_error_paths(a)
         modules = self.find_module_files()
-        modules.update({module_name: program_name})
+        modules.update({module_name: path for module_name, path, text in module_data})
         missing_paths = self.find_missing_cache_files(modules, manager)
         if missing_paths != error_paths:
             raise AssertionFailure("cache data discrepancy %s != %s" %
@@ -218,28 +224,40 @@ def find_missing_cache_files(self, modules: Dict[str, str],
                 missing[id] = path
         return set(missing.values())
 
-    def parse_module(self, program_text: str) -> Tuple[str, str, str]:
+    def parse_module(self, program_text: str, incremental: int = 0) -> List[Tuple[str, str, str]]:
         """Return the module and program names for a test case.
 
-        The default ('__main__') module name can be overridden by
-        using a comment like this in the test case input:
+        Normally, the unit tests will parse the default ('__main__')
+        module and follow all the imports listed there. You can override
+        this behavior and instruct the tests to check multiple modules
+        by using a comment like this in the test case input:
 
-          # cmd: mypy -m foo.bar
+          # cmd: mypy -m foo.bar foo.baz
 
-        Return tuple (main module name, main file name, main program text).
+        Return a list of tuples (module name, file name, program text).
         """
-        m = re.search('# cmd: mypy -m ([a-zA-Z0-9_.]+) *$', program_text, flags=re.MULTILINE)
+        m = re.search('# cmd: mypy -m ([a-zA-Z0-9_. ]+)$', program_text, flags=re.MULTILINE)
+        m2 = re.search('# cmd2: mypy -m ([a-zA-Z0-9_. ]+)$', program_text, flags=re.MULTILINE)
+        if m2 is not None and incremental == 2:
+            # Optionally return a different command if in the second
+            # stage of incremental mode, otherwise default to reusing
+            # the original cmd.
+            m = m2
+
         if m:
             # The test case wants to use a non-default main
             # module. Look up the module and give it as the thing to
             # analyze.
-            module_name = m.group(1)
-            path = build.find_module(module_name, [test_temp_dir])
-            with open(path) as f:
-                program_text = f.read()
-            return m.group(1), path, program_text
+            module_names = m.group(1)
+            out = []
+            for module_name in module_names.split(' '):
+                path = build.find_module(module_name, [test_temp_dir])
+                with open(path) as f:
+                    program_text = f.read()
+                out.append((module_name, path, program_text))
+            return out
         else:
-            return '__main__', 'main', program_text
+            return [('__main__', 'main', program_text)]
 
     def parse_options(self, program_text: str) -> Options:
         options = Options()
diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test
--- a/test-data/unit/check-incremental.test
+++ b/test-data/unit/check-incremental.test
@@ -274,3 +274,34 @@ bar(3)
 [out]
 main:1: note: In module imported here:
 tmp/m.py:4: error: Argument 1 to "bar" has incompatible type "int"; expected "str"
+
+[case testIncrementalUnsilencingModule]
+# cmd: mypy -m main package.subpackage.mod2
+# cmd2: mypy -m main package.subpackage.mod1
+# options: silent_imports
+
+[file main.py]
+from package.subpackage.mod1 import Class
+
+def handle(c: Class) -> None:
+    c.some_attribute
+
+[file package/__init__.py]
+# empty
+
+[file package/subpackage/__init__.py]
+# empty
+
+[file package/subpackage/mod1.py]
+import collections # Any previously unloaded package works here
+
+class Class: pass
+
+[file package/subpackage/mod2.py]
+# empty
+
+[builtins fixtures/args.pyi]
+[stale collections, main, package.subpackage.mod1]
+[out]
+tmp/main.py: note: In function "handle":
+tmp/main.py:4: error: "Class" has no attribute "some_attribute"
diff --git a/test-data/unit/check-modules.test b/test-data/unit/check-modules.test
--- a/test-data/unit/check-modules.test
+++ b/test-data/unit/check-modules.test
@@ -726,20 +726,20 @@ tmp/bar.py:1: error: Module has no attribute 'B'
 
 [case testImportSuppressedWhileAlmostSilent]
 # cmd: mypy -m main
-[file main.py]
 # options: silent_imports almost_silent
+[file main.py]
 import mod
 [file mod.py]
 [builtins fixtures/module.pyi]
 [out]
-tmp/main.py:2: note: Import of 'mod' silently ignored
-tmp/main.py:2: note: (Using --silent-imports, module not passed on command line)
-tmp/main.py:2: note: (This note courtesy of --almost-silent)
+tmp/main.py:1: note: Import of 'mod' silently ignored
+tmp/main.py:1: note: (Using --silent-imports, module not passed on command line)
+tmp/main.py:1: note: (This note courtesy of --almost-silent)
 
 [case testAncestorSuppressedWhileAlmostSilent]
 # cmd: mypy -m foo.bar
-[file foo/bar.py]
 # options: silent_imports almost_silent
+[file foo/bar.py]
 [file foo/__init__.py]
 [builtins fixtures/module.pyi]
 [out]
@@ -749,8 +749,8 @@ tmp/foo/bar.py: note: (This note brought to you by --almost-silent)
 
 [case testStubImportNonStubWhileSilent]
 # cmd: mypy -m main
-[file main.py]
 # options: silent_imports
+[file main.py]
 from stub import x # Permitted
 from other import y # Disallowed
 x + '' # Error here
@@ -763,7 +763,7 @@ x = 42
 y = 42
 [builtins fixtures/module.pyi]
 [out]
-tmp/main.py:4: error: Unsupported left operand type for + ("int")
+tmp/main.py:3: error: Unsupported left operand type for + ("int")
 
 [case testSuperclassInImportCycle]
 import a