Repository: python/mypy

Issue:
Fail fast if special module like typing not from typeshed
mypy generally crashes with an internal error if it winds up trying to handle a file named `typing.py`, apparently due to confusion with the built-in `typing` module. Here's the simplest test case I've found (tested on 0.4.3 and master):

```
(mypyve) vm:/tmp/test1$ cat typing.py 
from collections import namedtuple

x = namedtuple('x', [])
(mypyve) vm:/tmp/test1$ mypy -s typing.py 
Traceback (most recent call last):
  File "/tmp/mypyve/bin/mypy", line 6, in <module>
    main(__file__)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
    dispatch(sources, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
    process_graph(graph, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
    process_stale_scc(graph, scc)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
    graph[id].semantic_analysis()
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
    self.accept(d)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
    node.accept(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
    return visitor.visit_assignment_stmt(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue, name)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
    info = self.build_namedtuple_typeinfo(name, items, types)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
    info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

typing.py:3: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
```

This is the same error message reported in a comment in #1293. The root cause is that mypy ships a file called typing.py in site-packages, so if site-packages winds up on the path (even implicitly, even with -s), it dies:

```
(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ mypy -s typing.py 
Traceback (most recent call last):
...
(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ touch hello.py
(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ mypy -s hello.py 
Traceback (most recent call last):
...
```

This makes it impossible to analyze a package along with even a subset of its dependencies as the implicit addition to MYPYPATH results in an internal error.



Discussion:
We apologize for the crash and will look into producing a more meaningful error message.

Is the file typing.py that you have a copy of the official typing.py file or an unrelated file with an unfortunate name clash?

When analyzing a package and its dependencies it's usually best not to include site-packages -- there's an explicit warning about this in the docs (see the very bottom of http://mypy.readthedocs.io/en/latest/basics.html).

The first example simulates the "unfortunate name clash" case, contents of the file displayed. The second example it is the official typing.py taken from git master shortly before I posted this issue.

Note that in neither case did I explicitly set MYPYPATH. Just referencing a file in the site-packages directory (which implicitly adds it to the search path) is sufficient.

I ran into this while setting up mypy for the first time after having added the venv site-packages to the MYPYPATH variable in response to the tremendous number of import errors from the first run for libraries that I knew were installed. I only read `mypy --help` so I didn't see the warning in the docs until significantly later.

I thought it would be reasonable to "whitelist" site-packages that were known to be analyzable, with an invocation like `mypy -s src/ venv/.../site-packages/tiny_lib/`. If library authors add types going forward, this seems like a preferable approach to out-of-tree stubs. Unfortunately, today this adds the whole of site-packages to the path implicitly, and then this bug causes an internal error.

Yeah, the feature of automatically adding the parent directory to the import path fails spectacularly for site-packages -- perhaps we should just exclude site-packages from this behavior even if we preserve it for other locations (because it is often truly useful as well!).

We've also discussed the idea of a flag that adds a stdlib module to the packages to be analyzed (without this auto-path-setting behavior).

When you have a tremendous number of failing imports, you may also consider `--silent-imports` -- see here for the docs: http://mypy.readthedocs.io/en/latest/command_line.html#following-imports-or-not

Yeah, I started using `-s` after I found that page. Our codebase has a lot of dependencies, including sqlalchemy and a few other heavyweights so it looks like there's no other options for now. I was a little surprised to discover that even with `-s`, even if I specify only a single file to analyze, it will process other files in the path (like `typing.py`!) It makes sense but the semantics could be more clearly documented.

Actually typing.py is probably the exception. What does -v output for a single file? There's a little cycle of dependencies around builtins (at least builtins, typing, and abc) but typically these are gotten from typeshed stubs, not from .py files.

Is this what you wanted?

```
(mypyve) vm:/tmp/test5$ cat typing.py 
from collections import namedtuple
x = namedtuple('x', [])
(mypyve) vm:/tmp/test5$ mypy -vs typing.py 
0.000:LOG:  Using new dependency manager
0.000:LOG:  Parsing typing.py (typing)
0.001:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.010:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.089:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.090:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.098:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.109:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.112:LOG:  Loaded graph with 7 nodes
0.113:LOG:  Found 1 SCCs; largest has 7 nodes
0.113:LOG:  Processing SCC of size 7 (abc collections typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
  File "/tmp/mypyve/bin/mypy", line 6, in <module>
    main(__file__)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
    dispatch(sources, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
    process_graph(graph, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
    process_stale_scc(graph, scc)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
    graph[id].semantic_analysis()
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
    self.accept(d)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
    node.accept(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
    return visitor.visit_assignment_stmt(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue, name)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
    info = self.build_namedtuple_typeinfo(name, items, types)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
    info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.119:LOG:  Build finished with 7 modules, 0 types, and 57 errors
```

That one's not so surprising because you pass a bogus typing.py on the
command line -- this gets substituted for the stub in typeshed so that
whatever builtins needs from typing isn't there, and hence the crash.

I thought that you were reporting problems when passing foo.py on the
command line where there's a bogus typing.py in the same directory when
also passing -s.

Also, what version of mypy? I can repro this with 0.4.3-dev but not with
0.4.3.

Here's the case with that same typing.py alongside an empty hello.py with 0.4.3:

```
(app) vm:/tmp/test5$ cat typing.py 
from collections import namedtuple
x = namedtuple('x', [])
(app) vm:/tmp/test5$ cat hello.py 
(app) vm:/tmp/test5$ mypy --version
mypy 0.4.3
(app) vm:/tmp/test5$ mypy -sv hello.py 
0.000:LOG:  Using new dependency manager
0.000:LOG:  Parsing hello.py (hello)
0.001:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.077:LOG:  Parsing /tmp/test5/typing.py (typing)
0.078:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.079:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.087:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.096:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.106:LOG:  Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.109:LOG:  Loaded graph with 8 nodes
0.109:LOG:  Found 2 SCCs; largest has 7 nodes
0.110:LOG:  Processing SCC of size 7 (collections abc typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
  File "/home/adrake/ve/app/bin/mypy", line 6, in <module>
    main(__file__)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 177, in build
    dispatch(sources, manager)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1323, in dispatch
    process_graph(graph, manager)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1461, in process_graph
    process_stale_scc(graph, scc)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1534, in process_stale_scc
    graph[id].semantic_analysis()
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1288, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
    self.accept(d)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 2306, in accept
    node.accept(self)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
    return visitor.visit_assignment_stmt(self)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1075, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1415, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue, name)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1453, in check_namedtuple
    info = self.build_namedtuple_typeinfo(name, items, types)
  File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1542, in build_namedtuple_typeinfo
    info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

/tmp/test5/typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.116:LOG:  Build finished with 8 modules, 0 types, and 57 errors
(app) vm:/tmp/test5$ 
```

And here's git master, same file setup:

```
(mypyve) vm:/tmp/test5$ mypy --version
mypy 0.4.4-dev
(mypyve) vm:/tmp/test5$ mypy -sv hello.py 
0.000:LOG:  Using new dependency manager
0.000:LOG:  Parsing hello.py (hello)
0.001:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.079:LOG:  Parsing /tmp/test5/typing.py (typing)
0.080:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.081:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.089:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.096:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.107:LOG:  Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.110:LOG:  Loaded graph with 8 nodes
0.110:LOG:  Found 2 SCCs; largest has 7 nodes
0.110:LOG:  Processing SCC of size 7 (collections abc typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
  File "/tmp/mypyve/bin/mypy", line 6, in <module>
    main(__file__)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
    dispatch(sources, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
    process_graph(graph, manager)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
    process_stale_scc(graph, scc)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
    graph[id].semantic_analysis()
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
    self.accept(d)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
    node.accept(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
    return visitor.visit_assignment_stmt(self)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue, name)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
    info = self.build_namedtuple_typeinfo(name, items, types)
  File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
    info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

/tmp/test5/typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.116:LOG:  Build finished with 8 modules, 0 types, and 57 errors
```

Hm, the only reasonable way that I can think to avoid such crashes is to
explicitly check that the stub for typing, builtins and everything else in
that initial import cycle is actually loaded from typeshed. Hopefully you
can work around it now!

Was there a workaround for wanting to include a venv dependency in the analysis? Or is that future work

That's future work -- a cumbersome workaround would be to just download and
untar the tarball (or unzip the zipfile) for the package's source distro,
or clone the repo, and point mypy at the package in the source tree. For
most reasonable source distributions that should work.

See PR #1895 -- once that lands anybody can install typeshed additions.

I don't think finding a solution that avoids the initially-reported crash has much priority. When you feed mypy the wrong typing.py (or builtins.py, or whatever) it's going to be unhappy. So I'm moving the milestone up.

Random idea: I wonder if we should have a separate panic() function for reporting problems like this (notably things missing from the builtins), so that they are clearly distinguished from other crashes?

I propose to close this as "won't fix" since we're unlikely to be fixing crashes due to the presence of incorrect stubs for builtins and typing.
What about checking that typing and builtins come from typeshed?
Sure. (And their dependencies, I suppose -- I have "_importlib_modulespec,abc,types,builtins,typing,sys" listed for PY3.

If you want that in this same issue please rename the issue subject.
Ok, renamed. Let's continue to use this issue.
Bumping priority to high since this seems to happen pretty often.
Got bitten by this again. In this case it was a package that was missing `__init__.py` that contained the file `types.py`, which then was interpreted by mypy as the `types` stdlib module and replaced the typeshed stub. This didn't cause a crash but weird type checking results, which were a little tricky to debug.
Note this can happen not only with `typing`, or `types`, but also with `typing_extensions`, see https://github.com/python/mypy/issues/6874 for a recent example.

Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -36,7 +36,7 @@
 from mypy.errors import Errors, CompileError, ErrorInfo, report_internal_error
 from mypy.util import (
     DecodeError, decode_python_encoding, is_sub_path, get_mypy_comments, module_prefix,
-    read_py_file, hash_digest,
+    read_py_file, hash_digest, is_typeshed_file
 )
 if TYPE_CHECKING:
     from mypy.report import Reports  # Avoid unconditional slow import
@@ -66,6 +66,18 @@
 # that it's easy to enable this when running tests.
 DEBUG_FINE_GRAINED = False  # type: Final
 
+# These modules are special and should always come from typeshed.
+CORE_BUILTIN_MODULES = {
+    'builtins',
+    'typing',
+    'types',
+    'typing_extensions',
+    'mypy_extensions',
+    '_importlib_modulespec',
+    'sys',
+    'abc',
+}
+
 
 Graph = Dict[str, 'State']
 
@@ -2390,6 +2402,14 @@ def find_module_and_diagnose(manager: BuildManager,
                 if is_sub_path(path, dir):
                     # Silence errors in site-package dirs and typeshed
                     follow_imports = 'silent'
+        if (id in CORE_BUILTIN_MODULES
+                and not is_typeshed_file(path)
+                and not options.use_builtins_fixtures
+                and not options.custom_typeshed_dir):
+            raise CompileError([
+                'mypy: "%s" shadows library module "%s"' % (path, id),
+                'note: A user-defined top-level module with name "%s" is not supported' % id
+            ])
         return (path, follow_imports)
     else:
         # Could not find a module.  Typically the reason is a


Test patch:
diff --git a/mypy/test/testpythoneval.py b/mypy/test/testpythoneval.py
--- a/mypy/test/testpythoneval.py
+++ b/mypy/test/testpythoneval.py
@@ -88,6 +88,8 @@ def test_python_evaluation(testcase: DataDrivenTestCase, cache_dir: str) -> None
         if line.startswith(test_temp_dir + os.sep):
             output.append(line[len(test_temp_dir + os.sep):].rstrip("\r\n"))
         else:
+            # Normalize paths so that the output is the same on Windows and Linux/macOS.
+            line = line.replace(test_temp_dir + os.sep, test_temp_dir + '/')
             output.append(line.rstrip("\r\n"))
     if returncode == 0:
         # Execute the program.
diff --git a/test-data/unit/pythoneval.test b/test-data/unit/pythoneval.test
--- a/test-data/unit/pythoneval.test
+++ b/test-data/unit/pythoneval.test
@@ -1501,3 +1501,12 @@ from typing import List
 async def foo() -> None:
     f = []  # type: List[Future[None]]
     await wait(f)
+
+[case testShadowTypingModule]
+1 + ''
+[file typing.py]
+x = 0
+1 + ''
+[out]
+mypy: "tmp/typing.py" shadows library module "typing"
+note: A user-defined top-level module with name "typing" is not supported