Repository: python/mypy

Issue:
AssertionError in visit_member_expr with mypy caching
Since we upgraded to 0.4.6 a few weeks ago, I've been getting repeated reports of folks encountering this mypy caching bug when using mypy in the Zulip project.  It's reproducible running on just one file with the cache file.

```
$ ./tools/run-mypy zproject/local_settings_template.py
# Expands to:
$ mypy --fast-parser --silent-imports --check-untyped-defs --scripts-are-modules -i --cache-dir=var/mypy-cache --show-traceback --py2 --disallow-untyped-defs zproject/local_settings.py
zproject/local_settings.py:13: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues
Traceback (most recent call last):
  File "/srv/zulip-py3-venv/bin/mypy", line 6, in <module>
    main(__file__)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 41, in main
    res = type_check_only(sources, bin_dir, options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/main.py", line 86, in type_check_only
    options=options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 183, in build
    dispatch(sources, manager)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1512, in dispatch
    process_graph(graph, manager)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1692, in process_graph
    process_stale_scc(graph, scc)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1767, in process_stale_scc
    graph[id].semantic_analysis()
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/build.py", line 1425, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 246, in visit_file
    self.accept(d)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2758, in accept
    node.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 812, in accept
    return visitor.visit_assignment_stmt(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 1153, in visit_assignment_stmt
    s.rvalue.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1247, in accept
    return visitor.visit_call_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2230, in visit_call_expr
    expr.callee.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1202, in accept
    return visitor.visit_member_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2330, in visit_member_expr
    base.accept(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/nodes.py", line 1202, in accept
    return visitor.visit_member_expr(self)
  File "/srv/zulip-venv-cache/57f8666f53e8346f492eb35e8ffbcf05ee486a4f/zulip-py3-venv/lib/python3.5/site-packages/mypy/semanal.py", line 2337, in visit_member_expr
    assert isinstance(file, MypyFile)
AssertionError: 
zproject/local_settings.py:13: note: use --pdb to drop into pdb
```

That's the error I got on 0.4.6.  If I try using master instead, the error moves to running on `scripts/setup/generate_secrets.py`.

Here's a tarball of the mypy cache directory involved: [mypy-cache-bug.tar.gz](https://github.com/python/mypy/files/725898/mypy-cache-bug.tar.gz)

I don't know exactly what gets the cache in this state (running mypy twice after deleting the cache doesn't do it), but the issue does keep coming back for folks if one just deletes the cache.


Discussion:
Note: the latest version is `mypy==0.470`.
It's not on pypi though, so I just went to master rather than installing that release tarball: https://pypi.python.org/pypi/mypy-lang
Note that the PyPI package name was recently changed to `mypy` from `mypy-lang`.
(And `mypy-lang` won't be updated any more and will be stuck at 0.4.6.)
Ahh.  Is there an announcement list I can subscribe to in order to hear about things like this?
You can subscribe to the mypy blog or the mypy project twitter feed (the latter is very low-volume). We don't have an active mailing list right now.
Is this still an error on `mypy>=0.470`?
Yep, as I mentioned it fails on master as well with that cache, though with
a different file in the codebase.

-Tim Abbott (mobile)

On Jan 24, 2017 15:13, "David Fisher" <notifications@github.com> wrote:

Is this still an error on mypy>=0.470?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<https://github.com/python/mypy/issues/2751#issuecomment-274969849>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ACnm2tMqi3S8TEKonj5Kynmi1Cbb_LpOks5rVoWAgaJpZM4Lr8lq>
.

Ah, sorry that I missed that!
Thanks for the report. I'm not sure that the cache dump is enough to repro the issue, because the cache stores the mtimes of all files, and if these don't match for a specific file, its cache entry is ignored. But there's a clue in the line number given at the top of the traceback -- it's
```python
config_file = six.moves.configparser.RawConfigParser() # type: ignore # https://github.com/python/typeshed/pull/206
```
So I assume that the error you get with a newer mypy on scripts/setup/generate_secrets.py is for line 54:
```python
    secrets_file = six.moves.configparser.RawConfigParser() # type: ignore # https://github.com/python/typeshed/issues/307
```
I expect I can take it from here.
OK, so here's a repro. Using `mypy --incremental --ignore-missing-imports` run the following:
```python
import six.moves.configparser
```
Then edit the file to this instead:
```python
import six.moves.configparser
six.moves.configparser.RawConfigParser()  # type: ignore
```
and run again. You now get the traceback. Using `--pdb` we can see that `file` is `None`.

I will debug further when I can find some more time.

Resolving patch:
diff --git a/mypy/semanal.py b/mypy/semanal.py
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -2404,7 +2404,7 @@ def visit_member_expr(self, expr: MemberExpr) -> None:
             # In this case base.node is the module's MypyFile and we look up
             # bar in its namespace.  This must be done for all types of bar.
             file = base.node
-            assert isinstance(file, MypyFile)
+            assert isinstance(file, (MypyFile, type(None)))
             n = file.names.get(expr.name, None) if file is not None else None
             if n:
                 n = self.normalize_type_alias(n, expr)


Test patch: