Repository: python/mypy

Issue:
mypy fails importing my package with --silent-imports and -p
When I run mypy over [qutebrowser](https://github.com/The-Compiler/qutebrowser), things look good initially (apart from some use of type annotations for my own purposes):

```
$ mypy -m qutebrowser
qutebrowser/qutebrowser.py:27: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/checkpyver.py:29: error: No library stub file for standard library module 'tkinter'
qutebrowser/misc/checkpyver.py:29: note: (Stub files are from https://github.com/python/typeshed)
qutebrowser/misc/checkpyver.py:33: error: Cannot find module named 'Tkinter'
qutebrowser/misc/checkpyver.py:33: note: (Perhaps setting MYPYPATH or using the "--silent-imports" flag would help)
qutebrowser/misc/checkpyver.py:34: error: Cannot find module named 'tkMessageBox'
qutebrowser/qutebrowser.py:42: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/earlyinit.py:28: error: Cannot find module named 'hunter'
qutebrowser/misc/earlyinit.py:34: error: Cannot find module named 'faulthandler'
qutebrowser/misc/earlyinit.py:40: error: No library stub file for standard library module 'tkinter'
qutebrowser/misc/earlyinit.py:83: error: No library stub file for module 'PyQt5.QtWidgets'
qutebrowser/misc/earlyinit.py:84: error: No library stub file for module 'PyQt5.QtCore'
qutebrowser/misc/earlyinit.py:222: error: No library stub file for module 'PyQt5.QtNetwork'
qutebrowser/qutebrowser.py:156: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/app.py:34: error: No library stub file for module 'PyQt5.QtWidgets'
qutebrowser/app.py:35: error: No library stub file for module 'PyQt5.QtWebKit'
qutebrowser/app.py:36: error: No library stub file for module 'PyQt5.QtGui'
qutebrowser/app.py:37: error: No library stub file for module 'PyQt5.QtCore'
qutebrowser/app.py:40: error: Cannot find module named 'hunter'
qutebrowser/app.py:52: note: In module imported here,
qutebrowser/qutebrowser.py:156: note: ... from here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/sessions.py: note: In member "session_save" of class "SessionManager":
qutebrowser/misc/sessions.py:390: error: Parse error before {
qutebrowser/misc/sessions.py:391: error: Parse error before )
qutebrowser/misc/sessions.py: note: In class "SessionManager":
qutebrowser/misc/sessions.py:391: error: Parse error before :
qutebrowser/misc/sessions.py:392: error: Inconsistent indentation
qutebrowser/misc/sessions.py: note: At top level:
qutebrowser/misc/sessions.py:443: error: Inconsistent indentation
```

I then tried using `--silent-imports` as I don't care about the missing stubs for now:

```
$ mypy --silent-imports -m qutebrowser
qutebrowser.py:29: error: "module" has no attribute "main"
```

`--almost-silent` reveals a bit more:

```
$ mypy --silent-imports --almost-silent -m qutebrowser
qutebrowser.py:24: note: Import of 'qutebrowser.qutebrowser' silently ignored
qutebrowser.py:24: note: (Using --silent-imports, module not passed on command line)
qutebrowser.py:24: note: (This note courtesy of --almost-silent)
qutebrowser.py:29: error: "module" has no attribute "main"
```

Note my directory structure looks like this:

```
├── qutebrowser
│   ├── app.py
│   ├── [...]
│   ├── qutebrowser.py
│   ├── resources.py
│   └── utils
│       [...]
├── [...]
├── qutebrowser.py
```

That is, there's a toplevel `qutebrowser.py` which is a simple launcher script:

``` python
import qutebrowser.qutebrowser
import sys

if __name__ == '__main__':
    sys.exit(qutebrowser.qutebrowser.main())
```

Then there's the `qutebrowser` package, and a `qutebrowser.py` in it which
starts the actual application (in `main()`).

I should probably rename at least `qutebrowser/qutebrowser.py` and maybe the
toplevel `qutebrowser.py`, but I think there's still no reason mypy should get
confused by it with `--silent-imports`.



Discussion:
`-m` if used together with `--silent-imports` only type checks the target module (in this case, `qutebrowser/__init__.py`). If you want to recursively type check submodules, you can use `qutebrowser/` (without `-m`) or `-p qutebrowser`.

`-p` doesn't seem to change anything:

```
$ mypy --silent-imports --almost-silent -p qutebrowser
qutebrowser.py:24: note: Import of 'qutebrowser.qutebrowser' silently ignored
qutebrowser.py:24: note: (Using --silent-imports, module not passed on command line)
qutebrowser.py:24: note: (This note courtesy of --almost-silent)
qutebrowser.py:29: error: "module" has no attribute "main"
```

Using `qutebrowser/` seems to work, i.e. I get some more parse errors which will go away when I fix my annotations :wink:

This whole behaviour strikes me as unintuitive though. Why does `--silent-imports` change the behaviour of `-m` and `-p`? What's the difference between the two? Why couldn't `-m` check the whole package if a package is passed, similarily to what python does? It seems to do exactly that without `--silent-imports`?

The problem seems to be confusing between your toplevel `qutebrowser.py` and toplevel package `qutebrowser`. That looks like a genuine mypy bug, though I'm not sure exactly where. It seems the `-p` flag somehow manages to lie about the name of the file it actually reads vs. the file it claims to read; the `-v` flag shows no signs that the toplevel qutebrowser.py is ever read. How `-s` affects this is also not immediately clear (but there are definitely dragons there).

One work-around would be to skip the `-p` flag altogether and just say `mypy -s qutebrowser` (as of the 0.4 release the trailing slash is no longer needed). That seems to work. (It reveals problems with qutebrowser/browser/commands.py which you can silence using `@no_type_check`.)

Thanks! I'm actually working on getting rid of my own weird dict type annotations so I can consistently use PEP484 annotations.

(I remember we talked about this a while back and you recommended to use decorators, while I insisted a dict annotation wasn't as ugly as you feeled it was. Now that I switched to decorators I realized you were right. :laughing: )

I was wrong about one thing: mypy doesn't lie about which module it's reading (I was confused because there are so many things named qutebrowser here :-).

However the genuine bug is that when choosing between module `x.py` and package `x/` (containing `x/__init__.py`), mypy incorrectly prefers the module `x.py`, whereas the official algorithm prefers the package `x/`. I find this pretty serious.


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -599,10 +599,14 @@ def find():
         sepinit = os.sep + '__init__'
         for base_dir in candidate_base_dirs:
             base_path = base_dir + seplast  # so e.g. '/usr/lib/python3.4/foo/bar/baz'
+            # Prefer package over module, i.e. baz/__init__.py* over baz.py*.
+            for extension in PYTHON_EXTENSIONS:
+                path = base_path + sepinit + extension
+                if is_file(path) and verify_module(id, path):
+                    return path
+            # No package, look for module.
             for extension in PYTHON_EXTENSIONS:
                 path = base_path + extension
-                if not is_file(path):
-                    path = base_path + sepinit + extension
                 if is_file(path) and verify_module(id, path):
                     return path
         return None


Test patch:
diff --git a/mypy/test/data/check-modules.test b/mypy/test/data/check-modules.test
--- a/mypy/test/data/check-modules.test
+++ b/mypy/test/data/check-modules.test
@@ -769,3 +769,19 @@ class A:
 [file d.py]
 import a
 class D(a.C): pass
+
+[case testPreferPackageOverFile]
+import a
+[file a.py]
+/  # intentional syntax error -- this file shouldn't be parsed
+[file a/__init__.py]
+pass
+[out]
+
+[case testPreferPackageOverFile2]
+from a import x
+[file a.py]
+/  # intentional syntax error -- this file shouldn't be parsed
+[file a/__init__.py]
+x = 0
+[out]