Repository: python/mypy

Issue:
stubtest: Spurious warning related to symbols in '__all__'
<!--
  If you're new to mypy and you're not sure whether what you're experiencing is a mypy bug, please see the "Question and Help" form
  instead.
-->

**Bug Report**

<!--
Note: If the problem you are reporting is about a specific library function, then the typeshed tracker is better suited
for this report: https://github.com/python/typeshed/issues
-->

I'm getting an error from `stubtest` checking a package that lists a submodule in `__all__`.

**To Reproduce**

1. Create a package with a submodule
```shell
$ mkdir pkg
$ touch pkg/__init__.py
$ touch pkg/mod.py
```

2. So far, `stubtest` seems to be happy with it
```shell
$ PYTHONPATH=$PWD stubtest pkg
```
3. Now add `mod` to  `__all__` in `__init.py` and run `stubtest` again
```shell
$ echo "__all__=['mod']" > pkg/__init__.py
$ PYTHONPATH=$PWD stubtest pkg
error: pkg.mod is not present in stub
Stub:
MISSING
Runtime:
MISSING
```

Note, adding respective `*.pyi` files do not fix the problem.

- Mypy version used: latest master
- Python version used: 3.9
- Operating system and version: Linux (Fedora 33)



Discussion:
This example is confusing, since the "stub" stubtest is looking at is actually just the source file. This is a weird way of using stubtest — what are you trying to accomplish?

Moreover, the complaint is kind of true.
```
λ PYTHONPATH=$(pwd) python3 -c 'import pkg; pkg.mod'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'pkg' has no attribute 'mod'
```
Note if you were to add `from . import mod` to your `__init__.py`, things would work.

Finally, the flag `--ignore-missing-stub` will let you ignore errors caused by things missing from your stubs.
As I mentioned before, adding `*.pyi` files do not fix the problem.

Let's try again, using *.pyi stub files alongside the *.py files:

```shell
$ mkdir pkg
$ touch pkg/__init__.py
$ touch pkg/__init__.pyi
$ touch pkg/mod.py
$ touch pkg/mod.pyi
$ PYTHONPATH=$PWD stubtest pkg
```
```shell
$ echo "__all__=['mod']" > pkg/__init__.py
$ PYTHONPATH=$PWD stubtest pkg
error: pkg.mod is not present in stub
Stub:
MISSING
Runtime:
MISSING
```

IMHO, the complaint is not true, this is a perfectly valid Python package. The fact that `mod` is listed in `__all__` does not imply that the package's `__init__.py` should import the submodule. To the best of my knowledge, the purpose of `__all__` is to control how `from pkg import *` behaves, look yourself:

```shell
$ PYTHONPATH=$PWD python -c "from pkg import *; print(mod)"
<module 'pkg.mod' from '/tmp/pkg/mod.py'>
```
so there is no real requirement for `from . import mod` in `pkg/__init__.py`.

I'm not sure of this behavior of `stubtest` on handling `__all__` is a bug or it is intentional, but in any case it is incorrect. Again, listing `'something'` in `__all__` does not mean that `something` has to be present in the namespace. That's my understanding [from Python's documentation](https://docs.python.org/3/tutorial/modules.html#importing-from-a-package), and it has worked that way at least since the beginnings of this century, when we where using [Python 2.2](https://docs.python.org/release/2.2/tut/node8.html#SECTION008410000000000000000).

I'm trying to use `stubtest` to setup a sort of  "lint" for my manually maintained stubs. The flag `--ignore-missing-stub` is not an option, it does not operate at the stub file level, it will ignore actual missing stub entries.


Resolving patch:
diff --git a/mypy/stubtest.py b/mypy/stubtest.py
--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -161,6 +161,10 @@ def test_module(module_name: str) -> Iterator[Error]:
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
             runtime = importlib.import_module(module_name)
+            # Also run the equivalent of `from module import *`
+            # This could have the additional effect of loading not-yet-loaded submodules
+            # mentioned in __all__
+            __import__(module_name, fromlist=["*"])
     except Exception as e:
         yield Error([module_name], "failed to import: {}".format(e), stub, MISSING)
         return
@@ -200,15 +204,15 @@ def verify_mypyfile(
     to_check = set(
         m
         for m, o in stub.names.items()
+        # TODO: change `o.module_public` to `not o.module_hidden`
         if o.module_public and (not m.startswith("_") or hasattr(runtime, m))
     )
     runtime_public_contents = [
         m
         for m in dir(runtime)
         if not m.startswith("_")
-        # Ensure that the object's module is `runtime`, e.g. so that we don't pick up reexported
-        # modules and infinitely recurse. Unfortunately, there's no way to detect an explicit
-        # reexport missing from the stubs (that isn't specified in __all__)
+        # Ensure that the object's module is `runtime`, since in the absence of __all__ we don't
+        # have a good way to detect re-exports at runtime.
         and getattr(getattr(runtime, m), "__module__", None) == runtime.__name__
     ]
     # Check all things declared in module's __all__, falling back to runtime_public_contents
@@ -216,10 +220,13 @@ def verify_mypyfile(
     to_check.difference_update({"__file__", "__doc__", "__name__", "__builtins__", "__package__"})
 
     for entry in sorted(to_check):
-        stub_to_verify = stub.names[entry].node if entry in stub.names else MISSING
-        assert stub_to_verify is not None
+        stub_entry = stub.names[entry].node if entry in stub.names else MISSING
+        if isinstance(stub_entry, nodes.MypyFile):
+            # Don't recursively check exported modules, since that leads to infinite recursion
+            continue
+        assert stub_entry is not None
         yield from verify(
-            stub_to_verify,
+            stub_entry,
             getattr(runtime, entry, MISSING),
             object_path + [entry],
         )


Test patch: