Repository: python/mypy

Issue:
`stubgen --recursive` stubs entire package even when only subpackage was requested.
This is a request for a slight change of behavior for `stubgen`.  Right now, if I use `stubgen --recursive urllib.parse`, stubs will be generated for the entirety of `urllib`.

```sh
$ stubgen --recursive urllib.parse
Created out/urllib/__init__.pyi
Created out/urllib/error.pyi
Created out/urllib/parse.pyi
Created out/urllib/request.pyi
Created out/urllib/response.pyi
Created out/urllib/robotparser.pyi
```

This seems to be somewhat odd behavior -- I would have expected stubgen to only generate `out/urllib/parse.pyi`.  To generate all the stubs, I would expect `stubgen --recursive urllib` (currently there is no difference between passing `urllib.parse` and `urllib` as the module to be stubbed).

The behavior is because of the use of `__import__` rather than `importlib.import_module` in [`walk_packages`](https://github.com/python/mypy/blob/master/mypy/stubgen.py#L849).  

I'm not completely sure if this is an intended feature or a bug, but I figured that I'd report it and find out.


Discussion:
That looks like nobody really tested this before. I agree that it's odd behavior. Do you want to try sending a PR to fix it?

Resolving patch:
diff --git a/mypy/stubgen.py b/mypy/stubgen.py
--- a/mypy/stubgen.py
+++ b/mypy/stubgen.py
@@ -846,9 +846,13 @@ def get_qualified_name(o: Expression) -> str:
 
 def walk_packages(packages: List[str]) -> Iterator[str]:
     for package_name in packages:
-        package = __import__(package_name)
+        package = importlib.import_module(package_name)
         yield package.__name__
-        for importer, qualified_name, ispkg in pkgutil.walk_packages(package.__path__,
+        path = getattr(package, '__path__', None)
+        if path is None:
+            # It's a module inside a package.  There's nothing else to walk/yield.
+            continue
+        for importer, qualified_name, ispkg in pkgutil.walk_packages(path,
                                                                      prefix=package.__name__ + ".",
                                                                      onerror=lambda r: None):
             yield qualified_name


Test patch:
diff --git a/mypy/test/teststubgen.py b/mypy/test/teststubgen.py
--- a/mypy/test/teststubgen.py
+++ b/mypy/test/teststubgen.py
@@ -11,7 +11,9 @@
 from mypy.test.helpers import Suite, assert_equal, assert_string_arrays_equal
 from mypy.test.data import DataSuite, DataDrivenTestCase
 from mypy.errors import CompileError
-from mypy.stubgen import generate_stub, generate_stub_for_module, parse_options, Options
+from mypy.stubgen import (
+    generate_stub, generate_stub_for_module, parse_options, walk_packages, Options
+)
 from mypy.stubgenc import generate_c_type_stub, infer_method_sig
 from mypy.stubutil import (
     parse_signature, parse_all_signatures, build_signature, find_unique_signatures,
@@ -19,6 +21,26 @@
 )
 
 
+class StubgenCliParseSuite(Suite):
+    def test_walk_packages(self) -> None:
+        assert_equal(
+            set(walk_packages(["mypy.errors"])),
+            {"mypy.errors"})
+
+        assert_equal(
+            set(walk_packages(["mypy.errors", "mypy.stubgen"])),
+            {"mypy.errors", "mypy.stubgen"})
+
+        all_mypy_packages = set(walk_packages(["mypy"]))
+        self.assertTrue(all_mypy_packages.issuperset({
+            "mypy",
+            "mypy.errors",
+            "mypy.stubgen",
+            "mypy.test",
+            "mypy.test.helpers",
+        }))
+
+
 class StubgenUtilSuite(Suite):
     def test_parse_signature(self) -> None:
         self.assert_parse_signature('func()', ('func', [], []))