Repository: python/mypy
Pull request: Implement flag to allow typechecking of untyped modules
Add a flag and config ini options `"follow_untyped_imports"`. Setting it to `True` instructs mypy to typecheck also modules that do not have stubs or a `py.typed` marker.

Fixes #8545

Problem statement:
Allow/disallow pep561 packages
@alexander255 has requested a flag `--pep561-override` to allow listing packages as pep 561 compatible when they are not. This got me thinking and I think the best UX for this would be a pair of flags: `--allow-installed-package NAME` and `--disallow-installed-package NAME` (or something like that).

The idea is to solve two problems:

Some people publish PEP 561 packages that are either too unstable or flat out broken for some users. This allows the user to disable the package until things are fixed.

At the other end of the spectrum, some packagers are conservative on adding types, and while they exist, they are not public (yet). Some users would still like to use them.

This would eliminate `--no-site-packages`, which is a rather blunt instrument.

Hints from issue discussion:
I can see how this could be useful., and this shouldn't be hard to implement 

Random thoughts:

* What about `--use-installed-package X` as the option name? The inverse would be `--ignore-installed-package X`, perhaps. More ideas:
  * `--enable-installed-package X` and `--disable-installed-package X`
  * `--use-site-package X` and `--no-site-package X`
  * `--site-package X` and `--no-site-package X`
  * `--installed-package X` and `--no-installed-package X`
* We can start with `--use-site-package` (or whatever it would be called) and the inverse can be implemented later.  The inverse seems less commonly useful.
* Should this work for subpackages? What about things like `--use-site-package google.protobuf`?
I think I like either `--enable-installed-packages X` or `--use-installed-packages X`, ideally I think this should be a list of packages, as people may want to allowlist/denylist groups of packages.
I am a "-0" on this; I don't like adding more complexity to our import system. But also I'd be happy to merge it. Let's make a decision on it so we can resolve #8512.
Yeah I'm kinda -0 on this too. The complexity would be unfortunate, but I can kinda see the use case.
Just to clarify all selected flags will be usable via mypy.ini, correct?
:+1: for this.  I am using some packages that are fully typed and are PEP-561 compatible in principle (they have a `py.typed`) but due to a packaging bug (third-party package, outside my control) were missing the `py.typed` file when installed.

I can of course work around it by creating the `py.typed` file in the `site-packages/<package_name>` but it would be more convenient if I could tell mypy "yes, this package really does have type hints".
Adding my proposal from #10783, which concerns packages that have typing information but forgot to include a `py.typed` file. Many packages seem to make this mistake, and some are lightly maintained and haven't responded to bug reports or made new releases.

This should be a configuration option mirroring `ignore_missing_imports`, such as `allow_missing_pytyped`. Currently if a package is missing `py.typed`, my only recourse is to add

```ini
[mypy-missing.*]
ignore_missing_imports = True
```

Instead, since I know this package has typing information but forgot the `py.typed` file, I want to add

```ini
[mypy-missing.*]
allow_missing_pytyped = True
```
I can see the appeal of providing a way to work around library shortcomings like this, but it would serve the entire Python community better if you would work with the library maintainer to fix the problem at the source. If the library is no longer maintained, perhaps a fork is the best answer?
The library is maintained, just slowly because the maintainer is also a maintainer of many other packages. I've already reached out to projects and even provided PRs sometimes, but I have no control over when they have time to review, merge, and release new versions. For the same reason, I have no desire to fork, as I already maintain enough projects as it is.
I believe the disallow use case can already be accomplished using something like:
```
[mypy]
[mypy-numpy.*]
follow_imports = skip
follow_imports_for_stubs = True
```

Commit message: Change config option name to follow-untyped-imports

Patch to review:
diff --git a/mypy/main.py b/mypy/main.py
index ae3e7d75be7f..fb4a1f61a01d 100644
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -580,6 +580,11 @@ def add_invertible_flag(
         action="store_true",
         help="Silently ignore imports of missing modules",
     )
+    imports_group.add_argument(
+        "--follow-untyped-imports",
+        action="store_true",
+        help="Typecheck modules without stubs or py.typed marker",
+    )
     imports_group.add_argument(
         "--follow-imports",
         choices=["normal", "silent", "skip", "error"],
diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py
index bc11c1304776..a36f50c334c7 100644
--- a/mypy/modulefinder.py
+++ b/mypy/modulefinder.py
@@ -334,10 +334,11 @@ def _typeshed_has_version(self, module: str) -> bool:
         return version >= min_version and (max_version is None or version <= max_version)
 
     def _find_module_non_stub_helper(
-        self, components: list[str], pkg_dir: str
+        self, id: str, pkg_dir: str
     ) -> OnePackageDir | ModuleNotFoundReason:
         plausible_match = False
         dir_path = pkg_dir
+        components = id.split(".")
         for index, component in enumerate(components):
             dir_path = os_path_join(dir_path, component)
             if self.fscache.isfile(os_path_join(dir_path, "py.typed")):
@@ -350,6 +351,11 @@ def _find_module_non_stub_helper(
             if not self.fscache.isdir(dir_path):
                 break
         if plausible_match:
+            if self.options:
+                module_specific_options = self.options.clone_for_module(id)
+                if module_specific_options.follow_untyped_imports:
+                    # print("Returning ", id)
+                    return os.path.join(pkg_dir, *components[:-1]), False
             return ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS
         else:
             return ModuleNotFoundReason.NOT_FOUND
@@ -463,7 +469,7 @@ def _find_module(self, id: str, use_typeshed: bool) -> ModuleSearchResult:
                             third_party_stubs_dirs.append((path, True))
                     else:
                         third_party_stubs_dirs.append((path, True))
-            non_stub_match = self._find_module_non_stub_helper(components, pkg_dir)
+            non_stub_match = self._find_module_non_stub_helper(id, pkg_dir)
             if isinstance(non_stub_match, ModuleNotFoundReason):
                 if non_stub_match is ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS:
                     found_possible_third_party_missing_type_hints = True
diff --git a/mypy/options.py b/mypy/options.py
index 367267d1053a..561b23fec7d0 100644
--- a/mypy/options.py
+++ b/mypy/options.py
@@ -42,6 +42,7 @@ class BuildType:
     "extra_checks",
     "follow_imports_for_stubs",
     "follow_imports",
+    "follow_untyped_imports",
     "ignore_errors",
     "ignore_missing_imports",
     "implicit_optional",
@@ -113,6 +114,8 @@ def __init__(self) -> None:
         self.ignore_missing_imports = False
         # Is ignore_missing_imports set in a per-module section
         self.ignore_missing_imports_per_module = False
+        # Typecheck modules without stubs or py.typed marker
+        self.follow_untyped_imports = False
         self.follow_imports = "normal"  # normal|silent|skip|error
         # Whether to respect the follow_imports setting even for stub files.
         # Intended to be used for disabling specific stubs.
diff --git a/test-data/unit/pep561.test b/test-data/unit/pep561.test
index 9969c2894c36..fb303a8fb5ec 100644
--- a/test-data/unit/pep561.test
+++ b/test-data/unit/pep561.test
@@ -187,6 +187,13 @@ b.bf(1)
 testNamespacePkgWStubsWithNamespacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible type "int"; expected "bool"
 testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool"
 
+[case testMissingPytypedFlag]
+# pkgs: typedpkg_ns_b
+# flags: --namespace-packages --follow-untyped-imports
+import typedpkg_ns.b.bbb as b
+b.bf("foo", "bar")
+[out]
+testMissingPytypedFlag.py:4: error: Too many arguments for "bf"
 
 [case testTypedPkgNamespaceRegFromImportTwiceMissing]
 # pkgs: typedpkg_ns_a


Code review comments:
diff_hunk: @@ -350,6 +351,11 @@ def _find_module_non_stub_helper(
             if not self.fscache.isdir(dir_path):
                 break
         if plausible_match:
+            if self.options:
+                module_specific_options = self.options.clone_for_module(id)
+                if module_specific_options.follow_untyped_imports:
+                    # print("Returning ", id)
path: mypy/modulefinder.py
text: @user1:
No need for this comment

@author:
Thanks for pointing this out, overlooked it when I removed my debugging statements.