Repository: python/mypy

Issue:
in-project stubs cause mypy to bail in 0.971
<!--
  If you're new to mypy and you're not sure whether what you're experiencing is a mypy bug, please see the "Question and Help" form
  instead.
  Please also consider:

  - checking our common issues page: https://mypy.readthedocs.io/en/stable/common_issues.html
  - searching our issue tracker: https://github.com/python/mypy/issues to see if
    it's already been reported
  - asking on gitter chat: https://gitter.im/python/typing
-->

**Bug Report**

We have a project that includes type stubs for a python package that does not provide it's own typing. We followed guidance per https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-library-stubs-or-py-typed-marker number 3

```
(venv) vfazio@vfazio2 ~/development/xtf $ grep mypy_path mypy.ini
mypy_path=$MYPY_CONFIG_FILE_DIR/stubs
```

In mypy 0.961, type checks pass

```
(venv) vfazio@vfazio2 ~/development/xtf $ mypy -V; mypy xes/
mypy 0.961 (compiled: no)
Success: no issues found in 86 source files
```
In mypy 0.971, these now fail because the current path is being added to the path search 

```
(venv) vfazio@vfazio2 ~/development/xtf :( $ mypy -V; mypy xes/
mypy 0.971 (compiled: no)
['/mnt/development/mypy/venv/lib/python3.10/site-packages', '/mnt/development/xtf']
/mnt/development/xtf is in the MYPYPATH. Please remove it.
See https://mypy.readthedocs.io/en/stable/running_mypy.html#how-mypy-handles-imports for more info

```

If i modify `mypy/pyinfo.py::getsearchdirs` to exclude the current working directory (effectively reverting 2132036a019ed26d77389c0a9af9b928911a5846) the type checks "work" except I now get type failures for the stubs

```
(venv) vfazio@vfazio2 ~/development/xtf $ mypy -V; mypy xes/
mypy 0.971 (compiled: no)
stubs/redacted/path/file.pyi:49:9: error: Function is missing a return type annotation  [no-untyped-def]
stubs/redacted/path/file.pyi:49:9: note: Use "-> None" if function does not return a value
<snip>
Found 58 errors in 2 files (checked 86 source files)

```

<!--
Note: If the problem you are reporting is about a specific library function, then the typeshed tracker is better suited
for this report: https://github.com/python/typeshed/issues
-->

**Expected Behavior**

<!--
  How did you expect your project to behave?
  It’s fine if you’re not sure your understanding is correct.
  Write down what you thought would happen. If you just expected no errors, you can delete this section.
-->

Failures wouldn't occur or documentation would be updated to reflect how to accommodate this use case.

**Actual Behavior**

<!--
  Did something go wrong?
  Is something broken, or not behaving as you expected?
-->

mypy now fails to type check our code because we include stubs as part of our source tree

**Your Environment**

<!-- Include as many relevant details about the environment you experienced the bug in -->

- Mypy version used: 0.971
- Mypy command-line flags:
- Mypy configuration options from `mypy.ini` (and other config files):
- Python version used: 3.10.4
- Operating system and version: Ubuntu 20.04

<!--
You can freely edit this text, please remove all the lines
you believe are unnecessary.
-->



Discussion:
In mypy 0.960 and for long, long before, mypy only looked at site packages, with some logic on top to handle `.pth` files.
The big change here was https://github.com/python/mypy/pull/11143 , which changed that so mypy looks at most of `sys.path`.

The check you're running into was to prevent people from doing `MYPYPATH=/whatever/python/site-packages`, which would do terrible things (at least historically — that check is really really old and things might be better today).

If you see the before side of #11143, this check was only against `site_packages`, not `egg_dirs` (which is where the `.pth` stuff, such as an editably installed cwd, would go). The right side of #11143 smooshes all of `sys.path` together, which is why you now get this check. This seems bad.

The connection with errors being reported in your stubs comes from https://github.com/python/mypy/blob/a6166b2f7e8e4cf9d176107277da223e45c3a6a1/mypy/build.py#L2458
where mypy will automatically silence errors from things it's picked up from pyinfo and typeshed.

Just to be explicit and reduce confusion, I'm quite sure the current pyinfo / https://github.com/python/mypy/pull/13161 is correct (and it matches the behaviour of mypy 0.960 too, in terms of what ends up where on `SearchPaths`).
The issue described by OP isn't really with cwd — it can happen with any directory that is on `sys.path` and `MYPYPATH`.

I think the fix here is to either get rid of the check entirely, or to restrict it so that it only applies to actually site-package dirs, like it did in 0.960
And thanks for the report and the extra investigation, makes it much easier to figure out what's going on!
Ok, I think I kinda-sorta understand what's going on.

As a test, I reverted my hack to exclude `cwd` and dropped the `sys.exit(1)` from modulefinder after the MYPYPATH warning and I now see that the .pyi files no longer trigger warnings. 

For now, we'll stick to 0.961 until there's a patch to address this behavior.

It may make sense to add a unittest for this scenario since it's a documented configuration, that way any future regressions could be caught?
I think I've found the specific environment parts that are causing our error.

* We use poetry
* We have a namespace package and use the `packages` feature: https://python-poetry.org/docs/pyproject/#packages
* `mypy.ini` contains `mypy_path=$MYPY_CONFIG_FILE_DIR/stubs`

So our pyproject.toml looks like:
```
[tool.poetry]
name = "test_project"
...
packages = [
    { include = "namespace_pkg" }
]
```

Upon `poetry install`, a `test_project.pth` file is created in `<virtualenv path>/lib/python3.10/site-packages/`:
```
 $ ls <redacted>/lib/python3.10/site-packages/ -la                                                                                                                        
...                                                            
-rwxr-xr-x  1 jspeicher domain users 45790632 Jul 22 11:19 f4771e8a540c98c8e0df__mypyc.cpython-310-x86_64-linux-gnu.so
drwxr-xr-x  9 jspeicher domain users    12288 Jul 22 11:19 mypy
drwxr-xr-x  2 jspeicher domain users     4096 Jul 22 11:19 mypy-0.971.dist-info
drwxr-xr-x  2 jspeicher domain users     4096 Jul 22 11:19 mypy_extensions-0.4.3.dist-info
-rw-r--r--  1 jspeicher domain users     5078 Jul 22 11:19 mypy_extensions.py
drwxr-xr-x 14 jspeicher domain users     4096 Jul 22 11:19 mypyc
...
drwxr-xr-x  2 jspeicher domain users     4096 Jul 22 11:20 test_project-0.1.0.dist-info
-rw-r--r--  1 jspeicher domain users       30 Jul 22 11:20 test_project.pth
...
```

With this `test_project.pth` file, mypy errors as @vfazio commented.
```
$ mypy namespace_pkg
<project dir> is in the MYPYPATH. Please remove it.
```
If I comment-out the `packages` portion of `pyproject.toml` then the `test_project.pth` file is not created, and mypy works as expected.

---

I also tried this with a simple setuptools-based setup:

setup.py:
```python
from setuptools import setup

setup(
   name='test_project',
   version='2.0',
   packages=['namespace_pkg'],
)
```

Upon `pip install -e .`, a `venv/lib/python3.10/site-packages/test-project.egg-link` file is created, which causes the same mypy error:
```
$ mypy namespace_pkg
<project dir> is in the MYPYPATH. Please remove it.
```
To sum up my previous comment, I think that mypy is currently incompatible with an editable-installed package (`pip install -e .`) in combination with mypy config `mypy_path=$MYPY_CONFIG_FILE_DIR/<anything>`

Resolving patch:
diff --git a/mypy/main.py b/mypy/main.py
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -1051,11 +1051,10 @@ def set_strict_flags() -> None:
     # Set target.
     if special_opts.modules + special_opts.packages:
         options.build_type = BuildType.MODULE
-        search_dirs = get_search_dirs(options.python_executable)
-        search_paths = SearchPaths((os.getcwd(),),
-                                   tuple(mypy_path() + options.mypy_path),
-                                   tuple(search_dirs),
-                                   ())
+        sys_path, _ = get_search_dirs(options.python_executable)
+        search_paths = SearchPaths(
+            (os.getcwd(),), tuple(mypy_path() + options.mypy_path), tuple(sys_path), ()
+        )
         targets = []
         # TODO: use the same cache that the BuildManager will
         cache = FindModuleCache(search_paths, fscache, options)
diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py
--- a/mypy/modulefinder.py
+++ b/mypy/modulefinder.py
@@ -723,7 +723,7 @@ def default_lib_path(data_dir: str,
 
 
 @functools.lru_cache(maxsize=None)
-def get_search_dirs(python_executable: Optional[str]) -> List[str]:
+def get_search_dirs(python_executable: Optional[str]) -> Tuple[List[str], List[str]]:
     """Find package directories for given python.
 
     This runs a subprocess call, which generates a list of the directories in sys.path.
@@ -732,15 +732,15 @@ def get_search_dirs(python_executable: Optional[str]) -> List[str]:
     """
 
     if python_executable is None:
-        return []
+        return ([], [])
     elif python_executable == sys.executable:
         # Use running Python's package dirs
-        sys_path = pyinfo.getsearchdirs()
+        sys_path, site_packages = pyinfo.getsearchdirs()
     else:
         # Use subprocess to get the package directory of given Python
         # executable
         try:
-            sys_path = ast.literal_eval(
+            sys_path, site_packages = ast.literal_eval(
                 subprocess.check_output([python_executable, pyinfo.__file__, 'getsearchdirs'],
                 stderr=subprocess.PIPE).decode())
         except OSError as err:
@@ -748,7 +748,7 @@ def get_search_dirs(python_executable: Optional[str]) -> List[str]:
             raise CompileError(
                 [f"mypy: Invalid python executable '{python_executable}': {reason}"]
             ) from err
-    return sys_path
+    return sys_path, site_packages
 
 
 def add_py2_mypypath_entries(mypypath: List[str]) -> List[str]:
@@ -837,22 +837,26 @@ def compute_search_paths(sources: List[BuildSource],
     if options.python_version[0] == 2:
         mypypath = add_py2_mypypath_entries(mypypath)
 
-    search_dirs = get_search_dirs(options.python_executable)
-    for search_dir in search_dirs:
-        assert search_dir not in lib_path
-        if (search_dir in mypypath or
-                any(p.startswith(search_dir + os.path.sep) for p in mypypath) or
-                (os.path.altsep
-                    and any(p.startswith(search_dir + os.path.altsep) for p in mypypath))):
-            print(f"{search_dir} is in the MYPYPATH. Please remove it.", file=sys.stderr)
+    sys_path, site_packages = get_search_dirs(options.python_executable)
+    # We only use site packages for this check
+    for site in site_packages:
+        assert site not in lib_path
+        if (
+            site in mypypath
+            or any(p.startswith(site + os.path.sep) for p in mypypath)
+            or (os.path.altsep and any(p.startswith(site + os.path.altsep) for p in mypypath))
+        ):
+            print(f"{site} is in the MYPYPATH. Please remove it.", file=sys.stderr)
             print("See https://mypy.readthedocs.io/en/stable/running_mypy.html"
                   "#how-mypy-handles-imports for more info", file=sys.stderr)
             sys.exit(1)
 
-    return SearchPaths(python_path=tuple(reversed(python_path)),
-                       mypy_path=tuple(mypypath),
-                       package_path=tuple(search_dirs),
-                       typeshed_path=tuple(lib_path))
+    return SearchPaths(
+        python_path=tuple(reversed(python_path)),
+        mypy_path=tuple(mypypath),
+        package_path=tuple(sys_path + site_packages),
+        typeshed_path=tuple(lib_path),
+    )
 
 
 def load_stdlib_py_versions(custom_typeshed_dir: Optional[str]) -> StdlibVersions:
diff --git a/mypy/pyinfo.py b/mypy/pyinfo.py
--- a/mypy/pyinfo.py
+++ b/mypy/pyinfo.py
@@ -7,12 +7,13 @@
 possible.
 """
 import os
+import site
 import sys
 import sysconfig
 
 MYPY = False
 if MYPY:
-    from typing import List
+    from typing import Tuple, List
 
 if __name__ == '__main__':
     # HACK: We don't want to pick up mypy.types as the top-level types
@@ -24,7 +25,22 @@
     sys.path = old_sys_path
 
 
-def getsearchdirs():
+def getsitepackages():
+    # type: () -> List[str]
+    res = []
+    if hasattr(site, "getsitepackages"):
+        res.extend(site.getsitepackages())
+
+        if hasattr(site, "getusersitepackages") and site.ENABLE_USER_SITE:
+            res.insert(0, site.getusersitepackages())
+    else:
+        from distutils.sysconfig import get_python_lib
+
+        res = [get_python_lib()]
+    return res
+
+
+def getsyspath():
     # type: () -> List[str]
     # Do not include things from the standard library
     # because those should come from typeshed.
@@ -54,6 +70,14 @@ def getsearchdirs():
     return [p for p in abs_sys_path if p not in excludes]
 
 
+def getsearchdirs():
+    # type: () -> Tuple[List[str], List[str]]
+    return (
+        getsyspath(),
+        getsitepackages(),
+    )
+
+
 if __name__ == '__main__':
     if sys.argv[-1] == 'getsearchdirs':
         print(repr(getsearchdirs()))


Test patch: