Repository: python/mypy

Issue:
mypy doesn't recurse through directories, but -h says it does
According to `mypy -h`:
```
Mypy is a program that will type check your Python code.

Pass in any files or folders you want to type check. Mypy will
recursively traverse any provided folders to find .py files:

    $ mypy my_program.py my_src_folder
```


But it doesn't recurse, it only checks what's explicitly provided on the command line:
```
$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ pip install mypy
...
(venv) $ mkdir -p test/test/
(venv) $ touch test/test/a.py
(venv) $ mypy test
There are no .py[i] files in directory 'test'
(venv) $ touch test/b.py
(venv) $ mypy test
Success: no issues found in 1 source file
(venv) $ mypy test test/test
Success: no issues found in 2 source files
```

The fix might be to change the documentation.  This is easy to workaround with `find`.


Discussion:
I believe this is because you're using namespace packages instead of regular packages. If you try running:

    mypy --namespace-packages -p test

...mypy should detect that you have two source files. The [command line docs](https://mypy.readthedocs.io/en/stable/command_line.html) should have more details about both flags.

Alternatively, you could do:

    touch test/__init__.py
    touch test/test/__init__.py

...to convert your package back into a regular one. Then running `mypy test` should detect four files, as expected.
That might fix the above example.  But my actual use case is to check any python files regardless of where in the directory hierarchy they appear.  So I actually want the recursive behavior described in the help text.
@rvandegrift Can you give a more concrete example of a directory hierarchy where `-p FOO` doesn't work for you? You can give multiple `-p FOO` arguments.
Well, there's no concrete directory hierarchy that lead me here.

I wanted to enable mypy on all of our repos via a single script in a shared CI pipeline.  The script won't know which repo it's running for ahead of time.  And since we're human and messy, we have  different layouts in different repos.  So I can't supply any `-p FOO` arguments, since I don't know what `FOO` is.

One of my team members put together a quick hack to find package roots and run once for each: https://github.com/jceresini/mypy_r

I'm not sure that this is a valuable use case.  So maybe the only mypy issue is that the help text doesn't describe the actual operation.
The following directory hierarchy led me to this issue:

```
- setup.py
+ src
  + mypkg
    - __init__.py
    - module1.py
    - module2.py
+ tests
  - test_1.py
  - test_2.py
```

So this is a src layout, not a namespace package.

As @rvandegrift quoted from the help text:

> Mypy will recursively traverse any provided folders to find .py files

Based on this statement, I expect that invoking mypy via `mypy .` will cause mypy to check all the `.py` files in my hierarchy. However, `setup.py` is the only one it checks. To check everything, I need to invoke it as `mypy setup.py src/ tests/`. And that's fine, I have no problem invoking it that way, but the behavior doesn't match what I'd expect given the documentation.

The description from the [online docs](https://mypy.readthedocs.io/en/stable/running_mypy.html#specifying-code-to-be-checked) is similar, although it doesn't specifically say it finds `.py` files:

> In addition, mypy will recursively type check the entire contents of any provided directories.

I think the words "recursively" and "entire" are strong signals that it will descend down into _every_ directory looking for _every_ `.py` file, which isn't the case. A link to further details is helpfully provided right after this, where we learn how it actually works:

> Directories that don’t represent Python packages (i.e. not directly containing an `__init__.py[i]` file) are checked as follows:
> - All `*.py[i]` files contained directly therein are checked as toplevel Python modules;
> - All packages contained directly therein (i.e. immediate subdirectories with an `__init__.py[i]` file) are checked as toplevel Python packages.

In the case of `mypy .`, the current directory doesn't contain an `__init__.py`, so the above logic applies. `setup.py` is contained directly therein, so that one is checked. But neither of the immediate subdirectories `src` or `test` contain an `__init__.py`, so they aren't checked. Stopping after one subdirectory level is hardly what I'd call recursive.

I understand that I can address the issue by invoking mypy differently. But like @rvandegrift suggested, I think the solution to this particular issue is to update the help text and docs so that they aren't misleading. I suggest removing the references to "recursive", keeping things concise by leaving the directory rules a bit vague, and pointing to where the procedure is described in greater detail.

For example, the help text could be changed from:

> Pass in any files or folders you want to type check. Mypy will recursively traverse any provided folders to find .py files:

to:

> Pass in any files or folders you want to type check:

Links to further documentation are already provided right after the sample command, so no need to add those.

And the online docs could be changed from:

> The above command tells mypy it should type check all of the provided files together. In addition, mypy will recursively type check the entire contents of any provided directories.
> 
> For more details about how exactly this is done, see Mapping file paths to modules.

to:

> The above command tells mypy it should type check all of the provided files together. In addition, mypy will type check the contents of any provided directories. For more details about how exactly this is done, see Mapping file paths to modules.

If you wanted to provide more details about the directory checking up front, maybe something like this:

> For each provided directory, mypy will type check every package it finds in that directory and its immediate subdirectories, and it will type check all .py files it finds in that directory.

I find this behaviour highly surprising and unintuitive.

We just discovered that this caused a part of our source tree not to be checked for over a year...
I am convinced. We just need to fix this. Someone please send a PR that makes this properly recursive. There is quite a difference between `mypy <dir>` and `mypy -p <dir>` because the latter takes a package name (a valid Python identifier).

If it breaks people they can fix it by writing something like `mypy <dir>/*.py`. (Hm, does the files config flag allow globs?)
my layout:
```
- pyproject.toml
- setup.cfg
+ src/
  + my_package_name/
    - __init__.py
    + subm/
      - __init__.py
+ tests/
   + test_group1/
      - test_*.py
```

with this layout, the following works (I have not yet found a way to combine the two into one command, for example using the configuration file)
```
$ MYPYPATH=src mypy -p my_package_name
$ mypy tests/**/*.py
```

However, this really is a nuisance compared to `mypy .` - which I expect to work from:

"Pass in any files or folders you want to type check. Mypy will recursively traverse any provided folders to find .py files"

```
$ mypy .                  
There are no .py[i] files in directory '.'
```

I agree with rjmorris above, that at least the help and docs need to be updated. I would also appreciate a less cumbersome method to use mypy for a src-layout package tho.
There is no need to post more evidence to this issue. What we need is someone to volunteer a PR. The code is pretty straightforward (assuming you know how to iterate over directories recursively), it should all be contained in mypy/main.py.
I agree that this causes bad surprises and needs a solution.

I'll also mention that fixing this might be a good time to figure out / have a plan for #5759 and #8584 (assuming we want to be able to have some sort of support for checking namespace packages by passing directory arguments).

Resolving patch:
diff --git a/mypy/find_sources.py b/mypy/find_sources.py
--- a/mypy/find_sources.py
+++ b/mypy/find_sources.py
@@ -16,7 +16,7 @@ class InvalidSourceList(Exception):
     """Exception indicating a problem in the list of sources given to mypy."""
 
 
-def create_source_list(files: Sequence[str], options: Options,
+def create_source_list(paths: Sequence[str], options: Options,
                        fscache: Optional[FileSystemCache] = None,
                        allow_empty_dir: bool = False) -> List[BuildSource]:
     """From a list of source files/directories, makes a list of BuildSources.
@@ -26,22 +26,24 @@ def create_source_list(files: Sequence[str], options: Options,
     fscache = fscache or FileSystemCache()
     finder = SourceFinder(fscache)
 
-    targets = []
-    for f in files:
-        if f.endswith(PY_EXTENSIONS):
+    sources = []
+    for path in paths:
+        path = os.path.normpath(path)
+        if path.endswith(PY_EXTENSIONS):
             # Can raise InvalidSourceList if a directory doesn't have a valid module name.
-            name, base_dir = finder.crawl_up(os.path.normpath(f))
-            targets.append(BuildSource(f, name, None, base_dir))
-        elif fscache.isdir(f):
-            sub_targets = finder.expand_dir(os.path.normpath(f))
-            if not sub_targets and not allow_empty_dir:
-                raise InvalidSourceList("There are no .py[i] files in directory '{}'"
-                                        .format(f))
-            targets.extend(sub_targets)
+            name, base_dir = finder.crawl_up(path)
+            sources.append(BuildSource(path, name, None, base_dir))
+        elif fscache.isdir(path):
+            sub_sources = finder.find_sources_in_dir(path, explicit_package_roots=None)
+            if not sub_sources and not allow_empty_dir:
+                raise InvalidSourceList(
+                    "There are no .py[i] files in directory '{}'".format(path)
+                )
+            sources.extend(sub_sources)
         else:
-            mod = os.path.basename(f) if options.scripts_are_modules else None
-            targets.append(BuildSource(f, mod, None))
-    return targets
+            mod = os.path.basename(path) if options.scripts_are_modules else None
+            sources.append(BuildSource(path, mod, None))
+    return sources
 
 
 def keyfunc(name: str) -> Tuple[int, str]:
@@ -62,57 +64,82 @@ def __init__(self, fscache: FileSystemCache) -> None:
         # A cache for package names, mapping from directory path to module id and base dir
         self.package_cache = {}  # type: Dict[str, Tuple[str, str]]
 
-    def expand_dir(self, arg: str, mod_prefix: str = '') -> List[BuildSource]:
-        """Convert a directory name to a list of sources to build."""
-        f = self.get_init_file(arg)
-        if mod_prefix and not f:
-            return []
+    def find_sources_in_dir(
+        self, path: str, explicit_package_roots: Optional[List[str]]
+    ) -> List[BuildSource]:
+        if explicit_package_roots is None:
+            mod_prefix, root_dir = self.crawl_up_dir(path)
+        else:
+            mod_prefix = os.path.basename(path)
+            root_dir = os.path.dirname(path) or "."
+        if mod_prefix:
+            mod_prefix += "."
+        return self.find_sources_in_dir_helper(path, mod_prefix, root_dir, explicit_package_roots)
+
+    def find_sources_in_dir_helper(
+        self, dir_path: str, mod_prefix: str, root_dir: str,
+        explicit_package_roots: Optional[List[str]]
+    ) -> List[BuildSource]:
+        assert not mod_prefix or mod_prefix.endswith(".")
+
+        init_file = self.get_init_file(dir_path)
+        # If the current directory is an explicit package root, explore it as such.
+        # Alternatively, if we aren't given explicit package roots and we don't have an __init__
+        # file, recursively explore this directory as a new package root.
+        if (
+            (explicit_package_roots is not None and dir_path in explicit_package_roots)
+            or (explicit_package_roots is None and init_file is None)
+        ):
+            mod_prefix = ""
+            root_dir = dir_path
+
         seen = set()  # type: Set[str]
         sources = []
-        top_mod, base_dir = self.crawl_up_dir(arg)
-        if f and not mod_prefix:
-            mod_prefix = top_mod + '.'
-        if mod_prefix:
-            sources.append(BuildSource(f, mod_prefix.rstrip('.'), None, base_dir))
-        names = self.fscache.listdir(arg)
+
+        if init_file:
+            sources.append(BuildSource(init_file, mod_prefix.rstrip("."), None, root_dir))
+
+        names = self.fscache.listdir(dir_path)
         names.sort(key=keyfunc)
         for name in names:
             # Skip certain names altogether
-            if (name == '__pycache__' or name == 'py.typed'
-                    or name.startswith('.')
-                    or name.endswith(('~', '.pyc', '.pyo'))):
+            if name == '__pycache__' or name.startswith('.') or name.endswith('~'):
                 continue
-            path = os.path.join(arg, name)
+            path = os.path.join(dir_path, name)
+
             if self.fscache.isdir(path):
-                sub_sources = self.expand_dir(path, mod_prefix + name + '.')
+                sub_sources = self.find_sources_in_dir_helper(
+                    path, mod_prefix + name + '.', root_dir, explicit_package_roots
+                )
                 if sub_sources:
                     seen.add(name)
                     sources.extend(sub_sources)
             else:
-                base, suffix = os.path.splitext(name)
-                if base == '__init__':
+                stem, suffix = os.path.splitext(name)
+                if stem == '__init__':
                     continue
-                if base not in seen and '.' not in base and suffix in PY_EXTENSIONS:
-                    seen.add(base)
-                    src = BuildSource(path, mod_prefix + base, None, base_dir)
+                if stem not in seen and '.' not in stem and suffix in PY_EXTENSIONS:
+                    seen.add(stem)
+                    src = BuildSource(path, mod_prefix + stem, None, root_dir)
                     sources.append(src)
+
         return sources
 
-    def crawl_up(self, arg: str) -> Tuple[str, str]:
+    def crawl_up(self, path: str) -> Tuple[str, str]:
         """Given a .py[i] filename, return module and base directory
 
         We crawl up the path until we find a directory without
         __init__.py[i], or until we run out of path components.
         """
-        dir, mod = os.path.split(arg)
-        mod = strip_py(mod) or mod
-        base, base_dir = self.crawl_up_dir(dir)
-        if mod == '__init__' or not mod:
-            mod = base
+        parent, filename = os.path.split(path)
+        module_name = strip_py(filename) or os.path.basename(filename)
+        module_prefix, base_dir = self.crawl_up_dir(parent)
+        if module_name == '__init__' or not module_name:
+            module = module_prefix
         else:
-            mod = module_join(base, mod)
+            module = module_join(module_prefix, module_name)
 
-        return mod, base_dir
+        return module, base_dir
 
     def crawl_up_dir(self, dir: str) -> Tuple[str, str]:
         """Given a directory name, return the corresponding module name and base directory
@@ -124,7 +151,7 @@ def crawl_up_dir(self, dir: str) -> Tuple[str, str]:
 
         parent_dir, base = os.path.split(dir)
         if not dir or not self.get_init_file(dir) or not base:
-            res = ''
+            module = ''
             base_dir = dir or '.'
         else:
             # Ensure that base is a valid python module name
@@ -132,17 +159,16 @@ def crawl_up_dir(self, dir: str) -> Tuple[str, str]:
                 base = base[:-6]  # PEP-561 stub-only directory
             if not base.isidentifier():
                 raise InvalidSourceList('{} is not a valid Python package name'.format(base))
-            parent, base_dir = self.crawl_up_dir(parent_dir)
-            res = module_join(parent, base)
+            parent_module, base_dir = self.crawl_up_dir(parent_dir)
+            module = module_join(parent_module, base)
 
-        self.package_cache[dir] = res, base_dir
-        return res, base_dir
+        self.package_cache[dir] = module, base_dir
+        return module, base_dir
 
     def get_init_file(self, dir: str) -> Optional[str]:
         """Check whether a directory contains a file named __init__.py[i].
 
-        If so, return the file's name (with dir prefixed).  If not, return
-        None.
+        If so, return the file's name (with dir prefixed).  If not, return None.
 
         This prefers .pyi over .py (because of the ordering of PY_EXTENSIONS).
         """


Test patch:
diff --git a/test-data/unit/cmdline.test b/test-data/unit/cmdline.test
--- a/test-data/unit/cmdline.test
+++ b/test-data/unit/cmdline.test
@@ -45,29 +45,47 @@ pkg/subpkg/a.py:1: error: Name 'undef' is not defined
 # cmd: mypy dir
 [file dir/a.py]
 undef
-[file dir/subdir/a.py]
+[file dir/subdir/b.py]
 undef
 [out]
 dir/a.py:1: error: Name 'undef' is not defined
+dir/subdir/b.py:1: error: Name 'undef' is not defined
+
+[case testCmdlineNonPackageDuplicate]
+# cmd: mypy dir
+[file dir/a.py]
+undef
+[file dir/subdir/a.py]
+undef
+[out]
+dir/a.py: error: Duplicate module named 'a' (also at 'dir/subdir/a.py')
+dir/a.py: error: Are you missing an __init__.py?
+== Return code: 2
 
 [case testCmdlineNonPackageSlash]
 # cmd: mypy dir/
 [file dir/a.py]
 undef
-[file dir/subdir/a.py]
+import b
+[file dir/subdir/b.py]
 undef
+import a
 [out]
 dir/a.py:1: error: Name 'undef' is not defined
+dir/subdir/b.py:1: error: Name 'undef' is not defined
 
 [case testCmdlinePackageContainingSubdir]
 # cmd: mypy pkg
 [file pkg/__init__.py]
 [file pkg/a.py]
 undef
+import a
 [file pkg/subdir/a.py]
 undef
+import pkg.a
 [out]
 pkg/a.py:1: error: Name 'undef' is not defined
+pkg/subdir/a.py:1: error: Name 'undef' is not defined
 
 [case testCmdlineNonPackageContainingPackage]
 # cmd: mypy dir