Repository: python/mypy
Pull request: Allow files to be specified via ini config
## Summary
Allow target paths to be specified through config files as an alternative to the command line.

## Reason
To simplify the enforcement of which files/directories should be validated when working in a team. Rather than having to write a separate script that specifies the files, you can simply declare them in the existing `mypy.ini` (or alternative) config file.

## Behaviour
I've tried to copy the functionality that you might expect from the typical command line, so this includes recursive file globbing. There are two main differences:

1) paths are comma- or semicolon-separated rather than just using spaces
2) by using the `glob` module we get `**` resolving to zero or more intermediate directories. In bash (at least), `*/*.py` or `**/*.py` will only match python files in the directory below the current one.

If a path (wildcarded or not) doesn't match at least one file, it bubbles up and errors in the same way as when given on the command line. If any files *are* given on the command line, the value in the config file is ignored.

## Addendum
Closes: #5909

Problem statement:
Specify files/directories in `mypy.ini`
I'm posting this issue to gather some feedback around whether a feature I'd like would be accepted/well-received.

I'd like to be able to specify target files/directories as part of the `mypy.ini` config file, and I can't see a way to do this at the moment (if I'm missing something, hopefully this can be cleared up quickly 😄 ). This would, in my opinion, reduce the need to have a separate script that runs `mypy` with the correct arguments, as it should be easier to just rely on the settings in the config file.

I would propose something along the lines of a `files` key in the main `mypy` section, but I'm not sure if this is in line with how it works under the `special_opts` namespace. Open to any recommendation around the best section for this to sit in, if something else makes more sense.

Would a PR implementing this feature be merged/are there any reasons why this feature doesn't exist? I asked quickly in the `typing` slack and it wasn't shot down, so hoping to get more of a discussion here.

Cheers.

Hints from issue discussion:
You can’t expect a promise that your PR will be accepted. But a ‘files’
option in mypy.ini, only used when no files are given on the command line,
makes sense. Would it also include globbing?
-- 
--Guido (mobile)

Wasn't expecting a promise, definitely - more just clarification that there isn't an obvious/specific reason it *wouldn't* be accepted.

I'd intend to replicate the same treatment as the `files` argument on the command line (overridden by the command-line, as you say). The CLI does globbing, so yes, I would intend to make the config file do the same.
Technically the CLI does no globbing,the shell does. (Except on Windows
IIRC.) But yes, this makes sense.
-- 
--Guido (mobile)

Yep, I was a bit loose with my terminology there.

I will start work on putting a PR together.

Cheers.
I remembered this only recently, but we have a [`mypy_path`](https://mypy.readthedocs.io/en/stable/config_file.html#config-file-import-discovery-global) config file option that's intended to serve as a replacement for editing the MYPYPATH environment variable.

Would specifying the files you want to type check there solve your use case? If so, we should probably do a better job of documenting it. If not, it might at least serve as a good foundation for your PR.
@samuelhwilliams Any updates on the PR? E.g. Pytest has [similar functionality](https://docs.pytest.org/en/latest/reference.html#confval-testpaths) and it would be very convenient to have it in Mypy too.
I used to think this to be a silly feature, but I realize it’s actually pretty useful. Maybe someone else can create a PR. 
I dropped the ball on this. Sorry. I'll start looking at doing this again now.

Commit message: Allow files to specified via ini config

Adds support for declaring the files/directories to be type-checked
through the configuration file, with the `files` key. The behaviour is
intended to closely mimic the behaviour from the standard command-line,
and includes support for wildcard file globbing and erroring on unknown
files.

Closes: #5909

Patch to review:
diff --git a/docs/source/config_file.rst b/docs/source/config_file.rst
index 8575a3b38a8a..639c738c4fb3 100644
--- a/docs/source/config_file.rst
+++ b/docs/source/config_file.rst
@@ -333,6 +333,11 @@ a list of import discovery options that may be used
     Specifies the paths to use, after trying the paths from ``MYPYPATH`` environment
     variable.  Useful if you'd like to keep stubs in your repo, along with the config file.
 
+``files`` (string)
+    Specifies the paths which should be checked by mypy if none are given on the command
+    line. Supports recursive file globbing with `*` (eg `*.py` for files in the same directory)
+    and `**/` (eg `**/*.py`) for files in any directory below the current one.
+
 
 Platform configuration
 ----------------------
diff --git a/mypy/main.py b/mypy/main.py
index f9659701d571..7f44a13920db 100644
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -3,6 +3,7 @@
 import argparse
 import ast
 import configparser
+import glob as fileglob
 import os
 import re
 import subprocess
@@ -730,6 +731,11 @@ def add_invertible_flag(flag: str,
     if special_opts.no_executable:
         options.python_executable = None
 
+    # Paths listed in the config file will be ignored if any paths are passed on
+    # the command line.
+    if not special_opts.files and options.files:
+        special_opts.files = options.files
+
     # Check for invalid argument combinations.
     if require_targets:
         code_methods = sum(bool(c) for c in [special_opts.modules + special_opts.packages,
@@ -866,6 +872,26 @@ def process_cache_map(parser: argparse.ArgumentParser,
         options.cache_map[source] = (meta_file, data_file)
 
 
+def split_and_match_files(paths: str) -> List[str]:
+    """Take a string representing a list of files/directories (including globbing with */**).
+
+    Where a path/glob matches no file, we still include the raw path in the resulting list.
+
+    Returns a list of file paths
+    """
+    expanded_paths = []
+
+    for path in re.split('[,:]', paths):
+        path = path.strip()
+        globbed_files = fileglob.glob(path, recursive=True)
+        if globbed_files:
+            expanded_paths.extend(globbed_files)
+        else:
+            expanded_paths.append(path)
+
+    return expanded_paths
+
+
 # For most options, the type of the default value set in options.py is
 # sufficient, and we don't have to do anything here.  This table
 # exists to specify types for values initialized to None or container
@@ -876,6 +902,7 @@ def process_cache_map(parser: argparse.ArgumentParser,
     'custom_typing_module': str,
     'custom_typeshed_dir': str,
     'mypy_path': lambda s: [p.strip() for p in re.split('[,:]', s)],
+    'files': split_and_match_files,
     'quickstart_file': str,
     'junit_xml': str,
     # These two are for backwards compatibility
diff --git a/mypy/options.py b/mypy/options.py
index c498ad6fba50..a15a45aec13f 100644
--- a/mypy/options.py
+++ b/mypy/options.py
@@ -179,6 +179,10 @@ def __init__(self) -> None:
         # source hashes as often.
         self.quickstart_file = None  # type: Optional[str]
 
+        # A comma/semicolon-separated list of files/directories for mypy to type check;
+        # supports globbing
+        self.files = None  # type: Optional[List[str]]
+
         # Write junit.xml to given file
         self.junit_xml = None  # type: Optional[str]
 
diff --git a/test-data/unit/cmdline.test b/test-data/unit/cmdline.test
index 03d30be32836..62a9b6a62488 100644
--- a/test-data/unit/cmdline.test
+++ b/test-data/unit/cmdline.test
@@ -1268,3 +1268,39 @@ import d
 # cmd: mypy a.py --no-sqlite-cache --cache-map a.py a.meta.json a.data.json
 [file a.py]
 [out]
+
+[case testIniFiles]
+# cmd: mypy
+[file mypy.ini]
+[[mypy]
+files = a.py, b.py
+[file a.py]
+fail
+[file b.py]
+fail
+[out]
+b.py:1: error: Name 'fail' is not defined
+a.py:1: error: Name 'fail' is not defined
+
+[case testIniFilesGlobbing]
+# cmd: mypy
+[file mypy.ini]
+[[mypy]
+files = **/*.py
+[file a/b.py]
+fail
+[file c.py]
+fail
+[out]
+a/b.py:1: error: Name 'fail' is not defined
+c.py:1: error: Name 'fail' is not defined
+
+
+[case testIniFilesCmdlineOverridesConfig]
+# cmd: mypy override.py
+[file mypy.ini]
+[[mypy]
+files = config.py
+[out]
+mypy: can't read file 'override.py': No such file or directory
+== Return code: 2


Code review comments:
diff_hunk: @@ -333,6 +333,11 @@ a list of import discovery options that may be used
     Specifies the paths to use, after trying the paths from ``MYPYPATH`` environment
     variable.  Useful if you'd like to keep stubs in your repo, along with the config file.
 
+``files`` (string)
+    Specifies the paths which should be checked by mypy if none are given on the command
+    line. Supports recursive file globbing with `*` (eg `*.py` for files in the same directory)
path: docs/source/config_file.rst
text: @user1:
I think it might be good to add we pass globbing onto `glob`  (with `recursive=True`) and link https://docs.python.org/3/library/glob.html as glob supports more than `*` and `**`, though those are likely the most used constructs.

diff_hunk: @@ -866,6 +872,26 @@ def process_cache_map(parser: argparse.ArgumentParser,
         options.cache_map[source] = (meta_file, data_file)
 
 
+def split_and_match_files(paths: str) -> List[str]:
+    """Take a string representing a list of files/directories (including globbing with */**).
+
+    Where a path/glob matches no file, we still include the raw path in the resulting list.
+
+    Returns a list of file paths
+    """
+    expanded_paths = []
+
+    for path in re.split('[,:]', paths):
path: mypy/main.py
text: @user1:
Is the colon here due it being the path separator on Unix? If so, it might be good to also add semicolon, as that is the path separator on Windows.

@author:
I copied this re.split from the lambda on mypy_path a few lines down. Happy to add the semicolon as well; would it make sense to add it there too? 

@user2:
Honestly I wonder if it should just be comma, or space and comma? This isn't a search path like MYPYPATH, and when I tried to test this the very first mistake I made was using a space for a separator.

(I should add that for mypy_path, the behavior for `$MYPYPATH` is different than that for the config `mypy_path` variable -- the latter always uses `[,:]`, while the former uses `os.pathsep`, which is only `:` on *NIX and only `;` on Windows. So I don't think that's a great model.)

@author:
I'm a smidge nervous about using spaces as separators as that seems the most likely of all to be legitimately used in a path (albeit hopefully unlikely), and managing those within paths would introduce a lot more complexity. I think probably just commas are sufficient? It will always be processed slightly differently than by a shell, so making a clear distinction feels reasonable.

@user2:
I guess that might be the case on Windows... Agreed though, so let's do just comma. (I suppose colon is also sketchy on Windows, as in `C:\`.)