Repository: psf/black
Pull request: Add --extend-exclude parameter
Adding the `--extend-exclude` parameter to add additional exclude regexes on top of the `--exclude` regexes. This is especially useful if you want to specify _additional_ files/directories on top of the very useful default ones (as demonstrated by the change to our own `pyproject.toml`).

The parameter name also matches `flake8`'s parameter of `--extend-exclude`.

--- 

Fixes #1571

Problem statement:
Please offer a way to exclude dirs *in addition to the default exclusions*
**Is your feature request related to a problem? Please describe.**

I needed to exclude one auto-generated file so I naively specified `--exclude thefile.py`. Unfortunately that disabled the useful standard exclusions, which I did not realize until it broke a unit test running on Jenkins.

**Describe the solution you'd like**

I would like a way to exclude one or more files or directories *in addition to the standard exclusion defaults*.

**Describe alternatives you've considered**

The simplest UI change would be a new command-line option such as `--alsoexclude` meaning "exclude the specified files and directories in addition to other exclusions." The example above would read `--alsoexclude thefile.py`.

Please also consider allowing that option to be repeated. It seems a very natural thing and it saves the hassle of generating a regex to exclude multiple items. Thus: `--alsoexclude foo.py --alsoexclude .private --alsoexclude a/path/somewhere`. 

Ideally I would suggest that same capability for `--exclude` and `--include`, because regular expressions can be hard to read and can easily have subtle bugs. But I realize that is a stretch since it changes an existing interface.

Another option is to add a boolean option that means "exclude the standard defaults", such as `--excludedefaults`. Then the example above would read `--excludedefaults --exclude thefile.py`

Yet another option is to have some magic value that means "the standard exclusion defaults". But I don't know what it could be without risking collision with files and directory names.

**Additional context**

None.

Hints from issue discussion:
For comparison, Flake8 does this, named `--extend-exclude=<patterns>`

https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-extend-exclude
I'd be supportive of doing this, using the `--extend-exclude` name.
I can take a stab at it, with `--extend-exclude`.

Commit message: Fix flake8 issues

Patch to review:
diff --git a/README.md b/README.md
index 178f763c32d..df21d2b1f00 100644
--- a/README.md
+++ b/README.md
@@ -135,11 +135,17 @@ Options:
                                   hg|\.mypy_cache|\.nox|\.tox|\.venv|\.svn|_bu
                                   ild|buck-out|build|dist)/]
 
+  --extend-exclude TEXT           Like --exclude, but adds additional files
+                                  and directories on top of the excluded
+                                  ones. (Useful if you simply want to add to
+                                  the default)
+
   --force-exclude TEXT            Like --exclude, but files and directories
                                   matching this regex will be excluded even
                                   when they are passed explicitly as
                                   arguments.
 
+
   --stdin-filename TEXT           The name of the file when passing it through
                                   stdin. Useful to make sure Black will
                                   respect --force-exclude option on some
@@ -313,25 +319,10 @@ expressions by Black. Use `[ ]` to denote a significant space character.
 line-length = 88
 target-version = ['py37']
 include = '\.pyi?$'
-exclude = '''
+extend-exclude = '''
 # A regex preceded with ^/ will apply only to files and directories
 # in the root of the project.
-^/(
-  (
-      \.eggs         # exclude a few common directories in the
-    | \.git          # root of the project
-    | \.hg
-    | \.mypy_cache
-    | \.tox
-    | \.venv
-    | _build
-    | buck-out
-    | build
-    | dist
-  )/
-  | foo.py           # also separately exclude a file named foo.py in
-                     # the root of the project
-)
+^/foo.py  # exclude a file named foo.py in the root of the project (in addition to the defaults)
 '''
 ```
 
diff --git a/docs/change_log.md b/docs/change_log.md
index 066be76c06c..01c27553fca 100644
--- a/docs/change_log.md
+++ b/docs/change_log.md
@@ -28,6 +28,8 @@
 
 - use lowercase hex strings (#1692)
 
+- added `--extend-exclude` argument (#1571)
+
 #### _Packaging_
 
 - Self-contained native _Black_ binaries are now provided for releases via GitHub
diff --git a/docs/installation_and_usage.md b/docs/installation_and_usage.md
index ee45c934da8..5b4adb547ec 100644
--- a/docs/installation_and_usage.md
+++ b/docs/installation_and_usage.md
@@ -95,6 +95,11 @@ Options:
                                   when they are passed explicitly as
                                   arguments.
 
+  --extend-exclude TEXT           Like --exclude, but adds additional files
+                                  and directories on top of the excluded
+                                  ones. (Useful if you simply want to add to
+                                  the default)
+
   --stdin-filename TEXT           The name of the file when passing it through
                                   stdin. Useful to make sure Black will
                                   respect --force-exclude option on some
diff --git a/docs/pyproject_toml.md b/docs/pyproject_toml.md
index 453f533bf96..e71185cc036 100644
--- a/docs/pyproject_toml.md
+++ b/docs/pyproject_toml.md
@@ -54,25 +54,10 @@ expressions by Black. Use `[ ]` to denote a significant space character.
 line-length = 88
 target-version = ['py37']
 include = '\.pyi?$'
-exclude = '''
+extend-exclude = '''
 # A regex preceded with ^/ will apply only to files and directories
 # in the root of the project.
-^/(
-  (
-      \.eggs         # exclude a few common directories in the
-    | \.git          # root of the project
-    | \.hg
-    | \.mypy_cache
-    | \.tox
-    | \.venv
-    | _build
-    | buck-out
-    | build
-    | dist
-  )/
-  | foo.py           # also separately exclude a file named foo.py in
-                     # the root of the project
-)
+^/foo.py  # exclude a file named foo.py in the root of the project (in addition to the defaults)
 '''
 ```
 
diff --git a/pyproject.toml b/pyproject.toml
index 9d4da0bf692..7f632f2839d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -9,19 +9,8 @@
 line-length = 88
 target-version = ['py36', 'py37', 'py38']
 include = '\.pyi?$'
-exclude = '''
+extend-exclude = '''
 /(
-    \.eggs
-  | \.git
-  | \.hg
-  | \.mypy_cache
-  | \.tox
-  | \.venv
-  | _build
-  | buck-out
-  | build
-  | dist
-
   # The following are specific to Black, you probably don't want those.
   | blib2to3
   | tests/data
diff --git a/src/black/__init__.py b/src/black/__init__.py
index 6919468609c..8f6e3949600 100644
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -461,6 +461,14 @@ def target_version_option_callback(
     ),
     show_default=True,
 )
+@click.option(
+    "--extend-exclude",
+    type=str,
+    help=(
+        "Like --exclude, but adds additional files and directories on top of the"
+        " excluded ones. (Useful if you simply want to add to the default)"
+    ),
+)
 @click.option(
     "--force-exclude",
     type=str,
@@ -537,6 +545,7 @@ def main(
     verbose: bool,
     include: str,
     exclude: str,
+    extend_exclude: Optional[str],
     force_exclude: Optional[str],
     stdin_filename: Optional[str],
     src: Tuple[str, ...],
@@ -570,6 +579,7 @@ def main(
         verbose=verbose,
         include=include,
         exclude=exclude,
+        extend_exclude=extend_exclude,
         force_exclude=force_exclude,
         report=report,
         stdin_filename=stdin_filename,
@@ -602,6 +612,18 @@ def main(
     ctx.exit(report.return_code)
 
 
+def test_regex(
+    ctx: click.Context,
+    regex_name: str,
+    regex: Optional[str],
+) -> Optional[Pattern]:
+    try:
+        return re_compile_maybe_verbose(regex) if regex is not None else None
+    except re.error:
+        err(f"Invalid regular expression for {regex_name} given: {regex!r}")
+        ctx.exit(2)
+
+
 def get_sources(
     *,
     ctx: click.Context,
@@ -610,28 +632,18 @@ def get_sources(
     verbose: bool,
     include: str,
     exclude: str,
+    extend_exclude: Optional[str],
     force_exclude: Optional[str],
     report: "Report",
     stdin_filename: Optional[str],
 ) -> Set[Path]:
     """Compute the set of files to be formatted."""
-    try:
-        include_regex = re_compile_maybe_verbose(include)
-    except re.error:
-        err(f"Invalid regular expression for include given: {include!r}")
-        ctx.exit(2)
-    try:
-        exclude_regex = re_compile_maybe_verbose(exclude)
-    except re.error:
-        err(f"Invalid regular expression for exclude given: {exclude!r}")
-        ctx.exit(2)
-    try:
-        force_exclude_regex = (
-            re_compile_maybe_verbose(force_exclude) if force_exclude else None
-        )
-    except re.error:
-        err(f"Invalid regular expression for force_exclude given: {force_exclude!r}")
-        ctx.exit(2)
+
+    include_regex = test_regex(ctx, "include", include)
+    exclude_regex = test_regex(ctx, "exclude", exclude)
+    assert exclude_regex is not None
+    extend_exclude_regex = test_regex(ctx, "extend_exclude", extend_exclude)
+    force_exclude_regex = test_regex(ctx, "force_exclude", force_exclude)
 
     root = find_project_root(src)
     sources: Set[Path] = set()
@@ -672,6 +684,7 @@ def get_sources(
                     root,
                     include_regex,
                     exclude_regex,
+                    extend_exclude_regex,
                     force_exclude_regex,
                     report,
                     gitignore,
@@ -6110,17 +6123,27 @@ def normalize_path_maybe_ignore(
     return normalized_path
 
 
+def path_is_excluded(
+    normalized_path: str,
+    pattern: Optional[Pattern[str]],
+) -> bool:
+    match = pattern.search(normalized_path) if pattern else None
+    return bool(match and match.group(0))
+
+
 def gen_python_files(
     paths: Iterable[Path],
     root: Path,
     include: Optional[Pattern[str]],
     exclude: Pattern[str],
+    extend_exclude: Optional[Pattern[str]],
     force_exclude: Optional[Pattern[str]],
     report: "Report",
     gitignore: PathSpec,
 ) -> Iterator[Path]:
     """Generate all files under `path` whose paths are not excluded by the
-    `exclude_regex` or `force_exclude` regexes, but are included by the `include` regex.
+    `exclude_regex`, `extend_exclude`, or `force_exclude` regexes,
+    but are included by the `include` regex.
 
     Symbolic links pointing outside of the `root` directory are ignored.
 
@@ -6137,20 +6160,22 @@ def gen_python_files(
             report.path_ignored(child, "matches the .gitignore file content")
             continue
 
-        # Then ignore with `--exclude` and `--force-exclude` options.
+        # Then ignore with `--exclude` `--extend-exclude` and `--force-exclude` options.
         normalized_path = "/" + normalized_path
         if child.is_dir():
             normalized_path += "/"
 
-        exclude_match = exclude.search(normalized_path) if exclude else None
-        if exclude_match and exclude_match.group(0):
+        if path_is_excluded(normalized_path, exclude):
             report.path_ignored(child, "matches the --exclude regular expression")
             continue
 
-        force_exclude_match = (
-            force_exclude.search(normalized_path) if force_exclude else None
-        )
-        if force_exclude_match and force_exclude_match.group(0):
+        if path_is_excluded(normalized_path, extend_exclude):
+            report.path_ignored(
+                child, "matches the --extend-exclude regular expression"
+            )
+            continue
+
+        if path_is_excluded(normalized_path, force_exclude):
             report.path_ignored(child, "matches the --force-exclude regular expression")
             continue
 
@@ -6160,6 +6185,7 @@ def gen_python_files(
                 root,
                 include,
                 exclude,
+                extend_exclude,
                 force_exclude,
                 report,
                 gitignore,
diff --git a/tests/test_black.py b/tests/test_black.py
index 5d14ceda8f4..dbfb1622a4c 100644
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -1337,7 +1337,14 @@ def test_include_exclude(self) -> None:
         this_abs = THIS_DIR.resolve()
         sources.extend(
             black.gen_python_files(
-                path.iterdir(), this_abs, include, exclude, None, report, gitignore
+                path.iterdir(),
+                this_abs,
+                include,
+                exclude,
+                None,
+                None,
+                report,
+                gitignore,
             )
         )
         self.assertEqual(sorted(expected), sorted(sources))
@@ -1361,6 +1368,7 @@ def test_exclude_for_issue_1572(self) -> None:
                 verbose=False,
                 include=include,
                 exclude=exclude,
+                extend_exclude=None,
                 force_exclude=None,
                 report=report,
                 stdin_filename=None,
@@ -1383,6 +1391,7 @@ def test_get_sources_with_stdin(self) -> None:
                 verbose=False,
                 include=include,
                 exclude=exclude,
+                extend_exclude=None,
                 force_exclude=None,
                 report=report,
                 stdin_filename=None,
@@ -1406,6 +1415,7 @@ def test_get_sources_with_stdin_filename(self) -> None:
                 verbose=False,
                 include=include,
                 exclude=exclude,
+                extend_exclude=None,
                 force_exclude=None,
                 report=report,
                 stdin_filename=stdin_filename,
@@ -1433,6 +1443,35 @@ def test_get_sources_with_stdin_filename_and_exclude(self) -> None:
                 verbose=False,
                 include=include,
                 exclude=exclude,
+                extend_exclude=None,
+                force_exclude=None,
+                report=report,
+                stdin_filename=stdin_filename,
+            )
+        )
+        self.assertEqual(sorted(expected), sorted(sources))
+
+    @patch("black.find_project_root", lambda *args: THIS_DIR.resolve())
+    def test_get_sources_with_stdin_filename_and_extend_exclude(self) -> None:
+        # Extend exclude shouldn't exclude stdin_filename since it is mimicing the
+        # file being passed directly. This is the same as
+        # test_exclude_for_issue_1572
+        path = THIS_DIR / "data" / "include_exclude_tests"
+        include = ""
+        extend_exclude = r"/exclude/|a\.py"
+        src = "-"
+        report = black.Report()
+        stdin_filename = str(path / "b/exclude/a.py")
+        expected = [Path(f"__BLACK_STDIN_FILENAME__{stdin_filename}")]
+        sources = list(
+            black.get_sources(
+                ctx=FakeContext(),
+                src=(src,),
+                quiet=True,
+                verbose=False,
+                include=include,
+                exclude="",
+                extend_exclude=extend_exclude,
                 force_exclude=None,
                 report=report,
                 stdin_filename=stdin_filename,
@@ -1458,6 +1497,7 @@ def test_get_sources_with_stdin_filename_and_force_exclude(self) -> None:
                 verbose=False,
                 include=include,
                 exclude="",
+                extend_exclude=None,
                 force_exclude=force_exclude,
                 report=report,
                 stdin_filename=stdin_filename,
@@ -1542,7 +1582,14 @@ def test_gitignore_exclude(self) -> None:
         this_abs = THIS_DIR.resolve()
         sources.extend(
             black.gen_python_files(
-                path.iterdir(), this_abs, include, exclude, None, report, gitignore
+                path.iterdir(),
+                this_abs,
+                include,
+                exclude,
+                None,
+                None,
+                report,
+                gitignore,
             )
         )
         self.assertEqual(sorted(expected), sorted(sources))
@@ -1572,25 +1619,21 @@ def test_empty_include(self) -> None:
                 empty,
                 re.compile(black.DEFAULT_EXCLUDES),
                 None,
+                None,
                 report,
                 gitignore,
             )
         )
         self.assertEqual(sorted(expected), sorted(sources))
 
-    def test_empty_exclude(self) -> None:
+    def test_extend_exclude(self) -> None:
         path = THIS_DIR / "data" / "include_exclude_tests"
         report = black.Report()
         gitignore = PathSpec.from_lines("gitwildmatch", [])
-        empty = re.compile(r"")
         sources: List[Path] = []
         expected = [
-            Path(path / "b/dont_exclude/a.py"),
-            Path(path / "b/dont_exclude/a.pyi"),
             Path(path / "b/exclude/a.py"),
-            Path(path / "b/exclude/a.pyi"),
-            Path(path / "b/.definitely_exclude/a.py"),
-            Path(path / "b/.definitely_exclude/a.pyi"),
+            Path(path / "b/dont_exclude/a.py"),
         ]
         this_abs = THIS_DIR.resolve()
         sources.extend(
@@ -1598,7 +1641,8 @@ def test_empty_exclude(self) -> None:
                 path.iterdir(),
                 this_abs,
                 re.compile(black.DEFAULT_INCLUDES),
-                empty,
+                re.compile(r"\.pyi$"),
+                re.compile(r"\.definitely_exclude"),
                 None,
                 report,
                 gitignore,
@@ -1606,8 +1650,8 @@ def test_empty_exclude(self) -> None:
         )
         self.assertEqual(sorted(expected), sorted(sources))
 
-    def test_invalid_include_exclude(self) -> None:
-        for option in ["--include", "--exclude"]:
+    def test_invalid_cli_regex(self) -> None:
+        for option in ["--include", "--exclude", "--extend-exclude", "--force-exclude"]:
             self.invokeBlack(["-", option, "**()(!!*)"], exit_code=2)
 
     def test_preserves_line_endings(self) -> None:
@@ -1656,7 +1700,14 @@ def test_symlink_out_of_root_directory(self) -> None:
         try:
             list(
                 black.gen_python_files(
-                    path.iterdir(), root, include, exclude, None, report, gitignore
+                    path.iterdir(),
+                    root,
+                    include,
+                    exclude,
+                    None,
+                    None,
+                    report,
+                    gitignore,
                 )
             )
         except ValueError as ve:
@@ -1670,7 +1721,14 @@ def test_symlink_out_of_root_directory(self) -> None:
         with self.assertRaises(ValueError):
             list(
                 black.gen_python_files(
-                    path.iterdir(), root, include, exclude, None, report, gitignore
+                    path.iterdir(),
+                    root,
+                    include,
+                    exclude,
+                    None,
+                    None,
+                    report,
+                    gitignore,
                 )
             )
         path.iterdir.assert_called()


Code review comments:
diff_hunk: @@ -1433,6 +1443,35 @@ def test_get_sources_with_stdin_filename_and_exclude(self) -> None:
                 verbose=False,
                 include=include,
                 exclude=exclude,
+                extend_exclude=None,
+                force_exclude=None,
+                report=report,
+                stdin_filename=stdin_filename,
+            )
+        )
+        self.assertEqual(sorted(expected), sorted(sources))
+
+    @patch("black.find_project_root", lambda *args: THIS_DIR.resolve())
+    def test_get_sources_with_stdin_filename_and_extend_exclude(self) -> None:
+        # Extend exclude shouldn't exclude stdin_filename since it is mimicing the
path: tests/test_black.py
text: @user1:
typo :)
```suggestion
        # Extend exclude shouldn't exclude stdin_filename since it is mimicking the
```