Repository: psf/black

Issue:
gitignore rules are not prominently documented and ignore checked in files
My `.gitignore` is quite simple. It forces me to explicitly add new files or use `-u`; which is good practice and I don't have to worry about adding a bunch of exclusion rules as things pile up.

```
*
```

However black's "convenient" functionality of using the `.gitignore` to decide which files to exclude *without taking into account whether or not the file is checked in* means I cannot simply apply `black` to my source directory. 

```sh
> black src/
No Python files are present to be formatted. Nothing to do 😴
> rm .gitignore 
> black src/
All done! ✨ 🍰 ✨
6 files left unchanged.
> black --version
black, version 21.4b2
```

This is not the first large project I've seen where "convenient" features eventually weasel their way in. They might help the common case, but can totally defy other user's expectations and are *rarely* documented (\*cough cough\* `setuptools_scm` \*cough\*). It's mad. Resist the temptation please. Code formatting has *nothing* to do with version control. If someone thinks it's important, let it be a plugin and extra: `black[vcs]`


Discussion:
Hi,

> prominently documented 

I do agree that this feature was poorly documented and needs to be documented better. I'll definitely take this into account once I am done with the other things I'm doing. Matter of fact I've been planning for a little while to put together a "Common Issues & FAQ" page for situations just like this one! I'm sorry for the amount of confusion and frustration our lack of documentation has caused.

> "convenient"

> If someone thinks it's important

This is where I am going to seem harsh and inconsiderate but no, this _is a useful and convenient feature_. Is it for you? No, but we as maintainers must take into account the needs of all of our users when considering potential options. Since the year and a half this feature has been in the wild, this is the second complaint I've seen from users about this feature. On the other hand, the issue requesting this feature (https://github.com/psf/black/issues/475) has seen much more positive feedback from the community. Are issue counts and +1s a perfect representation of the userbase and their needs, wishes, and issues? No, but they are one of the limited ways we can get real world data.

> plugin and extra: black[vcs]

I'd argue that it would be better to make this feature configurable via a flag or option since turning this feature on whether `pathspec` is installed would lead to confusion. 

So that leaves me to consider two options (other maintainers may have different opinions):

1. Add a flag / option and **disable this feature by default**; or
2. Add a flag / option and **keep this feature enabled by default**

I personally prefer no. 2 because while disabling the feature by default might not lead to Black touching important files, it does lead Black to format files it most likely shouldn't, increasing runtime. That was the main argument for why this feature should be added anyway. Black has already gotten slower in some cases to avoid bugs, and I'd rather not the average runtime increase even more. Option 1 is not a deal breaker for me, but it's going to need someone else to come by in support for it.

Finally for you personally, a workaround is to use pre-commit. Pre-commit can handle the automatic file discovery Black does too and it passes the files directly to Black via the command line. Files and directories passed via the command line aren't affected by --include, --exclude, or .gitignore.

I hope that explains my viewpoint well enough. Thank you for reporting, it's nice to know more about our users' experiences.
I'm not saying it's totally bad feature. My point of view is that introducing a convenience (read optional, but positive) should not be optimizing the common *at the detriment of the uncommon case*. We aren't talking about total program performance here, where taking a small hit somewhere for the advantage of overall performance increase is okay; we are talking about people. 

On second inspection I think the issue is more that the implementation of this feature is incorrect, a file can be excluded while also being version controlled. The proper logic for exclusion is `ignored and not version_controlled`.

Of course I understand this will likely require the project to take on a dependency on `pygit` for a feature that not everyone will use, which isn't ideal. And of course other VCSs go without attention, maybe some user would like SVN support? Should that be added? And they will likely need lots of options to configure those features... With VCS support there is an unbounded interface on the project that will either balloon, or require some arbitrary line in the sand to be drawn that will peeve the users who end up on the opposite side of the line. It's not a good place to be.

If you do not intend to fix the logic, I would do the least intrusive thing, which is option 1. 

I've worked around the issue for now by changing my `.gitignore` to the following. I doubt there will be any Python source files in my project I do not intend to check in if I already haven't.
```
*
# to support black
!*.py
```

> use pre-commit

I am not a fan of modifying source files in pre-commit. I already use `black --check` in pre-commit.

And I hope you understand when I say "you" I don't mean you personally, but you "the project". Thank you for responding.
> convenience (read optional, but positive) should not be optimizing the common at the detriment of the uncommon case. 

My viewpoint is the opposite to a degree. Now optimizing for the common case and blocking the uncommon cases from ever working? That crosses my line in the sand, but in general, we can't get this perfect. Yes there's the people component to this issue, but it's about tradeoffs here, if there's a bunch of users that are all hitting this issue and find it annoying, sure, I wouldn't mind disrupting most our users and going with No. 1. And that's the next point I want to make, Option 1. is intrusive in its own way since I'm sure plenty of people are now dependant on this feature (including me! - I hope that covers my biases) and yanking it from under them would certainly be annoying and frustrating. Yes it's an "one time" (well sorta, some people may never notice the change and will face the sudden increase in runtime without knowing how to fix it) cost, but I am not convinced it's worth it.

> The proper logic for exclusion is `ignored and not version_controlled`

~~I mean, maybe? It totally makes sense to me to have files checked into version control but are still excluded from Black. Maybe the file is Python 2 and can't be formatted by Black. Or it purposefully has bad formatting or even invalid syntax (e.g, our `./tests/data` directory in this repo).~~ this comment was made under a wrong interpretation of what "ignored" and "checked into" meant, please ignore, it's wrong :(

Finally, I'm going to ping some other maintainers to see what their thoughts are about this /cc @JelleZijlstra @cooperlees 
One more argument is that `flake8` does not support this feature and just runs forever. Although it doesn't even include reasonable default exclusions like `.tox/`, `.nox/` and `.eggs/`, etc. which might be too far in the other direction.

> I mean, maybe? It totally makes sense to me to have files checked into version control but are still excluded from Black. Maybe the file is Python 2 and can't be formatted by Black.

This is not the purpose of a `.gitignore`. This is what a configuration file is for.

The previous comment was troubling so I decided to try configuring via `pyproject.toml` to set the exclude and include to what I would expect and found that apparently the `.gitignore` rules take precedence over actual configurations! This is really bad...
> This is not the purpose of a .gitignore. This is what a configuration file is for.

Sorry, I misunderstood what "ignored" and "checked in" meant in this context (spot the person who is an idiot when it comes to anything remotely advanced about source version control). Let's just move on about this topic between us two shall we :)  

> The previous comment was troubling so I decided to try configuring via pyproject.toml to set the exclude and include to what I would expect and found that apparently the .gitignore rules take precedence over actual configurations! This is really bad...

Taking a look at the code, we treat .gitignore as just another exclude type. Exclusions are calcuated first, then inclusions. This is documented in the help for --exclude and --include:

```text
  --include TEXT                  A regular expression that matches files and
                                  directories that should be included on
                                  recursive searches.  An empty value means
                                  all files are included regardless of the
                                  name.  Use forward slashes for directories
                                  on all platforms (Windows, too).  Exclusions
                                  are calculated first, inclusions later.
                                  [default: \.pyi?$]
```

I wouldn't consider it troubling per se. Not ideal but unless we want to redo how all of the file discovery works, I think that's a bit out of scope for this issue. FYI this is what was decided as the behaviour on that issue: 

>  Because we have --include, I think the most sensible option is to merge gitignore with the value of --exclude; if there's no --exclude, merge it with the default. Values in pyproject.toml basically replace the default value for command line options.

https://github.com/psf/black/issues/475#issuecomment-444278828
> Because we have --include, I think the most sensible option is to merge gitignore with the value of --exclude; if there's no --exclude, merge it with the default. Values in pyproject.toml basically replace the default value for command line options.

Oof... I absolutely disagree. Precedence is about *overriding* previous configurations. `.gitignore` should be a fallback if exclusions are not set, not a fundamental truth to be merged in. It's essentially turning all setting of exclusions in the configuration file and command line into extra exclusions. However, if you specify `--exclude` on the command line it *overrides* the configuration file, but neither override the `.gitignore`? It's just not clean... Hence why you need the flag you suggested. I think using `.gitignore` while not checking if the file is being version controlled is not great; but this is an even less palatable decision.

Also, are you sure about the exclusion/inclusion logic? I just tried the following.
```sh
> rm .gitignore
> black src/ --exclude ".*" --include "\.py$"
No Python files are present to be formatted. Nothing to do 😴
> black src/ --include "\.py$"
All done! ✨ 🍰 ✨
10 files left unchanged.
```
Honestly I'm so confused where I am going to check myself out of this conversation since I'm at high risk of saying something that is wrong and/or dumb again.

If you want to check yourself, here's most of the relevant code:

https://github.com/psf/black/blob/b39999da7f451c285befac217f1f9a685774b34d/src/black/__init__.py#L6206-L6269

edit:

if you're wondering what I'm confused by, here's what:

- What is even the problem? It sounds like all of our file discovery logic should be redone
- What solutions are we considering here?
- What is the scope here? The scope is seemingly growing in size, and I'm losing track of what's being discussed here

I might come back and see if with a clearer mind, I'll be able to make sense of it, but this has changed from some well defined into something that's too much for me to make sense of easily. It's a me issue btw, chances are that your explanations are just fine and it's me who is being dumb.
I think I may have gotten slightly confused on that last bit, I don't think the documentation is clear.

> Exclusions are calculated first, inclusions later.

The first time I reasoned about it as "if inclusion is considered later, that means it has precedence", but what it really means is "files are excluded first and remaining files are then checked for inclusion". Which plainly doesn't make sense if I stopped and thought about what the default patterns were. Still I think the documentation could be made more clear and concise: "exclusion has higher precedence than inclusion."

> What is the scope here? The scope is seemingly growing in size, and I'm losing track of what's being discussed here

Most of what I mentioned was just showcasing how playing around with version control can easily balloon in scope and complexity and it's a rock best left unturned (at least in the base project). But since then I've just been trying to find a way to work around the issue cleanly without having to modify my `.gitignore`. My workflow should not have to change to accommodate a single tool's assumptions.

> What is even the problem?

The next issue is that the `.gitignore` is *always* considered and has precedence over every other rule. I don't think black should be treating `.gitignore` as another configuration file, it was never intended as such. At best it is a convenient default in the case of explicit configuration not existing. If `pyproject.toml` or the commandline receive `exclude` it should *override* `.gitignore`. `extend_exclude` will extend `.gitignore` if no `exclude` is present.

> What solutions are we considering here?

Basically, the comment you linked (https://github.com/psf/black/issues/475#issuecomment-444278828) I think is totally wrong and we should do the opposite. If `exclude` is passed to that function, `gitignore` is not considered.
Thank you very much @ktbarrett! It's pretty late where I am right now so I'll respond later with my proper (counter)arguments of my own (well maybe, you've done a decent job of warming me up to accepting more disruption to clean up the semantics - although please don't take that as "I'm accepting your solution 100%"). I just wanted to say my thanks for clarifying when you didn't have to, I appreciate it a lot. I should do better and try not to get these things wrong as a maintainer, but oh well, I suppose you gotta live and learn sometimes... Even if we two aren't able to reach an acceptable middle ground, you've certainly expanded my view on this topic so thank you for engaging me in some meaningful discussion :)

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -492,15 +492,15 @@ def validate_regex(
 @click.option(
     "--exclude",
     type=str,
-    default=DEFAULT_EXCLUDES,
     callback=validate_regex,
     help=(
         "A regular expression that matches files and directories that should be"
         " excluded on recursive searches. An empty value means no paths are excluded."
         " Use forward slashes for directories on all platforms (Windows, too)."
-        " Exclusions are calculated first, inclusions later."
+        " Exclusions are calculated first, inclusions later. [default:"
+        f" {DEFAULT_EXCLUDES}]"
     ),
-    show_default=True,
+    show_default=False,
 )
 @click.option(
     "--extend-exclude",
@@ -587,7 +587,7 @@ def main(
     quiet: bool,
     verbose: bool,
     include: Pattern,
-    exclude: Pattern,
+    exclude: Optional[Pattern],
     extend_exclude: Optional[Pattern],
     force_exclude: Optional[Pattern],
     stdin_filename: Optional[str],
@@ -662,7 +662,7 @@ def get_sources(
     quiet: bool,
     verbose: bool,
     include: Pattern[str],
-    exclude: Pattern[str],
+    exclude: Optional[Pattern[str]],
     extend_exclude: Optional[Pattern[str]],
     force_exclude: Optional[Pattern[str]],
     report: "Report",
@@ -673,7 +673,12 @@ def get_sources(
     root = find_project_root(src)
     sources: Set[Path] = set()
     path_empty(src, "No Path provided. Nothing to do 😴", quiet, verbose, ctx)
-    gitignore = get_gitignore(root)
+
+    if exclude is None:
+        exclude = re_compile_maybe_verbose(DEFAULT_EXCLUDES)
+        gitignore = get_gitignore(root)
+    else:
+        gitignore = None
 
     for s in src:
         if s == "-" and stdin_filename:
@@ -6215,12 +6220,12 @@ def path_is_excluded(
 def gen_python_files(
     paths: Iterable[Path],
     root: Path,
-    include: Optional[Pattern[str]],
+    include: Pattern[str],
     exclude: Pattern[str],
     extend_exclude: Optional[Pattern[str]],
     force_exclude: Optional[Pattern[str]],
     report: "Report",
-    gitignore: PathSpec,
+    gitignore: Optional[PathSpec],
 ) -> Iterator[Path]:
     """Generate all files under `path` whose paths are not excluded by the
     `exclude_regex`, `extend_exclude`, or `force_exclude` regexes,
@@ -6236,8 +6241,8 @@ def gen_python_files(
         if normalized_path is None:
             continue
 
-        # First ignore files matching .gitignore
-        if gitignore.match_file(normalized_path):
+        # First ignore files matching .gitignore, if passed
+        if gitignore is not None and gitignore.match_file(normalized_path):
             report.path_ignored(child, "matches the .gitignore file content")
             continue
 


Test patch:
diff --git a/tests/data/include_exclude_tests/.gitignore b/tests/data/include_exclude_tests/.gitignore
new file mode 100644
--- /dev/null
+++ b/tests/data/include_exclude_tests/.gitignore
@@ -0,0 +1 @@
+dont_exclude/
diff --git a/tests/data/include_exclude_tests/pyproject.toml b/tests/data/include_exclude_tests/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/data/include_exclude_tests/pyproject.toml
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=41.0", "setuptools-scm", "wheel"]
+build-backend = "setuptools.build_meta"
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -1418,6 +1418,32 @@ def test_include_exclude(self) -> None:
         )
         self.assertEqual(sorted(expected), sorted(sources))
 
+    def test_gitingore_used_as_default(self) -> None:
+        path = Path(THIS_DIR / "data" / "include_exclude_tests")
+        include = re.compile(r"\.pyi?$")
+        extend_exclude = re.compile(r"/exclude/")
+        src = str(path / "b/")
+        report = black.Report()
+        expected: List[Path] = [
+            path / "b/.definitely_exclude/a.py",
+            path / "b/.definitely_exclude/a.pyi",
+        ]
+        sources = list(
+            black.get_sources(
+                ctx=FakeContext(),
+                src=(src,),
+                quiet=True,
+                verbose=False,
+                include=include,
+                exclude=None,
+                extend_exclude=extend_exclude,
+                force_exclude=None,
+                report=report,
+                stdin_filename=None,
+            )
+        )
+        self.assertEqual(sorted(expected), sorted(sources))
+
     @patch("black.find_project_root", lambda *args: THIS_DIR.resolve())
     def test_exclude_for_issue_1572(self) -> None:
         # Exclude shouldn't touch files that were explicitly given to Black through the
@@ -1705,6 +1731,8 @@ def test_empty_include(self) -> None:
             Path(path / "b/.definitely_exclude/a.pie"),
             Path(path / "b/.definitely_exclude/a.py"),
             Path(path / "b/.definitely_exclude/a.pyi"),
+            Path(path / ".gitignore"),
+            Path(path / "pyproject.toml"),
         ]
         this_abs = THIS_DIR.resolve()
         sources.extend(