Repository: psf/black

Issue:
Black stops looking for config at first pyproject.toml file even if there's no black section in it
<!--
Please make sure that the bug is not already fixed either in newer versions or the
current development version. To confirm this, you have three options:

1. Update Black's version if a newer release exists: `pip install -U black`
2. Use the online formatter at <https://black.vercel.app/?version=main>, which will use
   the latest main branch.
3. Or run _Black_ on your machine:
   - create a new virtualenv (make sure it's the same Python version);
   - clone this repository;
   - run `pip install -e .[d]`;
   - run `pip install -r test_requirements.txt`
   - make sure it's sane by running `python -m pytest`; and
   - run `black` like you did last time.
-->
Disclaimer: I was hesitant whether I should report this as a bug or rather a "Feature request".

**Describe the bug**

<!-- A clear and concise description of what the bug is. -->
In a project dealing with multiple sub-projects in a single repository (aka. mono-repo style) having more than one `pyproject.toml` file confuses black.
In our example we wanted to have one top-level black configuration,
while also having sub-project specific `pyproject.toml` files (without any black config) in each sub-directory.
Unfortunately, running black on a sub-directory
uses default configuration instead of the one defined in the top-level config file.

**To Reproduce**

<!--
Minimal steps to reproduce the behavior with source code and Black's configuration.
-->
Given this file structure:
```
.
|-- pyproject.toml
|-- sub1
|   |-- pyproject.toml
|   `-- test.py
`-- sub2
    `-- test.py
```

With the following contents:
```
==> ./pyproject.toml <==
[tool.black]
line-length = 20

==> ./sub1/pyproject.toml <==
[tool.isort]
profile = "black"

==> ./sub1/test.py <==
a_very_long_line = {"that should be wrapped": "to fit 20 characters line length"}

==> ./sub2/test.py <==
a_very_long_line = {"that should be wrapped": "to fit 20 characters line length"}
```

Running black on both test.py files:
```sh
$ black -v sub1/test.py
Identified `/private/tmp/black-many-configs/sub1` as project root containing a pyproject.toml.
Sources to be formatted: "test.py"
sub1/test.py already well formatted, good job.

All done! ✨ 🍰 ✨
1 file left unchanged.

$ black -v sub2/test.py
Identified `/private/tmp/black-many-configs` as project root containing a pyproject.toml.
Sources to be formatted: "sub2/test.py"
Using configuration from project root.
reformatted sub2/test.py

All done! ✨ 🍰 ✨
1 file reformatted.
```

Modifies only the file inside `sub2` directory (the one without `pyproject.toml` file)
even though `sub1/pyproject.toml` doesn't contain any black config section.

**Expected behavior**

<!-- A clear and concise description of what you expected to happen. -->
Both files should be modified
and black should continue looking for a file
that is relevant for its configuration (in this case, the one that has a `[tool.black]` section).

**Environment**

<!-- Please complete the following information: -->

- Black's version: black, 22.1.0 (compiled: yes)
- OS and Python version: OSX - Python 3.9.9

**Additional context**

<!-- Add any other context about the problem here. -->

A variation of this problem has already been reported in https://github.com/psf/black/issues/1826, where the proposed solution is to introduce support for the `config` option inside the sub-project's `pyproject.toml` config file.

Other (potentially) relevant issues: https://github.com/psf/black/issues/2537, https://github.com/psf/black/issues/2850.


Discussion:
Hi! This is indeed intended behavior. I'm not sure if we should go beyond if we don't find a Black section, but I'm not totally against it either. But would #1826 be enough for your use case? It has an active PR.
> But would https://github.com/psf/black/issues/1826 be enough for your use case?

Thanks. I think it would, yes.
Although IMHO this behaviour is confusing, given that `pyproject.toml` is becoming (or even already is?) the de-facto standard for tooling configuration.
> Although IMHO this behaviour is confusing

Could you elaborate a bit? The way I see it, #1826 would make the association explicit. If you wanted to leave one of your projects' configuration empty, you can now leave it empty, but you would have to have a `tool.black` with nothing in it. That's fine, but forgetting to put it could be confusing. Honestly, not sure, but I like the explicitness and lack of guessing what the intent is.
Hey 👋🏻,

Firstly, according to me none of your linked relevant issues is really relevant to this, so that could be edited way. Secondly, yeah for now #2525 (PR for 1826) should work by adding `config = "../pyproject.toml` to `sub1/pyproject.toml`.

This is also an issue when you are working with mono repo, say you have `black` as a repository with `blib2to3` as separate repositories within `black`. Now, if you are in `blib2to3` and the black config is outside in the root directory `black`, it would never go beyond `blib2to3` as it exists after finding a `.git` folder inside it.

A possible solution to this would be to go through all the `common_base.parents` and find the ones which fulfil the condition of `.git/.hg/pyproject.toml`. Now you check for `pyproject.toml` in them and see if they are valid, and you return the first path which has valid `pyproject.toml`. Okay, this is a bit confusing ig, so say if you have:
```
/, /home/, /home/shivansh/, /home/shivansh/oss/, /home/shivansh/oss/black, /home/shivansh/oss/black/blib2to3
```
as the `common_base.parents`, now if you check for those which fulfil the conditions, and you get
```
/home/shivansh/oss/, /home/shivansh/oss/black, /home/shivansh/oss/black/blib2to3
```
you check for valid `pyproject.toml` and get `oss/` and `oss/black` have valid configs, but since `oss/black` is "higher" you return that. If none of them is valid, you return the as-it-is root which would be `blib2to3`, if you have no path fulfilling the condition, the root is `/` as we have now.

Note: This is only for finding the config, the src root would stay the same as now.

Well yeah, this has the same problem as Felix mentioned, where you would need to have an empty black config in case you want to have an empty config.

--- 

The second solution, I can think of is to have no special case for this and just make the users link `config=` (#2525) to the "actual" root config. I am slightly leaning towards this for simplicity.


I feel we should only find the first pyproject.toml in the hierarchy and just error that there is no black config in the file we find. Then the user can fix their repo. I don't think black should do anything clever here. I'm a big KISS (Keep it simple stupid) person. The second solution to allow specifying the config sounds a good middle ground.

Am I missing something here?

E.g. .flake8 does this, but is more explicit as it uses a non shared config file, but if an an empty / incomplete config file exists it will cause unexpected behaviors ... I'm not saying this is ideal just comparing to other tools in the ecosystem
> Could you elaborate a bit?

Oh, sorry @felix-hilden, I didn't mean the behaviour/solution in #1826. It would improve the current situation and it is explicit (even if slightly more verbose).

What I meant was that the assumption that black config is and should be in the first encountered `pyproject.toml` might be surprising to the user. In this vain raising an error as @cooperlees is proposing is also surprising, at least to me. I mean, wouldn't it be surprising to you when you have your tool X config in exactly one place in the same directory you're invoking that tool from and it doesn't get picked up?

I guess (and I might be entirely off here, so correct me if needed or ignore this paragraph) the problem origins from the facts that (a) mono-repo layouts weren't considered when black was being conceived and (b) back in the day black was one of the few tools that was using `pyproject.toml` file for config, so I guess its assumption was "oh, I see this file, no one else probably cares about it, so my config must be in there". Yes, I've just anthropomorphised black 😄.

On the other hand, it's hard to find consistency in other tools for this behaviour. I personally like the way `isort` discovers config. Mypy for example doesn't even go beyond current working directory when searching for config. So... 🤷 
Just looked at how `isort` handles config, and I really like the whole config component. It's somewhat similar to what I proposed above in the first solution, not sure how it handles the empty config problem though

> Well yeah, this has the same problem as Felix mentioned, where you would need to have an empty black config in case you want to have an empty config.
Looks like [Where Black looks for the file](https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#where-black-looks-for-the-file) does not make this special situation (multiple configs) better:
```
By default Black looks for pyproject.toml starting from the common base directory of all files and directories passed on the command line.
```
Had that issue just now. I would propose to
- only search for a config starting from each file independently. This way it is always clear which file is chosen, independently how black is called (one or multiple files)
- skip configs that do not contain a black tool section until one is found (in `/` 🙈 )
This issue seems related to #2598 (just came across it)
I also find it pretty problematic in a monorepo setup:
- pre-commit hook doesn't work as you can expect unless you provide `--config` flag explicitly, which can be really confusing (it took me a while to understand why files are formatted with no respect to the config), but when you specify it eventually - it works fine
- VS Code is more problematic - here you also have to specify `Python › Formatting: Black Args` explicitly, but that's more tricky because there's no obvious way to share the config with your team - workspace-level settings.json is not universal, so you need to create sth like settings.json.default and tell people to include manually / with some custom script the settings specified there in their own config. That's ofc doable, but it's a pretty annoying experience for new-comers.

If Black would scan all the `pyproject.toml` files up to the root of a monorepo - nothing from the above would be needed. And empty Black section in `pyproject.toml` as a way to say "don't scan further, just use Black defaults" should provide pretty smooth experience.

EDIT: I just had a look at #2525 - that should also be fine, I can handle specifying the path during project generation with `cookiecutter` then. It would be nice to be able to specify the path also from the project root, not only relatively, but I understand that would complicate the setup. @Shivansh-007 - is that PR expected to be merged anytime soon?
@Shivansh-007 I did a quick test with `isort` - if there's intermediate `pyproject.toml`, but there's no `isort` section there - it searches further and finally pick up the config from the root, so that's exactly the behaviour we would like to see also in Black.
Yeah, that PR is expected to merge sometime soon, it is waiting for @ichard26's review, which would probably be small changes if not an approval.

Yeah thanks for looking into that, so it's similar to what I proposed originally.
@Shivansh-007 any updates? We have already centralised almost all the configs in our monorepo, but `black` is still a problem 😕
I have been super busy for the past month or two due to school, so I haven't been able to respond to any of the open projects, so no updates as of now.

I just had a look at the PR and looks like it's still waiting for @ichard26's review.


It doesn't seem like #2525 is going to be merged soon. I'm curious to hear how others are working around this
Ruff does this https://docs.astral.sh/ruff/configuration/#config-file-discovery

Resolving patch:
diff --git a/src/black/files.py b/src/black/files.py
--- a/src/black/files.py
+++ b/src/black/files.py
@@ -42,6 +42,12 @@
     import colorama  # noqa: F401
 
 
+@lru_cache
+def _load_toml(path: Union[Path, str]) -> Dict[str, Any]:
+    with open(path, "rb") as f:
+        return tomllib.load(f)
+
+
 @lru_cache
 def find_project_root(
     srcs: Sequence[str], stdin_filename: Optional[str] = None
@@ -84,7 +90,9 @@ def find_project_root(
             return directory, ".hg directory"
 
         if (directory / "pyproject.toml").is_file():
-            return directory, "pyproject.toml"
+            pyproject_toml = _load_toml(directory / "pyproject.toml")
+            if "black" in pyproject_toml.get("tool", {}):
+                return directory, "pyproject.toml"
 
     return directory, "file system root"
 
@@ -117,8 +125,7 @@ def parse_pyproject_toml(path_config: str) -> Dict[str, Any]:
 
     If parsing fails, will raise a tomllib.TOMLDecodeError.
     """
-    with open(path_config, "rb") as f:
-        pyproject_toml = tomllib.load(f)
+    pyproject_toml = _load_toml(path_config)
     config: Dict[str, Any] = pyproject_toml.get("tool", {}).get("black", {})
     config = {k.replace("--", "").replace("-", "_"): v for k, v in config.items()}
 


Test patch:
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -1668,9 +1668,9 @@ def test_find_project_root(self) -> None:
             src_dir.mkdir()
 
             root_pyproject = root / "pyproject.toml"
-            root_pyproject.touch()
+            root_pyproject.write_text("[tool.black]", encoding="utf-8")
             src_pyproject = src_dir / "pyproject.toml"
-            src_pyproject.touch()
+            src_pyproject.write_text("[tool.black]", encoding="utf-8")
             src_python = src_dir / "foo.py"
             src_python.touch()
 
@@ -1693,6 +1693,20 @@ def test_find_project_root(self) -> None:
                     (src_dir.resolve(), "pyproject.toml"),
                 )
 
+            src_sub = src_dir / "sub"
+            src_sub.mkdir()
+
+            src_sub_pyproject = src_sub / "pyproject.toml"
+            src_sub_pyproject.touch()  # empty
+
+            src_sub_python = src_sub / "bar.py"
+
+            # we skip src_sub_pyproject since it is missing the [tool.black] section
+            self.assertEqual(
+                black.find_project_root((src_sub_python,)),
+                (src_dir.resolve(), "pyproject.toml"),
+            )
+
     @patch(
         "black.files.find_user_pyproject_toml",
     )