Repository: psf/black

Issue:
Apply excludes before normalising symlinks
**Is your feature request related to a problem? Please describe.**

Our project contains symlinks like this:

```
project
+- a
   +- b -> ../x/y
   +- other files
+- x
   +- y
      +- lots of files
```

We would like to avoid black spending time formatting/complaining about the same files twice, so we make the excludes regex match `/a/b/`.
This doesn't help because the excludes are evaluated after normalising the path to `/x/y/`.
We cannot put that in the excludes because we do want to format the files.

**Describe the solution you'd like**

Excludes could be evaluated before normalising the path, or at least before evaluating symlinks.

**Describe alternatives you've considered**

Perhaps it would be best to evaluate excludes both before and after normalising the path.
There is also a use case for the way it is now, if we did want to exclude the files completely without listing all paths to them in the excludes.
Force Exclude does not work with symlinks
**Describe the bug**
I use the `force-exclude` feature to exclude files living in symlinked directories.
However, it looks like `black` uses the regexp from `force-exclude` **after** the symlink is resolved (not the symlinked path).

```
$ black -v /path_2_file/application/api/lib/dir_symlinked/__init__.py
Identified `/path_2_file` as project root containing a .git directory.
Using configuration from project root.
line_length: 135
target_version: ['py39']
force_exclude: (?:.*?application/api(?:-ext)?/lib/dir_symlinked/.+)

skip_string_normalization: True
Found input source: "application/lib/ref_dir/__init__.py"
/path_2_file/application/api/lib/dir_symlinked/__init__.py wasn't modified on disk since last run.

All done! ✨ 🍰 ✨
1 file left unchanged.
```

The resulting error is:

It should have ignored the file based on this path `/path_2_file/application/api/lib/dir_symlinked/__init__.py` rather than including it based on this path `application/lib/ref_dir/__init__.py`.
Said differently it used the resolved/resulting path rather than the path through the symlink

**Environment**

- Black's version: 23.7.0
- OS and Python version: MacOS Ventura 13.4.1, Python 3.9.12, 

**Additional context**

I would be equally happy if I could simply tell `black` **not** to follow symlinks.

Thank you in advance for your help!


Discussion:
This is a tough one to get right. I wonder if we could just simplify it add a config option (in pyproject.toml) to not follow symlink at all? I feel that would help your use case here. Thoughts?
Given the fact that symlinks pointing outside the project are rejected already, not following symlinks at all would be the easiest option. There should be no need to follow symlinks since black has no need to get the module hierarchy right (one file at a time).

Even without following symlinks, it should be possible to use the current configuration options to reach any file within the project.
I'm in favour of this, at least for the gitignore excludes; I was surprised this was the current behaviour when making https://github.com/psf/black/pull/3751


Resolving patch:
diff --git a/src/black/files.py b/src/black/files.py
--- a/src/black/files.py
+++ b/src/black/files.py
@@ -330,35 +330,37 @@ def gen_python_files(
 
     assert root.is_absolute(), f"INTERNAL ERROR: `root` must be absolute but is {root}"
     for child in paths:
-        normalized_path = normalize_path_maybe_ignore(child, root, report)
-        if normalized_path is None:
-            continue
+        root_relative_path = child.absolute().relative_to(root).as_posix()
 
         # First ignore files matching .gitignore, if passed
         if gitignore_dict and _path_is_ignored(
-            normalized_path, root, gitignore_dict, report
+            root_relative_path, root, gitignore_dict, report
         ):
             continue
 
         # Then ignore with `--exclude` `--extend-exclude` and `--force-exclude` options.
-        normalized_path = "/" + normalized_path
+        root_relative_path = "/" + root_relative_path
         if child.is_dir():
-            normalized_path += "/"
+            root_relative_path += "/"
 
-        if path_is_excluded(normalized_path, exclude):
+        if path_is_excluded(root_relative_path, exclude):
             report.path_ignored(child, "matches the --exclude regular expression")
             continue
 
-        if path_is_excluded(normalized_path, extend_exclude):
+        if path_is_excluded(root_relative_path, extend_exclude):
             report.path_ignored(
                 child, "matches the --extend-exclude regular expression"
             )
             continue
 
-        if path_is_excluded(normalized_path, force_exclude):
+        if path_is_excluded(root_relative_path, force_exclude):
             report.path_ignored(child, "matches the --force-exclude regular expression")
             continue
 
+        normalized_path = normalize_path_maybe_ignore(child, root, report)
+        if normalized_path is None:
+            continue
+
         if child.is_dir():
             # If gitignore is None, gitignore usage is disabled, while a Falsey
             # gitignore is when the directory doesn't have a .gitignore file.


Test patch:
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -492,9 +492,7 @@ def test_false_positive_symlink_output_issue_3384(self) -> None:
         project_root = Path(THIS_DIR / "data" / "nested_gitignore_tests")
         working_directory = project_root / "root"
         target_abspath = working_directory / "child"
-        target_contents = (
-            src.relative_to(working_directory) for src in target_abspath.iterdir()
-        )
+        target_contents = list(target_abspath.iterdir())
 
         def mock_n_calls(responses: List[bool]) -> Callable[[], bool]:
             def _mocked_calls() -> bool:
@@ -2387,38 +2385,48 @@ def test_extend_exclude(self) -> None:
         )
 
     @pytest.mark.incompatible_with_mypyc
-    def test_symlink_out_of_root_directory(self) -> None:
+    def test_symlinks(self) -> None:
         path = MagicMock()
         root = THIS_DIR.resolve()
-        child = MagicMock()
         include = re.compile(black.DEFAULT_INCLUDES)
         exclude = re.compile(black.DEFAULT_EXCLUDES)
         report = black.Report()
         gitignore = PathSpec.from_lines("gitwildmatch", [])
-        # `child` should behave like a symlink which resolved path is clearly
-        # outside of the `root` directory.
-        path.iterdir.return_value = [child]
-        child.resolve.return_value = Path("/a/b/c")
-        child.as_posix.return_value = "/a/b/c"
-        try:
-            list(
-                black.gen_python_files(
-                    path.iterdir(),
-                    root,
-                    include,
-                    exclude,
-                    None,
-                    None,
-                    report,
-                    {path: gitignore},
-                    verbose=False,
-                    quiet=False,
-                )
+
+        regular = MagicMock()
+        outside_root_symlink = MagicMock()
+        ignored_symlink = MagicMock()
+
+        path.iterdir.return_value = [regular, outside_root_symlink, ignored_symlink]
+
+        regular.absolute.return_value = root / "regular.py"
+        regular.resolve.return_value = root / "regular.py"
+        regular.is_dir.return_value = False
+
+        outside_root_symlink.absolute.return_value = root / "symlink.py"
+        outside_root_symlink.resolve.return_value = Path("/nowhere")
+
+        ignored_symlink.absolute.return_value = root / ".mypy_cache" / "symlink.py"
+
+        files = list(
+            black.gen_python_files(
+                path.iterdir(),
+                root,
+                include,
+                exclude,
+                None,
+                None,
+                report,
+                {path: gitignore},
+                verbose=False,
+                quiet=False,
             )
-        except ValueError as ve:
-            pytest.fail(f"`get_python_files_in_dir()` failed: {ve}")
+        )
+        assert files == [regular]
+
         path.iterdir.assert_called_once()
-        child.resolve.assert_called_once()
+        outside_root_symlink.resolve.assert_called_once()
+        ignored_symlink.resolve.assert_not_called()
 
     @patch("black.find_project_root", lambda *args: (THIS_DIR.resolve(), None))
     def test_get_sources_with_stdin(self) -> None: