Repository: pantsbuild/pants

Issue:
Black writes to shared `$XDG_CACHE_DIR/black` cache directory
**Describe the bug**

Black has its own global caching mechanism, to avoid reformatting files that have already been formatted. 
- https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#black-cache-dir
- https://black.readthedocs.io/en/stable/usage_and_configuration/file_collection_and_discovery.html#ignoring-unmodified-files


Pants doesn't currently seem to consider this, it seems like we should either:

1. explicitly support Black writing to a global one, by passing through the `XDG_CACHE_DIR` and/or `BLACK_CACHE_DIR` env vars automatically(?)
2. writing it to a Pants-managed named/append-only cache
3. disabling it entirely, and just rely on Pants' process level caching


Reproducer:

```shell
cd $(mktemp -d)

default_cache_dir=~/Library/Caches/black # or ~/.cache/black, or something else

# WARNING: this is mutates your disk! Check before running
clear_global_cache() {
    echo "clearing '$default_cache_dir'"
    rm -rf "$default_cache_dir"
}

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.19.0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.black",
]

[python]
interpreter_constraints = ["==3.*"]
EOF

echo 'python_sources(name="src")' > BUILD

echo 'a' > file.py

# This bug relates to when black is actually executed, so let's get the Pants caches out of the way
export PANTS_PANTSD=false PANTS_LOCAL_CACHE=false

# BUG 1: writes to cache with default settings
clear_global_cache
pants fmt ::
tree "$default_cache_dir"

# BUG 2: writes to cache even with (attempted) override
tmp_cache_dir="temp-cache"
clear_global_cache
BLACK_CACHE_DIR="$tmp_cache_dir" pants fmt ::

tree "$default_cache_dir"
tree "$tmp_cache_dir"
```

This sets up pants to run black in two configurations, starting with an empty global/default black cache each time. In both configurations, that global cache is written to:

1. default settings
2. with an explicit override via `BLACK_CACHE_DIR`

**Pants version**
2.19.0

**OS**
macOS

**Additional info**
N/A


Discussion:
This cache seems to use a few "all-encompassing" pickle files, that get loaded and then updated with any new files (see https://black.readthedocs.io/en/stable/usage_and_configuration/file_collection_and_discovery.html#ignoring-unmodified-files)... it appears that the writes after an update are at least (attempted) to be done atomically (https://github.com/psf/black/blob/d1d4fc58d3b744db35ac832cddf1d076ba5f5e7f/src/black/cache.py#L138-L148).

I think this likely means that concurrent executions:

- may overwriting on each other's updates
- won't totally corrupt the file itself
 
(For instance: process A formats `a.py` and adds that to the pickle, while B formats `b.py` and adds that to the pickle. Depending on the order of the reads and writes, the resulting file will still be a sensible pickle, without torn writes or other corruption, but might have info on `a.py` and not `b.py`, `b.py` and not `a.py`, or both.)

Given Pants' caching, I'd be inclined to disable the cache entirely. It doesn't look like there's an option for this, but we could always set `BLACK_CACHE_DIR` to point to a local path within the sandbox, that isn't persisted after execution.

Thoughts?
I'm happy to take this on - and I agree that the best option is probably to disable/neutralize Black's caching since it might conflict with Pants and we have a potential race condition when Black is run in parallel.

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/black/rules.py b/src/python/pants/backend/python/lint/black/rules.py
--- a/src/python/pants/backend/python/lint/black/rules.py
+++ b/src/python/pants/backend/python/lint/black/rules.py
@@ -57,6 +57,10 @@ async def _run_black(
             ),
             input_digest=input_digest,
             output_files=request.files,
+            # Note - the cache directory is not used by Pants,
+            # and we pass through a temporary directory to neutralize
+            # Black's caching behavior in favor of Pants' caching.
+            extra_env={"BLACK_CACHE_DIR": "__pants_black_cache"},
             concurrency_available=len(request.files),
             description=f"Run Black on {pluralize(len(request.files), 'file')}.",
             level=LogLevel.DEBUG,


Test patch: