Repository: pantsbuild/pants

Issue:
Coarsened target calculation for `pylint` pulls in more files than it should on v2.15.0a0
**Describe the bug**

While testing `v2.15.0a0`, I saw that `./pants lint` would consistently freeze / be OOM killed. Through `py-spy` and logging I found that:

* Much time was being spent merging together source digests when setting up `pylint` runs
* The number of source digests in each of ^^^ setups was much larger than expected (tens of thousands of files for an input batch size of a few hundred)
* `pylint` batches of different sizes always ended up with the same number of source digests

While looking through the `pylint` changes in v2.15, I found that coarsened target calculation is currently running in the "partitioner" rule (see [here](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/lint/pylint/rules.py#L89-L92)). This will result in too many targets being associated with each `pylint` batch, because the partitions returned by that rule are re-batched into smaller chunks according to `[lint].batch_size`, and there's no support in the re-batching logic for subsetting the partition metadata.

We should push the calculation of coarsened targets into the "runner" rule for `pylint`, so we only compute & hydrate the transitive dependencies that are relevant for the specific inputs used in each batch.

**Pants version**

v2.15.0a0

**OS**

Both


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/pylint/rules.py b/src/python/pants/backend/python/lint/pylint/rules.py
--- a/src/python/pants/backend/python/lint/pylint/rules.py
+++ b/src/python/pants/backend/python/lint/pylint/rules.py
@@ -118,10 +118,20 @@ async def run_pylint(
 ) -> LintResult:
     assert request.partition_metadata is not None
 
+    # The coarsened targets in the incoming request are for all targets in the request's original
+    # partition. Since the core `lint` logic re-batches inputs according to `[lint].batch_size`,
+    # this could be many more targets than are actually needed to lint the specific batch of files
+    # received by this rule. Subset the CTs one more time here to only those that are relevant.
+    all_coarsened_targets_by_address = request.partition_metadata.coarsened_targets.by_address()
+    coarsened_targets = CoarsenedTargets(
+        all_coarsened_targets_by_address[field_set.address] for field_set in request.elements
+    )
+    coarsened_closure = tuple(coarsened_targets.closure())
+
     requirements_pex_get = Get(
         Pex,
         RequirementsPexRequest(
-            (target.address for target in request.partition_metadata.coarsened_targets.closure()),
+            (target.address for target in coarsened_closure),
             # NB: These constraints must be identical to the other PEXes. Otherwise, we risk using
             # a different version for the requirements than the other two PEXes, which can result
             # in a PEX runtime error about missing dependencies.
@@ -140,7 +150,7 @@ async def run_pylint(
 
     sources_get = Get(
         PythonSourceFiles,
-        PythonSourceFilesRequest(request.partition_metadata.coarsened_targets.closure()),
+        PythonSourceFilesRequest(coarsened_closure),
     )
     # Ensure that the empty report dir exists.
     report_directory_digest_get = Get(Digest, CreateDigest([Directory(REPORT_DIR)]))


Test patch: