Repository: pantsbuild/pants

Issue:
allow excluding transitive jars from `jvm_artifact` targets
Pants v1 supported excluding certain jars from the transitive dependency graph of a `jar_libary` target. The `jvm_artifact` target type should also support exclusions. If done in a similar manner to v1, it would be via an `exclusions` field on `jvm_artifact`.

It also supported an `intransitive=True` flag, which excluded all transitive dependencies.


Discussion:
Can you use !! ignores in the dependencies field instead? 

The dependencies toolbox at https://www.pantsbuild.org/docs/targets
> Can you use !! ignores in the dependencies field instead?
> 
> The dependencies toolbox at https://www.pantsbuild.org/docs/targets

I'm not sure that would cover this case. Given `jvm_artifact` does not support `dependencies`, the user would have to write a `jvm_artifact` target and then reference it with `!!` on every target that pulled in the original dependency where we need that exclusion. The issue here is to filter that jar(s) from the classpath entirely with respect to the transitive dependencies of a specific dependency. The excluded jar at a different version might appear (and be wanted) to come in from another part of the transitive dependency graph. Easier to specify the exclusion on the `jvm_artifact` that represents the part of the dependency graph where we want the filtering.
Taking this on --  it looks like `cs fetch` has an excludes option which lets you specify partially qualified coordinates (i.e. `group:artifact` without `version`). Let's see what it would take to make that work.
I see at least two similar but slightly different use cases for exclusions:

1. Exclude a jar from a resolve entirely.
2. Exclude a jar that is a transitive dependency of a `jvm_artifact` but allow it to come in via another `jvm_artifact` (presumably at a different version). That is, influence the selection of jars by selectively pruning parts of the jar dependency  graph. This use case may be moot with lockfiles though since the user could maybe pin the transitive jar via an explicit `jvm_artifact`?

(1) points at exclusions being a per-resolve item (like Scala version), while (2) points at it being a per-`jvm_artifact` field. But depends on how the use case in (2) should be accomplished.
I had the use case in (2) in the past in Pants v1 to resolve conflicting versions of `netty` by pruning netty from one of the `jar_library` v1 targets. v1 didn't have lockfiles though, so for v2, how we would handle pinning netty in such a case probably will point at how to model this.
@tdyas I'm going to aim for solving (2) initially, since it's the easier option, which _should_ be easy enough, given that we fetch artifacts on an individual basis. We may need to think a bit harder about how to model (1).

I haven't tested coursier's behaviour if you specify a specific version to fetch, but it'd be interesting to see what happens if a higher version shows up as a transitive dependency. Let me go try that one out.
We fetch individually but we resolve lockfiles collectively by passing in all of the coordinates: https://github.com/pantsbuild/pants/blob/ad85e9bd77bd11baf257c3f0de99fb663a15c8eb/src/python/pants/jvm/resolve/coursier_fetch.py#L371

Thinking "aloud", this brings up some points about when exclusion can be applied (with some regard to whether the user writes it in a `jvm_artifact` field or in a per-resolve exclusions option):
1. If the exclusion is applied during lockfile resolution, it would be removed from resolve entirely unless Coursier had a way to prune for a specific coordinate. And given how we pass coordinate strings during lockfile resolution, not sure if that is the case.
2. If the exclusion is applied when constructing a classpath, then it could influence what subset of jars to take from the resolve.  Not sure if this corresponds to a real use case though. Thinking to the `netty` use case, if I used the `jvm_artifact` with the exclusion as a dependency and not the other `jvm_artifact` with a netty dep, then `netty` would be missing when instead I wanted to influence what `netty` version to include.
Per below, it looks like Coursier's default way to handle a complex resolve is to use the newest version of the artifact. This means we'll need to force our version of the artifact when we do the resolve (re-specifying the versions from our `jvm_artifact` targets as a conflict resolution preference should be sufficient here, tbqh).

_We will still need to specify excludes at fetch time to ensure that the transitive dependency isn't fetched (this is actually even more important than before, since we won't have a SHA digest for the excluded dependency._

```chrisjrn@chrisjrns-MacBook-Pro example-jvm % cs fetch "com.fasterxml.jackson.core:jackson-databind:2.12.2" "com.fasterxml.jackson.core:jackson-core:2.11.4" --json-output-file=coursier_report.json
chrisjrn@chrisjrns-MacBook-Pro example-jvm % less coursier_report.json | jq .
{
  "conflict_resolution": {
    "com.fasterxml.jackson.core:jackson-core:2.11.4": "com.fasterxml.jackson.core:jackson-core:2.12.2"
  },
  "dependencies": [
    {
      "coord": "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.2/jackson-annotations-2.12.2.jar",
      "directDependencies": [],
      "dependencies": []
    },
    {
      "coord": "com.fasterxml.jackson.core:jackson-core:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.2/jackson-core-2.12.2.jar",
      "directDependencies": [],
      "dependencies": []
    },
    {
      "coord": "com.fasterxml.jackson.core:jackson-databind:2.12.2",
      "file": "/Users/chrisjrn/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.2/jackson-databind-2.12.2.jar",
      "directDependencies": [
        "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
        "com.fasterxml.jackson.core:jackson-core:2.12.2"
      ],
      "dependencies": [
        "com.fasterxml.jackson.core:jackson-annotations:2.12.2",
        "com.fasterxml.jackson.core:jackson-core:2.12.2"
      ]
    }
  ],
  "version": "0.1.0"
}
```
Might be relevant: I had to disable "strict" resolution of versions due to a bug described in https://github.com/pantsbuild/pants/issues/13496. Not directly on-point with exclusions but we may want to revisit strict dependency resolution again.
So for starters, which version of `jackson-core` should we be returning by default here?

```
rule_runner.request(
        CoursierResolvedLockfile,
        [
            ArtifactRequirements.from_coordinates(
                [
                    Coordinate(group="com.fasterxml.jackson.core", artifact="jackson-databind", version="2.12.1"),
                    Coordinate(group="com.fasterxml.jackson.core", artifact="jackson-core", version="2.11.1"),
                ]
            ),
        ],
    )
```
The flag used for a local exclude in v1 was: https://github.com/pantsbuild/pants/blob/674df55a3dda870e78997965e617993b9d10d2eb/src/python/pants/backend/jvm/tasks/coursier_resolve.py#L509-L516

Resolving patch:
diff --git a/src/python/pants/jvm/resolve/common.py b/src/python/pants/jvm/resolve/common.py
--- a/src/python/pants/jvm/resolve/common.py
+++ b/src/python/pants/jvm/resolve/common.py
@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import dataclasses
 import re
 from dataclasses import dataclass
 from typing import Iterable
@@ -12,6 +13,7 @@
 from pants.engine.target import Target
 from pants.jvm.target_types import (
     JvmArtifactArtifactField,
+    JvmArtifactExcludeDependenciesField,
     JvmArtifactFieldSet,
     JvmArtifactGroupField,
     JvmArtifactJarSourceField,
@@ -149,6 +151,7 @@ class ArtifactRequirement:
 
     url: str | None = None
     jar: JvmArtifactJarSourceField | None = None
+    excludes: frozenset[str] | None = None
 
     @classmethod
     def from_jvm_artifact_target(cls, target: Target) -> ArtifactRequirement:
@@ -157,6 +160,7 @@ def from_jvm_artifact_target(cls, target: Target) -> ArtifactRequirement:
                 "`ArtifactRequirement.from_jvm_artifact_target()` only works on targets with "
                 "`JvmArtifactFieldSet` fields present."
             )
+
         return ArtifactRequirement(
             coordinate=Coordinate(
                 group=target[JvmArtifactGroupField].value,
@@ -169,6 +173,17 @@ def from_jvm_artifact_target(cls, target: Target) -> ArtifactRequirement:
                 if target[JvmArtifactJarSourceField].value
                 else None
             ),
+            excludes=frozenset(target[JvmArtifactExcludeDependenciesField].value or []) or None,
+        )
+
+    def with_extra_excludes(self, *excludes: str) -> ArtifactRequirement:
+        """Creates a copy of this `ArtifactRequirement` with `excludes` provided.
+
+        Mostly useful for testing (`Coordinate(...).as_requirement().with_extra_excludes(...)`).
+        """
+
+        return dataclasses.replace(
+            self, excludes=self.excludes.union(excludes) if self.excludes else frozenset(excludes)
         )
 
     def to_coord_arg_str(self) -> str:
@@ -177,12 +192,14 @@ def to_coord_arg_str(self) -> str:
         )
 
     def to_metadata_str(self) -> str:
-        return self.coordinate.to_coord_arg_str(
-            {
-                "url": self.url or "not_provided",
-                "jar": self.jar.address.spec if self.jar else "not_provided",
-            }
-        )
+        attrs = {
+            "url": self.url or "not_provided",
+            "jar": self.jar.address.spec if self.jar else "not_provided",
+        }
+        if self.excludes:
+            attrs["excludes"] = ",".join(self.excludes)
+
+        return self.coordinate.to_coord_arg_str(attrs)
 
 
 # TODO: Consider whether to carry classpath scope in some fashion via ArtifactRequirements.
@@ -197,7 +214,7 @@ def from_coordinates(cls, coordinates: Iterable[Coordinate]) -> ArtifactRequirem
 @dataclass(frozen=True)
 class GatherJvmCoordinatesRequest:
     """A request to turn strings of coordinates (`group:artifact:version`) and/or addresses to
-    `jar_artifact` targets into `ArtifactRequirements`."""
+    `jvm_artifact` targets into `ArtifactRequirements`."""
 
     artifact_inputs: FrozenOrderedSet[str]
     option_name: str
diff --git a/src/python/pants/jvm/resolve/coursier_fetch.py b/src/python/pants/jvm/resolve/coursier_fetch.py
--- a/src/python/pants/jvm/resolve/coursier_fetch.py
+++ b/src/python/pants/jvm/resolve/coursier_fetch.py
@@ -5,6 +5,7 @@
 
 import dataclasses
 import importlib.resources
+import itertools
 import json
 import logging
 import os
@@ -22,15 +23,18 @@
 from pants.engine.collection import Collection
 from pants.engine.fs import (
     AddPrefix,
+    CreateDigest,
     Digest,
     DigestContents,
     DigestSubset,
+    FileContent,
     FileDigest,
     MergeDigests,
     PathGlobs,
     RemovePrefix,
     Snapshot,
 )
+from pants.engine.internals.native_engine import EMPTY_DIGEST
 from pants.engine.process import ProcessResult
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import CoarsenedTargets, Target, Targets
@@ -289,8 +293,17 @@ def classpath_dest_filename(coord: str, src_filename: str) -> str:
 @dataclass(frozen=True)
 class CoursierResolveInfo:
     coord_arg_strings: FrozenSet[str]
+    extra_args: tuple[str, ...]
     digest: Digest
 
+    @property
+    def argv(self) -> Iterable[str]:
+        """Return coursier arguments that can be used to compute or fetch this resolve.
+
+        Must be used in concert with `digest`.
+        """
+        return itertools.chain(self.coord_arg_strings, self.extra_args)
+
 
 @rule
 async def prepare_coursier_resolve_info(
@@ -300,6 +313,9 @@ async def prepare_coursier_resolve_info(
     # URLs, and put the files in the place specified by the URLs.
     no_jars: List[ArtifactRequirement] = []
     jars: List[Tuple[ArtifactRequirement, JvmArtifactJarSourceField]] = []
+    extra_args: List[str] = []
+
+    LOCAL_EXCLUDE_FILE = "PANTS_RESOLVE_EXCLUDES"
 
     for req in artifact_requirements:
         jar = req.jar
@@ -308,6 +324,23 @@ async def prepare_coursier_resolve_info(
         else:
             jars.append((req, jar))
 
+    excludes = [
+        (req.coordinate, exclude)
+        for req in artifact_requirements
+        for exclude in (req.excludes or [])
+    ]
+
+    excludes_digest = EMPTY_DIGEST
+    if excludes:
+        excludes_file_content = FileContent(
+            LOCAL_EXCLUDE_FILE,
+            "\n".join(
+                f"{coord.group}:{coord.artifact}--{exclude}" for (coord, exclude) in excludes
+            ).encode("utf-8"),
+        )
+        excludes_digest = await Get(Digest, CreateDigest([excludes_file_content]))
+        extra_args += ["--local-exclude-file", LOCAL_EXCLUDE_FILE]
+
     jar_files = await Get(SourceFiles, SourceFilesRequest(i[1] for i in jars))
     jar_file_paths = jar_files.snapshot.files
 
@@ -320,9 +353,12 @@ async def prepare_coursier_resolve_info(
 
     to_resolve = chain(no_jars, resolvable_jar_requirements)
 
+    digest = await Get(Digest, MergeDigests([jar_files.snapshot.digest, excludes_digest]))
+
     return CoursierResolveInfo(
         coord_arg_strings=frozenset(req.to_coord_arg_str() for req in to_resolve),
-        digest=jar_files.snapshot.digest,
+        digest=digest,
+        extra_args=tuple(extra_args),
     )
 
 
@@ -368,7 +404,7 @@ async def coursier_resolve_lockfile(
         CoursierFetchProcess(
             args=(
                 coursier_report_file_name,
-                *coursier_resolve_info.coord_arg_strings,
+                *coursier_resolve_info.argv,
             ),
             input_digest=coursier_resolve_info.digest,
             output_directories=("classpath",),
@@ -523,7 +559,7 @@ async def coursier_fetch_one_coord(
             args=(
                 coursier_report_file_name,
                 "--intransitive",
-                *coursier_resolve_info.coord_arg_strings,
+                *coursier_resolve_info.argv,
             ),
             input_digest=coursier_resolve_info.digest,
             output_directories=("classpath",),
diff --git a/src/python/pants/jvm/target_types.py b/src/python/pants/jvm/target_types.py
--- a/src/python/pants/jvm/target_types.py
+++ b/src/python/pants/jvm/target_types.py
@@ -168,6 +168,19 @@ class JvmProvidesTypesField(StringSequenceField):
     )
 
 
+class JvmArtifactExcludeDependenciesField(StringSequenceField):
+    alias = "excludes"
+    help = (
+        "A list of unversioned coordinates (i.e. `group:artifact`) that should be excluded "
+        "as dependencies when this artifact is resolved.\n\n"
+        "This does not prevent this artifact from being included in the resolve as a dependency "
+        "of other artifacts that depend on it, and is currently intended as a way to resolve "
+        "version conflicts in complex resolves.\n\n"
+        "These values are passed directly to Coursier, and if specified incorrectly will show a "
+        "parse error from Coursier."
+    )
+
+
 class JvmArtifactResolveField(JvmResolveField):
     help = (
         "The resolve from `[jvm].resolves` that this artifact should be included in.\n\n"
@@ -204,6 +217,7 @@ class JvmArtifactTarget(Target):
         JvmArtifactUrlField,  # TODO: should `JvmArtifactFieldSet` have an `all_fields` field?
         JvmArtifactJarSourceField,
         JvmArtifactResolveField,
+        JvmArtifactExcludeDependenciesField,
     )
     help = (
         "A third-party JVM artifact, as identified by its Maven-compatible coordinate.\n\n"


Test patch:
diff --git a/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py b/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
--- a/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
+++ b/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
@@ -606,3 +606,27 @@ def test_user_repo_order_is_respected(rule_runner: RuleRunner) -> None:
             ArtifactRequirements.from_coordinates([jai_core]),
         ],
     )
+
+
+@maybe_skip_jdk_test
+def test_transitive_excludes(rule_runner: RuleRunner) -> None:
+    resolve = rule_runner.request(
+        CoursierResolvedLockfile,
+        [
+            ArtifactRequirements(
+                [
+                    Coordinate(
+                        group="com.fasterxml.jackson.core",
+                        artifact="jackson-databind",
+                        version="2.12.1",
+                    )
+                    .as_requirement()
+                    .with_extra_excludes("com.fasterxml.jackson.core:jackson-core")
+                ]
+            ),
+        ],
+    )
+
+    entries = resolve.entries
+    assert any(i for i in entries if i.coord.artifact == "jackson-databind")
+    assert not any(i for i in entries if i.coord.artifact == "jackson-core")