Repository: pantsbuild/pants

Issue:
coursier post processing script choking on null dependency file in report json
**Describe the bug**
One of our 3rdparty jars (`org.geotools:gt-main:9.2` from the `https://repo.osgeo.org/repository/release` repositiory) transitively depends on a `javax.media:jai_core:1.1.3` jar. For whatever reason, Coursier's resolve report json lists `"file": null` for that jar, which the post processing script doesn't currently handle.

**Pants version**
2.9.0

**OS**
Encountered on MacOS, though expect we would see this on Linux as well

**Additional info**
Coursier config:
```
[coursier]
repos = [
  "https://maven-central.storage-download.googleapis.com/maven2",
  "https://repo1.maven.org/maven2",
  "https://repo.osgeo.org/repository/release",
]
```

affected jvm_artifact:
```
jvm_artifact(
  group="org.geotools",
  artifact="gt-main",
  version="9.2",
)
```

[slack discussion](https://pantsbuild.slack.com/archives/C046T6T9U/p1645598534013879) with links to full resolve output and json



Discussion:
Note that [`jai_core:1.1.3` exists in the osgeo.org repository](https://repo.osgeo.org/#browse/search=keyword%3Djavax.media:3f5cae01760233b6403868dadb9402be) -- however, there are entries for that same package in maven central, which contain a pom but no jar.

My working hypothesis is that Coursier is preferring the definition from `maven-central` for whatever reason.
Observations thus far:

• Coursier appears to download from [repo1.maven.org](http://repo1.maven.org/), regardless of `-r` arguments
• Coursier appears to use cached POM definitions even if you change the `-r` arguments and the `COURSIER_REPOSITORIES` envvar (so you have to blow away `~/.cache/pants` to get coursier to try new repos)
I've managed to get the `jai_core` package to resolve properly. The issue is that Coursier will always look in its default repos before user-specified repos unless the `--no-default` option is specified to Coursier.

I'll get a fix for this written presently.
Unfortunately @jvandew reports that this may not fix the issue. 

https://pantsbuild.slack.com/archives/C046T6T9U/p1645672200019649?thread_ts=1645598534.013879&cid=C046T6T9U
Specifically, the coursier command:

```
cs fetch -r=https://repo.osgeo.org/repository/release -r=https://maven-central.storage-download.googleapis.com/maven2 -r=https://repo1.maven.org/maven2 --no-default --json-output-file=coursier_report.json org.geotools:gt-main:9.2 com.google.geometry:s2-geometry:2.0.0 com.twitter:util-logging_2.11:20.1.0 org.mongodb:bson:3.11.2 joda-time:joda-time:2.10.3 com.fasterxml.jackson.core:jackson-core:2.6.7 com.twitter:util-core_2.11:20.1.0 com.fasterxml.jackson.core:jackson-databind:2.6.7 com.vividsolutions:jts:1.12 com.fasterxml.jackson.module:jackson-module-scala_2.11:2.6.7.1 org.apache.thrift:libthrift:0.9.0 org.codehaus.jackson:jackson-core-asl:1.9.8
```

when run by Pants fails for them with the same error as before.

However when I run that same command outside Pants it succeeds.

I wasn't able to get Coursier to re-attempt downloading from the correct repos until I blew away the entirety of my Pants cache (`~/.cache/pants`). Not the cleanest of fixes, but 🤷
Per Slack, the re-emergence of this issue reported by @jvandew is due to a Coursier cache poisoning issue.

I'm going to see what it would take to fix this in Pants, and then report upstream.
Here's how to reproduce the nut of the issue outside Pants:

1. 
Work in a tmpdir: `mkdir /tmp/coursier && cd /tmp/coursier`

2. 
[Install](https://get-coursier.io/docs/cli-installation) coursier . E.g., on MacOS: `brew install coursier/formulas/coursier && cs setup`. 

I said "no" to the kind offer to add coursier to my PATH in my .bash_profile ...

3.
Nuke the coursier download cache, to start with clean state. E.g., on MacOS: `rm -rf ~/Library/Caches/Coursier/v1/`.

4. 
Fetch the broken artifact from the bad repository:
```
$ cs fetch -r=https://repo1.maven.org/maven2 --no-default --json-output-file=coursier_report.json javax.media:jai_core:1.1.3

$ cat coursier_report.json | jq -M .dependencies[0].file
null
```

5.
Try and refetch with the good repository:
```
$ rm -f coursier_report.json  # Just to be sure that isn't confusing us somehow
$ cs fetch -r=https://repo.osgeo.org/repository/release -r=https://repo1.maven.org/maven2 --no-default --json-output-file=coursier_report.json javax.media:jai_core:1.1.3

$ cat coursier_report.json | jq -M .dependencies[0].file
null
```
6.
Nuke the coursier cache again, as above.

7.
Refetch with the good repository, as above:
```
$ rm -f coursier_report.json 
$ cs fetch -r=https://repo.osgeo.org/repository/release -r=https://repo1.maven.org/maven2 --no-default --json-output-file=coursier_report.json javax.media:jai_core:1.1.3

$ cat coursier_report.json | jq -M .dependencies[0].file
"/Users/benjyw/Library/Caches/Coursier/v1/https/repo.osgeo.org/repository/release/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar"
```
So this is a two-part Coursier bug:

1) Coursier accepts a missing JAR as legit, if there is a POM, and doesn't look further in other repos.
2) The artifact with a missing JAR poisons the cache, so even changing your repositories doesn't help until you nuke the cache.
The workaround, after updating the repositories, is to nuke the coursier cache in ~/.pants/cache , which is not ideal but should work. @jvandew when you get a moment can you see if this works?

Resolving patch:
diff --git a/src/python/pants/jvm/resolve/coursier_fetch.py b/src/python/pants/jvm/resolve/coursier_fetch.py
--- a/src/python/pants/jvm/resolve/coursier_fetch.py
+++ b/src/python/pants/jvm/resolve/coursier_fetch.py
@@ -49,7 +49,7 @@
     Coordinates,
     GatherJvmCoordinatesRequest,
 )
-from pants.jvm.resolve.coursier_setup import Coursier, CoursierWrapperProcess
+from pants.jvm.resolve.coursier_setup import Coursier, CoursierFetchProcess
 from pants.jvm.resolve.key import CoursierResolveKey
 from pants.jvm.resolve.lockfile_metadata import JVMLockfileMetadata, LockfileContext
 from pants.jvm.subsystems import JvmSubsystem
@@ -365,7 +365,7 @@ async def coursier_resolve_lockfile(
 
     process_result = await Get(
         ProcessResult,
-        CoursierWrapperProcess(
+        CoursierFetchProcess(
             args=(
                 coursier_report_file_name,
                 *coursier_resolve_info.coord_arg_strings,
@@ -519,7 +519,7 @@ async def coursier_fetch_one_coord(
 
     process_result = await Get(
         ProcessResult,
-        CoursierWrapperProcess(
+        CoursierFetchProcess(
             args=(
                 coursier_report_file_name,
                 "--intransitive",
diff --git a/src/python/pants/jvm/resolve/coursier_setup.py b/src/python/pants/jvm/resolve/coursier_setup.py
--- a/src/python/pants/jvm/resolve/coursier_setup.py
+++ b/src/python/pants/jvm/resolve/coursier_setup.py
@@ -7,6 +7,7 @@
 import shlex
 import textwrap
 from dataclasses import dataclass
+from hashlib import sha256
 from typing import ClassVar, Iterable, Tuple
 
 from pants.core.util_rules import external_tool
@@ -21,6 +22,8 @@
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.python.binaries import PythonBinary
 from pants.util.logging import LogLevel
+from pants.util.memo import memoized_property
+from pants.util.ordered_set import FrozenOrderedSet
 
 COURSIER_POST_PROCESSING_SCRIPT = textwrap.dedent(
     """\
@@ -125,7 +128,13 @@ def register_options(cls, register) -> None:
                 "https://maven-central.storage-download.googleapis.com/maven2",
                 "https://repo1.maven.org/maven2",
             ],
-            help=("Maven style repositories to resolve artifacts from."),
+            help=(
+                "Maven style repositories to resolve artifacts from."
+                "\n\n"
+                "Coursier will resolve these repositories in the order in which they are "
+                "specifed, and re-ordering repositories will cause artifacts to be "
+                "re-downloaded. This can result in artifacts in lockfiles becoming invalid."
+            ),
         )
 
     def generate_exe(self, plat: Platform) -> str:
@@ -140,6 +149,7 @@ class Coursier:
 
     coursier: DownloadedExternalTool
     _digest: Digest
+    repos: FrozenOrderedSet[str]
 
     bin_dir: ClassVar[str] = "__coursier"
     fetch_wrapper_script: ClassVar[str] = f"{bin_dir}/coursier_fetch_wrapper_script.sh"
@@ -157,13 +167,26 @@ def args(self, args: Iterable[str], *, wrapper: Iterable[str] = ()) -> tuple[str
             *args,
         )
 
+    @memoized_property
+    def _coursier_cache_prefix(self) -> str:
+        """Returns a key for `COURSIER_CACHE` determined by the configured repositories.
+
+        This helps us avoid a cache poisoning issue that we uncovered in #14577.
+        """
+        sha = sha256()
+        for repo in self.repos:
+            sha.update(repo.encode("utf-8"))
+        return sha.digest().hex()
+
     @property
     def env(self) -> dict[str, str]:
         # NB: These variables have changed a few times, and they change again on `main`. But as of
         # `v2.0.16+73-gddc6d9cc9` they are accurate. See:
         #  https://github.com/coursier/coursier/blob/v2.0.16+73-gddc6d9cc9/modules/paths/src/main/java/coursier/paths/CoursierPaths.java#L38-L48
         return {
-            "COURSIER_CACHE": f"{self.cache_dir}/jdk",
+            # Maven artifacts and JDK tarballs go here
+            "COURSIER_CACHE": f"{self.cache_dir}/{self._coursier_cache_prefix}/jdk",
+            # extracted JDK tarballs go here
             "COURSIER_ARCHIVE_CACHE": f"{self.cache_dir}/arc",
             "COURSIER_JVM_CACHE": f"{self.cache_dir}/v1",
         }
@@ -178,7 +201,7 @@ def immutable_input_digests(self) -> dict[str, Digest]:
 
 
 @dataclass(frozen=True)
-class CoursierWrapperProcess:
+class CoursierFetchProcess:
 
     args: Tuple[str, ...]
     input_digest: Digest
@@ -191,7 +214,7 @@ class CoursierWrapperProcess:
 async def invoke_coursier_wrapper(
     bash: BashBinary,
     coursier: Coursier,
-    request: CoursierWrapperProcess,
+    request: CoursierFetchProcess,
 ) -> Process:
 
     return Process(
@@ -271,6 +294,7 @@ async def setup_coursier(
                 ]
             ),
         ),
+        repos=FrozenOrderedSet(coursier_subsystem.options.repos),
     )
 
 


Test patch:
diff --git a/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py b/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
--- a/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
+++ b/src/python/pants/jvm/resolve/coursier_fetch_integration_test.py
@@ -570,3 +570,39 @@ def test_fetch_one_coord_with_mismatched_coord(rule_runner: RuleRunner) -> None:
     )
     with pytest.raises(ExecutionError, match=expected_exception_msg):
         rule_runner.request(ClasspathEntry, [lockfile_entry])
+
+
+@maybe_skip_jdk_test
+def test_user_repo_order_is_respected(rule_runner: RuleRunner) -> None:
+    """Tests that the repo resolution order issue found in #14577 is avoided."""
+
+    jai_core = Coordinate(group="javax.media", artifact="jai_core", version="1.1.3")
+
+    # `repo1.maven.org` has a bogus POM that Coursier hits first
+    # `repo.osgeo.org` has a valid POM and should succeed
+    rule_runner.set_options(
+        args=[
+            """--coursier-repos=['https://repo1.maven.org/maven2', 'https://repo.osgeo.org/repository/release']"""
+        ],
+        env_inherit=PYTHON_BOOTSTRAP_ENV,
+    )
+    with engine_error(ProcessExecutionFailure):
+        rule_runner.request(
+            CoursierResolvedLockfile,
+            [
+                ArtifactRequirements.from_coordinates([jai_core]),
+            ],
+        )
+
+    rule_runner.set_options(
+        args=[
+            """--coursier-repos=['https://repo.osgeo.org/repository/release', 'https://repo1.maven.org/maven2']"""
+        ],
+        env_inherit=PYTHON_BOOTSTRAP_ENV,
+    )
+    rule_runner.request(
+        CoursierResolvedLockfile,
+        [
+            ArtifactRequirements.from_coordinates([jai_core]),
+        ],
+    )