internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pantsbuild/pants

Issue:
pyenv fallback directory is incorrect
pyenv default directory is set to `$HOME/.cache` but should be `$HOME/.pyenv`.

docs: https://github.com/pyenv/pyenv#environment-variables

Discovered via: https://github.com/pantsbuild/pants/issues/12496#issuecomment-892798436


Discussion:

Resolving patch:
diff --git a/src/python/pants/python/python_setup.py b/src/python/pants/python/python_setup.py
--- a/src/python/pants/python/python_setup.py
+++ b/src/python/pants/python/python_setup.py
@@ -504,5 +504,5 @@ def get_pyenv_root(env: Environment) -> str | None:
         return from_env
     home_from_env = env.get("HOME")
     if home_from_env:
-        return os.path.join(home_from_env, ".cache")
+        return os.path.join(home_from_env, ".pyenv")
     return None


Test patch:
diff --git a/src/python/pants/python/python_setup_test.py b/src/python/pants/python/python_setup_test.py
--- a/src/python/pants/python/python_setup_test.py
+++ b/src/python/pants/python/python_setup_test.py
@@ -116,7 +116,7 @@ def rule_runner() -> RuleRunner:
 
 def test_get_pyenv_root() -> None:
     home = "/♡"
-    default_root = f"{home}/.cache"
+    default_root = f"{home}/.pyenv"
     explicit_root = f"{home}/explicit"
 
     assert explicit_root == get_pyenv_root(Environment({"PYENV_ROOT": explicit_root}))
Repository: pantsbuild/pants

Issue:
`shfmt` does not work on macOS ARM due to bad Platform handling
This needs to be updated to handle the new Platform support:

https://github.com/pantsbuild/pants/blob/4d5e46f7ebcd2d3d05c7bdf2fe6419253c1043f6/src/python/pants/backend/shell/lint/shfmt/subsystem.py#L61-L63

It looks like this also needs updating:

https://github.com/pantsbuild/pants/blob/4d5e46f7ebcd2d3d05c7bdf2fe6419253c1043f6/src/python/pants/engine/desktop.py#L40-L55


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/shell/lint/shfmt/subsystem.py b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
--- a/src/python/pants/backend/shell/lint/shfmt/subsystem.py
+++ b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
@@ -59,8 +59,8 @@ def register_options(cls, register):
         )
 
     def generate_exe(self, plat: Platform) -> str:
-        plat_str = "linux" if plat == Platform.linux else "darwin"
-        return f"./shfmt_{self.version}_{plat_str}_amd64"
+        plat_str = self.default_url_platform_mapping[plat.value]
+        return f"./shfmt_{self.version}_{plat_str}"
 
     @property
     def skip(self) -> bool:
diff --git a/src/python/pants/engine/desktop.py b/src/python/pants/engine/desktop.py
--- a/src/python/pants/engine/desktop.py
+++ b/src/python/pants/engine/desktop.py
@@ -37,7 +37,7 @@ async def find_open_program(
     plat: Platform,
     complete_env: CompleteEnvironment,
 ) -> OpenFiles:
-    open_program_name = "open" if plat == Platform.darwin else "xdg-open"
+    open_program_name = "open" if plat.is_macos else "xdg-open"
     open_program_paths = await Get(
         BinaryPaths,
         BinaryPathRequest(binary_name=open_program_name, search_path=("/bin", "/usr/bin")),
@@ -52,7 +52,7 @@ async def find_open_program(
         logger.error(error)
         return OpenFiles(())
 
-    if plat == Platform.darwin:
+    if plat.is_macos:
         processes = [
             InteractiveProcess(
                 argv=(open_program_paths.first_path.path, *(str(f) for f in request.files)),
diff --git a/src/python/pants/engine/platform.py b/src/python/pants/engine/platform.py
--- a/src/python/pants/engine/platform.py
+++ b/src/python/pants/engine/platform.py
@@ -41,6 +41,10 @@ def darwin(cls) -> Platform:
         Platform.deprecated_due_to_no_architecture()
         return Platform.macos_x86_64
 
+    @property
+    def is_macos(self) -> bool:
+        return self in [Platform.macos_arm64, Platform.macos_x86_64]
+
     def matches(self, value):
         """Returns true if the provided value is the value for this platform, or if the provided
         value is the value for the deprecated platform symbol from before we qualified based on


Test patch:
Repository: pantsbuild/pants

Issue:
pyenv fallback directory is incorrect
pyenv default directory is set to `$HOME/.cache` but should be `$HOME/.pyenv`.

docs: https://github.com/pyenv/pyenv#environment-variables

Discovered via: https://github.com/pantsbuild/pants/issues/12496#issuecomment-892798436


Discussion:

Resolving patch:
diff --git a/src/python/pants/python/python_setup.py b/src/python/pants/python/python_setup.py
--- a/src/python/pants/python/python_setup.py
+++ b/src/python/pants/python/python_setup.py
@@ -287,5 +287,5 @@ def get_pyenv_root(env: Environment) -> str | None:
         return from_env
     home_from_env = env.get("HOME")
     if home_from_env:
-        return os.path.join(home_from_env, ".cache")
+        return os.path.join(home_from_env, ".pyenv")
     return None


Test patch:
diff --git a/src/python/pants/python/python_setup_test.py b/src/python/pants/python/python_setup_test.py
--- a/src/python/pants/python/python_setup_test.py
+++ b/src/python/pants/python/python_setup_test.py
@@ -67,7 +67,7 @@ def rule_runner() -> RuleRunner:
 
 def test_get_pyenv_root() -> None:
     home = "/♡"
-    default_root = f"{home}/.cache"
+    default_root = f"{home}/.pyenv"
     explicit_root = f"{home}/explicit"
 
     assert explicit_root == get_pyenv_root(Environment({"PYENV_ROOT": explicit_root}))
Repository: pantsbuild/pants

Issue:
The release script does not work for PyPI token authentication
The error looks like so:
```console
$ ./build-support/bin/release.sh publish
[=== 00:00 Releasing to PyPI and GitHub ===]
[=== 00:00 Checking for a clean Git branch ===]
[=== 00:00 Checking PGP setup ===]
Found the following key for release signing:

pub   rsa4096 2014-04-21 [SC]
      A1FE765B15233EAD18FA6ABB93E55CB567B5C626
uid           [ unknown] John Sirois (CODE SIGNING KEY) <jsirois@apache.org>
uid           [ unknown] John Sirois <john.sirois@gmail.com>
uid           [ unknown] John Sirois (https://people.apache.org/~jsirois/) <jsirois@apache.org>
sub   rsa4096 2014-04-21 [E]


Is this the correct key? [Y/n]: 
[=== 00:01 Checking current user. ===]
User __token__ not authorized to publish.
That __token__ user is what you have in .pypirc for 2-factor auth setups.
```



Discussion:

Resolving patch:
diff --git a/build-support/bin/_release_helper.py b/build-support/bin/_release_helper.py
--- a/build-support/bin/_release_helper.py
+++ b/build-support/bin/_release_helper.py
@@ -722,7 +722,11 @@ def check_roles() -> None:
     # Check that the packages we plan to publish are correctly owned.
     banner("Checking current user.")
     username = get_pypi_config("server-login", "username")
-    if username not in _expected_owners and username not in _expected_maintainers:
+    if (
+        username != "__token__"  # See: https://pypi.org/help/#apitoken
+        and username not in _expected_owners
+        and username not in _expected_maintainers
+    ):
         die(f"User {username} not authorized to publish.")
     banner("Checking package roles.")
     validator = PackageAccessValidator()


Test patch:
Repository: pantsbuild/pants

Issue:
`DigestEntries` returns a `Directory` instead of an empty vector for non-matching digest



Discussion:

Resolving patch:
diff --git a/src/python/pants/jvm/util_rules.py b/src/python/pants/jvm/util_rules.py
--- a/src/python/pants/jvm/util_rules.py
+++ b/src/python/pants/jvm/util_rules.py
@@ -18,8 +18,7 @@ class ExtractFileDigest:
 @rule
 async def digest_to_file_digest(request: ExtractFileDigest) -> FileDigest:
     digest = await Get(Digest, DigestSubset(request.digest, PathGlobs([request.file_path])))
-    files_or_directories = await Get(DigestEntries, Digest, digest)
-    digest_entries = [entry for entry in files_or_directories if isinstance(entry, FileEntry)]
+    digest_entries = await Get(DigestEntries, Digest, digest)
 
     if len(digest_entries) == 0:
         raise Exception(f"ExtractFileDigest: '{request.file_path}' not found in {request.digest}.")
@@ -29,6 +28,12 @@ async def digest_to_file_digest(request: ExtractFileDigest) -> FileDigest:
         )
 
     file_info = digest_entries[0]
+
+    if not isinstance(file_info, FileEntry):
+        raise AssertionError(
+            f"Unexpected error: '{request.file_path}' refers to a directory, not a file."
+        )
+
     return file_info.file_digest
 
 


Test patch:
diff --git a/src/rust/engine/fs/store/src/tests.rs b/src/rust/engine/fs/store/src/tests.rs
--- a/src/rust/engine/fs/store/src/tests.rs
+++ b/src/rust/engine/fs/store/src/tests.rs
@@ -12,7 +12,7 @@ use bytes::{Bytes, BytesMut};
 use fs::{DigestEntry, FileEntry};
 use grpc_util::prost::MessageExt;
 use grpc_util::tls;
-use hashing::{Digest, Fingerprint};
+use hashing::{Digest, Fingerprint, EMPTY_DIGEST};
 use mock::StubCAS;
 
 use crate::{EntryType, FileContent, Store, UploadSummary, MEGABYTES};
@@ -1267,6 +1267,13 @@ async fn entries_for_directory() {
       }),
     ],
   );
+
+  let empty_digest_entries = store
+    .entries_for_directory(EMPTY_DIGEST)
+    .await
+    .expect("Getting EMTPY_DIGEST");
+
+  assert_same_digest_entries(empty_digest_entries, vec![]);
 }
 
 fn assert_same_digest_entries(left: Vec<DigestEntry>, right: Vec<DigestEntry>) {
Repository: pantsbuild/pants

Issue:
Flake in coursier-resolve integration tests
https://github.com/pantsbuild/pants/runs/4028069406?check_suite_focus=true#step:13:420


Discussion:

Resolving patch:
diff --git a/src/python/pants/testutil/rule_runner.py b/src/python/pants/testutil/rule_runner.py
--- a/src/python/pants/testutil/rule_runner.py
+++ b/src/python/pants/testutil/rule_runner.py
@@ -322,14 +322,14 @@ def run_goal_rule(
         specs = SpecsParser(self.build_root).parse_specs(raw_specs)
 
         stdout, stderr = StringIO(), StringIO()
-        console = Console(stdout=stdout, stderr=stderr, use_colors=False)
+        console = Console(stdout=stdout, stderr=stderr, use_colors=False, session=self.scheduler)
 
         exit_code = self.scheduler.run_goal_rule(
             goal,
             Params(
                 specs,
                 console,
-                Workspace(self.scheduler, _enforce_effects=False),
+                Workspace(self.scheduler),
             ),
         )
 


Test patch:
Repository: pantsbuild/pants

Issue:
Evaluate removing the invalidation of parent directories when files change
Since using `watchman` (which was deprecated in favor of `notify` in #9714), we have had a special case in place to invalidate the parent directory of a changed file, because in some cases we would not be notified about parent directory changes when a file was created or removed.

https://github.com/pantsbuild/pants/blob/5d9d725bcb10dc141c11e1a701d222c31d15e62b/src/rust/engine/watch/src/lib.rs#L191-L196

It's possible that we could remove this case for _some_ file change events: for example, if we can tell from the type of an event that only the content has changed.


Discussion:

Resolving patch:
diff --git a/src/python/pants/engine/internals/scheduler.py b/src/python/pants/engine/internals/scheduler.py
--- a/src/python/pants/engine/internals/scheduler.py
+++ b/src/python/pants/engine/internals/scheduler.py
@@ -277,10 +277,7 @@ def rule_graph_consumed_types(
             self.py_scheduler, root_subject_types, product_type
         )
 
-    def invalidate_files(self, direct_filenames: Iterable[str]) -> int:
-        filenames = set(direct_filenames)
-        # TODO(#11707): Evaluate removing the invalidation of parent directories.
-        filenames.update(os.path.dirname(f) for f in direct_filenames)
+    def invalidate_files(self, filenames: Iterable[str]) -> int:
         return native_engine.graph_invalidate_paths(self.py_scheduler, tuple(filenames))
 
     def invalidate_all_files(self) -> int:


Test patch:
Repository: pantsbuild/pants

Issue:
Update `pylint` to v2.8.2 or later
Initially intended to include this in #12135 as part of #12133 but the newer version broke tests so reverted to upgrading to v2.6.2 instead. Upgrading should be as simple as changing `src/python/pants/backend/python/lint/pylint/subsystem.py` and `src/python/pants/backend/python/lint/pylint/rules_integration_test.py` and then fixing tests.


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/pylint/subsystem.py b/src/python/pants/backend/python/lint/pylint/subsystem.py
--- a/src/python/pants/backend/python/lint/pylint/subsystem.py
+++ b/src/python/pants/backend/python/lint/pylint/subsystem.py
@@ -67,7 +67,7 @@ class Pylint(PythonToolBase):
     options_scope = "pylint"
     help = "The Pylint linter for Python code (https://www.pylint.org/)."
 
-    default_version = "pylint>=2.6.2,<2.7"
+    default_version = "pylint>=2.11.0,<2.12"
     default_main = ConsoleScript("pylint")
 
     register_lockfile = True


Test patch:
diff --git a/src/python/pants/backend/python/lint/pylint/rules_integration_test.py b/src/python/pants/backend/python/lint/pylint/rules_integration_test.py
--- a/src/python/pants/backend/python/lint/pylint/rules_integration_test.py
+++ b/src/python/pants/backend/python/lint/pylint/rules_integration_test.py
@@ -323,7 +323,7 @@ def test_source_plugin(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                python_requirement(name='pylint', requirements=['pylint>=2.6.2,<2.7'])
+                python_requirement(name='pylint', requirements=['pylint>=2.11.0,<2.12'])
                 python_requirement(name='colors', requirements=['ansicolors'])
                 """
             ),
diff --git a/src/python/pants/backend/python/lint/pylint/subsystem_test.py b/src/python/pants/backend/python/lint/pylint/subsystem_test.py
--- a/src/python/pants/backend/python/lint/pylint/subsystem_test.py
+++ b/src/python/pants/backend/python/lint/pylint/subsystem_test.py
@@ -49,7 +49,7 @@ def test_first_party_plugins(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                python_requirement(name='pylint', requirements=['pylint==2.6.2'])
+                python_requirement(name='pylint', requirements=['pylint==2.11.1'])
                 python_requirement(name='colors', requirements=['ansicolors'])
                 """
             ),
@@ -83,7 +83,7 @@ def test_first_party_plugins(rule_runner: RuleRunner) -> None:
     )
     first_party_plugins = rule_runner.request(PylintFirstPartyPlugins, [])
     assert first_party_plugins.requirement_strings == FrozenOrderedSet(
-        ["ansicolors", "pylint==2.6.2"]
+        ["ansicolors", "pylint==2.11.1"]
     )
     assert first_party_plugins.interpreter_constraints_fields == FrozenOrderedSet(
         [
Repository: pantsbuild/pants

Issue:
`jvm_artifact` `jar` field incorrectly validated during `tailor`
During `./pants tailor`, the `jvm_artifact(jar=..)` field is validated as having 1 file, even when it has not been set for the target. That fails with:

```
Engine traceback:
  in select
  in pants.core.goals.tailor.tailor
  in pants.backend.scala.goals.tailor.find_putative_targets
  in pants.core.goals.tailor.determine_all_owned_sources
  in pants.engine.internals.graph.resolve_source_paths (3rdparty/jvm/com/fasterxml/jackson/core:jackson-databind)
Traceback (most recent call last):
  File "src/python/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "src/python/pants/engine/internals/graph.py", line 853, in resolve_source_paths
    sources_field.validate_resolved_files(paths.files)
  File "src/python/pants/engine/target.py", line 1613, in validate_resolved_files
    f"The {repr(self.alias)} field in target {self.address} must have "
pants.engine.target.InvalidFieldException: The 'jar' field in target 3rdparty/jvm/com/fasterxml/jackson/core:jackson-databind must have 1 file, but it had 0 files.
```

Ideally, this validation would not trigger for a field which is set to `required=False` unless it was actually set. Much less ideal would be setting:
```diff
diff --git a/src/python/pants/jvm/target_types.py b/src/python/pants/jvm/target_types.py
index 27dbeebb66..9d3045c9f3 100644
--- a/src/python/pants/jvm/target_types.py
+++ b/src/python/pants/jvm/target_types.py
@@ -60,6 +60,7 @@ class JvmArtifactUrlField(StringField):
 class JvmArtifactJarSourceField(SingleSourceField):
     alias = "jar"
     expected_file_extensions = (".jar",)
+    expected_num_files = range(0, 2)
     required = False
     help = "A JAR file that provides this artifact to the lockfile resolver, instead of a maven repository."
 
```



Discussion:
Thanks! I'll look into this soon.

Resolving patch:
diff --git a/src/python/pants/engine/target.py b/src/python/pants/engine/target.py
--- a/src/python/pants/engine/target.py
+++ b/src/python/pants/engine/target.py
@@ -1576,6 +1576,12 @@ def validate_resolved_files(self, files: Sequence[str]) -> None:
         To enforce that there are only a certain number of resulting files, such as binary targets
         checking for only 0-1 sources, set the class property `expected_num_files`.
         """
+
+        if not self.required and not self.value:
+            # If this field isn't required or set, validation is done against the default value
+            # which is probably not valuable (See #13851).
+            return None
+
         if self.expected_file_extensions is not None:
             bad_files = [
                 fp for fp in files if not PurePath(fp).suffix in self.expected_file_extensions


Test patch:
Repository: pantsbuild/pants

Issue:
Fix illegal reflective access in Google Java Format on JDK11+
See https://github.com/google/google-java-format#jdk-16. With JDK11 we see:
```
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer (file:/private/var/folders/bg/_r10hqp14kjcpv68yzdk5svc0000gn/T/.tmpjDeqDJ/4709a472310186fc49e11bd6e6834da83cb4ab66c4d9fb94408faf6702e354dd/com.google.googlejavaformat_google-java-format_1.13.0.jar) to field com.sun.tools.javac.parser.JavaTokenizer.reader
WARNING: Please consider reporting this to the maintainers of com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
```


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/java/lint/google_java_format/rules.py b/src/python/pants/backend/java/lint/google_java_format/rules.py
--- a/src/python/pants/backend/java/lint/google_java_format/rules.py
+++ b/src/python/pants/backend/java/lint/google_java_format/rules.py
@@ -87,9 +87,9 @@ async def setup_google_java_format(
         toolcp_relpath: tool_classpath.digest,
     }
 
-    maybe_java16_or_higher_options = []
-    if jdk_setup.jre_major_version >= 16:
-        maybe_java16_or_higher_options = [
+    maybe_java11_or_higher_options = []
+    if jdk_setup.jre_major_version >= 11:
+        maybe_java11_or_higher_options = [
             "--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED",
             "--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED",
             "--add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED",
@@ -99,7 +99,7 @@ async def setup_google_java_format(
 
     args = [
         *jdk_setup.args(bash, tool_classpath.classpath_entries(toolcp_relpath)),
-        *maybe_java16_or_higher_options,
+        *maybe_java11_or_higher_options,
         "com.google.googlejavaformat.java.Main",
         *(["--aosp"] if tool.aosp else []),
         "--dry-run" if setup_request.check_only else "--replace",
diff --git a/src/python/pants/backend/scala/lint/scalafmt/rules.py b/src/python/pants/backend/scala/lint/scalafmt/rules.py
--- a/src/python/pants/backend/scala/lint/scalafmt/rules.py
+++ b/src/python/pants/backend/scala/lint/scalafmt/rules.py
@@ -170,6 +170,8 @@ async def setup_scalafmt_partition(
     ]
     if request.check_only:
         args.append("--list")
+    else:
+        args.append("--quiet")
     args.extend(request.files)
 
     process = Process(


Test patch:
Repository: pantsbuild/pants

Issue:
coursier post processing script choking on null dependency file in report json
**Describe the bug**
One of our 3rdparty jars (`org.geotools:gt-main:9.2` from the `https://repo.osgeo.org/repository/release` repositiory) transitively depends on a `javax.media:jai_core:1.1.3` jar. For whatever reason, Coursier's resolve report json lists `"file": null` for that jar, which the post processing script doesn't currently handle.

**Pants version**
2.9.0

**OS**
Encountered on MacOS, though expect we would see this on Linux as well

**Additional info**
Coursier config:
```
[coursier]
repos = [
  "https://maven-central.storage-download.googleapis.com/maven2",
  "https://repo1.maven.org/maven2",
  "https://repo.osgeo.org/repository/release",
]
```

affected jvm_artifact:
```
jvm_artifact(
  group="org.geotools",
  artifact="gt-main",
  version="9.2",
)
```

[slack discussion](https://pantsbuild.slack.com/archives/C046T6T9U/p1645598534013879) with links to full resolve output and json



Discussion:
Note that [`jai_core:1.1.3` exists in the osgeo.org repository](https://repo.osgeo.org/#browse/search=keyword%3Djavax.media:3f5cae01760233b6403868dadb9402be) -- however, there are entries for that same package in maven central, which contain a pom but no jar.

My working hypothesis is that Coursier is preferring the definition from `maven-central` for whatever reason.
Observations thus far:

• Coursier appears to download from [repo1.maven.org](http://repo1.maven.org/), regardless of `-r` arguments
• Coursier appears to use cached POM definitions even if you change the `-r` arguments and the `COURSIER_REPOSITORIES` envvar (so you have to blow away `~/.cache/pants` to get coursier to try new repos)
I've managed to get the `jai_core` package to resolve properly. The issue is that Coursier will always look in its default repos before user-specified repos unless the `--no-default` option is specified to Coursier.

I'll get a fix for this written presently.

Resolving patch:
diff --git a/src/python/pants/jvm/resolve/coursier_setup.py b/src/python/pants/jvm/resolve/coursier_setup.py
--- a/src/python/pants/jvm/resolve/coursier_setup.py
+++ b/src/python/pants/jvm/resolve/coursier_setup.py
@@ -215,7 +215,10 @@ async def setup_coursier(
     coursier_subsystem: CoursierSubsystem,
     python: PythonBinary,
 ) -> Coursier:
-    repos_args = " ".join(f"-r={shlex.quote(repo)}" for repo in coursier_subsystem.options.repos)
+    repos_args = (
+        " ".join(f"-r={shlex.quote(repo)}" for repo in coursier_subsystem.options.repos)
+        + " --no-default"
+    )
     coursier_wrapper_script = COURSIER_FETCH_WRAPPER_SCRIPT.format(
         repos_args=repos_args,
         coursier_working_directory=Coursier.working_directory_placeholder,


Test patch: