internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pantsbuild/pants

Issue:
pantsd log files gets trimmed too frequently
this is mostly problematic in CI scenarios when the log is uploaded as an artifact.
missing information makes it hard to debug and troubleshoot any pantsd issue that occur in CI.
https://pantsbuild.slack.com/archives/C046T6T9U/p1642104451293500

 


Discussion:
@stuhood 

Resolving patch:
diff --git a/src/python/pants/pantsd/pants_daemon.py b/src/python/pants/pantsd/pants_daemon.py
--- a/src/python/pants/pantsd/pants_daemon.py
+++ b/src/python/pants/pantsd/pants_daemon.py
@@ -32,6 +32,7 @@
 from pants.util.contextutil import argv_as, hermetic_environment_as
 from pants.util.dirutil import safe_open
 from pants.util.strutil import ensure_text
+from pants.version import VERSION
 
 
 class PantsDaemon(PantsDaemonProcessManager):
@@ -145,7 +146,7 @@ def _close_stdio(self, log_path: PurePath):
             fd.close()
 
             # Open the new.
-            temp_fd = safe_open(log_path, "w") if writable else open(os.devnull)
+            temp_fd = safe_open(log_path, "a") if writable else open(os.devnull)
             os.dup2(temp_fd.fileno(), fileno)
             setattr(sys, attr, os.fdopen(fileno, mode=("w" if writable else "r")))
         sys.__stdin__, sys.__stdout__, sys.__stderr__ = sys.stdin, sys.stdout, sys.stderr  # type: ignore[assignment]
@@ -162,7 +163,7 @@ def _initialize_metadata(self) -> None:
         self.write_pid()
         self.write_process_name()
         self.write_fingerprint(ensure_text(self.options_fingerprint))
-        self._logger.debug(f"pantsd running with PID: {self.pid}")
+        self._logger.info(f"pantsd {VERSION} running with PID: {self.pid}")
         self.write_socket(self._server.port())
 
     def run_sync(self):


Test patch:
Repository: pantsbuild/pants

Issue:
Please add Payhere to the public list of Pants Users
### Company name

Payhere

### Company website

https://payhere.in

### Company logo

![payhere_logo_basic](https://user-images.githubusercontent.com/19988590/210507676-6d036b4c-f105-4723-97f5-7c4879e3ac07.jpg)



Discussion:

Resolving patch:
diff --git a/build-support/bin/generate_user_list.py b/build-support/bin/generate_user_list.py
--- a/build-support/bin/generate_user_list.py
+++ b/build-support/bin/generate_user_list.py
@@ -92,6 +92,7 @@ class Org:
         "https://files.readme.io/e87f6c5-Orca_Security-small.png",
     ),
     Org("Pave", "https://www.pave.dev/", "https://files.readme.io/924aa3e-pave-small.png"),
+    Org("Payhere", "https://payhere.in/", "https://files.readme.io/d263cfd-payhere-small.jpg"),
     Org(
         "People Data Labs",
         "https://www.peopledatalabs.com/",


Test patch:
Repository: pantsbuild/pants

Issue:
Options value history for pass-through args is wrong
**Describe the bug**
Pass-through options always reports having a value from the command line, even when there were none.

**Pants version**
`main`

**OS**
MacOS.

**Additional info**
This has no functional impact, only presentational in the help, as demonstrated in these two pictures:

Correctly reports the source of the pass-through args:
<img width="393" alt="image" src="https://user-images.githubusercontent.com/72965/213452308-32a9c89f-620f-4e18-8e9c-403404ddf1ba.png">

Incorrectly reports a value from the command line, although there was none:
<img width="383" alt="image" src="https://user-images.githubusercontent.com/72965/213452473-a6447792-631a-4ffa-9a03-c4927b766602.png">



Discussion:
N.B. this is a generic issue, not specific to any particular subsystem. Subsystem used here to serve as example only.

Resolving patch:
diff --git a/src/python/pants/option/parser.py b/src/python/pants/option/parser.py
--- a/src/python/pants/option/parser.py
+++ b/src/python/pants/option/parser.py
@@ -624,7 +624,7 @@ def merge_in_rank(vals):
 
         # Get value from cmd-line flags.
         flag_vals = list(flag_val_strs)
-        if kwargs.get("passthrough"):
+        if kwargs.get("passthrough") and passthru_arg_strs:
             # NB: Passthrough arguments are either of type `str` or `shell_str`
             # (see self._validate): the former never need interpretation, and the latter do not
             # need interpretation when they have been provided directly via `sys.argv` as the


Test patch:
Repository: pantsbuild/pants

Issue:
`shfmt` assumes downloaded executable will be named `shfmt_{version}_{platform}`, and breaks if it isn't
**Describe the bug**

To reduce network transfer & flakiness during CI, we've pre-cached all the "external" tools used by Pants in our executor container. As part of this I've overridden the `url_template` for each tool to use a `file://` URL. The URL-paths I ended up using in the image were "simplified" from the defaults - for example, I have:
```toml
[shfmt]
url_template = "file:///opt/pants-tools/shfmt/{version}/shfmt"
```

When CI runs with this config, it fails with:
```
Error launching process: Os { code: 2, kind: NotFound, message: "No such file or directory" }
```

I `ssh`'d into one of the executors that hit this failure, and looked inside the failing sandbox. There I saw:
1. The `shfmt` binary _was_ in the sandbox, and runnable
2. According to `__run.sh`, Pants was trying to invoke `./shfmt_v3.2.4_linux_amd64` instead of plain `./shfmt`

I believe this is happening because the `shfmt` subsystem defines `generate_exe` to hard-code the same naming pattern as is used in the default `url_pattern`: https://github.com/pantsbuild/pants/blob/ac9e27b142b14f079089522c1175a9e380291100/src/python/pants/backend/shell/lint/shfmt/subsystem.py#L56-L58

I think things would operate as expected if we deleted that `generate_exe` override, since the `shfmt` download is the executable itself.

**Pants version**

2.15.0rc4

**OS**

Observed on Linux

**Additional info**

https://app.toolchain.com/organizations/color/repos/color/builds/pants_run_2023_02_15_12_48_26_897_660d20c55cc041fbb63374c79a4402b0/



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/shell/lint/shfmt/subsystem.py b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
--- a/src/python/pants/backend/shell/lint/shfmt/subsystem.py
+++ b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
@@ -8,7 +8,6 @@
 
 from pants.core.util_rules.config_files import ConfigFilesRequest
 from pants.core.util_rules.external_tool import TemplatedExternalTool
-from pants.engine.platform import Platform
 from pants.option.option_types import ArgsListOption, BoolOption, SkipOption
 from pants.util.strutil import softwrap
 
@@ -53,10 +52,6 @@ class Shfmt(TemplatedExternalTool):
         ),
     )
 
-    def generate_exe(self, plat: Platform) -> str:
-        plat_str = self.default_url_platform_mapping[plat.value]
-        return f"./shfmt_{self.version}_{plat_str}"
-
     def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:
         # Refer to https://editorconfig.org/#file-location for how config files are discovered.
         candidates = (os.path.join(d, ".editorconfig") for d in ("", *dirs))


Test patch:
Repository: pantsbuild/pants

Issue:
`shfmt` assumes downloaded executable will be named `shfmt_{version}_{platform}`, and breaks if it isn't
**Describe the bug**

To reduce network transfer & flakiness during CI, we've pre-cached all the "external" tools used by Pants in our executor container. As part of this I've overridden the `url_template` for each tool to use a `file://` URL. The URL-paths I ended up using in the image were "simplified" from the defaults - for example, I have:
```toml
[shfmt]
url_template = "file:///opt/pants-tools/shfmt/{version}/shfmt"
```

When CI runs with this config, it fails with:
```
Error launching process: Os { code: 2, kind: NotFound, message: "No such file or directory" }
```

I `ssh`'d into one of the executors that hit this failure, and looked inside the failing sandbox. There I saw:
1. The `shfmt` binary _was_ in the sandbox, and runnable
2. According to `__run.sh`, Pants was trying to invoke `./shfmt_v3.2.4_linux_amd64` instead of plain `./shfmt`

I believe this is happening because the `shfmt` subsystem defines `generate_exe` to hard-code the same naming pattern as is used in the default `url_pattern`: https://github.com/pantsbuild/pants/blob/ac9e27b142b14f079089522c1175a9e380291100/src/python/pants/backend/shell/lint/shfmt/subsystem.py#L56-L58

I think things would operate as expected if we deleted that `generate_exe` override, since the `shfmt` download is the executable itself.

**Pants version**

2.15.0rc4

**OS**

Observed on Linux

**Additional info**

https://app.toolchain.com/organizations/color/repos/color/builds/pants_run_2023_02_15_12_48_26_897_660d20c55cc041fbb63374c79a4402b0/



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/shell/lint/shfmt/subsystem.py b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
--- a/src/python/pants/backend/shell/lint/shfmt/subsystem.py
+++ b/src/python/pants/backend/shell/lint/shfmt/subsystem.py
@@ -8,7 +8,6 @@
 
 from pants.core.util_rules.config_files import ConfigFilesRequest
 from pants.core.util_rules.external_tool import TemplatedExternalTool
-from pants.engine.platform import Platform
 from pants.option.option_types import ArgsListOption, BoolOption, SkipOption
 from pants.util.strutil import softwrap
 
@@ -49,10 +48,6 @@ class Shfmt(TemplatedExternalTool):
         ),
     )
 
-    def generate_exe(self, plat: Platform) -> str:
-        plat_str = self.default_url_platform_mapping[plat.value]
-        return f"./shfmt_{self.version}_{plat_str}"
-
     def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:
         # Refer to https://editorconfig.org/#file-location for how config files are discovered.
         candidates = (os.path.join(d, ".editorconfig") for d in ("", *dirs))


Test patch:
Repository: pantsbuild/pants

Issue:
attrs module mapping needs to be updated?
**Describe the bug**
Pants fails to resolve the dependency for `import attrs` with the following error, when `attrs` is listed in `requirements.txt` and `import attr` works:
```
14:19:42.52 [WARN] Pants cannot infer owners for the following imports in the target helloworld/main.py:lib:

  * attrs (line: 4)
```

The *attrs* class now provides two import packages (as of [v21.3.0](https://www.attrs.org/en/stable/changelog.html#id7)): `attr` and `attrs`.  Thus, I think the `DEFAULT_MODULE_MAPPING` for `attrs` in `src/python/pants/backend/python/dependency_inference/default_module_mapping.py` needs to be updated to include both.  I was able to workaround the issue by adding the following directive to the `BUILD` file:

```
python_requirement(
    name="attrs",
    requirements=["attrs>=21.3.0"],
    modules=["attrs"],
)
```

**Pants version**
2.15.0

**OS**
Ubuntu Linux (WSL).

**Additional info**
I was able to replicate the issue by cloning https://github.com/pantsbuild/example-python and making the changes published in [this gist](https://gist.github.com/smsearcy/20c4c6a4e18132a5fda0c654317d5369).



Discussion:
@smsearcy you've done everything except submitting the PR. Do you want to do that?
@jsirois thank you, I hadn't actually tested the fix either.  I had just started looking at Pants, so I wanted to confirm I was understanding this correctly.  I'll go ahead and open the PR.

Resolving patch:
diff --git a/src/python/pants/backend/python/dependency_inference/default_module_mapping.py b/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
--- a/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
+++ b/src/python/pants/backend/python/dependency_inference/default_module_mapping.py
@@ -9,7 +9,7 @@
     "ansicolors": ("colors",),
     "apache-airflow": ("airflow",),
     "atlassian-python-api": ("atlassian",),
-    "attrs": ("attr",),
+    "attrs": ("attr", "attrs"),
     # Azure
     "azure-common": ("azure.common",),
     "azure-core": ("azure.core",),


Test patch:
Repository: pantsbuild/pants

Issue:
Pants fails to discover ruff configuration file when [tool.ruff] section is missing
**Describe the bug**
When the command `pants fmt fix lint ::` is run with a pyproject.toml file that only contains a ruff sub-section config (e.g. [tool.ruff.per-file-ignores]) and no top-level [tool.ruff] section, Pants fails to discover the configuration file and misses the configurations in the sub-section. 

**Pants version**
2.16.0.dev7

**OS**
Ubuntu 22.10

**Additional info**
[A minimal example](https://github.com/siggi84/pants_ruff_bug/tree/pyproject_bug) that reproduces the issue is available. It shows how the this bug causes a removal of an unused imports, even when the rule causing the removal should be disabled in a config sub-section.

This issue is likely due to a simplistic config discovery mechanism, as indicated by this line of code
https://github.com/pantsbuild/pants/blob/c9def21de8a4f95d6929af227b605b3f1adfa9d2/src/python/pants/backend/python/lint/ruff/subsystem.py#L108


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/ruff/subsystem.py b/src/python/pants/backend/python/lint/ruff/subsystem.py
--- a/src/python/pants/backend/python/lint/ruff/subsystem.py
+++ b/src/python/pants/backend/python/lint/ruff/subsystem.py
@@ -105,7 +105,7 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:
             specified_option_name=f"[{self.options_scope}].config",
             discovery=self.config_discovery,
             check_existence=["ruff.toml", *(os.path.join(d, "ruff.toml") for d in ("", *dirs))],
-            check_content={"pyproject.toml": b"[tool.ruff]"},
+            check_content={"pyproject.toml": b"[tool.ruff"},
         )
 
 


Test patch:
Repository: pantsbuild/pants

Issue:
Only expose the equiv of `from typing import *` in prelude files
              `dir` on a module appears to include all of the top-level names including any imported modules. For `typing` this includes `sys` (and `__builtins__`). Is that intentional?

_Originally posted by @huonw in https://github.com/pantsbuild/pants/pull/18442#discussion_r1130083359_
            


Discussion:

Resolving patch:
diff --git a/src/python/pants/engine/internals/build_files.py b/src/python/pants/engine/internals/build_files.py
--- a/src/python/pants/engine/internals/build_files.py
+++ b/src/python/pants/engine/internals/build_files.py
@@ -94,7 +94,7 @@ async def evaluate_preludes(
     )
     globals: dict[str, Any] = {
         **{name: getattr(builtins, name) for name in dir(builtins) if name.endswith("Error")},
-        **{name: getattr(typing, name) for name in dir(typing)},
+        **{name: getattr(typing, name) for name in typing.__all__},
         # Ensure the globals for each prelude includes the builtin symbols (E.g. `python_sources`)
         # and any build file aliases (e.g. from plugins)
         **parser.symbols,


Test patch:
Repository: pantsbuild/pants

Issue:
Add support for authentication for docker environments
Currently, the `bollard` crate [does not support](https://github.com/fussybeaver/bollard/issues/155) consuming `~/.docker/config.json` or interacting with [docker credential helpers](https://docs.docker.com/engine/reference/commandline/login/#credential-helper-protocol). This means that images from Azure/AWS/etc private registries cannot be used as `docker_environment` targets.

To fix this, it looks like we can integrate the https://crates.io/crates/docker_credential crate.


Discussion:

Resolving patch:
diff --git a/src/python/pants/pantsd/pants_daemon.py b/src/python/pants/pantsd/pants_daemon.py
--- a/src/python/pants/pantsd/pants_daemon.py
+++ b/src/python/pants/pantsd/pants_daemon.py
@@ -44,6 +44,10 @@
     "DOCKER_HOST",
     "DOCKER_CONFIG",
     "DOCKER_CERT_PATH",
+    # Environment variables consumed (indirectly) by the `docker_credential` crate as of
+    # https://github.com/keirlawson/docker_credential/commit/0c42d0f3c76a7d5f699d4d1e8b9747f799cf6116
+    "HOME",
+    "PATH",
 ]
 
 


Test patch:
Repository: pantsbuild/pants

Issue:
Add support for authentication for docker environments
Currently, the `bollard` crate [does not support](https://github.com/fussybeaver/bollard/issues/155) consuming `~/.docker/config.json` or interacting with [docker credential helpers](https://docs.docker.com/engine/reference/commandline/login/#credential-helper-protocol). This means that images from Azure/AWS/etc private registries cannot be used as `docker_environment` targets.

To fix this, it looks like we can integrate the https://crates.io/crates/docker_credential crate.


Discussion:

Resolving patch:
diff --git a/src/python/pants/pantsd/pants_daemon.py b/src/python/pants/pantsd/pants_daemon.py
--- a/src/python/pants/pantsd/pants_daemon.py
+++ b/src/python/pants/pantsd/pants_daemon.py
@@ -43,6 +43,10 @@
     "DOCKER_HOST",
     "DOCKER_CONFIG",
     "DOCKER_CERT_PATH",
+    # Environment variables consumed (indirectly) by the `docker_credential` crate as of
+    # https://github.com/keirlawson/docker_credential/commit/0c42d0f3c76a7d5f699d4d1e8b9747f799cf6116
+    "HOME",
+    "PATH",
 ]
 
 


Test patch:
Repository: pantsbuild/pants

Issue:
`Failed to write to remote cache` when running `ruff`
**Describe the bug**

When running `pants lint --only=ruff ::` I see many `WARN` logs like:
```
14:18:48.60 [WARN] Failed to write to remote cache (1 occurrences so far): Declared output directory path RelativePath("src/django_apps/preventive_care/consults/automatic_refer_out/__init__.py") in output digest Digest { hash: Fingerprint<b4c4a7e44c3d23b8eac247bfdd5ed723d054947915bbfb808e0ee16e4fa75430>, size_bytes: 77 } contained a file instead.
```

**Pants version**

`PANTS_SHA=254f69b3f111fb620206bbfe72b262520849484f` (on the 2.16.x branch)

**OS**

MacOS


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/python/lint/ruff/rules.py b/src/python/pants/backend/python/lint/ruff/rules.py
--- a/src/python/pants/backend/python/lint/ruff/rules.py
+++ b/src/python/pants/backend/python/lint/ruff/rules.py
@@ -45,7 +45,7 @@ async def ruff_fix(request: RuffRequest.Batch, ruff: Ruff) -> FixResult:
             ruff_pex,
             argv=("--fix", *conf_args, *ruff.args, *request.files),
             input_digest=input_digest,
-            output_directories=request.files,
+            output_files=request.files,
             description=f"Run ruff on {pluralize(len(request.elements), 'file')}.",
             level=LogLevel.DEBUG,
         ),


Test patch:
Repository: pantsbuild/pants

Issue:
JVM resource jar creation is broken on Mac by variation in `touch` command.
BSD touch uses `-t` to set the timestamp whereas GNU touch uses `-d`. We use / assume the GNU binary as noted here: https://github.com/pantsbuild/pants/pull/16950#discussion_r1157196330

As discovered here: https://pantsbuild.slack.com/archives/C046T6T9U/p1680604327733559?thread_ts=1680604327.733559&cid=C046T6T9U

Where the error looks like:
```
 12:24:56.74 [ERROR] 1 Exception encountered:
 
 Engine traceback:
   in select
   in pants.core.goals.check.check
   in pants.backend.scala.goals.check.scalac_check (scalac)
   in pants.backend.scala.compile.scalac.compile_scala_source
   in pants.jvm.compile.compile_classpath_entries
   in pants.jvm.resources.assemble_resources_jar
   in pants.engine.process.fallible_to_exec_result_or_raise
 Traceback (most recent call last):
   File "/Users/jbenito/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.1Nnv7r/install/lib/python3.9/site-packages/pants/engine/process.py", line 275, in fallible_to_exec_result_or_raise
     raise ProcessExecutionFailure(
 pants.engine.process.ProcessExecutionFailure: Process 'Build resources JAR for sdk/transport-security-web-lib/src/test/resources:resources' failed with exit code 1.
 stdout:
 
 stderr:
 /usr/bin/touch: illegal option -- d
 usage:
 touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...
```

It appears #16950 was cherry picked back to 2.13.1 and 2.14.0; so Pants has been broken for JVM resource jars since 2.13.1.




Discussion:
Good catch.
@jsirois `man touch` is showing `-d` as a valid option on my machine:

```
TOUCH(1)                     General Commands Manual                    TOUCH(1)

NAME
     touch – change file access and modification times

SYNOPSIS
     touch [-A [-][[hh]mm]SS] [-achm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]]
           [-d YYYY-MM-DDThh:mm:SS[.frac][tz]] file ...
```

At the end of the manpage is:

```
macOS 12.6                        June 1, 2018                        macOS 12.6
```

Probably need a better idea of exactly when this was broken.

That said, it looks like BSD touch and GNU touch both support `-t`, so 🤷 

Resolving patch:
diff --git a/src/python/pants/jvm/resources.py b/src/python/pants/jvm/resources.py
--- a/src/python/pants/jvm/resources.py
+++ b/src/python/pants/jvm/resources.py
@@ -98,8 +98,9 @@ async def assemble_resources_jar(
                 "-c",
                 " ".join(
                     [
+                        "TZ=UTC",
                         touch.path,
-                        "-d 1980-01-01T00:00:00Z",
+                        "-t 198001010000.00",
                         input_filenames,
                         "&&",
                         "TZ=UTC",


Test patch:
Repository: pantsbuild/pants

Issue:
`help backends` and `help-advanced backends` show the same content (both include preview/experimental backends)
**Describe the bug**
Under the `help` goal: `help backends` and `help-advanced backends` show the same content (both include preview/experimental backends). Only the `-advanced` variant should show the preview/experimental backends (according to the `help` output).

```
Usage:
  ...
  ./pants help backends                      List all available backends.
  ./pants help-advanced backends             List all backends, including experimental/preview.
  ...
```

Either remove the experimental/preview backends from the `help backends` view, or just remove the `help-advanced` version.

**Pants version**
2.16.0a0

**OS**
Both


Discussion:

Resolving patch:
diff --git a/src/python/pants/help/help_printer.py b/src/python/pants/help/help_printer.py
--- a/src/python/pants/help/help_printer.py
+++ b/src/python/pants/help/help_printer.py
@@ -227,7 +227,7 @@ def _print_top_level_help(self, thing: str, show_advanced: bool) -> None:
         elif thing == "api-types":
             self._print_all_api_types()
         elif thing == "backends":
-            self._print_all_backends()
+            self._print_all_backends(show_advanced)
         elif thing == "symbols":
             self._print_all_symbols(show_advanced)
 
@@ -351,7 +351,7 @@ def _print_all_api_types(self) -> None:
             f"Use `{self.maybe_green(api_help_cmd)}` to get help for a specific API type or rule.\n"
         )
 
-    def _print_all_backends(self) -> None:
+    def _print_all_backends(self, include_experimental: bool) -> None:
         self._print_title("Backends")
         print(
             softwrap(
@@ -368,6 +368,8 @@ def _print_all_backends(self) -> None:
         provider_col_width = 3 + max(map(len, (info.provider for info in backends.values())))
         enabled_col_width = 4
         for info in backends.values():
+            if not (include_experimental or info.enabled) and ".experimental." in info.name:
+                continue
             enabled = "[*] " if info.enabled else "[ ] "
             name_col_width = max(
                 len(info.name) + 1, self._width - enabled_col_width - provider_col_width


Test patch: