internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pantsbuild/pants

Issue:
Failure to running requirement's binary scripts using subprocess.run or os.system or ... (FileNotFoundError: [Errno 2] No such file or directory: '...')
**Bug description**
I believe this issue is very similar to the https://github.com/pantsbuild/pants/issues/12651 issue.

Consider the following example:

```
.
├── BUILD
├── main.py
├── pants.toml
└── test_main.py
```

```toml
# pants.toml
[GLOBAL]
pants_version = '2.10.0'
backend_packages = ['pants.backend.python']
```

```starlark
# BUILD
python_requirement(name='python_requirement', requirements=['cowsay'])
python_sources(name='python_sources')
python_tests(name='python_tests')
```

```python
# main.py
import subprocess
import cowsay

def cowsay_hello_using_subprocess():
    subprocess.run(['cowsay', 'hello'])

def cowsay_hello_using_module():
    cowsay.cow('hello')
```

```python
# test_main.py
import main

def test_cowsay_hello_using_subprocess():
    main.cowsay_hello_using_subprocess()

def test_cowsay_hello_using_module():
    main.cowsay_hello_using_module()
```
Running the `./pants test :python_tests` results `1 failed, 1 passed`. The `test_cowsay_hello_using_subprocess` test fails:

```FileNotFoundError: [Errno 2] No such file or directory: 'cowsay'```

Actually, a similar issue occurred if we define `pex_binary` for the source code. But, thanks to https://github.com/pantsbuild/pants/issues/12651, using `execution_mode='venv'` solves the issue there.

In my opinion, there should be a similar solution for the intermediate pex files created for other goals and targets. Am I thinking right?


**Pants version**: 2.10.0
**OS**: Ubuntu 20.04.3 LTS



Discussion:
> In my opinion, there should be a similar solution for the intermediate pex files created for other goals and targets. Am I thinking right?

I think so. That would just mean changing `"--venv"` here to `"--venv", "prepend"` (or "append"):
https://github.com/pantsbuild/pants/blob/000be2f3aa40ae1545f2dd70b3517298595b9aaf/src/python/pants/backend/python/util_rules/pex.py#L772-L785

I can't think of any likely trouble this may cause; so it probably could just be added with no ability to toggle needed.
I am also interested in this. We currently have python tests that rely on scripts from 3rdparty deps being on the path.
@mpcusack-color do you want to try tackling the suggested change above?
My current workaround is using `[join(sys.prefix, 'bin', 'command')]` inside `subprocess.run`. The problem remains if the sub-process uses sub-process too (but it works most of the time).

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex.py b/src/python/pants/backend/python/util_rules/pex.py
--- a/src/python/pants/backend/python/util_rules/pex.py
+++ b/src/python/pants/backend/python/util_rules/pex.py
@@ -1030,6 +1030,7 @@ async def create_venv_pex(
         additional_args=pex_request.additional_args
         + (
             "--venv",
+            "prepend",
             "--seed",
             "verbose",
             pex_environment.venv_site_packages_copies_option(


Test patch:
Repository: pantsbuild/pants

Issue:
Can not use target type in visibility rule selector.
**Describe the bug**
> the example given in the docs, {"type": python_sources}, but the parsing breaks with an error of "TypeError: decoding to str: need a bytes-like object, Registrar found",

**Pants version**
2.17.0rc1

**OS**
n/a

**Additional info**
https://pantsbuild.slack.com/archives/C046T6T9U/p1687953070098239




Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/visibility/glob.py b/src/python/pants/backend/visibility/glob.py
--- a/src/python/pants/backend/visibility/glob.py
+++ b/src/python/pants/backend/visibility/glob.py
@@ -225,9 +225,9 @@ def parse(cls: type[TargetGlob], spec: str | Mapping[str, Any], base: str) -> Ta
             raise ValueError(f"Target spec must not be empty. {spec!r}")
 
         return cls.create(  # type: ignore[call-arg]
-            type_=spec_dict.get("type"),
-            name=spec_dict.get("name"),
-            path=spec_dict.get("path"),
+            type_=str(spec_dict["type"]) if "type" in spec_dict else None,
+            name=str(spec_dict["name"]) if "name" in spec_dict else None,
+            path=str(spec_dict["path"]) if "path" in spec_dict else None,
             base=base,
             tags=cls._parse_tags(spec_dict.get("tags")),
         )


Test patch:
diff --git a/src/python/pants/backend/visibility/rule_types_test.py b/src/python/pants/backend/visibility/rule_types_test.py
--- a/src/python/pants/backend/visibility/rule_types_test.py
+++ b/src/python/pants/backend/visibility/rule_types_test.py
@@ -931,3 +931,24 @@ def test_single_rules_declaration_per_build_file(rule_runner: RuleRunner) -> Non
             rule_runner,
             "test",
         )
+
+
+def test_target_type_in_verbose_selector_issue_19390(rule_runner: RuleRunner) -> None:
+    """This test is logically a better fit in `./glob_test.py:test_target_glob_parse_spec` but I
+    want to avoid hard coding any assumptions regarding a target types representation in the BUILD
+    file (the `Registrar` type)."""
+    rule_runner.write_files(
+        {
+            "test/BUILD": dedent(
+                """
+                target(name="test")
+
+                __dependencies_rules__(
+                  ({"type": files}, "*"),
+                )
+                """
+            ),
+        },
+    )
+
+    assert_dependency_rules(rule_runner, "test")
Repository: pantsbuild/pants

Issue:
Can not use target type in visibility rule selector.
**Describe the bug**
> the example given in the docs, {"type": python_sources}, but the parsing breaks with an error of "TypeError: decoding to str: need a bytes-like object, Registrar found",

**Pants version**
2.17.0rc1

**OS**
n/a

**Additional info**
https://pantsbuild.slack.com/archives/C046T6T9U/p1687953070098239




Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/visibility/glob.py b/src/python/pants/backend/visibility/glob.py
--- a/src/python/pants/backend/visibility/glob.py
+++ b/src/python/pants/backend/visibility/glob.py
@@ -225,9 +225,9 @@ def parse(cls: type[TargetGlob], spec: str | Mapping[str, Any], base: str) -> Ta
             raise ValueError(f"Target spec must not be empty. {spec!r}")
 
         return cls.create(  # type: ignore[call-arg]
-            type_=spec_dict.get("type"),
-            name=spec_dict.get("name"),
-            path=spec_dict.get("path"),
+            type_=str(spec_dict["type"]) if "type" in spec_dict else None,
+            name=str(spec_dict["name"]) if "name" in spec_dict else None,
+            path=str(spec_dict["path"]) if "path" in spec_dict else None,
             base=base,
             tags=cls._parse_tags(spec_dict.get("tags")),
         )


Test patch:
diff --git a/src/python/pants/backend/visibility/rule_types_test.py b/src/python/pants/backend/visibility/rule_types_test.py
--- a/src/python/pants/backend/visibility/rule_types_test.py
+++ b/src/python/pants/backend/visibility/rule_types_test.py
@@ -927,3 +927,24 @@ def test_single_rules_declaration_per_build_file(rule_runner: RuleRunner) -> Non
             rule_runner,
             "test",
         )
+
+
+def test_target_type_in_verbose_selector_issue_19390(rule_runner: RuleRunner) -> None:
+    """This test is logically a better fit in `./glob_test.py:test_target_glob_parse_spec` but I
+    want to avoid hard coding any assumptions regarding a target types representation in the BUILD
+    file (the `Registrar` type)."""
+    rule_runner.write_files(
+        {
+            "test/BUILD": dedent(
+                """
+                target(name="test")
+
+                __dependencies_rules__(
+                  ({"type": files}, "*"),
+                )
+                """
+            ),
+        },
+    )
+
+    assert_dependency_rules(rule_runner, "test")
Repository: pantsbuild/pants

Issue:
`ruff` backend missing from 2.16.x docs
**Describe the bug**
See https://www.pantsbuild.org/v2.16/docs/reference-all-subsystems. I think it's just missing from docs because I see it in the wheel.



Discussion:
guess it's another list it needs to be a member of for that:
https://github.com/pantsbuild/pants/blob/ad20ba4d303cf293b0bf0ed5d18280f725b11c27/build-support/bin/generate_docs.py#L232-L241

we really ought to generate these lists from an authoritative source where we list all backends and have options like:
* [ ] include in dist
* [ ] include in docs
* [ ] ....



Resolving patch:
diff --git a/build-support/bin/generate_docs.py b/build-support/bin/generate_docs.py
--- a/build-support/bin/generate_docs.py
+++ b/build-support/bin/generate_docs.py
@@ -196,8 +196,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.experimental.python",
         "pants.backend.experimental.python.framework.stevedore",
         "pants.backend.experimental.python.lint.add_trailing_comma",
-        "pants.backend.experimental.python.lint.autoflake",
-        "pants.backend.experimental.python.lint.pyupgrade",
+        "pants.backend.experimental.python.lint.ruff",
         "pants.backend.experimental.python.packaging.pyoxidizer",
         "pants.backend.experimental.scala",
         "pants.backend.experimental.scala.lint.scalafmt",
@@ -207,6 +206,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.google_cloud_function.python",
         "pants.backend.plugin_development",
         "pants.backend.python",
+        "pants.backend.python.lint.autoflake",
         "pants.backend.python.lint.bandit",
         "pants.backend.python.lint.black",
         "pants.backend.python.lint.docformatter",
@@ -214,6 +214,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.python.lint.isort",
         "pants.backend.python.lint.pydocstyle",
         "pants.backend.python.lint.pylint",
+        "pants.backend.python.lint.pyupgrade",
         "pants.backend.python.lint.yapf",
         "pants.backend.python.mixed_interpreter_constraints",
         "pants.backend.python.typecheck.mypy",


Test patch:
Repository: pantsbuild/pants

Issue:
`ruff` backend missing from 2.16.x docs
**Describe the bug**
See https://www.pantsbuild.org/v2.16/docs/reference-all-subsystems. I think it's just missing from docs because I see it in the wheel.



Discussion:
guess it's another list it needs to be a member of for that:
https://github.com/pantsbuild/pants/blob/ad20ba4d303cf293b0bf0ed5d18280f725b11c27/build-support/bin/generate_docs.py#L232-L241

we really ought to generate these lists from an authoritative source where we list all backends and have options like:
* [ ] include in dist
* [ ] include in docs
* [ ] ....



Resolving patch:
diff --git a/build-support/bin/generate_docs.py b/build-support/bin/generate_docs.py
--- a/build-support/bin/generate_docs.py
+++ b/build-support/bin/generate_docs.py
@@ -196,8 +196,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.experimental.python",
         "pants.backend.experimental.python.framework.stevedore",
         "pants.backend.experimental.python.lint.add_trailing_comma",
-        "pants.backend.experimental.python.lint.autoflake",
-        "pants.backend.experimental.python.lint.pyupgrade",
+        "pants.backend.experimental.python.lint.ruff",
         "pants.backend.experimental.python.packaging.pyoxidizer",
         "pants.backend.experimental.scala",
         "pants.backend.experimental.scala.lint.scalafmt",
@@ -206,6 +205,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.google_cloud_function.python",
         "pants.backend.plugin_development",
         "pants.backend.python",
+        "pants.backend.python.lint.autoflake",
         "pants.backend.python.lint.bandit",
         "pants.backend.python.lint.black",
         "pants.backend.python.lint.docformatter",
@@ -213,6 +213,7 @@ def run_pants_help_all() -> dict[str, Any]:
         "pants.backend.python.lint.isort",
         "pants.backend.python.lint.pydocstyle",
         "pants.backend.python.lint.pylint",
+        "pants.backend.python.lint.pyupgrade",
         "pants.backend.python.lint.yapf",
         "pants.backend.python.mixed_interpreter_constraints",
         "pants.backend.python.typecheck.mypy",


Test patch:
Repository: pantsbuild/pants

Issue:
Missing `pants/backend/experimental/tools/yamllint/__init__.py` file
```
Can not check requirements for backend: 'pants.backend.experimental.tools.yamllint'. A __init__.py file is probably missing.
```

Tagging as internal as I don't believe it has a practical impact.


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/experimental/tools/yamllint/__init__.py b/src/python/pants/backend/experimental/tools/yamllint/__init__.py
new file mode 100644


Test patch:
Repository: pantsbuild/pants

Issue:
Validate benchmarks run successfully in CI
It seems like the benchmark tests aren't currently run in CI, and thus can degrade. For instance #20057.

I believe that `cargo test --benches` (or something along those lines) allows running benchmarks in "test mode" where they only execute once to provide some sort of assurance that they're functional. I wonder if we could do that too and how big of an impact on CI time it would be.

_Originally posted by @huonw in https://github.com/pantsbuild/pants/pull/20057#pullrequestreview-1687414306_
            


Discussion:

Resolving patch:
diff --git a/src/python/pants_release/generate_github_workflows.py b/src/python/pants_release/generate_github_workflows.py
--- a/src/python/pants_release/generate_github_workflows.py
+++ b/src/python/pants_release/generate_github_workflows.py
@@ -620,14 +620,14 @@ def bootstrap_jobs(
         human_readable_job_name += ", test and lint Rust"
         human_readable_step_name = "Test and lint Rust"
         # We pass --tests to skip doc tests because our generated protos contain
-        # invalid doc tests in their comments.
+        # invalid doc tests in their comments, and --benches to ensure that the
+        # benchmarks can at least execute once correctly
         step_cmd = "\n".join(
             [
                 "./build-support/bin/check_rust_pre_commit.sh",
                 helper.maybe_append_cargo_test_parallelism(
-                    "./cargo test --locked --all --tests -- --nocapture"
+                    "./cargo test --locked --all --tests --benches -- --nocapture"
                 ),
-                "./cargo check --benches",
                 "./cargo doc",
             ]
         )


Test patch:
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -198,9 +198,7 @@ jobs:
 
         ./build-support/bin/check_rust_pre_commit.sh
 
-        ./cargo test --locked --all --tests -- --nocapture
-
-        ./cargo check --benches
+        ./cargo test --locked --all --tests --benches -- --nocapture
 
         ./cargo doc'
     timeout-minutes: 60
Repository: pantsbuild/pants

Issue:
helm 2.18.0 interpolation DEPRECATED warnings appear regardless actual use
**Describe the bug**
When using the helm backend these warnings are emitted
```
11:11:03.64 [WARN] DEPRECATED: pants.backend.helm.target_types.HelmDeploymentValuesField.format_with() is scheduled to be removed in version 2.19.0.dev0.
 ==> /home/ecsb/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/util/memo.py:123
                return memoized_results[key]
            result = func(*args, **kwargs)
            memoized_results[key] = result

11:11:03.64 [WARN] DEPRECATED: Using the {env.VAR_NAME} interpolation syntax is scheduled to be removed in version 2.19.0.dev0.
```

Even when the `{env.VAR_NAME}` interpolation syntax is not used anywhere in the target repository.  (Besides `grep`, I also tried upgrading to the 2.19.0 series to make sure everything still worked.)

**Pants version**
2.18.0

**OS**
Linux

**Additional info**
I remain confused *when* pants chooses to emit DEPRECATED as I can't get it print consistently with `pants --no-local-cache --no-pantsd`

https://pantsbuild.slack.com/archives/C046T6T9U/p1700496601965609



Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/helm/target_types.py b/src/python/pants/backend/helm/target_types.py
--- a/src/python/pants/backend/helm/target_types.py
+++ b/src/python/pants/backend/helm/target_types.py
@@ -498,6 +498,11 @@ class HelmDeploymentValuesField(DictStringToStringField, AsyncFieldMixin):
     @deprecated("2.19.0.dev0", start_version="2.18.0.dev0")
     def format_with(
         self, interpolation_context: InterpolationContext, *, ignore_missing: bool = False
+    ) -> dict[str, str]:
+        return self._format_with(interpolation_context, ignore_missing=ignore_missing)
+
+    def _format_with(
+        self, interpolation_context: InterpolationContext, *, ignore_missing: bool = False
     ) -> dict[str, str]:
         source = InterpolationContext.TextSource(
             self.address,
@@ -625,7 +630,7 @@ class HelmDeploymentFieldSet(FieldSet):
     def format_values(
         self, interpolation_context: InterpolationContext, *, ignore_missing: bool = False
     ) -> dict[str, str]:
-        return self.values.format_with(interpolation_context, ignore_missing=ignore_missing)
+        return self.values._format_with(interpolation_context, ignore_missing=ignore_missing)
 
 
 class AllHelmDeploymentTargets(Targets):
diff --git a/src/python/pants/backend/helm/util_rules/renderer.py b/src/python/pants/backend/helm/util_rules/renderer.py
--- a/src/python/pants/backend/helm/util_rules/renderer.py
+++ b/src/python/pants/backend/helm/util_rules/renderer.py
@@ -326,7 +326,7 @@ async def setup_render_helm_deployment_process(
         request.field_set.release_name.value
         or request.field_set.address.target_name.replace("_", "-")
     )
-    inline_values = request.field_set.values.format_with(
+    inline_values = request.field_set.values._format_with(
         interpolation_context, ignore_missing=is_render_cmd
     )
 


Test patch:
Repository: pantsbuild/pants

Issue:
Type error on FileContent results in infinite loop
**Describe the bug**
I'm working on a custom plugin and I did this:

```
content = FileContent(path=versions_file_path, content="some string")
digest = await Get(Digest, CreateDigest([content]))
```

That resulted in the infamous `Filesystem changed during run: retrying 'MyGoal'` infinite loop. I was really confused. Upon further inspection of the run logs, I found this:
```
09:14:56.95 [31m[ERROR][0m panic at 'called `Result::unwrap()` on an `Err` value: "Field `content` was not convertible to type alloc::vec::Vec<u8>: PyErr { type: <class 'TypeError'>, value: TypeError(\"'str' object cannot be interpreted as an integer\"), traceback: None }"', src/intrinsics.rs:422
```
So I updated the code above to call `.encode()` on the string before passing it as content and things started working. However, I feel like my mistake should have resulted in an error / failure instead of the infinite `Filesystem changed` loop as this just made things harder to debug.

**Pants version**
2.16.0

**OS**
MacOS

**Additional info**
I like pants



Discussion:
Good catch @hopper-signifyd . We'd welcome a PR to detect the wrong arg type in the FileContent ctor and error sensibly.

Resolving patch:
diff --git a/src/python/pants/engine/fs.py b/src/python/pants/engine/fs.py
--- a/src/python/pants/engine/fs.py
+++ b/src/python/pants/engine/fs.py
@@ -53,6 +53,12 @@ class FileContent:
     content: bytes
     is_executable: bool = False
 
+    def __post_init__(self):
+        if not isinstance(self.content, bytes):
+            raise TypeError(
+                f"Expected 'content' to be bytes, but got {type(self.content).__name__}"
+            )
+
     def __repr__(self) -> str:
         return (
             f"FileContent(path={self.path}, content=(len:{len(self.content)}), "


Test patch:
diff --git a/src/python/pants/engine/fs_test.py b/src/python/pants/engine/fs_test.py
--- a/src/python/pants/engine/fs_test.py
+++ b/src/python/pants/engine/fs_test.py
@@ -142,6 +142,18 @@ def try_with_backoff(assertion_fn: Callable[[], bool], count: int = 4) -> bool:
     return False
 
 
+# -----------------------------------------------------------------------------------------------
+# `FileContent`
+# -----------------------------------------------------------------------------------------------
+
+
+def test_file_content_non_bytes():
+    with pytest.raises(TypeError) as exc:
+        FileContent(path="4.txt", content="four")
+
+    assert str(exc.value) == "Expected 'content' to be bytes, but got str"
+
+
 # -----------------------------------------------------------------------------------------------
 # `PathGlobs`, including `GlobMatchErrorBehavior` and symlink handling
 # -----------------------------------------------------------------------------------------------
Repository: pantsbuild/pants

Issue:
Empty TOML section `[]` in GLOBAL options docs
**Describe the bug**

The toml suggestion for options in the GLOBAL section end up with `[]` as their TOML section, e.g. https://www.pantsbuild.org/v2.19/docs/reference-global#level

<img width="224" alt="image" src="https://github.com/pantsbuild/pants/assets/1203825/97654b4c-eefe-479a-a20b-7f4c664d5c7e">

That should show:

```toml
[GLOBAL]
level = <LogLevel>
```

**Pants version**
2.19

**OS**
N/A
**Additional info**

#19718 


Discussion:

Resolving patch:
diff --git a/build-support/bin/generate_docs.py b/build-support/bin/generate_docs.py
--- a/build-support/bin/generate_docs.py
+++ b/build-support/bin/generate_docs.py
@@ -35,6 +35,7 @@
 
 from pants.base.build_environment import get_buildroot, get_pants_cachedir
 from pants.help.help_info_extracter import to_help_str
+from pants.option.scope import GLOBAL_SCOPE, GLOBAL_SCOPE_CONFIG_SECTION
 from pants.util.strutil import softwrap
 from pants.version import MAJOR_MINOR
 
@@ -333,6 +334,10 @@ def _generate_toml_snippet(option_data, scope) -> str:
             # This option is not configurable.
             return ""
 
+        if scope == GLOBAL_SCOPE:
+            # make sure we're rendering with the section name as appears in pants.toml
+            scope = GLOBAL_SCOPE_CONFIG_SECTION
+
         toml_lines = []
         example_cli = option_data["display_args"][0]
         config_key = option_data["config_key"]


Test patch: