internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pantsbuild/pants

Issue:
Pants not resolving/building Mac dependencies if it finds Linux dependency
I'm currently running Pants 1.13.0 in my company and was preparing to upgrade to 1.14 a while back and ran into this issue then. Link to the Slack conversation: https://pantsbuild.slack.com/archives/C046T6T9U/p1551189787029200

My conclusion was that this issue was introduced with merging of https://github.com/pantsbuild/pants/pull/7156 but I'm not sure if that's correct or not, just seemed to fit in time. :)

I have dug into this a bit more and [created a minimal repro][repro] with a run log. 

From what I can tell the issue is:

1. I have a binary with a platform specified
2. When resolving all dependencies Pants happily downloads the manylinux1 wheel of `cffi` for Linux
3. But Pants doesn't build/download the equivalent wheel for macOS which would be required when running the test, because it has found a package but it's not the correct platform

[repro]: https://github.com/bjorn-spire/pants-multiplatform-build-issues


Discussion:
Thanks for the excellent bug report here!

I see the problem, and I'll put up a fix shortly :)

Resolving patch:
diff --git a/src/python/pants/backend/python/subsystems/pex_build_util.py b/src/python/pants/backend/python/subsystems/pex_build_util.py
--- a/src/python/pants/backend/python/subsystems/pex_build_util.py
+++ b/src/python/pants/backend/python/subsystems/pex_build_util.py
@@ -55,14 +55,24 @@ def has_python_requirements(tgt):
   return isinstance(tgt, PythonRequirementLibrary)
 
 
-def can_have_python_platform(tgt):
+def always_uses_default_python_platform(tgt):
+  return isinstance(tgt, PythonTests)
+
+
+def may_have_explicit_python_platform(tgt):
   return isinstance(tgt, (PythonBinary, PythonDistribution))
 
 
 def targets_by_platform(targets, python_setup):
   d = defaultdict(OrderedSet)
   for target in targets:
-    if can_have_python_platform(target):
+    if always_uses_default_python_platform(target):
+       # There are currently no tests for this because they're super platform specific and it's hard
+       # for us to express that on CI, but https://github.com/pantsbuild/pants/issues/7616 has an
+       # excellent repro case for why this is necessary.
+      for platform in python_setup.platforms:
+        d[platform].add(target)
+    elif may_have_explicit_python_platform(target):
       for platform in target.platforms if target.platforms else python_setup.platforms:
         d[platform].add(target)
   return d


Test patch:
Repository: pantsbuild/pants

Issue:
Use S3 Sync for CI deploys to avoid recopying files
We currently copy all of `dist/deploy` into S3, even if the file is an exact copy of something already in S3. This poses a real cost, both monetary and time.

Instead, here we use `aws s3 sync` to only transfer files that are different from what is currently in the bucket.

Will close https://github.com/pantsbuild/pants/issues/7258.


Discussion:
Going to run all CI via the branch build, not the PR build, since this impacts only non-PRs. I ran yesterday one shard on the PR build to confirm that this won't break PRs uploading `pants.pex`.
Big thanks to @benjyw for all the help to land this!

Resolving patch:
diff --git a/build-support/bin/deploy_to_s3.py b/build-support/bin/deploy_to_s3.py
--- a/build-support/bin/deploy_to_s3.py
+++ b/build-support/bin/deploy_to_s3.py
@@ -12,7 +12,7 @@
 def main() -> None:
   if shutil.which("aws") is None:
     install_aws_cli()
-  check_authentication_setup()
+  setup_authentication()
   deploy()
 
 
@@ -20,11 +20,13 @@ def install_aws_cli() -> None:
   subprocess.run(["./build-support/bin/install_aws_cli_for_ci.sh"], check=True)
 
 
-def check_authentication_setup() -> None:
-  access_key_id = "AWS_ACCESS_KEY_ID"
+def setup_authentication() -> None:
+  access_key_id = "AWS_ACCESS_KEY_ID__TO_BE_REEXPORTED_ON_DEPLOYS"
   secret_access_key = "AWS_SECRET_ACCESS_KEY"
   if access_key_id not in os.environ or secret_access_key not in os.environ:
     die(f"Caller of the script must set both {access_key_id} and {secret_access_key}.")
+  # Properly export the value so that AWS picks it up.
+  os.environ["AWS_ACCESS_KEY_ID"] = os.environ[access_key_id]
 
 
 def deploy() -> None:


Test patch:
Repository: pantsbuild/pants

Issue:
Speculation should fail only if the "primary" fails
The speculation implemented in #8050 (and described in #7949) will fail fast if either the primary or secondary attempt fail.

After more discussion, we think that it is likely the case that only failure of the "primary" attempt to run a command should trigger failure of the entire speculated run.

We should expand the documentation of the `SpeculatingCommandRunner` to describe its behavior (and confirm that the change described here would be "reasonable" in both a remote-first and local-first case).


Discussion:

Resolving patch:
diff --git a/build-support/bin/ci.py b/build-support/bin/ci.py
--- a/build-support/bin/ci.py
+++ b/build-support/bin/ci.py
@@ -223,13 +223,10 @@ def pants_command(
                         "--no-v1",
                         "--v2",
                         "--pants-config-files=pants.remote.ini",
-                        # We turn off speculation to reduce the risk of flakiness, where a test
-                        # passes either locally or remotely and fails in the other environment.
-                        "--process-execution-speculation-strategy=none",
                         f"--remote-oauth-bearer-token-path={oauth_token_path}",
                         "test",
                         *sorted(targets),
-                      ],
+                      ]
     }[self]
     if shard is not None and self in [self.v1_no_chroot, self.v1_chroot]:  # type: ignore
       result.insert(2, f"--test-pytest-test-shard={shard}")
diff --git a/src/python/pants/option/global_options.py b/src/python/pants/option/global_options.py
--- a/src/python/pants/option/global_options.py
+++ b/src/python/pants/option/global_options.py
@@ -97,7 +97,7 @@ def from_bootstrap_options(cls, bootstrap_options):
     process_execution_local_parallelism=multiprocessing.cpu_count()*2,
     process_execution_remote_parallelism=128,
     process_execution_cleanup_local_dirs=True,
-    process_execution_speculation_delay=.1,
+    process_execution_speculation_delay=1,
     process_execution_speculation_strategy='local_first',
     process_execution_use_local_cache=True,
     remote_execution_process_cache_namespace=None,


Test patch:
Repository: pantsbuild/pants

Issue:
Narrow or remove the hole that pantsd punches in the ignore patterns for `.pids`
In #9636, we punched a hole in the ignore patterns in order to allow `pantsd` to watch its own pidfile. But in cases where users had previously used `pantsd`, this resulted in a failure to deal with an orphaned `watchman` socket file (`watchman` is now disabled by default, so this only impacts users who) which had previous been ignored.

```
22:32:58:201 [ERROR] 1 Exception encountered:
  Exception: Failed to scan directory "$workspace/.pids/watchman": Expected File, Dir or Link, but ".pids/watchman/watchman.sock" (relative to "$workspace") was a FileType(FileType { mode: 49152 })
```

Narrowing the pattern to just the `.pids/pantsd` subdirectory is challenging (possibly not an option) because gitignore patterns do not allow re-including the children (`.pids/pantsd` in this case) of ignored parents (`.pids`, due to `.*`).


Discussion:

Resolving patch:
diff --git a/src/python/pants/init/options_initializer.py b/src/python/pants/init/options_initializer.py
--- a/src/python/pants/init/options_initializer.py
+++ b/src/python/pants/init/options_initializer.py
@@ -118,9 +118,12 @@ def add(absolute_path, include=False):
 
         add(global_options.pants_workdir)
         add(global_options.pants_distdir)
-        # NB: We punch a hole in the ignore patterns to allow pants to directly watch process
-        # metadata that is written to disk.
+        # TODO: We punch a hole in the ignore patterns to allow pantsd to directly watch process
+        # metadata that is written to disk, but we re-ignore the watchman directory (which
+        # contains a named pipe). Over time, as more of the pantsd server components are ported to
+        # rust, we will be able to remove this special case.
         add(global_options.pants_subprocessdir, include=True)
+        add(os.path.join(global_options.pants_subprocessdir, "watchman"))
 
         return pants_ignore
 


Test patch:
Repository: pantsbuild/pants

Issue:
Invalid project names of projects generated with `./pants idea-plugin` 
Since the switch to `.idea` project type (https://github.com/pantsbuild/pants/commit/bc8b6d2de121458aeab41c6b9afb45dea2b450f6), the name of IntelliJ project generated with `./pants idea-plugin` is the same as the folder name where the generated project files are located. Before the change, it was more descriptive, same as the imported target spec


Discussion:

Resolving patch:
diff --git a/src/python/pants/backend/project_info/tasks/idea_plugin_gen.py b/src/python/pants/backend/project_info/tasks/idea_plugin_gen.py
--- a/src/python/pants/backend/project_info/tasks/idea_plugin_gen.py
+++ b/src/python/pants/backend/project_info/tasks/idea_plugin_gen.py
@@ -138,6 +138,7 @@ def __init__(self, *args, **kwargs):
         safe_mkdir(output_dir)
 
         with temporary_dir(root_dir=output_dir, cleanup=False) as output_project_dir:
+            self.project_name = self.get_project_name(self.context.options.specs)
             self.gen_project_workdir = output_project_dir
             self.idea_workspace_filename = os.path.join(
                 self.gen_project_workdir, ".idea", "workspace.xml"
@@ -145,6 +146,7 @@ def __init__(self, *args, **kwargs):
             self.idea_modules_filename = os.path.join(
                 self.gen_project_workdir, ".idea", "modules.xml"
             )
+            self.idea_name_filename = os.path.join(self.gen_project_workdir, ".idea", ".name")
             self.intellij_output_dir = os.path.join(self.gen_project_workdir, "out")
             self.intellij_idea_dir = os.path.join(self.gen_project_workdir, ".idea")
 
@@ -197,9 +199,11 @@ def gen_file(template_file_name, **mustache_kwargs):
 
         idea_ws = gen_file(self.idea_workspace_template, workspace=configured_workspace)
         idea_modules = gen_file(self.idea_modules_template, project=configured_project)
+        idea_dotname = self._write_to_tempfile(self.project_name)
 
         shutil.move(idea_ws, self.idea_workspace_filename)
         shutil.move(idea_modules, self.idea_modules_filename)
+        shutil.move(idea_dotname, self.idea_name_filename)
 
         return self.gen_project_workdir
 
@@ -212,6 +216,12 @@ def _generate_to_tempfile(self, generator):
             generator.write(output)
             return output.name
 
+    def _write_to_tempfile(self, content):
+        """Writes coontent to a temp file and returns the path to that file."""
+        with temporary_file(cleanup=False, binary_mode=False) as output:
+            output.write(content)
+            return output.name
+
     def console_output(self, _targets):
         if not self.context.options.specs:
             raise TaskError("No targets specified.")


Test patch:
Repository: pantsbuild/pants

Issue:
 Uncaught exception in nailgun handler
```
15:41:50:645 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".pids/pantsd/fingerprint", ".pids/pantsd"}
15:41:50:714 [INFO] handling request: `--pants-config-files=+["/Users/asher/projects/toolchain/toolchain.toml"] --pants-bin-name=./tc test src/python/toolchain/buildsense/:: src/python/toolchain/django/site/::`
15:42:24:076 [INFO] request completed: `--pants-config-files=+["/Users/asher/projects/toolchain/toolchain.toml"] --pants-bin-name=./tc test src/python/toolchain/buildsense/:: src/python/toolchain/django/site/::`
15:42:24:079 [INFO] handling request: `--pants-config-files=+["/Users/asher/projects/toolchain/toolchain.toml"] --pants-bin-name=./tc fmt src/python/toolchain/buildsense/::`
15:42:33:225 [INFO] request completed: `--pants-config-files=+["/Users/asher/projects/toolchain/toolchain.toml"] --pants-bin-name=./tc fmt src/python/toolchain/buildsense/::`
15:42:33:229 [ERROR] Uncaught exception in nailgun handler: Throw {
    val: [Errno 9] Bad file descriptor,
    python_traceback: "Traceback (most recent call last):\n  File \"/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0.dev1_py38/lib/python3.8/site-packages/pants/engine/internals/native.py\", line 288, in call\n    val = func(*args)\n  File \"/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0.dev1_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py\", line 173, in __call__\n    return self._run(working_directory.decode())\n  File \"/Users/asher/.pyenv/versions/3.8.2/lib/python3.8/contextlib.py\", line 120, in __exit__\n    next(self.gen)\n  File \"/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0.dev1_py38/lib/python3.8/site-packages/pants/util/contextutil.py\", line 150, in stdio_as\n    yield\n  File \"/Users/asher/.pyenv/versions/3.8.2/lib/python3.8/contextlib.py\", line 120, in __exit__\n    next(self.gen)\n  File \"/Users/asher/.cache/pants/setup/bootstrap-Darwin-x86_64/1.29.0.dev1_py38/lib/python3.8/site-packages/pants/util/contextutil.py\", line 126, in _stdio_stream_as\n    new_dst.close()\nOSError: [Errno 9] Bad file descriptor\n",
    engine_traceback: [],
}


```

@stuhood 


Discussion:
Thanks... was this visible in the client at all, or only in the server's log?
just the server log. didn't see anything in the client

Resolving patch:
diff --git a/src/python/pants/util/contextutil.py b/src/python/pants/util/contextutil.py
--- a/src/python/pants/util/contextutil.py
+++ b/src/python/pants/util/contextutil.py
@@ -121,9 +121,9 @@ def _stdio_stream_as(src_fd: int, dst_fd: int, dst_sys_attribute: str, mode: str
                 termios.tcdrain(dst_fd)
             else:
                 new_dst.flush()
+            new_dst.close()
         except BaseException:
             pass
-        new_dst.close()
 
         # Restore the python and os level file handles.
         os.dup2(old_dst_fd, dst_fd)


Test patch:
Repository: pantsbuild/pants

Issue:
pantsd leaks memory on ubuntu linux box
We have an EC2 machine running ubuntu
`Linux ip-10-1-201-194 5.3.0-1028-aws #30~18.04.1-Ubuntu SMP Mon Jun 22 15:48:21 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux`
Pantsd leaks memory which causes the machine to hang a few times a week.
<img width="1244" alt="Screen Shot 2020-07-06 at 8 48 52 AM" src="https://user-images.githubusercontent.com/1268088/86612805-86a84c00-bf65-11ea-828e-bfe69eada9fc.png">
<img width="1593" alt="Screen Shot 2020-07-06 at 8 49 16 AM" src="https://user-images.githubusercontent.com/1268088/86612846-945dd180-bf65-11ea-8389-02ec6f1204ee.png">


@stuhood FYI


Discussion:
This was previously resolved in #10003, but it looks like the default value is too high for this box. 4GB still feels like a reasonable default, although we could maybe try to do a better job of choosing it automatically with something like https://crates.io/crates/effective-limits

In this case, it feels like lowering the default in this environment might be reasonable? A shared dev box with 4GB total ram is unusual.
obviously, I don't have enough context. However, as a user, 4gb seems extremely excessive. 
so I don't think that is a reasonable default. We need to keep memory free so we can run slack & chrome ;-)


Resolving patch:
diff --git a/src/python/pants/option/global_options.py b/src/python/pants/option/global_options.py
--- a/src/python/pants/option/global_options.py
+++ b/src/python/pants/option/global_options.py
@@ -547,7 +547,7 @@ def register_bootstrap_options(cls, register):
             "--pantsd-max-memory-usage",
             advanced=True,
             type=int,
-            default=2 ** 32,
+            default=2 ** 30,
             help=(
                 "The maximum memory usage of a pantsd process (in bytes). There is at most one "
                 "pantsd process per workspace."


Test patch:
Repository: pantsbuild/pants

Issue:
Pex TMPDIR is relative
@etiennedupont noticed the following in Pex CLI subprocesses after upgrading from `2.0.0a3` to `2.0.0b0`
```
nvcc fatal   : Could not open output file '.tmp/tmpxft_00018c96_00000000'
```

That release included this bad fix:
https://github.com/pantsbuild/pants/blob/dfb2a85245a9e4cc8d8ba39455350adb985138f3/src/python/pants/backend/python/util_rules/pex_cli.py#L99-L107

The problem is that a subprocess is free to change its working directory in which case TMPDIR, being relative to the Pex CLI working dir, is no longer a valid path.



Discussion:
We can fix this with an upgrade to Pex 2.1.18 which will contain a fix for pantsbuild/pex#1067 that we can leverage.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -32,7 +32,7 @@ class PexBinary(ExternalTool):
 
     options_scope = "download-pex-bin"
     name = "pex"
-    default_version = "v2.1.17"
+    default_version = "v2.1.18"
 
     @classproperty
     def default_known_versions(cls):
@@ -41,8 +41,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "df836ddd102d8fc9fb390cc4c7fc478caf5afa1600cbdeb7753b185e27a6a8b0",
-                    "2673293",
+                    "3a47dcbcf49294c06439fdda50675546edc9ea09a116c4c57ee5a8e337a63509",
+                    "2678219",
                 )
             )
             for plat in ["darwin", "linux"]
@@ -133,17 +133,20 @@ async def setup_pex_cli_process(
         *cert_args,
         "--pex-root",
         pex_root_path,
-        python=request.python,
-    )
-    env = {
         # Ensure Pex and its subprocesses create temporary files in the the process execution
         # sandbox. It may make sense to do this generally for Processes, but in the short term we
         # have known use cases where /tmp is too small to hold large wheel downloads Pex is asked to
         # perform. Making the TMPDIR local to the sandbox allows control via
         # --local-execution-root-dir for the local case and should work well with remote cases where
         # a remoting implementation has to allow for processes producing large binaries in a
-        # sandbox to support reasonable workloads.
-        "TMPDIR": tmpdir,
+        # sandbox to support reasonable workloads. Communicating TMPDIR via --tmpdir instead of via
+        # environment variable allows Pex to absolutize the path ensuring subprocesses that change
+        # CWD can find the TMPDIR.
+        "--tmpdir",
+        tmpdir,
+        python=request.python,
+    )
+    env = {
         **pex_env.environment_dict(python_configured=request.python is not None),
         **python_native_code.environment_dict,
         **(request.extra_env or {}),


Test patch:
Repository: pantsbuild/pants

Issue:
Ctrl+C delayed when pantsd is disabled
#11223 broke `Ctrl+C` when `pantsd` is disabled, because we no longer explicitly poll the Python code inside the `Scheduler::execute` loop.


Discussion:

Resolving patch:
diff --git a/src/python/pants/base/exception_sink.py b/src/python/pants/base/exception_sink.py
--- a/src/python/pants/base/exception_sink.py
+++ b/src/python/pants/base/exception_sink.py
@@ -15,7 +15,6 @@
 import psutil
 import setproctitle
 
-from pants.engine.internals.native_engine import session_cancel_all
 from pants.util.dirutil import safe_mkdir, safe_open
 from pants.util.osutil import Pid
 
@@ -75,7 +74,6 @@ def _send_signal_to_children(self, received_signal: int, signame: str) -> None:
             child_process.send_signal(received_signal)
 
     def handle_sigint(self, signum: int, _frame):
-        session_cancel_all()
         self._send_signal_to_children(signum, "SIGINT")
         raise KeyboardInterrupt("User interrupted execution with control-c!")
 
@@ -102,12 +100,10 @@ def __init__(self, signum, signame):
                 )
 
     def handle_sigquit(self, signum, _frame):
-        session_cancel_all()
         self._send_signal_to_children(signum, "SIGQUIT")
         raise self.SignalHandledNonLocalExit(signum, "SIGQUIT")
 
     def handle_sigterm(self, signum, _frame):
-        session_cancel_all()
         self._send_signal_to_children(signum, "SIGTERM")
         raise self.SignalHandledNonLocalExit(signum, "SIGTERM")
 


Test patch:
Repository: pantsbuild/pants

Issue:
Filename too long when invoking PEX with --pex-verbosity >=3
Any PEX-using run of Pants at 44017419d (PEX 2.1.33) with `--pex-verbosity=3` (i.e. `./pants test --pex-verbosity=3 src/python/pants/util/::`) fails reproducibly with a line like:
```
/Users/stuhood/.cache/pants/named_caches/pex_root/venvs/34bbe9579368d3ee50226ae26ab367a6fe827866/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: File name too long
```
Notably: `--pex-verbosity=1` does not trigger the issue.

Capturing the sandbox of the run using `--no-process-execution-cleanup-local-dirs` allows for reproducing the issue with the `__run.sh` script (have captured a repro). The complete log of `./__run.sh` is attached.

[ftl.log](https://github.com/pantsbuild/pants/files/6127740/ftl.log)

----

The content of the `pants_plugins.pex_pex_shim.sh` file (attached as [pants_plugins.pex_pex_shim.sh.txt](https://github.com/pantsbuild/pants/files/6127756/pants_plugins.pex_pex_shim.sh.txt)) seems to indicate that some of the verbose output gets accidentally embedded in the generated shim script.


Discussion:
It looks like the computed `venv_dir` is capturing the verbose stdout here, and that's getting prepended onto other paths:
https://github.com/pantsbuild/pants/blob/7781df51294b20e71859c9d5dd380d567ec5dde5/src/python/pants/backend/python/util_rules/pex.py#L792-L799
This is a Pex bug: https://github.com/pantsbuild/pex/pull/1268

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,7 +37,7 @@ class PexBinary(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.33"
+    default_version = "v2.1.34"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
 
     @classproperty
@@ -47,8 +47,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "7f9f6168691fe83f38fbc248bd8629a152cf2a8833be1afdc06219f70fbb6064",
-                    "3596348",
+                    "9b1a959ccb61b3deb64ffeed43a735c7115e414f4de6f96e66adc9e7fc7a757f",
+                    "3597768",
                 )
             )
             for plat in ["darwin", "linux"]


Test patch: