Repository: pantsbuild/pants

Issue:
Wrong version of Python used to build `pex_binary` targets in `2.16.0rc0`
**Describe the bug**

* Our CI image contains both Python 3.8 and 3.9.
* We set `[python].interpreter_constraints = ["==3.8.*"]` in `pants.toml`.
* At least one `pex_binary` depends on a version of `numpy` that (for reasons we haven't dug into) only works with Python 3.8, not Python 3.9
  * We haven't investigated the build failures because we expect everything to run against Python 3.8 as configured by `[python].interpreter_constraints`

After upgrading to Pants 2.16.0rc0 we see failures building the `pex_binary` in CI, with errors that indicate the build process is trying to build a dependency (`numpy`) against Python 3.9 instead of the expected/configured Python 3.8

This is very concerning. We still run Python 3.8 everywhere in production, so I don't want Pexes to be building against Python 3.9. I've downgraded us back to 2.16.0a1 for now and confirmed this fixes the problem.

**Pants version**

2.16.0rc0

**OS**

Linux

**Additional info**

I am suspicious of https://github.com/pantsbuild/pants/commit/d3d325777952435186be42443fb28fde6771fae7 and https://github.com/pantsbuild/pants/commit/e8d387ba6b4d4502e3b6db5ae68ffe7beeeb10a7


Discussion:
BuildSense link for the CI job that failed: https://app.toolchain.com/organizations/color/repos/color/builds/pants_run_2023_04_03_10_02_39_985_33ee4b30c097463ca5578c297745722d/
(Updated the issue description after taking a closer look at the full log - I had some details wrong, but my core concern is the same)
@danxmoran this is a critical bug! Is there a simple repro you can create on github? Would really need to bisect this.
@benjyw I'll try to get something set up
I've set up a repro at https://github.com/danxmoran/pants-pex-wrong-python-repro

It was slightly more complicated than I first expected! I needed all of:
1. A crusty `git+` requirement
2. An override to the `pex` version
3. Pants 2.16.0rc0

Dropping the `git+` requirement or the `pex` version-override seems to fix the behavior.

I've actually already dropped the req in Color's repo after hitting this behavior yesterday and will upgrade Pex today, so maybe we're unblocked 😅 though I wonder if Pants should bump the minimum "supported" version of Pex to catch this problem up-front.
@danxmoran you found a mitigation. I contend you identified the correct commits beforehand though and no one actually understands the mechanism still, from the 1st commit you identified forward. If my contention is correct it would be good to continue to pause until this is fully sussed, and, presumably, the real issue fixed or the combination of things you found is actually determined to be the issue because of X, Y, Z.
My interpretation is:
* This bug is actually a duplicate of https://github.com/pantsbuild/pex/issues/2092
* That issue was fixed in Pex v2.1.129, but (until today) Color has been pinning Pex v2.1.128
* We'd avoided bumping into the Pex issue because the "ambient" Pex interpreter was Python 3.8, matching the interpreter constraints we use for our code
* https://github.com/pantsbuild/pants/commit/e8d387ba6b4d4502e3b6db5ae68ffe7beeeb10a7 changed the ambient interpreter to always be the scie-provided Python 3.9, so we "suddenly" started hitting the Pex bug
To confirm - upgrading Pex with no other changes fixes the issue? If so then I assume we can close this as a dup?
Confirmed upgrading Pex fixes the issue.

Before we close, I would ask: Is https://github.com/pantsbuild/pants/commit/e8d387ba6b4d4502e3b6db5ae68ffe7beeeb10a7 a significant enough change to justify bumping the [minimum](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/util_rules/pex_cli.py#L40) supported Pex version to v2.1.129? I expect Color is not unique in having the "ambient" Pex interpreter be the same as the interpreter targeted by user code. After upgrading to Pants 2.16 the ambient interpreter will silently switch to 3.9, potentially causing more head-scratchers like this one.
Good point, want to send a PR for that bump?

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,7 +37,7 @@ class PexCli(TemplatedExternalTool):
 
     default_version = "v2.1.131"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
-    version_constraints = ">=2.1.124,<3.0"
+    version_constraints = ">=2.1.129,<3.0"
 
     @classproperty
     def default_known_versions(cls):


Test patch: