Repository: pantsbuild/pants

Issue:
Binary location and interpreter selection issues
Two issues:
1. ~~the `Find binary` process that is used to locate a bootstrap interpreter limits the PATH to the search path for the tool that it is looking for. For search paths which do not include `bash` (which will happen in the case described on #9760), bash will not be found, causing the script to fail with `env: bash: No such file or directory`~~ - fixed in #10858 
2. The `Find interpreter for constraints` process that is used to test interpreters is not marked fallible `FallibleProcessResult`, and so poisoned interpreters can kill it rather than just causing the poisoned interpreters to be skipped ([full error](https://gist.github.com/stuhood/f572b2b10d56ee882815b9b5ceeb742f)).
```
pex.executor.ExecutableNotFound: caught FileNotFoundError(2, "No such file or directory: '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/bin/python3'") while trying to execute 
```


Discussion:
Posted a fix for the first issue here: #10858. It should be sufficient to allow someone to work around the second issue by setting:
```
interpreter_search_path=["<PYENV>"]
```

----

I might have misdiagnosed the second issue though: `Find interpreter for constraints` is relying on PEX successfully filtering interpreters before using them, and so we should not try to recover from PEX failing to do that. And whichever code PEX is using to eliminate bad interpreters is not eliminating the problematic interpreter in this case. See the full error for the stack: https://gist.github.com/stuhood/8443ad481a65dd23b83020847ef91712

So to fix the second portion, we'd either need to make PEX resistant to this type of interpreter issue, or have pants take over more of the interpreter discovery/selection process.
> So to fix the second portion, we'd either need to make PEX resistant to this type of interpreter issue, or have pants take over more of the interpreter discovery/selection process.

Does this seem like an issue that affects Pex usage in general? Or is it specifically because of some weirdness around how pants invokes it here?
> > So to fix the second portion, we'd either need to make PEX resistant to this type of interpreter issue, or have pants take over more of the interpreter discovery/selection process.
> 
> Does this seem like an issue that affects Pex usage in general? Or is it specifically because of some weirdness around how pants invokes it here?

Unclear... it's possible that Pants is explicitly passing an interpreter that PEX would not otherwise use. But it's also possible that PEX would fail in this user's environment as well. Can follow up with the user to ask them to try using PEX directly I suppose, but will hold off on doing that unless it turns out that the stack is not sufficient to fix the issue.
> it's possible that Pants is explicitly passing an interpreter that PEX would not otherwise use. 

I don't think this is possible. We use Pex to do the interpreter selection still:

https://github.com/pantsbuild/pants/blob/5f46286117bbabebca4edbae7a2ea030f2e2070f/src/python/pants/backend/python/util_rules/pex.py#L395-L440
Yeah, this has to be a bug in the interpreter cache somehow, I'm just not seeing it yet. OSX does something very weird here and I'm floundering a bit not having the OS in hand to repro / debug with. I'll give this one more hard throw logic out the window code scan a bit later.
~~Noting that even with fixing the interpreter cache to handle moved files (and maybe handle files since made unexecutable, etc ...) we'll need to switch to a FallibleProcessResult. Without actually re-running the interpreter identification process and thus defeating the point of the interpreter cache in the 1st place, we'll never be able to detect - for example - that the interpreter has had a shared library it needs nuked, etc. IE: There is more to check than the single python binary path Pex knows about to ensure the cached interpreter package is unmodified in full from when it was cached.~~

Since we use an UncacheableProcess to discover interpreters subject to interpreter constraints there is no need to mark the Process fallible. If there is something correctable on the filesystem, that can be corrected and pantsd re-started.
Alright, I've found an OSX-only mechanism to trip up Pex via this Python ecosystem bug https://bugs.python.org/issue22490#msg283859 which we are vulnerable to. More here: https://github.com/python/cpython/pull/9516

Re-opened an old unreproducible Pex issue that was probably dues to this same issue: https://github.com/pantsbuild/pex/issues/1009

I'll implement the fix over there.
I'll close this upon upgrade to Pex 2.1.17 https://github.com/pantsbuild/pex/issues/1013.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -32,7 +32,7 @@ class PexBinary(ExternalTool):
 
     options_scope = "download-pex-bin"
     name = "pex"
-    default_version = "v2.1.16"
+    default_version = "v2.1.17"
 
     @classproperty
     def default_known_versions(cls):
@@ -41,8 +41,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "38712847654254088a23394728f9a5fb93c6c83631300e7ab427ec780a88f653",
-                    "2662638",
+                    "df836ddd102d8fc9fb390cc4c7fc478caf5afa1600cbdeb7753b185e27a6a8b0",
+                    "2673293",
                 )
             )
             for plat in ["darwin", "linux"]


Test patch: