Repository: pex-tool/pex

Issue:
No such file for interpreter's binary name
With `2.1.12`, @asherf encountered a case where `pex` detected an interpreter with a bad binary name:
```
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Users/asher/.pyenv/versions/3.8.2/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/bin/python3.7
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/bin/python3.7
```

The file `/Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7` does not exist (`python3.8` does, as might be expected).

This bug was reproduced in the context of a long lived `pex_root`, which @asherf can provide if need be.


Discussion:
This is hard to understand since FWICT all interpreter resolution funnels through here:
https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/interpreter.py#L429-L433

IOW the path `/Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7` has been normalized (via `os.realpath`) so we've read through all symlinks to a hard link and we've not raised `InterpreterNotFound`.

@asherf is there any chance you can re-run against the saved `PEX_ROOT` with `-vvvvvvvvv` (9 `v`s)?
For example:

With Pex 2.1.12:
```
jsirois@gill ~/dev/pantsbuild/pex (master) $ git co v2.1.12 
HEAD is now at 2ed8ed9 Prepare the 2.1.12 release.
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex --version
__main__.py 2.1.12
```

Using a fresh interpreter cache for sanity sake:
```
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Hashing pex: 26.2ms
pex: Isolating pex: 51.3ms                                                                                    
pex:   Extracting pex to /tmp/pex-issues-1009/isolated/6bd62cbe8c9f1ae82b35f976967538908799cbd6: 51.1ms
pex: Executing: PYTHONPATH=/tmp/pex-issues-1009/isolated/6bd62cbe8c9f1ae82b35f976967538908799cbd6 /opt/pypy/bin/pypy -s -c import os
import sys

from pex.common import atomic_directory, safe_open
from pex.interpreter import PythonIdentity


encoded_identity = PythonIdentity.get().encode()
sys.stdout.write(encoded_identity)
with atomic_directory('/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/917a091d50cf40d0d7f4a0de5a935db28edbaf59/opt.pypy.bin.pypy') as cache_dir:
  if cache_dir:
    with safe_open(os.path.join(cache_dir, 'INTERP-INFO'), 'w') as fp:
      fp.write(encoded_identity)

...
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1 | head -6
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.6
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])
```

So here's a similar base case. That leaves these cached interpreters from the PATH scan:
```
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ find ~/tmp/pex-issues-1009/interpreters -type f | while read ii; do echo "$ii:" && jq .binary $ii && echo; done
/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/718a50d6552a2112904130b42af2c59767214f73/usr.bin.python3.7/INTERP-INFO:
"/usr/bin/python3.7"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/718a50d6552a2112904130b42af2c59767214f73/usr.bin.python3.7m/INTERP-INFO:
"/usr/bin/python3.7m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/917a091d50cf40d0d7f4a0de5a935db28edbaf59/opt.pypy.bin.pypy/INTERP-INFO:
"/opt/pypy/bin/pypy"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/c3d7ea9d48d046c47d558adc82ead9af0e885bcb/usr.bin.python3.6m/INTERP-INFO:
"/usr/bin/python3.6m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/c3d7ea9d48d046c47d558adc82ead9af0e885bcb/usr.bin.python3.6/INTERP-INFO:
"/usr/bin/python3.6"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/3ffacb7e0e1938412e210a942e1036ec64ac9355/usr.bin.python3.4/INTERP-INFO:
"/usr/bin/python3.4"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/3ffacb7e0e1938412e210a942e1036ec64ac9355/usr.bin.python3.4m/INTERP-INFO:
"/usr/bin/python3.4m"

/home/jsirois/tmp/pex-issues-1009/interpreters/487fcfe7682222815a3ccd74c0e2a81f0c51e710/2e8e8915ef45f0cf40098ab616d817fb3db5ebc1/usr.bin.python2.7/INTERP-INFO:
"/usr/bin/python2.7"
```

Now on to simulating a once present but now gone interpreter:
``` 
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ sudo mv /usr/bin/python3.6 /usr/bin/disable-python3.6
[sudo] password for jsirois: 
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.6m
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ /usr/bin/python3.6m --version
Python 3.6.10
```

And really removing the 3.6 by removing its other hard link:
```
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ sudo mv /usr/bin/python3.6m /usr/bin/disable-python3.6m
jsirois@gill ~/dev/pantsbuild/pex ((v2.1.12)) $ python -mpex -vvv --pex-root=/tmp/pex-issues-1009 --interpreter-constraint='CPython>=3.6' -- -c '' 2>&1 | head -6
pex: Building pex
pex: Building pex :: Resolving interpreters
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.8
pex: Constraints on interpreters: ['CPython>=3.6'], Matching Interpreter: /usr/bin/python3.7
pex: Building pex :: Resolving distributions ([])
pex: Building pex: 24.7ms
```

So it looks like the existence check is working in this mocked up simulation.
@asherf : If you're still able to repro this, working with John on it would be helpful.
doesn't repo anymore since I reinstalled my homebrew env.

I'm going to close this since its both not reproduceable and the exercise above seems to indicate interpreter caching is robust to this sort of thing.
> This is hard to understand since FWICT all interpreter resolution funnels through here:
> https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/interpreter.py#L429-L433
> 
> IOW the path `/Users/asher/.toolchain/homebrew/Cellar/python@3.8/3.8.4/bin/python3.7` has been normalized (via `os.realpath`) so we've read through all symlinks to a hard link and we've not raised `InterpreterNotFound`.
> 
> @asherf is there any chance you can re-run against the saved `PEX_ROOT` with `-vvvvvvvvv` (9 `v`s)?

OK, we've had another instance of this come up over in https://github.com/pantsbuild/pants/issues/10855
I've re-inspected the code paths, and under the assumption sys.executable actually == the path of the currently running python interpreter, there seems to be no way this can happen. However:
+ https://bugs.python.org/issue22490#msg283859
+ https://github.com/python/cpython/pull/9516

I'll remove reliance on this assumption.


Resolving patch:
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -68,11 +68,22 @@ class UnknownRequirement(Error):
     }
 
     @classmethod
-    def get(cls):
+    def get(cls, binary=None):
+        # N.B.: We should not need to look past `sys.executable` to learn the current interpreter's
+        # executable path, but on OSX there has been a bug where the `sys.executable` reported is
+        # _not_ the path of the current interpreter executable:
+        # https://bugs.python.org/issue22490#msg283859
+        if binary and binary != sys.executable:
+            TRACER.log(
+                "Identifying interpreter found at {} which reports an incorrect sys.executable of "
+                "{}.".format(binary, sys.executable),
+                V=9,
+            )
+
         supported_tags = tuple(tags.sys_tags())
         preferred_tag = supported_tags[0]
         return cls(
-            binary=sys.executable,
+            binary=binary or sys.executable,
             python_tag=preferred_tag.interpreter,
             abi_tag=preferred_tag.abi,
             platform_tag=preferred_tag.platform,
@@ -400,11 +411,20 @@ def _execute(cls, binary, args=None, pythonpath=None, env=None, stdin_payload=No
 
     @classmethod
     def _spawn_from_binary_external(cls, binary):
-        def create_interpreter(stdout):
+        def create_interpreter(stdout, check_binary=False):
             identity = stdout.decode("utf-8").strip()
             if not identity:
-                raise cls.IdentificationError("Could not establish identity of %s" % binary)
-            return cls(PythonIdentity.decode(identity))
+                raise cls.IdentificationError("Could not establish identity of {}.".format(binary))
+            interpreter = cls(PythonIdentity.decode(identity))
+            # We should not need to check this since binary == interpreter.binary should always be
+            # true, but historically this could be untrue as noted in `PythonIdentity.get`.
+            if check_binary and not os.path.exists(interpreter.binary):
+                raise cls.InterpreterNotFound(
+                    "Cached interpreter for {} reports a binary of {}, which could not be found".format(
+                        binary, interpreter.binary
+                    )
+                )
+            return interpreter
 
         # Part of the PythonInterpreter data are environment markers that depend on the current OS
         # release. That data can change when the OS is upgraded but (some of) the installed interpreters
@@ -435,7 +455,7 @@ def create_interpreter(stdout):
         if os.path.isfile(cache_file):
             try:
                 with open(cache_file, "rb") as fp:
-                    return SpawnedJob.completed(create_interpreter(fp.read()))
+                    return SpawnedJob.completed(create_interpreter(fp.read(), check_binary=True))
             except (IOError, OSError, cls.Error, PythonIdentity.Error):
                 safe_rmtree(cache_dir)
                 return cls._spawn_from_binary_external(binary)
@@ -454,14 +474,14 @@ def create_interpreter(stdout):
                         from pex.interpreter import PythonIdentity
 
 
-                        encoded_identity = PythonIdentity.get().encode()
+                        encoded_identity = PythonIdentity.get(binary={binary!r}).encode()
                         sys.stdout.write(encoded_identity)
                         with atomic_directory({cache_dir!r}) as cache_dir:
                             if cache_dir:
                                 with safe_open(os.path.join(cache_dir, {info_file!r}), 'w') as fp:
                                     fp.write(encoded_identity)
                         """.format(
-                            cache_dir=cache_dir, info_file=cls.INTERP_INFO_FILE
+                            binary=binary, cache_dir=cache_dir, info_file=cls.INTERP_INFO_FILE
                         )
                     ),
                 ],


Test patch: