Repository: pex-tool/pex

Issue:
Interpreter discovery and pyenv don't interact well
At pex runtime, interpreter discovery looks on the PATH for any binary with a name that looks like it might be a python interpreter, and executes it to get its version (and verify that it is in fact a python interpreter). 

Unfortunately, pyenv can create a large number of such files in `~/.pyenv/shims`. E.g.,

```
> ls /Users/benjyw/.pyenv/shims/ | grep -E "python\d?(.\d)?m?$"
python
python2
python2.7
python3
python3.5
python3.5m
python3.6
python3.6m
python3.7
python3.7m
>
```
Note that these aren't symlinks, but wrapper scripts, so pex's realpath deduping doesn't help.

So, pex will execute every one of these, plus whatever other python interpreters it finds, which adds up to non trivial overhead (~1 second in the above example) on every run of the pex. Note that not all of these are actually valid python interpreters - some are shim stubs that fail on purpose.  


Discussion:
I don't have an obvious solution here. Some heuristics to discuss:
1. Checking the interpreter pex is running under first. It may be likely to be the right one?
2. Stopping as soon as we find a valid interpreter, instead of first executing all of them and then comparing. This won't help in the worst case but probably will the average case.  This will effectively implement 1.
3. An on-disk interpreter cache, in some well-known location, to speed up subsequent runs.  This won't help much in container scenarios.
4. Be smarter about matching names to versions. E.g., if the user specified Python>=3.7, we can eliminate many versions just by naming convention.
It would definitely be good to implement 1 directly, this is already nodded towards in some of the machinations in `maybe_reexec_pex`, but the treatment does not include the cases where PEX_PYTHON or PEX_PYTHON_PATH are set:
https://github.com/pantsbuild/pex/blob/e5317cbc94a49d5129144806ce4117ccd926fb6a/pex/pex_bootstrapper.py#L165-L168
Presumably this catches the majority of cases - the current interpreter is fine.

Instead of 2 a multiprocessing approach could probe all interpreters in parallel. Perhaps too complex.

Item 4 makes sense, but I think 1 and maybe 2 might do it.


I took a whack at point 1 in #783.
#835 released as part of Pex 2.1.0 only made interpreter identification slower per interpreter; so finishing out a few of the other mitigations in this issue has come to the fore. Parallelization is now pretty straightforward given the groundwork in `pex.jobs` so I'll knock that out first and then likely an interpreter cache.
Items 1-3 above are now implemented. Item 4 is problematic and very much less important now that paralellization and caching are in effect.

As a postscript, item 4 turns out to be particularly problematic due to pyenv shims. These, when on the `PATH`, have the behavior of re-directing `python*` to the "correct" python as configured by `pyenv (local|global)`. As such, executing the python3 shim "binary" can land you in a python2 interpreter, defeating any hueristic that does not also special case binaries residing inside the pyenv shims dir.

Resolving patch:
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -5,20 +5,25 @@
 
 from __future__ import absolute_import
 
+import hashlib
 import json
 import os
+import platform
 import re
 import subprocess
 import sys
 from textwrap import dedent
 
 from pex import third_party
+from pex.common import safe_rmtree
 from pex.compatibility import string
 from pex.executor import Executor
 from pex.jobs import Job, SpawnedJob, execute_parallel
 from pex.third_party.packaging import markers, tags
 from pex.third_party.pkg_resources import Distribution, Requirement
 from pex.tracer import TRACER
+from pex.util import CacheHelper
+from pex.variables import ENV
 
 
 class PythonIdentity(object):
@@ -64,12 +69,8 @@ def decode(cls, encoded):
     supported_tags = values.pop('supported_tags')
 
     def iter_tags():
-      for supported_tag in supported_tags:
-        yield tags.Tag(
-          interpreter=supported_tag['interpreter'],
-          abi=supported_tag['abi'],
-          platform=supported_tag['platform']
-        )
+      for (interpreter, abi, platform) in supported_tags:
+        yield tags.Tag(interpreter=interpreter, abi=abi, platform=platform)
 
     return cls(supported_tags=iter_tags(), **values)
 
@@ -98,10 +99,7 @@ def encode(self):
       abi_tag=self._abi_tag,
       platform_tag=self._platform_tag,
       version=self._version,
-      supported_tags=[
-        dict(interpreter=tag.interpreter, abi=tag.abi, platform=tag.platform)
-        for tag in self._supported_tags
-      ],
+      supported_tags=[(tag.interpreter, tag.abi, tag.platform) for tag in self._supported_tags],
       env_markers=self._env_markers
     )
     return json.dumps(values)
@@ -290,39 +288,74 @@ def _execute(cls, binary, args=None, pythonpath=None, env=None, stdin_payload=No
     stdout, stderr = Executor.execute(cmd, stdin_payload=stdin_payload, env=env, **kwargs)
     return cmd, stdout, stderr
 
+  INTERP_INFO_FILE = 'INTERP-INFO'
+
   @classmethod
   def _spawn_from_binary_external(cls, binary):
-    pythonpath = third_party.expose(['pex'])
-    cmd, env = cls._create_isolated_cmd(
-      binary,
-      args=[
-        '-c',
-        dedent("""\
-        import sys
-        from pex.interpreter import PythonIdentity
-
-        sys.stdout.write(PythonIdentity.get().encode())
-        """)
-      ],
-      pythonpath=pythonpath
-    )
-    process = Executor.open_process(cmd, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    job = Job(command=cmd, process=process)
-
     def create_interpreter(stdout):
       identity = stdout.decode('utf-8').strip()
       if not identity:
         raise cls.IdentificationError('Could not establish identity of %s' % binary)
       return cls(binary, PythonIdentity.decode(identity))
 
-    return SpawnedJob.stdout(job, result_func=create_interpreter)
+    # Part of the PythonInterpreter data are environment markers that depend on the current OS
+    # release. That data can change when the OS is upgraded but (some of) the installed interpreters
+    # remain the same. As such, include the OS in the hash structure for cached interpreters.
+    os_digest = hashlib.sha1()
+    for os_identifier in platform.release(), platform.version():
+      os_digest.update(os_identifier.encode('utf-8'))
+    os_hash = os_digest.hexdigest()
+
+    interpreter_cache_dir = os.path.join(ENV.PEX_ROOT, 'interpreters')
+    os_cache_dir = os.path.join(interpreter_cache_dir, os_hash)
+    if os.path.isdir(interpreter_cache_dir) and not os.path.isdir(os_cache_dir):
+      with TRACER.timed('GCing interpreter cache from prior OS version'):
+        safe_rmtree(interpreter_cache_dir)
+
+    interpreter_hash = CacheHelper.hash(binary)
+    cache_dir = os.path.join(os_cache_dir, interpreter_hash)
+    cache_file = os.path.join(cache_dir, cls.INTERP_INFO_FILE)
+    if os.path.isfile(cache_file):
+      try:
+        with open(cache_file, 'rb') as fp:
+          return SpawnedJob.completed(create_interpreter(fp.read()))
+      except (IOError, OSError, cls.Error, PythonIdentity.Error):
+        safe_rmtree(cache_dir)
+        return cls._spawn_from_binary_external(binary)
+    else:
+      pythonpath = third_party.expose(['pex'])
+      cmd, env = cls._create_isolated_cmd(
+        binary,
+        args=[
+          '-c',
+          dedent("""\
+          import os
+          import sys
+
+          from pex.common import atomic_directory, safe_open
+          from pex.interpreter import PythonIdentity
+
+
+          encoded_identity = PythonIdentity.get().encode()
+          sys.stdout.write(encoded_identity)
+          with atomic_directory({cache_dir!r}) as cache_dir:
+            if cache_dir:
+              with safe_open(os.path.join(cache_dir, {info_file!r}), 'w') as fp:
+                fp.write(encoded_identity)
+          """.format(cache_dir=cache_dir, info_file=cls.INTERP_INFO_FILE))
+        ],
+        pythonpath=pythonpath
+      )
+      process = Executor.open_process(cmd, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+      job = Job(command=cmd, process=process)
+      return SpawnedJob.stdout(job, result_func=create_interpreter)
 
   @classmethod
   def _expand_path(cls, path):
     if os.path.isfile(path):
       return [path]
     elif os.path.isdir(path):
-      return [os.path.join(path, fn) for fn in os.listdir(path)]
+      return sorted(os.path.join(path, fn) for fn in os.listdir(path))
     return []
 
   @classmethod


Test patch: