Repository: pex-tool/pex

Issue:
The Pex CLI can fail to find `pex.bin`.
As originally reported in https://github.com/pantsbuild/pants/issues/11211, if the timing is right and the set of available interpreters that conform to Pex's interpreter constraints is right it appears that the interpreter identification / selection that runs in parallel in the PEX bootstrap phase can lead to Pex PEX failing to run.

Interpreter selection is started here in `maybe_reexec_pex`:
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L471-L474

If the current interpreter:
1. Conforms to Pex interpreter constraints (i.e.: Has a version in ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.10").
2. Is not the 1st interpreter on the PEX_PYTHON_PATH.

Then this code will short circuit some time _after_ the 1st interpreter is yielded (i.e.: on the second interpreter or greater):
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L158-L175

Since the short circuit happens on the second or later interpreter, this execute_parallel code is engaged by the lazy iterator and it spawns a single thread to manage concurrent shell-outs:
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L103-L108
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/interpreter.py#L776-L796
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/jobs.py#L384-L404

That iterator neither auto-cancels nor is cancelled by this Pex bootstrap code. The upshot, is when the code continues on from `maybe_reexec_pex` to `PEX.execute()` there can still be background work going on identifying candidate interpreters that will never be picked:
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/pex_bootstrapper.py#L471-L474

This is fine in and of itself, but it appears it can cause an import of `pex` to interleave between the unimports here and the `sys.path` re-arrangement below:
https://github.com/pantsbuild/pex/blob/1111cae72e4eea2c1681745925e50c38ec7cb3e8/pex/bootstrap.py#L36-L52

This causes `pex` to be re-imported from the `.bootstrap/` instead of from the `pex` distribution for the Pex PEX, which is disastrous since the `.bootstrap/` only contains enough of `pex` to bootstrap a PEX file and not the whole CLI. The result is something like:
```
Traceback (most recent call last):
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 446, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 378, in _wrap_coverage
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 409, in _wrap_profiling
    return runner(*args)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 508, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 610, in execute_entry
    return runner(entry_point)
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/pex.py", line 625, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/ziyadedher/.cache/pants/named_caches/pex_root/unzipped_pexes/42d4f75d046397e5f014ec8afd1a39b3ba8787c6/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'pex.bin'
```

Note that in the chain of events described above, the critical event - the re-import of `pex`from a background thread - is neither observed when instrumenting imports with a `sys.path_importer_hook` (only a single import of `encodings.ascii` by `encodings` via the `__import__` function is observed) nor does it appear this should ever happen by inspection. In other words it appears all pex imports happen _before_ the thread is ever spawned.


Discussion:

Resolving patch:
diff --git a/pex/bootstrap.py b/pex/bootstrap.py
--- a/pex/bootstrap.py
+++ b/pex/bootstrap.py
@@ -41,14 +41,18 @@ def demote(self):
         """
         import sys  # Grab a hold of `sys` early since we'll be un-importing our module in this process.
 
+        # N.B.: We mutate the sys.path before un-importing modules so that any re-imports triggered
+        # by concurrent code will pull from the desired sys.path ordering.
+        # See here for how this situation might arise: https://github.com/pantsbuild/pex/issues/1272
+
+        sys.path[:] = [path for path in sys.path if os.path.realpath(path) != self._realpath]
+        sys.path.append(self._sys_path_entry)
+
         unimported_modules = []
         for name, module in reversed(sorted(sys.modules.items())):
             if self.imported_from_bootstrap(module):
                 unimported_modules.append(sys.modules.pop(name))
 
-        sys.path[:] = [path for path in sys.path if os.path.realpath(path) != self._realpath]
-        sys.path.append(self._sys_path_entry)
-
         return unimported_modules
 
     def imported_from_bootstrap(self, module):


Test patch: