Repository: pex-tool/pex

Issue:
pex creation fails under pants w/ `No such file or directory` error
```
23:06:05.36 [INFO] Starting: Resolving 3rdparty/python/constraints.txt
23:06:05.38 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/base/exception_sink.py:313: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),
23:06:05.38 [ERROR] 1 Exception encountered:
Engine traceback:
  in select
  in `typecheck` goal
  in Typecheck using MyPy
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building mypy.pex with 1 requirement: mypy==0.782' failed with exit code 1.
stdout:
stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 256, in run
    engine_result = self._run_v2()
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 168, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 185, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 125, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 527, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
Engine traceback:
  in select
  in `typecheck` goal
  in Typecheck using MyPy
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building mypy.pex with 1 requirement: mypy==0.782' failed with exit code 1.
stdout:
stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'

```

seeing this randomly from time to time in different CI runs.


Discussion:
Pants now invokes multiple instances of mypy concurrently: it's possible that this is related. But the contents of the `__pycache__` should be concurrency safe, afaik.
Looking very closely at the last line, I think this may be caused by the `sys.executable` value that lies issue on OSX framework builds (#1009 fixed by #1049) Since what pex thinks is a Python 3.8 interpreter is actually launching a Python 3.7 interpreter (the file not found is a 3.7 bytecode file `Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'`.

I can't provide a causal chain reason yet though so I'll leave this open to work on a bit harder.
this happens in CI (circleci), not OSX.
Aha, ok - almost certainly something else then. I'll take a look.
Notes:

1. If this is a straight-up concurrency problem, the relevant 3.8 code is here:
https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L120-L139
And the relevant 3.7 code is here and the same:
https://github.com/python/cpython/blob/4e02981de0952f54bf87967f8e10d169d6946b40/Lib/importlib/_bootstrap_external.py#L105-L124
As commented the code is best effort. Two independent processes could have the same object id for the string representing the path of the final pyc file and could thus race. This seems really unlikely though and I'm not sure how we'd fix this without turning off bytecode compilation.

2. This may not be a straight-up concurrency issue since the interpreter (3.8) vs bytecode path (37) indicates something unexpected.
Narrowing down a bit more, this error involves running the Pex PEX which is the one released by the Pex proejct and marked as `--unzip`. This means for the corresponding points above:
1. Since the PEX bootstrap is doing the unzipping into `PEX_ROOT/unzipped_pexes` using `atomic_directory` which is UUID4 best-effort it could also eagerly bytecode compile all files while under that lock to circumvent the much weaker best effort of CPython which just relies on object id - aka memory address - uniqueness.
2. This could explain the interpreter mismatch if we're running the Pex PEX with 3.8 but it then re-execs itself with 3.7. That needs more investigation though since the Pex PEX interpreter constraints are `">=2.7,<3.9,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"` which allows for 3.8 and since Pex supposedly always uses the current interpreter instead of re-execing if the current interpreter matches constraints.
Alright, explanation in hand. The issue here is `AtomicDirectory`. That code is robust but should only be used when the contents of a directory you want to create atomically are immutable. In the case of `--unzip` (`~/.pex/unzipped_pexes/...`) and `--not-zip-safe` (`~/.pex/code/...`) and the unpacked wheel cache (`~/.pex/installed_wheels/...`) where the directory is populated with python code that will be executed later, the directory contents is appended to later by python bytecode compilation. Since `AtomicDirectory` has at least once semantics, in a race involving more than one process filling the directory atomically simultaneously we can have:
1. 1st process creates the directory.
2. Another process executes against that directory and creates a temporary bytecode file but does not yet rename it: https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L126-L132
3. 2nd process (re) creates the directory, losing the temprary bytecode file from 2.
4. The process in 2 finds the temporary bytecode file it wrote is missing when attempting to atomically rename it: https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L133

For atomic mutable directory creation we'll need an interprocess lock. Another approach would be to continue with `AtomicDirectory` but append an identifier to the path unique to the bytecode compilation, say `cpython-37`. This has the disadvantage though of keeping a copy of source code for each unique interpreter run against it.
Do either of these solutions need to be applied unconditionally, or we could try to detect when it's unsafe and only use these fixes then? 
> Do either of these solutions need to be applied unconditionally, or we could try to detect when it's unsafe and only use these fixes then?

They need to be applied unconditionally since detecting when its safe amounts to detecting if there are other similar processes racing which amounts to either implementing an interprocess lock (the proposed solution) or else eliminating the need for a lock (the alternative solution).
yay! thanks for fixing @jsirois !
@jsirois need to reopen this... Since I am seeing this with the latest release.
```

Engine traceback:
  in select
  in pants.core.goals.typecheck.typecheck
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Resolving 3rdparty/python/constraints.txt' failed with exit code 1.
stdout:

stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/d8bfff1518d5d211d4ec95cf18305ff3939eef68/.deps/pex-2.1.18-py2.py3-none-any.whl/pex/__pycache__/platforms.cpython-37.pyc.139727263750640'


Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 281, in run
    engine_result = self._run_v2()
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 193, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 210, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 126, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 528, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

```
Ok. Assuming Linux posix APIs are not broken, which is a very safe bet, the observation of temporary pyc files created by Python can only be via `os.walk`. Pex has at least one of those.

Resolving patch:
diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -5,6 +5,7 @@
 
 import contextlib
 import os
+import re
 import sys
 import tempfile
 from hashlib import sha1
@@ -133,20 +134,25 @@ def _compute_hash(cls, names, stream_factory):
         return digest.hexdigest()
 
     @classmethod
-    def _iter_files(cls, directory):
+    def _iter_non_pyc_files(cls, directory):
         # type: (str) -> Iterator[str]
         normpath = os.path.realpath(os.path.normpath(directory))
-        for root, _, files in os.walk(normpath):
+        for root, dirs, files in os.walk(normpath):
+            dirs[:] = [d for d in dirs if d != "__pycache__"]
             for f in files:
-                yield os.path.relpath(os.path.join(root, f), normpath)
+                # For Python 2.7, `.pyc` files are compiled as siblings to `.py` files (there is no
+                # __pycache__ dir. We rely on the fact that the temporary files created by CPython
+                # have object id (integer) suffixes to avoid picking up either finished `.pyc` files
+                # or files where Python bytecode compilation is in-flight; i.e.:
+                # `.pyc.0123456789`-style files.
+                if not re.search(r"\.pyc(?:\.[0-9]+)?$", f):
+                    yield os.path.relpath(os.path.join(root, f), normpath)
 
     @classmethod
     def pex_hash(cls, d):
         # type: (str) -> str
-        """Return a reproducible hash of the contents of a directory."""
-        names = sorted(
-            f for f in cls._iter_files(d) if not (f.endswith(".pyc") or f.startswith("."))
-        )
+        """Return a reproducible hash of the contents of a loose PEX; excluding all `.pyc` files."""
+        names = sorted(f for f in cls._iter_non_pyc_files(d) if not f.startswith("."))
 
         def stream_factory(name):
             # type: (str) -> IO
@@ -157,8 +163,8 @@ def stream_factory(name):
     @classmethod
     def dir_hash(cls, d):
         # type: (str) -> str
-        """Return a reproducible hash of the contents of a directory."""
-        names = sorted(f for f in cls._iter_files(d) if not f.endswith(".pyc"))
+        """Return a reproducible hash of the contents of a directory; excluding all `.pyc` files."""
+        names = sorted(cls._iter_non_pyc_files(d))
 
         def stream_factory(name):
             # type: (str) -> IO


Test patch:
diff --git a/tests/test_util.py b/tests/test_util.py
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -6,7 +6,7 @@
 from hashlib import sha1
 from textwrap import dedent
 
-from pex.common import safe_mkdir, temporary_dir
+from pex.common import safe_mkdir, safe_open, temporary_dir, touch
 from pex.compatibility import nested, to_bytes
 from pex.pex import PEX
 from pex.pex_builder import PEXBuilder
@@ -74,14 +74,38 @@ def test_hash():
         assert hash_output == empty_hash.hexdigest()
 
 
-CONTENT = {
-    "__main__.py": 200,
-    ".deps/morfgorf": 10000,
-    "twitter/__init__.py": 0,
-    "twitter/common/python/foo.py": 4000,
-    "twitter/common/python/bar.py": 8000,
-    "twitter/common/python/bar.pyc": 6000,
-}
+def test_dir_hash():
+    # type: () -> None
+    with temporary_dir() as tmp_dir:
+        safe_mkdir(os.path.join(tmp_dir, "a", "b"))
+        with safe_open(os.path.join(tmp_dir, "c", "d", "e.py"), "w") as fp:
+            fp.write("contents1")
+        with safe_open(os.path.join(tmp_dir, "f.py"), "w") as fp:
+            fp.write("contents2")
+        hash1 = CacheHelper.dir_hash(tmp_dir)
+
+        os.rename(os.path.join(tmp_dir, "c"), os.path.join(tmp_dir, "c-renamed"))
+        assert hash1 != CacheHelper.dir_hash(tmp_dir)
+
+        os.rename(os.path.join(tmp_dir, "c-renamed"), os.path.join(tmp_dir, "c"))
+        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+
+        touch(os.path.join(tmp_dir, "c", "d", "e.pyc"))
+        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        touch(os.path.join(tmp_dir, "c", "d", "e.pyc.123456789"))
+        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+
+        pycache_dir = os.path.join(tmp_dir, "__pycache__")
+        safe_mkdir(pycache_dir)
+        touch(os.path.join(pycache_dir, "f.pyc"))
+        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        touch(os.path.join(pycache_dir, "f.pyc.123456789"))
+        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+
+        touch(os.path.join(pycache_dir, "f.py"))
+        assert hash1 == CacheHelper.dir_hash(
+            tmp_dir
+        ), "All content under __pycache__ directories should be ignored."
 
 
 try: