Repository: pex-tool/pex

Issue:
Pex use of `pip wheel` to build sdists into wheels can be non-atomic.
Pex re-routes the Pip cache to the PEX_ROOT. This leads to two Pip-controlled directories there, `http/` and `wheels/`. The latter is a wheel cache where built wheels are moved to:
https://github.com/pantsbuild/pex/blob/ddd3bd807856788d32037a539b5a36f28198dbf9/pex/vendor/_vendored/pip/pip/_internal/wheel_builder.py#L245-L284

Since that move will be non-atomic when the wheel is built in a directory residing on a different filesystem from the Pip `wheels/` cache, it can lead to errors like so under concurrent use of Pex:
```
  ERROR: Command errored out with exit status 1:
   command: /home/jsirois/.pex/venvs/s/99506507/venv/bin/python3.11 /home/jsirois/.pex/venvs/s/99506507/venv/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bcsxoi58/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (21 lines):
  Collecting setuptools>=40.8.0
    Using cached setuptools-62.3.2.tar.gz (2.6 MB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting wheel
    Using cached wheel-0.37.1.tar.gz (66 kB)
  Skipping wheel build for wheel, due to binaries being disabled for it.
  Building wheels for collected packages: setuptools
    Building wheel for setuptools (PEP 517): started
    Building wheel for setuptools (PEP 517): finished with status 'done'
    Created wheel for setuptools: filename=setuptools-62.3.2-py3-none-any.whl size=1225929 sha256=bb23f2e58e6137a8bdfc42b97cc8b77e3b0c11df8387ac7ab1c2745015e14964
    Stored in directory: /home/jsirois/.cache/pip/wheels/70/a3/cf/340de2f5eb2aa7abaf1059081a1f19f96bf7e1f4bb8644c0cb
    WARNING: Built wheel for setuptools is invalid: setuptools has an invalid wheel, could not read 'setuptools-62.3.2.dist-info/WHEEL' file: BadZipFile('Truncated file header')
  Failed to build setuptools
  ERROR: Could not build wheels for setuptools which use PEP 517 and cannot be installed directly
  WARNING: You are using pip version 20.3.4; however, version 22.1.1 is available.
  You should consider upgrading via the '/home/jsirois/.pex/venvs/s/99506507/venv/bin/python3.11 -m pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/jsirois/.pex/venvs/s/99506507/venv/bin/python3.11 /home/jsirois/.pex/venvs/s/99506507/venv/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-bcsxoi58/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
pid 76513 -> /home/jsirois/.pex/venvs/97931ae45d86d091c7ecbdc5f93be710c50be86e/4e65055febaf795bcb2334d120ef464ddf6b4963/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex download --dest /tmp/tmpwaxtu38q/home.jsirois..pyenv.versions.3.11.0b1.bin.python3.11 --no-binary :all: psutil --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None
```

Pex should ensure that this can never happen. It already guards similar issues in its own codebase using `atomic_directory` and all that work is effectively made moot by this non-atomicity.


Discussion:

Resolving patch:
diff --git a/pex/pip/tool.py b/pex/pip/tool.py
--- a/pex/pip/tool.py
+++ b/pex/pip/tool.py
@@ -15,7 +15,7 @@
 
 from pex import dist_metadata, targets, third_party
 from pex.auth import PasswordEntry
-from pex.common import atomic_directory, safe_mkdtemp
+from pex.common import atomic_directory, safe_mkdir, safe_mkdtemp
 from pex.compatibility import unquote, urlparse
 from pex.dist_metadata import ProjectNameAndVersion, Requirement
 from pex.interpreter import PythonInterpreter
@@ -690,6 +690,15 @@ def _spawn_pip_isolated(
         if package_index_configuration:
             extra_env.update(package_index_configuration.env)
 
+        # Ensure the pip cache (`http/` and `wheels/` dirs) is housed in the same partition as the
+        # temporary directories it creates. This is needed to ensure atomic filesystem operations
+        # since Pip relies upon `shutil.move` which is only atomic when `os.rename` can be used.
+        # See https://github.com/pantsbuild/pex/issues/1776 for an example of the issues non-atomic
+        # moves lead to in the `pip wheel` case.
+        pip_tmpdir = os.path.join(cache or ENV.PEX_ROOT, "tmp")
+        safe_mkdir(pip_tmpdir)
+        extra_env.update(TMPDIR=pip_tmpdir)
+
         with ENV.strip().patch(
             PEX_ROOT=cache or ENV.PEX_ROOT,
             PEX_VERBOSE=str(ENV.PEX_VERBOSE),


Test patch: