Repository: pantsbuild/pants

Issue:
CreatePex attempts to write to the default pex cache (~/.pex).
Seen in a remote build execution platform that only presents a read-only filesystem except for the EPR chroot:
```
$ ./pants --no-v1 --v2-ui --pants-config-files=pants.local-bf.toml --remote-execution-extra-platform-properties=execution-policy=debug test src/python/toolchain/satresolver:tests
20:38:20 [WARN] Error loading access token: AuthError('Failed to load auth token (no default file or environment variable). Run `./tc auth-acquire` to set up authentication.')
20:38:20 [INFO] Starting tests: src/python/toolchain/satresolver:tests
20:38:23 [WARN] /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.27.0.dev0_py38/lib/python3.8/site-packages/pants/base/exception_sink.py:431: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),
ERROR: process 'Create a requirements PEX: ansicolors==1.1.8, coloredlogs, pex==2.1.6, pytest>=5.4.1, termcolor' failed with exit code 129.
stdout:
stderr:
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Enabling debug
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Called in PWD=/worker/operations/default_memory_instance/operations/0843afa8-7036-4f43-a954-791a37846457 with: network=true container-image=283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0 debug=true python ./pex --disable-cache --output-file requirements.pex --interpreter-constraint CPython>=3.8,<4 --not-zip-safe --manylinux manylinux2014 --constraints 3rdparty/python/constraints.txt --sources-directory=source_files ansicolors==1.1.8 coloredlogs pex==2.1.6 pytest>=5.4.1 termcolor
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	total 2564
drwxr-xr-x 3 worker worker    4096 Mar 20 20:38 3rdparty
-rwxr--r-- 3 worker worker 2614280 Mar 20 19:58 pex
drwxr-xr-x 2 worker worker    4096 Mar 20 20:38 source_files
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Performing runt.sh exec on image 283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Enabling network
2020-03-20T20:38:21+00:00 runt.sh DEBUG
	Wrapping process with waitnet: /usr/bin/waitnet -d /slirp4netns/run python ./pex --disable-cache --output-file requirements.pex --interpreter-constraint CPython>=3.8,<4 --not-zip-safe --manylinux manylinux2014 --constraints 3rdparty/python/constraints.txt --sources-directory=source_files ansicolors==1.1.8 coloredlogs pex==2.1.6 pytest>=5.4.1 termcolor
2020-03-20T20:38:21+00:00 runt.sh DEBUG
	Executing /worker/exec/283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0/283194185447_dkr_ecr_us_east_1_amazonaws_com_ci_2020_03_18_00_32_05_4255fb8f57f0_4c3fa4a1_d153_4694_9c85_1ee4cbd5f40f/config.json as uid=1000(worker) gid=1000(worker) groups=1000(worker)
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Called as /worker/bin/start-network.sh -d true -s /usr/bin/slirp4netns /worker/exec/283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0/283194185447_dkr_ecr_us_east_1_amazonaws_com_ci_2020_03_18_00_32_05_4255fb8f57f0_4c3fa4a1_d153_4694_9c85_1ee4cbd5f40f/slirp4netns/run
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Starting slirp4netns daemon for container process 452...
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Started slirp4netns daemon @ 465.
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Starting quit daemon for slirp4netns daemon @ 465...
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Started quit daemon @ 473.
waitnet DEBUG
	Received 1 on /slirp4netns/run/ready
waitnet DEBUG
	Executing python...
Traceback (most recent call last):
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 393, in execute
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 325, in _wrap_coverage
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 356, in _wrap_profiling
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 438, in _execute
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 535, in execute_entry
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 550, in execute_pkg_resources
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/bin/pex.py", line 661, in main
    pex_builder = build_pex(reqs, options)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/bin/pex.py", line 578, in build_pex
    resolveds = resolve_multi(requirements=reqs,
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 802, in resolve_multi
    return list(resolve_request.resolve_distributions(ignore_errors=ignore_errors))
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 497, in resolve_distributions
    for resolve_result in self._run_parallel(inputs=to_resolve,
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 366, in _run_parallel
    for result in execute_parallel(
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/jobs.py", line 219, in execute_parallel
    raise error
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/jobs.py", line 202, in spawn_jobs
    result = spawn_func(item)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 376, in _spawn_resolve
    download_job = get_pip().spawn_download_distributions(
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/pip.py", line 256, in get_pip
    _PIP = Pip.create()
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/pip.py", line 30, in create
    isolated_pip_environment = third_party.pkg_resources.Environment(search_path=pythonpath)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1016, in __init__
    self.scan(search_path)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1047, in scan
    for item in search_path:
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/third_party/__init__.py", line 442, in expose
    for path in VendorImporter.expose(dists, root=isolated()):
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/third_party/__init__.py", line 363, in isolated
    with atomic_directory(isolated_dir) as chroot:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/common.py", line 303, in atomic_directory
    safe_mkdir(atomic_dir.work_dir)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/common.py", line 185, in safe_mkdir
    os.makedirs(directory)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/toolchain/.pex'
waitnet DEBUG
	Sent quit on /slirp4netns/run/quit
(Use --print-exception-stacktrace to see more error details.)
```


Discussion:
It looks like newer Pex is what is broken here, #8335 via #8283 fixed this by always passing `--disable-cache` - which no longer appears to fully work. A workaround would be to set the PEX_ROOT like `HermeticPex` does.
OK - this is a problem unique to PEXed Pex. It, and only it, extracts itself to `$PEX_ROOT/isolated`, but all PEXes have `PEX_` env vars stripped before running, which leaves PEXed Pex using the default cache even though Pants tries to set that to `./pex_root`.
This will need to be solved over in Pex. Forked off this issue which Pants will block on via a Pex 2.1.7 release: https://github.com/pantsbuild/pex/issues/926

Resolving patch:
diff --git a/src/python/pants/backend/python/rules/download_pex_bin.py b/src/python/pants/backend/python/rules/download_pex_bin.py
--- a/src/python/pants/backend/python/rules/download_pex_bin.py
+++ b/src/python/pants/backend/python/rules/download_pex_bin.py
@@ -37,16 +37,16 @@ def directory_digest(self) -> Digest:
     class Factory(Script):
         options_scope = "download-pex-bin"
         name = "pex"
-        default_version = "v2.1.6"
+        default_version = "v2.1.7"
 
         # Note: You can compute the digest and size using:
         # curl -L https://github.com/pantsbuild/pex/releases/download/vX.Y.Z/pex | tee >(wc -c) >(shasum -a 256) >/dev/null
         default_versions_and_digests = {
             PlatformConstraint.none: ToolForPlatform(
                 digest=Digest(
-                    "73e692f9a67a8d8b3f8b246076a3fd99f29fd5cbe18126c69657ac8c99c277fc", 2614280
+                    "375ab4a405a6db57f3afd8d60eca666e61931b44f156dc78ac7d8e47bddc96d6", 2620451
                 ),
-                version=ToolVersion("v2.1.6"),
+                version=ToolVersion("v2.1.7"),
             ),
         }
 


Test patch:
diff --git a/src/python/pants/backend/python/rules/pex_test.py b/src/python/pants/backend/python/rules/pex_test.py
--- a/src/python/pants/backend/python/rules/pex_test.py
+++ b/src/python/pants/backend/python/rules/pex_test.py
@@ -4,9 +4,11 @@
 import json
 import os.path
 import zipfile
-from typing import Dict, List, Optional, Tuple, cast
+from dataclasses import dataclass
+from typing import Dict, Iterable, Iterator, List, Optional, Tuple, cast
 
 import pytest
+from pkg_resources import Requirement
 
 from pants.backend.python.rules import download_pex_bin
 from pants.backend.python.rules.pex import (
@@ -123,6 +125,31 @@ def assert_merged(*, input: List[List[str]], expected: List[str]) -> None:
     )
 
 
+@dataclass(frozen=True)
+class ExactRequirement:
+    project_name: str
+    version: str
+
+    @classmethod
+    def parse(cls, requirement: str) -> "ExactRequirement":
+        req = Requirement.parse(requirement)
+        assert len(req.specs) == 1, (
+            "Expected an exact requirement with only 1 specifier, given {requirement} with "
+            "{count} specifiers".format(requirement=requirement, count=len(req.specs))
+        )
+        operator, version = req.specs[0]
+        assert operator == "==", (
+            "Expected an exact requirement using only the '==' specifier, given {requirement} "
+            "using the {operator!r} operator".format(requirement=requirement, operator=operator)
+        )
+        return cls(project_name=req.project_name, version=version)
+
+
+def parse_requirements(requirements: Iterable[str]) -> Iterator[ExactRequirement]:
+    for requirement in requirements:
+        yield ExactRequirement.parse(requirement)
+
+
 class PexTest(TestBase):
     @classmethod
     def rules(cls):
@@ -227,7 +254,9 @@ def test_resolves_dependencies(self) -> None:
         pex_info = self.create_pex_and_get_pex_info(requirements=requirements)
         # NB: We do not check for transitive dependencies, which PEX-INFO will include. We only check
         # that at least the dependencies we requested are included.
-        assert set(requirements.requirements).issubset(pex_info["requirements"]) is True
+        assert set(parse_requirements(requirements.requirements)).issubset(
+            set(parse_requirements(pex_info["requirements"]))
+        )
 
     def test_requirement_constraints(self) -> None:
         # This is intentionally old; a constraint will resolve us to a more modern version.
@@ -245,7 +274,9 @@ def test_requirement_constraints(self) -> None:
             requirements=PexRequirements([direct_dep]),
             additional_pants_args=("--python-setup-requirement-constraints=constraints.txt",),
         )
-        assert set(pex_info["requirements"]) == set(constraints)
+        assert set(parse_requirements(pex_info["requirements"])) == set(
+            parse_requirements(constraints)
+        )
 
     def test_entry_point(self) -> None:
         entry_point = "pydoc"