Repository: pantsbuild/pants

Issue:
ModuleNotFoundError: No module named 'pex'
**Describe the bug**
When running pants in our CI system, we can have a container that has multiple clones of our repository. At various points, they will be running goals like `./pants export` or `./pants run`. At some point in the chain, we see errors like this
```
10:42:12.54 [INFO] Initialization options changed: reinitializing scheduler...
10:42:13.10 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/7907cee4a80399696d31dbee748702a492b91a80/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'
```
The only way to remedy this seems to be clearing out `~/.pex` and trying the goal again.

**Pants version**
2.13.0

**OS**
Ubuntu 20.04

**Additional info**
We've tried adding some steps to manage concurrency in the system and automatically wipe `~/.pex` when this error is detected, but that's unreliable to implement everywhere. Some initially debugging was kindly provided by @jsirois who filed https://github.com/pantsbuild/pants/issues/17176 to resolve.


Discussion:
I won't be back at a keyboard until the evening of 10/19 to look at this, but with Pants out of the way in this case (it specifically involves "normal" PEXes using the default `PEX_ROOT`), it lends clinching evidence to the conclusion in #17176 that the Pants shim script `rm -rf` was not the only crease for this bug. Something, somehow, is clearly broken with Pex's atomic_directory / it's use.

@engnatha what version of Pants did you upgrade from to get to 2.13.0? That helps set a range of Pex releases any bugs could have been introduced in.
We jumped from 2.11.0 directly to 2.13.0
It looks like `./pants export` is certainly a culprit prior to @benjyw's fix here: https://github.com/pantsbuild/pants/pull/17177#discussion_r996333078

This may be enough to explain / solve the totality of this issue. The only remaining task afaict is confirming 2.11.0 used the old venv symlinking code which did not do the ~`rm -rf ~/.pex` that @benjyw's fix removes.
Hmm, on the one hand that does seem very likely to be the culprit. 

On the other hand, 2.11.0 does include #14640 (see `git tag --contains 3d7f6a666776a4`), so I dunno...
#17177 will be in the upcoming 2.14.0rc4, so worth a try?
We weren’t using the export functionality with 2.11 because we were still using pip-tools to generate our constraints file and didn’t need the special tooling for setting up a developer environment. With the upgrade, we switched to pex lock files and started using the export feature. 
Ah, OK, so this is back on as the leading culprit. Is 2.14.0rc4 (when we publish it) an option for you to try?
It is. While I prefer to keep us on a stable release, I think it's better to bump to this if it resolves the instability we're seeing in our CI system.
Do you know when that candidate is planned to be published?
> will be in the upcoming 2.14.0rc4

I think this is likely to help, but the issue still exists. See https://github.com/pantsbuild/pants/actions/runs/3283455197/jobs/5408557053#step:11:11743 from today's 2.14 CI.

> Do you know when that candidate is planned to be published?

Today

That is not the same issue. There is a difference between the `~/.pex` and named caches cases.
Perhaps the test failures higher up? Multiple test files failed in this PR. https://github.com/pantsbuild/pants/actions/runs/3283455197/jobs/5408557053#step:11:11527
From my phone, but afaict no. The issues there are all named caches related. The issue @engnatha experienced was completely different where Pants was foisted on it's own petard when running ~`rm -rf` on `~/.pex` during export - decidedly not in named caches. @benjyw fixed the foisting issue.
@benjyw  caused the issue, and then, eventually, fixed it...
@Eric-Arellano I've filed https://github.com/pantsbuild/pex/issues/1955, https://github.com/pantsbuild/pex/issues/1956 and https://github.com/pantsbuild/pex/issues/1957 for the 3 variants I found using your links. That's crazy though to see so many different types of missing file errors at once. I really have no clue what's up here. Have you seen this sort of thing regularly or just with that PR or ???.
@jsirois I've never seen so many similar errors in the same run. Note that the second run of the job also had several similar errors: https://github.com/pantsbuild/pants/actions/runs/3283455197/jobs/5409915111. The third run of the job worked.
@Eric-Arellano OK. Thanks, that's good to know.

I've got https://github.com/pantsbuild/pex/pull/1961 spun up to hopefully make errors more clear for the #16778 style case, although that change fixes nothing afaict since I still can't see the underlying bug for that case.

For the 3 cases I extracted from your failed CI runs so far (more work to do looking at the last run you posted in https://github.com/pantsbuild/pants/actions/runs/3283455197/jobs/5409915111) that change may or may not help make the underlying issue more clear. I still need to do analysis on all those backtraces against 2.1.108 code.
Ok, busted 2.14.x CP: https://github.com/pantsbuild/pants/pull/17200#discussion_r1002093435
I'll fix that up and send out a review. So this is all still different from #16778 and just more petards. We'll eventually get this right!

Resolving patch:
diff --git a/src/python/pants/backend/python/goals/export.py b/src/python/pants/backend/python/goals/export.py
--- a/src/python/pants/backend/python/goals/export.py
+++ b/src/python/pants/backend/python/goals/export.py
@@ -141,7 +141,7 @@ async def export_virtualenv(
                     "venv",
                     "--pip",
                     "--collisions-ok",
-                    "--remove=all",
+                    "--remove=pex",
                     f"{{digest_root}}/{py_version}",
                 ],
                 {
@@ -192,7 +192,7 @@ async def export_tool(
                     os.path.join("{digest_root}", pex.name),
                     "venv",
                     "--collisions-ok",
-                    "--remove=all",
+                    "--remove=pex",
                     f"{{digest_root}}/{request.resolve_name}",
                 ],
                 {


Test patch:
diff --git a/src/python/pants/backend/python/goals/export_test.py b/src/python/pants/backend/python/goals/export_test.py
--- a/src/python/pants/backend/python/goals/export_test.py
+++ b/src/python/pants/backend/python/goals/export_test.py
@@ -84,7 +84,7 @@ def run(enable_resolves: bool) -> ExportResults:
                 "venv",
                 "--pip",
                 "--collisions-ok",
-                "--remove=all",
+                "--remove=pex",
                 f"{{digest_root}}/{current_interpreter}",
             )
             assert ppc0.extra_env["PEX_MODULE"] == "pex.tools"