Repository: pantsbuild/pants

Issue:
Nondeterministic `ImportError: No module named __pants_df_parser` while computing changed targets
(NOTE: There's been a lot of discussion on this in Slack [here](https://pantsbuild.slack.com/archives/C046T6T9U/p1658412143380059), but I realized I never filed an issue for it)

**Describe the bug**

A few times per day (out of ~hundreds of jobs), our CI jobs will fail when trying to build `dockerfile_parser.pex`. The error always hits at the beginning of goal execution, and affects all our goals (`test`, `lint`, `check`). The error looks like:
```
13:28:44.08 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:56.61 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:57.16 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
13:29:17.25 [INFO] Starting: Building dockerfile_parser.pex from dockerfile-parser_default.lock
13:29:18.32 [INFO] Canceled: Building dockerfile_parser.pex from dockerfile-parser_default.lock
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 159, in create
    specs = calculate_specs(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 94, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.util_rules.dependencies.inject_docker_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.subsystems.dockerfile_parser.parse_dockerfile
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Parse Dockerfile.' failed with exit code 1.
stdout:
stderr:
Traceback (most recent call last):
  File "/tmp/pants-sandbox-kVePcL/.cache/pex_root/venvs/e2b00a6620ed72ee63184ca7a64555f5551da8c5/f679563b742633f5edaba61ca7d405616ffc49b3/pex", line 234, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 138, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named __pants_df_parser
```

**Pants version**

* The error started appearing on `PANTS_SHA=5d8a328d72209863986c8959b20305505bc068ba`, and has continued appearing as we've upgraded along the `2.13.x` branch.
* As far as I can tell, the error was _not_ appearing on `v2.13.0a0`

**OS**

Linux

**Additional info**

Through debugging on Slack, we've found:
1. `__pants_df_parser.py` truly isn't in the venv when this error occurs
2. The error doesn't occur if we use `--no-process-cleanup`
3. The sandbox PEX created for the dockerfile parser contains the file, even when the pex_root venv does not. Running the PEX creates a venv that contains the file.


Discussion:
I just saw a new error for the first time, still in the `Parse Dockerfile` stage 😮 
```
/tmp/pants_named_caches/pex_root/venvs/f92f4a34a585f3429d24d32fb8349b7f900ea322/cf26e88f2412df9ce2b3f6b83a6b129bc3e3bb55/bin/python: can't open file '/tmp/pants_named_caches/pex_root/venvs/f92f4a34a585f3429d24d32fb8349b7f900ea322/cf26e88f2412df9ce2b3f6b83a6b129bc3e3bb55/pex': [Errno 2] No such file or directory
```
Ok, I started re-reviewing this from the top instead of focusing in on Pex. That leads here:
https://github.com/pantsbuild/pants/blob/0bbf1fbb6078cacb1e3a75f5ea67a14d0398910c/src/python/pants/backend/python/util_rules/pex.py#L750-L757

Clearly? the rm -rf can be interrupted and since `rm -r` is implemented as a depth-1st delete, that can leave the root `venv/pex` file present while portions of its subtree have been removed. So, given that, imagine this:
*1. A process runs on a machine with no venv in the PEX_ROOT yet, and enters the outer guard.
2. Another process enters the guard and builds the venv and starts `exec`'uting it.
3. The original process starts processing the rm -rf on the venv the process from step 2 is using.

*N.B. 1 can only happen when the PEX in question is not built locally, since we pre `--seed` the PEX venv when building it. In CI, that means either the LMDB store is cached but the named_caches (PEX_ROOT) are not or that remote caching is enabled and that's where the pre-built PEX comes from.

@danxmoran can you check me on this logic / if it fully describes / comports with your setup where you are seeing failures?
I think if the above is the explanation, then just switching to Pex's `--sh-boot` mode and eliminating the wrapper script will do the trick and allow Pants to keep all the hard earned milliseconds saved by this old wrapper script pre-dating `--sh-boot`.
@jsirois the explanation makes sense for our CI context - we use ephemeral runners so the Pants cache and PEX root are empty at the start of each run, and we use remote caching.
Alrighty, thanks for the confirmation. I think the issue is now fully understood. I want to stew a bit more on the `--sh-boot` fix vs just removing the `rm -rf` as a smaller step. I need to review why / when I added the `rm -rf` in the 1st place.
Ok, the `rm -rf` was in there from the get-go a year and 1/2 ago: https://github.com/pantsbuild/pants/pull/11557/files#r575485648

I seem to recall during development of that PR that I initially didn't have it and ran into issues, but I'll give that path a thorough analysis before committing to the longer term `--sh-boot` fix.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex.py b/src/python/pants/backend/python/util_rules/pex.py
--- a/src/python/pants/backend/python/util_rules/pex.py
+++ b/src/python/pants/backend/python/util_rules/pex.py
@@ -749,8 +749,7 @@ def _create_venv_script(
 
             # If the seeded venv has been removed from the PEX_ROOT, we re-seed from the original
             # `--venv` mode PEX file.
-            if [ ! -e "${{target_venv_executable}}" ]; then
-                rm -rf "${{venv_dir}}" || true
+            if [ ! -e "${{venv_dir}}" ]; then
                 PEX_INTERPRETER=1 ${{execute_pex_args}} -c ''
             fi
 


Test patch: