Repository: pantsbuild/pants
Issue: Pyenv test failing over directory already existing

On my macbook, running `./pants test src/python/pants/backend/python/providers/pyenv/rules_integration_test.py` consistently errors with the underlying issue being:

```
E       pants.engine.process.ProcessExecutionFailure: Process 'Install Python 3.9' failed with exit code 1.
E       stdout:
E       
E       stderr:
E       pyenv: /Users/benjyw/.cache/pants/named_caches/pyenv/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/versions/3.9.16 already exists
E       Traceback (most recent call last):
E         File "/private/tmp/pants-sandbox-TjtzIT/install_python_shim.py", line 29, in <module>
E           main()
E         File "/private/tmp/pants-sandbox-TjtzIT/install_python_shim.py", line 21, in main
E           subprocess.run(["./pyenv-2.3.13/bin/pyenv", "install", SPECIFIC_VERSION], check=True)
E         File "/Users/benjyw/.pyenv/versions/3.10.6/lib/python3.10/subprocess.py", line 524, in run
E           raise CalledProcessError(retcode, process.args,
E       subprocess.CalledProcessError: Command '['./pyenv-2.3.13/bin/pyenv', 'install', '3.9.16']' returned non-zero exit status 1.
```

The key line is `/Users/benjyw/.cache/pants/named_caches/pyenv/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/versions/3.9.16 already exists` - which is true! In fact it has to be true since we create the DONE file and its lock file in there. 

In that dir there is a DONE.lock but no DONE. SO I guess a previous pyenv install failed and left things in an intermediate state. 

Probably we need to keep the DONE and lockfile outside that dir, so we can clean it prior to attempting to pyenv-install into it?

Changed files:
src/python/pants/backend/python/providers/pyenv/rules.py

diff --git a/src/python/pants/backend/python/providers/pyenv/rules.py b/src/python/pants/backend/python/providers/pyenv/rules.py
index d61b7541c4..6cd391ae2f 100644
--- a/src/python/pants/backend/python/providers/pyenv/rules.py
+++ b/src/python/pants/backend/python/providers/pyenv/rules.py
@@ -144,12 +144,10 @@ async def get_pyenv_install_info(
                         #!/usr/bin/env bash
                         set -e
                         export PYENV_ROOT=$(readlink {PYENV_NAMED_CACHE})/{installation_fingerprint}
-                        DEST="$PYENV_ROOT"/versions/$1
-                        if [ ! -f "$DEST"/DONE ]; then
-                            mkdir -p "$DEST" 2>/dev/null || true
+                        if [ ! -f "$PYENV_ROOT"/pants_donefiles/$1.DONE ]; then
                             {python_binary.path} install_python_shim.py $1
                         fi
-                        echo "$DEST"/bin/python
+                        echo "$PYENV_ROOT"/versions/$1/bin/python
                         """
                     ).encode(),
                     is_executable=True,
@@ -159,25 +157,30 @@ async def get_pyenv_install_info(
                     dedent(
                         f"""\\
                         import fcntl
+                        import os
                         import pathlib
+                        import shutil
                         import subprocess
                         import sys
 
                         PYENV_ROOT = pathlib.Path("{PYENV_NAMED_CACHE}", "{installation_fingerprint}").resolve()
                         SPECIFIC_VERSION = sys.argv[1]
                         SPECIFIC_VERSION_PATH = PYENV_ROOT / "versions" / SPECIFIC_VERSION
-                        DONEFILE_PATH = SPECIFIC_VERSION_PATH / "DONE"
-                        DONEFILE_LOCK_PATH = SPECIFIC_VERSION_PATH / "DONE.lock"
-                        DONEFILE_LOCK_FD = DONEFILE_LOCK_PATH.open(mode="w")
+                        DONEFILE_DIR = PYENV_ROOT / "pants_donefiles"
+                        DONEFILE_PATH = DONEFILE_DIR / (SPECIFIC_VERSION + ".DONE")
+                        DONEFILE_LOCK_PATH = DONEFILE_DIR / "DONE.lock"
 
                         def main():
+                            os.makedirs(DONEFILE_DIR, exist_ok=True)
                             if DONEFILE_PATH.exists():
                                 return
-                            fcntl.lockf(DONEFILE_LOCK_FD, fcntl.LOCK_EX)
+                            fcntl.lockf(DONEFILE_LOCK_PATH.open(mode="w"), fcntl.LOCK_EX)
                             # Use double-checked locking to ensure that we really need to do the work
                             if DONEFILE_PATH.exists():
                                 return
-
+                            # If a previous install failed this directory may exist in an intermediate
+                            # state, and pyenv may choke trying to install into it, so we remove it.
+                            shutil.rmtree(SPECIFIC_VERSION_PATH, ignore_errors=True)
                             subprocess.run(["{pyenv.exe}", "install", SPECIFIC_VERSION], check=True)
                             # Removing write perms helps ensure users aren't accidentally modifying
                             # Python or the site-packages