Repository: pex-tool/pex

Issue:
Venv pex script enters an infinite re-exec loop.
At least two known cases of this happened over in Pants for --venv mode PEXes. One case has the following data:

The inifinite loop:
```
$ /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -sE /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-execution6Y2Rzi/.cache/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/pex --check --config pyproject.toml src/python/pants/util/strutil.py
Re-execing from /opt/homebrew/opt/python@3.9/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
```

The `pex` venv script:
```
#!/Users/eric/.cache/pants/named_caches/pex_root/venvs/short/c3eb1b0a/bin/python3.9 -sE
if __name__ == "__main__":
    import os
    import sys
    venv_dir = os.path.abspath(os.path.dirname(__file__))
    venv_bin_dir = os.path.join(venv_dir, "bin")
    shebang_python = '/Users/eric/.cache/pants/named_caches/pex_root/venvs/short/c3eb1b0a/bin/python3.9'
    python = os.path.join(venv_bin_dir, os.path.basename(shebang_python))
    if sys.executable not in (python, shebang_python):
        sys.stderr.write("Re-execing from {}\n".format(sys.executable))
        os.execv(python, [python, "-sE"] + sys.argv)
        ...
```

Some debug print data about the script above:
```        
python: /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-execution18NBX4/.cache/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
python real_path: /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9
sys.executable: /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
sys.executable real_path: /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9
```


Discussion:
A few things to note here:
1. The tuple of Pythons the `sys.executable` is checked against should include the `venvs/short` path resolved once. os.path.realpath won't do since the link may be double and dereference to a system Python which won't have the needed site-libraries of the venv interpreter.
2. The re-exec should probably use `shebang_python` since this is known to be the correct python and it's in the tuple.
3. Although 2 probably obviates the need for this, exporting an env var to indicate a re-exec is in-progress would allow th re-execed process to know it should not re-exec again.

Resolving patch:
diff --git a/pex/tools/commands/venv.py b/pex/tools/commands/venv.py
--- a/pex/tools/commands/venv.py
+++ b/pex/tools/commands/venv.py
@@ -152,8 +152,24 @@ def record_provenance(src_to_dst):
             venv_bin_dir = os.path.join(venv_dir, "bin")
             shebang_python = {shebang_python!r}
             python = os.path.join(venv_bin_dir, os.path.basename(shebang_python))
-            if sys.executable not in (python, shebang_python):
+
+            def iter_valid_venv_pythons():
+                # Allow for both the known valid venv pythons and their fully resolved venv path
+                # version in the case their parent directories contain symlinks.
+                for python_binary in (python, shebang_python):
+                    yield python_binary
+                    yield os.path.join(
+                        os.path.realpath(os.path.dirname(python_binary)),
+                        os.path.basename(python_binary)
+                    )
+
+            current_interpreter_blessed_env_var = "_PEX_SHOULD_EXIT_VENV_REEXEC"
+            if (
+                not os.environ.pop(current_interpreter_blessed_env_var, None)
+                and sys.executable not in tuple(iter_valid_venv_pythons())
+            ):
                 sys.stderr.write("Re-execing from {{}}\\n".format(sys.executable))
+                os.environ[current_interpreter_blessed_env_var] = "1"
                 os.execv(python, [python, "-sE"] + sys.argv)
 
             os.environ["VIRTUAL_ENV"] = venv_dir


Test patch: