Repository: pantsbuild/pants

Issue:
2.3.1rc0 fails to run/complete pytests
Just tried upgrading from 2.3.0 to 2.3.1rc0 (using Python 3.7.9 and 3.8.8 as environments).

On 2.3.0, I'm able to successfully run `./pants test ::` and it completes in about 10-15 seconds. When I wipe all my caches, pids, etc and then upgrade pants to 2.3.1rc0 - the console tells me that pytest is running, but they don't seem to end. I have to manually terminate the testing to exit. 

My workflow for testing this is:

```bash
rm -r ~/.cache/pants
rm -r ./pants.d
rm -r ./pids
./pants version
./pants test ::
```

2.3.1rc0 installs successfully, compiles everything and then gets to this point - and the time continuously racks up:

```bash
platform darwin -- Python 3.7.9, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /Users/sj/.cache/pants/named_caches/pex_root/venvs/short/8ac57bed/bin/python3.7
cachedir: .pytest_cache
rootdir: /private/var/folders/18/q1r7phps28nc9rx5j_0t3jmm0000gp/T/process-execution7sTDoG
plugins: cov-2.11.1, asyncio-0.14.0, icdiff-0.5
collecting ... collected 0 items / 1 skipped

============================== 1 skipped in 0.09s ==============================

⠈ 53.93s Run Pytest for foo1_test.py:../../tests
⠈ 53.12s Run Pytest for foo2_test.py:../../../tests
⠈ 53.83s Run Pytest for foo3_test.py:../../../tests
⠈ 53.24s Run Pytest for bar1_test.py:../../../tests
⠈ 53.83s Run Pytest for bar2_test.py:../../tests
⠈ 53.83s Run Pytest for bar3_test.py:../../tests
```


Discussion:
Thanks for reporting this @sureshjoshi! I've encountered this with running Flake8, isort, and Black, but not Pytest. I've been trying to figure it out. It was bisected back to https://github.com/pantsbuild/pants/pull/11765. FYI, Pex is getting into an infinite loop where it tries to re-exec with the correct interpreter, but it's not the right one and keeps looping to re-exec.

Can you please share more about your environment, e.g. your OS? None of the other maintainers have been able to reproduce, and I'm trying to figure out what the culprit is.

If possible, please do preserve `~/.cache/pants` - I may have some questions. (Not a huge deal if you need to purge it because I can reproduce.) Reverting to 2.3.0 should fix this.

This will be my top priority tomorrow.
NP @Eric-Arellano - I can keep the cache. 

I should note that I also do have black, isort, and a couple other items in there - but yeah, it looks like the problem occurs on the pytest part. 

I figured it was related to the append-cache PR, as there wasn't much else changed between 2.3.0 and 2.3.1rc0

```bash
macOS Big Sur (11.2.3)

Python 3.7.9 (default, Jan 22 2021, 16:36:49) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Python 3.8.8 (default, Mar 24 2021, 21:12:19) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
```

pants.toml
```toml
[GLOBAL]
pants_version = "2.3.1rc0"
pantsd = true
pants_ignore_use_gitignore = false
backend_packages = [
    "pants.backend.codegen.protobuf.python",
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.mixed_interpreter_constraints",
    "pants.backend.python.typecheck.mypy",
]
build_patterns = ["BUILD.pants"]

[source]
# The Python source root is the repo root. See https://www.pantsbuild.org/docs/source-roots.
#root_patterns = ["/"]
marker_filenames = ["BUILD.pants"]

[python-setup]
interpreter_constraints = [">=3.9"]
requirement_constraints = "requirements.lock.txt"

[coverage-py]
report = ["html", "console"]

[pytest]
version = "pytest>=6.2"
args=["-vv"]
pytest_plugins.add = [
    "pytest-asyncio>=0.14",
    "pytest-icdiff>=0.5",
]

[test]
extra_env_vars = [
    "APP_MODE=development",
]
```
@Eric-Arellano 
 
Also, one thing I don't entirely understand how it works (not really having spent much time in Pex source), 

https://github.com/pantsbuild/pants/pull/11765/files

Line 131: PEX_ROOT goes from (on my machine): 
`/Users/sj/.cache/pants/named_caches/pex_root` (which exists and has stuff in it)

to just:
`.cache/pex_root` (which doesn't exist - if that is supposed to represent /Users/sj/.cache/pex_root). 

There is probably more to it than that, but at the very least - that's strange... Shouldn't the new PEX_ROOT exist? There is no other `pex_root` folder in my `.cache` directory.

Forklifting good data Eric had out of the Slack black hole as near as I can tell:

Output from ~executing a `__run.sh`:
```
$ /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python -sE /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-execution6Y2Rzi/.cache/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/pex --check --config pyproject.toml src/python/pants/util/strutil.py
Re-execing from /opt/homebrew/opt/python@3.9/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
Re-execing from /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
```

Snip of the re-exec loop script:
```
#!/Users/eric/.cache/pants/named_caches/pex_root/venvs/short/c3eb1b0a/bin/python3.9 -sE
if __name__ == "__main__":
    import os
    import sys
    venv_dir = os.path.abspath(os.path.dirname(__file__))
    venv_bin_dir = os.path.join(venv_dir, "bin")
    shebang_python = '/Users/eric/.cache/pants/named_caches/pex_root/venvs/short/c3eb1b0a/bin/python3.9'
    python = os.path.join(venv_bin_dir, os.path.basename(shebang_python))
    if sys.executable not in (python, shebang_python):
        sys.stderr.write("Re-execing from {}\n".format(sys.executable))
        os.execv(python, [python, "-sE"] + sys.argv)
        ...
```

Debug prints added to above:
```    
python: /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-execution18NBX4/.cache/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
python real_path: /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9
sys.executable: /Users/eric/.cache/pants/named_caches/pex_root/venvs/3c89ae006bab8f37897fd91a0a2d26ee196c0977/837c093c06bd686fc4cd8a7c11bdb7b9cb3598d4/bin/python3.9
sys.executable real_path: /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/bin/python3.9
```

In this case the `sys.executable` is the non-short version `shebang_python` as near as I can tell from afar which points to an obvious fix on the Pex side: ensure the set of interpreters `sys.executable` is tested against includes both abspath and realpath variants for each acceptable python.
I've filed https://github.com/pantsbuild/pex/issues/1285 for the above to start. It also looks like at least two bits of hardening could prevent infinite re-exec loops in all cases. If I break out issues for those, I'll link here.
@sureshjoshi the `.cache/pex_root` is a path in the process execution sandbox that's a symlink out to `/Users/sj/.cache/pants/named_caches/pex_root`. Like so:
```
$ ./pants --no-process-execution-local-cleanup test src/python/pants/util/strutil_test.py
22:29:36.35 [INFO] preserving local process execution dir `"/tmp/process-executionRlii1N"` for "Building pytest.pex with 6 requirements: ipdb, pygments, pytest-cov>=2.10.1,<2.12, pytest-html, pytest-icdiff, pytest>=6.0.1,<6.3"
22:29:36.35 [INFO] preserving local process execution dir `"/tmp/process-execution6213al"` for "Resolving 3rdparty/python/constraints.txt"
22:29:56.19 [INFO] Completed: Building pytest.pex with 6 requirements: ipdb, pygments, pytest-cov>=2.10.1,<2.12, pytest-html, pytest-icdiff, pytest>=6.0.1,<6.3
22:30:06.24 [INFO] Completed: Resolving 3rdparty/python/constraints.txt
22:30:06.24 [INFO] preserving local process execution dir `"/tmp/process-execution4DHoIg"` for "Building pytest_runner.pex"
22:30:08.55 [INFO] Completed: Building pytest_runner.pex
22:30:08.55 [INFO] preserving local process execution dir `"/tmp/process-executiontP99rm"` for "Run Pytest for src/python/pants/util/strutil_test.py:tests"
22:30:09.10 [INFO] Completed: test - src/python/pants/util/strutil_test.py:tests succeeded.
...
```

And looking in the pytest chroot:
```
$ tree -a -L 2 /tmp/process-executiontP99rm/
/tmp/process-executiontP99rm/
├── .cache
│   └── pex_root -> /home/jsirois/.cache/pants/named_caches/pex_root
├── .coveragerc
├── extra-output
├── .pytest_cache
│   ├── CACHEDIR.TAG
│   ├── .gitignore
│   ├── README.md
│   └── v
├── pytest.pex
├── pytest_runner.pex
├── pytest_runner.pex_bin_python_shim.sh
├── pytest_runner.pex_pex_shim.sh
├── requirements.pex
├── __run.sh
└── src
    └── python
```
So when the equivalent of the `__run.sh` script executes:
```
$ cat /tmp/process-executiontP99rm/__run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export HOME=/home/jsirois PATH=$'/home/jsirois/.cache/pants/pants_dev_deps/Linux.py37.venv/bin:/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/opt/cuda/bin:/opt/cuda/integration/nsight-compute:/opt/cuda/integration/nsight-systems:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin:/home/jsirois/.bash.d/bin' PEX_EXTRA_SYS_PATH=src/python PYTEST_ADDOPTS=$'--color=yes' RUST_BACKTRACE=1

./pytest_runner.pex_pex_shim.sh -vvs src/python/pants/util/strutil_test.py
```

The `./pytest_runner.pex_pex_shim.sh` executes from the chroot as CWD:
```
$ cat /tmp/process-executiontP99rm/pytest_runner.pex_pex_shim.sh
#!/usr/bin/bash
set -euo pipefail

export PATH=/home/jsirois/.cache/pants/pants_dev_deps/Linux.py37.venv/bin:/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/opt/cuda/bin:/opt/cuda/integration/nsight-compute:/opt/cuda/integration/nsight-systems:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin PEX_INHERIT_PATH=false PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root LANG=en_US.UTF-8

# Let PEX_TOOLS invocations pass through to the original PEX file since venvs don't come
# with tools support.
if [ -n "${PEX_TOOLS:-}" ]; then
  exec /usr/bin/python3.7 pytest_runner.pex "$@"
fi

# If the seeded venv has been removed from the PEX_ROOT, we re-seed from the original
# `--venv` mode PEX file.
if [ ! -e .cache/pex_root/venvs/10a3b2d1e54d0cea62577cd42a1437fd01bacacb/08a7d74fb0b242808f942b2d0f4d72808f379c87/pex ]; then
    rm -rf .cache/pex_root/venvs/10a3b2d1e54d0cea62577cd42a1437fd01bacacb/08a7d74fb0b242808f942b2d0f4d72808f379c87 || true
    PEX_INTERPRETER=1 /usr/bin/python3.7 pytest_runner.pex -c ''
fi

exec .cache/pex_root/venvs/10a3b2d1e54d0cea62577cd42a1437fd01bacacb/08a7d74fb0b242808f942b2d0f4d72808f379c87/pex "$@"
```

And so the `PEX_ROOT` of `.cache/pex_root` is a correct relative path pointing off to the global named_caches directory.
@jsirois Ah okay, thanks - it was definitely odd to read  :)

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,7 +37,7 @@ class PexBinary(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.34"
+    default_version = "v2.1.35"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
 
     @classproperty
@@ -47,8 +47,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "9b1a959ccb61b3deb64ffeed43a735c7115e414f4de6f96e66adc9e7fc7a757f",
-                    "3597768",
+                    "3b1c97ebd79a8b650c81f0afbdbefef17c3d1a128219a8e95b5f35c2c9dfd532",
+                    "3600775",
                 )
             )
             for plat in ["darwin", "linux"]


Test patch: