Repository: pex-tool/pex

Issue:
python<=3.8 symlink with a suffix (eg `3.7m`) can create a venv without a `pythonX.Y` symlink which breaks pex assumptions that `pythonX.Y` is always available
I get this error when running `./pants version` on pants 2.16.x branch.

Pants is using pex v2.1.131 because I have overriden `[pex-cli].version = "v2.1.131"` and `[ex-cli].known_versions` in `~/.pants.rc`

```
$ ./pants --keep-sandboxes=always version
11:48:27.51 [INFO] waiting for pantsd to start...
11:48:30.01 [INFO] pantsd started
11:48:30.13 [INFO] Preserving local process execution dir /tmp/pants-sandbox-tP7l8W for Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin
11:48:30.14 [INFO] Preserving local process execution dir /tmp/pants-sandbox-X9MYvB for Test binary /bin/bash.
11:48:30.16 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:30.16 [INFO] Preserving local process execution dir /tmp/pants-sandbox-M3MXvy for Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:32.16 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:32.16 [ERROR] 1 Exception encountered:

[snip]

pants.engine.process.ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0' failed with exit code 1.
stdout:

stderr:
Failed to spawn a job for /home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python3.7: pid 1531475 -> /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python -sE /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/cognifloyd/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-M3MXvy/.tmp/pex-pip-log.4f462xf5/pip.log download --dest /home/cognifloyd/.cache/pants/named_caches/pex_root/downloads/resolver_download.ho5or1_z/home.cognifloyd..cache.pants.pants_dev_deps.Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv.bin.python3.7 pip==23.0.1 setuptools==67.4.0 wheel==0.38.4 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
Re-execing from /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python
Traceback (most recent call last):
  File "/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex", line 50, in <module>
    os.execv(python, argv)
FileNotFoundError: [Errno 2] No such file or directory
```

Looking in `/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex`, it lists a python binary that does not exist.

```
$ cd /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/
$ ls -l pex __main__.py bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd    10 Apr 17 12:04 bin/python -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd    10 Apr 17 12:04 bin/python3 -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd    19 Apr 17 12:04 bin/python3.7m -> /usr/bin/python3.7m
-rwxr-xr-x 1 cognifloyd cognifloyd 11256 Apr 17 12:04 __main__.py
lrwxrwxrwx 1 cognifloyd cognifloyd    11 Apr 17 12:04 pex -> __main__.py
```

The shebang python is wrong in `__main__.py` because it is using `python3.7` when the virtualenv contains `python3.7m`:
```
#!/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/s/0a5a863d/venv/bin/python3.7 -sE
...
    shebang_python = '/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/s/0a5a863d/venv/bin/python3.7'
```

### Debugging results

Using `--keep-sandboxes=always` I got into the pants sandbox for pex to see what's going on: `/tmp/pants-sandbox-M3MXvy`

```
$ cd /tmp/pants-sandbox-M3MXvy
$ ./pex --version
2.1.131
```

Here is the full pex command line pants is running (`__run.sh`):
```bash
#!/bin/bash
# This command line should execute the same process as pants did internally.
export CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' PEX_ROOT=.cache/pex_root
cd /tmp/pants-sandbox-M3MXvy
$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' ./pex --tmpdir .tmp --jobs 2 --pip-version 23.0.1 --python-path $'/home/cognifloyd/g/github/pyenv/pyenv.git/versions/3.4.10/bin:/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' --output-file pants_plugins.pex --no-emit-warnings --venv --seed verbose --no-venv-site-packages-copies --python $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' $'--sources-directory=source_files' hdrhistogram $'toolchain.pants.plugin==0.27.0' --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --resolver-version pip-2020-resolver --constraints __constraints.txt --layout packed
```

So, pex is using the pants bootstrap venv, the interpreters available are:
```
$ cd $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin'
$ ls -l python*
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python3 -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd 18 Apr 16 00:08 python3.7 -> /usr/bin/python3.7
```

#### python versions available on my machine

Here are all of the versions of python I have available:
```
$ ls -l /usr/bin/python3.[1-9]{,[0-9m]}
-rwxr-xr-x 1 root root 14304 Mar 15 12:46 /usr/bin/python3.10
-rwxr-xr-x 1 root root 14304 Mar 15 12:51 /usr/bin/python3.11
lrwxrwxrwx 1 root root    10 Aug  8  2022 /usr/bin/python3.6 -> python3.6m
-rwxr-xr-x 1 root root 14280 Aug  8  2022 /usr/bin/python3.6m
lrwxrwxrwx 1 root root    10 Aug  8  2022 /usr/bin/python3.7 -> python3.7m
-rwxr-xr-x 1 root root 14224 Aug  8  2022 /usr/bin/python3.7m
-rwxr-xr-x 1 root root 14304 Dec 12 04:09 /usr/bin/python3.8
-rwxr-xr-x 1 root root 14304 Mar 15 12:49 /usr/bin/python3.9
```

I have some older builds of python3.6 and python3.7 on my gentoo linux machine. 3.7 is the oldest that pants supports, so it is using that.

(If you're familiar with gentoo: I have ebuilds for these in my local overlay that I copied from `::gentoo` when they were dropped because I still needed 3.6 and 3.7)

These versions of python include the optional `m` abi tag described in https://peps.python.org/pep-3149/ (`m` means `--with-pymalloc`), so the binaries are `python3.6m` and `python3.7m`:

#### What's wrong with the pex venv?

As noted above, the venv is getting built with `bin/python3.7m` but `__main__.py` tries to use it as `bin/python3.7`, which doesn't exist.

Next I ran this to force a rebuild of the venv and see how it is getting built:
```
$ rm -rf ~/.cache/pants/named_caches/pex_root/
$ export PEX_VERBOSE=6
$ ./__run.sh
```

Sifting through the output, this is how it is creating the venv:
```
pex: Executing: /usr/bin/python3.7m -s -E -m venv --without-pip /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708.lck.work --prompt pex
```

So, I ran a version of that myself to see how it creates the symlinks:
```
$ /usr/bin/python3.7m -s -E -m venv --without-pip /tmp/pex-py37m-venv --prompt pex
$ ls -l /tmp/pex-py37m-venv/bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd 10 Apr 17 13:01 /tmp/pex-py37m-venv/bin/python -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd 10 Apr 17 13:01 /tmp/pex-py37m-venv/bin/python3 -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd 19 Apr 17 13:01 /tmp/pex-py37m-venv/bin/python3.7m -> /usr/bin/python3.7m
```

So, it is the `venv` module that is creating the symlink as `python3.7m`.

The next log entry (below) shows that pex is using the correct symlink in that venv to finish creating it, so it's only after the venv is populated and pex tries to reexecute that it gets lost.
```
pex: Executing: PYTHONPATH=/home/cognifloyd/.cache/pants/named_caches/pex_root/isolated/424a1c62c3a17be0bfeeb465d9221d7e2f0cb3ff /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708.lck.work/bin/python3.7m -s -c import os
```

#### Why is pex hard-coding `python3.7` as the `shebang_python` instead of `python3.7m`?

I think pex is calculating the binary name instead of looking it up:
https://github.com/pantsbuild/pex/blob/2534f1fd2d4fc9f2021b3a0dea9af757baccc027/pex/interpreter.py#L66-L75

Which eventually makes its way into the template here:
https://github.com/pantsbuild/pex/blob/2534f1fd2d4fc9f2021b3a0dea9af757baccc027/pex/venv/pex.py#L343-L354

The `--python` option pointed to the pants venv python `.../bin/python3.7` which had the symlink `python3.7` instead of `python3.7m`. So, I'm not sure how that pants bootstrap venv is getting created, but it is probably not using `-m venv` or I would expect it to have a symlink with the same name (`python3.7m`).

I'm not sure where to fix handling/detection of abi-tagged python binaries.


Discussion:
@cognifloyd do you have this same issue with Pants prior to https://github.com/pantsbuild/pants/pull/18610? So either 2.16.0a1 or 2.15.x?
I think the fix is `s/pex.interpreter.binary/virtualenv.interpreter.binary/` here:
https://github.com/pantsbuild/pex/blob/9c972c2b2653f63624f162ad5e594f24169b39ea/pex/pex_bootstrapper.py#L545-L551

I suspect https://github.com/pantsbuild/pants/pull/18610 triggers `pex.interpreter` being different from `virtualenv.interpreter` with its new use of `sys.executable` to run the Pex PEX, and here, your `sys.executable` is non-malloc.
I get the same on 2.15.x branch, including after `rm -rf ~/.cache/pants/named_caches/pex_root/`

```
stderr:
pid 1857243 -> /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python -sE /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/cognifloyd/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-J8Lcsv/.tmp/pex-pip-log.ynpj4ki6/pip.log download --dest /home/cognifloyd/.cache/pants/named_caches/pex_root/downloads/resolver_download.bmr_xspm/home.cognifloyd..cache.pants.pants_dev_deps.Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv.bin.python3.7 --constraint __constraints.txt hdrhistogram toolchain.pants.plugin==0.27.0 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
Re-execing from /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python
Traceback (most recent call last):
  File "/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex", line 50, in <module>
    os.execv(python, argv)
FileNotFoundError: [Errno 2] No such file or directory
```

Here is `__run.sh` from that sandbox:
```bash
#!/bin/bash
# This command line should execute the same process as pants did internally.
export CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root
cd /tmp/pants-sandbox-WEmOAJ
$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' ./pex --tmpdir .tmp --jobs 2 --python-path $'/home/cognifloyd/g/github/pyenv/pyenv.git/versions/3.4.10/bin:/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' --output-file pants_plugins.pex --no-emit-warnings --venv --seed verbose --no-venv-site-packages-copies --python $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' $'--sources-directory=source_files' hdrhistogram $'toolchain.pants.plugin==0.27.0' --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --resolver-version pip-2020-resolver --constraints __constraints.txt --layout packed
```

So, the 2.15.x branch is still adding the `--python` arg. I'll try the `release_2.16.0a1` tag next.
with 2.16.0a1:
```
stderr:
Failed to spawn a job for /home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python3.7: pid 1905700 -> /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python -sE /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/cognifloyd/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-0eLzwZ/.tmp/pex-pip-log.2g5rrsao/pip.log download --dest /home/cognifloyd/.cache/pants/named_caches/pex_root/downloads/resolver_download.ftih8lua/home.cognifloyd..cache.pants.pants_dev_deps.Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv.bin.python3.7 pip==23.0.1 setuptools==67.4.0 wheel==0.38.4 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
Re-execing from /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python
Traceback (most recent call last):
  File "/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex", line 50, in <module>
    os.execv(python, argv)
FileNotFoundError: [Errno 2] No such file or directory
```
sandbox `__run.sh`
```
#!/bin/bash
# This command line should execute the same process as pants did internally.
export CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON_PATH=$'/home/cognifloyd/g/github/pyenv/pyenv.git/versions/3.4.10/bin:/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' PEX_ROOT=.cache/pex_root
cd /tmp/pants-sandbox-Lpm768
$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' ./pex --tmpdir .tmp --jobs 2 --pip-version 23.0.1 --python-path $'/home/cognifloyd/g/github/pyenv/pyenv.git/versions/3.4.10/bin:/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' --output-file pants_plugins.pex --no-emit-warnings --venv --seed verbose --no-venv-site-packages-copies --python $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' $'--sources-directory=source_files' hdrhistogram $'toolchain.pants.plugin==0.26.0' --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --resolver-version pip-2020-resolver --constraints __constraints.txt --layout packed
```
Ok, thanks for testing @cognifloyd. I'll try to come up with a repro IT. I'm pretty sure the variance is between `pex.interpreter` and `virtualenv.interpreter` as I pointed out, but I have no step-by-step explanation for the mechanism quite yet.
Yeah. I'm not sure how to repro this outside of this context. :cry:

I get the same error on pants `release_2.16.0a0` tag which still passes the same value for `--python`.
Now trying the `release_2.15.0` tag.
Same results (and `--python` arg) on pants `release_2.15.0` and `release_2.14.1` tags. I've gone back this far to rule out any of the recent pyenv or PexPEX changes.

This is not a new problem. I haven't recorded the errors I got before now though, as I went with the more expedient options. For most pants development, I've had to move over to my Mac or just rely on CI to handle running everything for me. I'm reporting now because I'm finally diving in to try and fix things so I can develop pants on my gentoo linux laptop.

Now for another odd wrinkle: I can run pants in the st2.git checkout just fine. I'm using pants 2.16.0rc1 right now, and I've used various versions before that. It's only in the pants repo that I'm getting these errors.

Note: I get different errors on pants `main` / 2.17.x that are unrelated to this, or that are at least masking this error.

--------------------------------

So, then I dug into how pants creates its dev venv. It does in fact use the `venv` module, but it does so with `/usr/bin/python3.7` instead of `/usr/bin/python3.7m`. And I prefixed this array with `3.7m` to force the dev virtualenv to include a binary named `python3.7m` instead of `python3.7`. That worked!

Then I did something that is really ugly to see how the `venv` module treats the python it is called with:
```
$ sudo ln -s python3.7m /usr/bin/python3.7abc
$ /usr/bin/python3.7abc -m venv /tmp/py37abc_venv
$ ls -l /tmp/py37abc_venv/bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd 12 Apr 17 15:47 /tmp/py37abc_venv/bin/python -> python3.7abc
lrwxrwxrwx 1 cognifloyd cognifloyd 12 Apr 17 15:47 /tmp/py37abc_venv/bin/python3 -> python3.7abc
lrwxrwxrwx 1 cognifloyd cognifloyd 21 Apr 17 15:47 /tmp/py37abc_venv/bin/python3.7abc -> /usr/bin/python3.7abc
```

And then something a little less ugly:
```
$ pwd
/home/cognifloyd/g/github/pantsbuild/pants.git
$ ln -s /usr/bin/python3.7m python3.7foobar
$ ./python3.7foobar -m venv /tmp/py37foobar_venv
$ ls -l /tmp/py37foobar_venv/bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd 15 Apr 17 15:52 /tmp/py37foobar_venv/bin/python -> python3.7foobar
lrwxrwxrwx 1 cognifloyd cognifloyd 15 Apr 17 15:52 /tmp/py37foobar_venv/bin/python3 -> python3.7foobar
lrwxrwxrwx 1 cognifloyd cognifloyd 62 Apr 17 15:52 /tmp/py37foobar_venv/bin/python3.7foobar -> /home/cognifloyd/g/github/pantsbuild/pants.git/python3.7foobar
```
So, I think I can work around the bug in pants by adding `realpath` for the dev venv. But I'd still like to find the root cause and a repro for the issue in pex if possible.

The bug in pex seems to be the assumption that virtualenvs will always have a `pythonX.Y` binary or symlink. But, it seems that the venv module can skip that name in some circumstances including:
- <=python3.8
- A symlinked python binary with some kind of suffix

The `venv` module in python3.9 has been updated so that pex can rely on the assumption that `pythonX.Y` will be present. I was able to build a venv that does not include that with python3.8 and python3.7. The ABI tag, however, is a red herring - it doesn't matter what the python interpreter reports as the abi because my python3.8 python does not report `py38m` in the platform tags list like my `python3.7m` interpreter does. Consider:

```
$ ls -l /usr/bin/python3.8*
-rwxr-xr-x 1 root root 14304 Dec 12 04:09 /usr/bin/python3.8
-rwxr-xr-x 1 root root  2857 Dec 12 04:08 /usr/bin/python3.8-config
$ cd /tmp
$ ln -s /usr/bin/python3.8 python3.8foobar
$ ./python3.8foobar -m venv py38foobar_venv
$ ls -l py38foobar_venv/bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd 15 Apr 17 16:31 py38foobar_venv/bin/python -> python3.8foobar
lrwxrwxrwx 1 cognifloyd cognifloyd 15 Apr 17 16:31 py38foobar_venv/bin/python3 -> python3.8foobar
lrwxrwxrwx 1 cognifloyd cognifloyd 20 Apr 17 16:31 py38foobar_venv/bin/python3.8foobar -> /tmp/python3.8foobar
```

Hoepfully that's enough to build a repro for this issue.
The pants workaround: https://github.com/pantsbuild/pants/pull/18755
Ok, the OP error path runs through here:
https://github.com/pantsbuild/pants/blob/7398971f83db4877e34dd48f002920e4906997fb/src/python/pants/init/plugin_resolver.py#L67-L86

I verified the production code path does not set interpreter constraints for the PluginRequest; so the `python` derived from `sys.executable` is used. This explains the python3.7 from the Pants bootstrap venv being used to run the venv PEX as opposed to the python3.7m on your PATH. That also explains why this has ~always been a problem - this plugin code path is a bit be-spoke compared to other venv PEX uses in Pants that have had more recent changes.

This still leaves me figuring out a test / fix for `python3.7 venv.pex` leading to a a bad venv PEX shebang, but it does fully explain the mechanism behind the Pants bootstrap venv python3.7 getting used here.
Ok, it looks like the root weirdness is here (from a captured `__run.sh` for the Pants "Resolving plugins:" step):
```
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-Lfp3Bq
env -i CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin:/home/jsirois/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/mnt/c/Program Files/Python35/Scripts/:/mnt/c/Program Files/Python35/:/mnt/c/Program Files/Python36/Scripts/:/mnt/c/Program Files/Python36/:/mnt/c/Program Files/Python37/Scripts/:/mnt/c/Program Files/Python37/:/mnt/c/Program Files/Python38/Scripts/:/mnt/c/Program Files/Python38/:/mnt/c/Python27/:/mnt/c/Python27/Scripts:/mnt/c/Program Files/Python39/Scripts/:/mnt/c/Program Files/Python39/:/mnt/c/Program Files/Python310/Scripts/:/mnt/c/Program Files/Python310/:/mnt/c/Program Files/Python311/Scripts/:/mnt/c/Program Files/Python311/:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/GnuPG/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/Meld/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/PowerShell/7/:/mnt/c/Users/John Sirois/scoop/apps/openssl/current/bin:/mnt/c/Users/John Sirois/bin:/mnt/c/Users/John Sirois/scoop/shims:/mnt/c/Users/John Sirois/.cargo/bin:/mnt/c/Users/John Sirois/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/John Sirois/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python PEX_ROOT=.cache/pex_root /home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python ./pex --tmpdir .tmp --jobs 2 --pip-version latest --python-path $'/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.7/bin:/home/jsirois/.pyenv/versions/3.11.0rc2/bin:/home/jsirois/.pyenv/versions/3.11.1/bin:/home/jsirois/.pyenv/versions/3.5.10/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.14/bin:/home/jsirois/.pyenv/versions/3.7.15/bin:/home/jsirois/.pyenv/versions/3.8.0/bin:/home/jsirois/.pyenv/versions/3.8.14/bin:/home/jsirois/.pyenv/versions/3.9.14/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.9/bin:/home/jsirois/.pyenv/versions/pypy3.9-7.3.9/bin:/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin:/home/jsirois/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/mnt/c/Program Files/Python35/Scripts/:/mnt/c/Program Files/Python35/:/mnt/c/Program Files/Python36/Scripts/:/mnt/c/Program Files/Python36/:/mnt/c/Program Files/Python37/Scripts/:/mnt/c/Program Files/Python37/:/mnt/c/Program Files/Python38/Scripts/:/mnt/c/Program Files/Python38/:/mnt/c/Python27/:/mnt/c/Python27/Scripts:/mnt/c/Program Files/Python39/Scripts/:/mnt/c/Program Files/Python39/:/mnt/c/Program Files/Python310/Scripts/:/mnt/c/Program Files/Python310/:/mnt/c/Program Files/Python311/Scripts/:/mnt/c/Program Files/Python311/:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/GnuPG/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/Meld/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/PowerShell/7/:/mnt/c/Users/John Sirois/scoop/apps/openssl/current/bin:/mnt/c/Users/John Sirois/bin:/mnt/c/Users/John Sirois/scoop/shims:/mnt/c/Users/John Sirois/.cargo/bin:/mnt/c/Users/John Sirois/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/John Sirois/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin' --output-file pants_plugins.pex --no-emit-warnings --venv --seed verbose --no-venv-site-packages-copies --python /home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python $'--sources-directory=source_files' hdrhistogram $'toolchain.pants.plugin==0.27.0' --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --resolver-version pip-2020-resolver --constraints __constraints.txt --layout packed
```

Making the key bits more readable by eliding the rest, that's:
```
PEX_PYTHON=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python \
/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python \
./pex \
--python-path /home/jsirois/.pyenv/versions/2.7.18/bin:... \
--python /home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.x86_64.py37.venv/bin/python
```

That's contradictory since `--python` is not a member of `--python-path`; it comes from the Pants bootstrap venv instead which is not on the `--python-path`.

So this leads to the question of what Pex does when given a contradiction like this - clearly it doesn't fail fast though - which is probably the right thing to do:
https://github.com/pantsbuild/pex/blob/2ed43a53081e1ffb291281cc60d2e27953296aa8/pex/resolve/target_configuration.py#L47-L64

That code can select a different Python than the current Python. Still more work to do to see how the `pex` / `__main__.py` shebang comes out wrong in this scenario...
Aha, the `--python` / `--python-path` contradiction continues to be worrying, but that's not the issue here. I think I can connect the dots here and they do, in fact, land at `s/pex.interpreter.binary/virtualenv.interpreter.binary/` and the link requires `--python` is the `sys.executable` of a venv, which is the case here with the origination being a `pants_dev_deps` venv `sys.executable`.
Ok, the key to all of this is the following on your machine:
```
$ ls -l /usr/bin/python3.[1-9]{,[0-9m]}
...
lrwxrwxrwx 1 root root    10 Aug  8  2022 /usr/bin/python3.7 -> python3.7m
-rwxr-xr-x 1 root root 14224 Aug  8  2022 /usr/bin/python3.7m
...
```

It's the symlink from 3.7 to 3.7m that the `./pants` script picks out here to get the ball rolling:
https://github.com/pantsbuild/pants/blob/f6147cee973ba7bd50e26dfdca6f3110986ad609/build-support/common.sh#L55-L72

That leads to a pants_dev_deps venv with bin dir like you showed above:
```
$ cd $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin'
$ ls -l python*
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python3 -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd 18 Apr 16 00:08 python3.7 -> /usr/bin/python3.7
```

The critical entry is `python3.7` which is a symlink to `/usr/bin/python3.7` which is a symlink to `/usr/bin/python3.7m`. Pants then runs with `sys.executable` of `/home/cognifloyd/.cache/pants/pants_dev_deps/bin/python3.7`. Since that `sys.executable` is hosted in a venv, this resolve code does _not_ realpath and escape the venv (quite on purpose):
https://github.com/pantsbuild/pex/blob/dbd4c138f318619c00f5f1ffa4bb30bb3c585b80/pex/interpreter.py#L1059
https://github.com/pantsbuild/pex/blob/dbd4c138f318619c00f5f1ffa4bb30bb3c585b80/pex/interpreter.py#L584-L593

This leads to `pex.interpreter.binary` having a basename of `python3.7` - the `pants_dev_deps` canonical Python symlink out of the venv to the system interpreter symlink.

When that interpreter is used to build a venv via `/home/cognifloyd/.cache/pants/pants_dev_deps/bin/python3.7 -mvenv ...` you get a bin structure of:
```
python -> python3.7m
python3 -> python3.7m
python3.7m -> /usr/bin/python3.7m
```

IOW the `sys.executable` symlink is fully resolved by the `venv` module. Now we have the mismatch.

My machine also has a python3.7m (three of them, one system 3.7.15, one pyenv 3.7.14 and one pyenv 3.7.15), but _none_ of them has your symlink structure. They each have both `python3.7` and `python3.7m` as copies of the same binary. To repro all this i moved aside the pyenv 3.7.15 `python3.7` binary and made it be a symlink to the `python3.7m` sibling binary. I was then able to repro, and then test a fix.

I'll probably end up giving up on an IT for this though.
@cognifloyd is the machine your having this trouble on generic or personal? I.E.: is this just affecting you right now or also your colleagues? I ask, because - although its great to stamp out bugs like this - it's also a ton of debugging work for a fairly exotic system IIUC. I.E. it harkens to the Termux issue. If so, I'd just encourage you to state loudly and clearly up-front in these sorts of issue that that is the case if the issue is going to involve others helping out.

Resolving patch:
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -535,7 +535,6 @@ def _find_pyvenv_cfg(cls, maybe_venv_python_binary):
     @classmethod
     def _resolve_pyvenv_canonical_python_binary(
         cls,
-        real_binary,  # type: str
         maybe_venv_python_binary,  # type: str
     ):
         # type: (...) -> Optional[str]
@@ -581,16 +580,11 @@ def canonicalize_path(cls, path):
         N.B.: If the path is a venv symlink it will not be fully de-referenced in order to maintain
         fidelity with the requested venv Python binary choice.
         """
-        real_binary = os.path.realpath(path)
-
-        # If the path is a PEP-405 venv interpreter symlink we do not want to resolve outside of the
+        # If the path is a PEP-405 venv interpreter symlink we do not want to resolve outside the
         # venv in order to stay faithful to the binary path choice.
-        return (
-            cls._resolve_pyvenv_canonical_python_binary(
-                real_binary=real_binary, maybe_venv_python_binary=path
-            )
-            or real_binary
-        )
+        return cls._resolve_pyvenv_canonical_python_binary(
+            maybe_venv_python_binary=path
+        ) or os.path.realpath(path)
 
     class Error(Exception):
         pass
diff --git a/pex/pex_bootstrapper.py b/pex/pex_bootstrapper.py
--- a/pex/pex_bootstrapper.py
+++ b/pex/pex_bootstrapper.py
@@ -547,7 +547,10 @@ def ensure_venv(
                         pex,
                         bin_path=pex_info.venv_bin_path,
                         python=os.path.join(
-                            short_venv_dir, "venv", "bin", os.path.basename(pex.interpreter.binary)
+                            short_venv_dir,
+                            "venv",
+                            "bin",
+                            os.path.basename(virtualenv.interpreter.binary),
                         ),
                         collisions_ok=collisions_ok,
                         symlink=symlink,


Test patch: