Repository: pantsbuild/pants

Issue:
2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2
Hi all, 

I just upgraded from `2.3.0.dev3` to `2.3.0rc0` and running `./pants version` wouldn't finish without an error. I can successfully compile and test with `2.3.0.dev1` and `2.3.0.dev3` (didn't test dev2) - but rc0 fails before it finishes installing/setting up.

Note: It DOES work on Mac, but fails on Debian.

**Mac Environment** (works)
```bash
macOS Big Sur (11.2.1)

Python 3.7.9 (default, Jan 22 2021, 16:36:49) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Python 3.9.1 (default, Jan 22 2021, 16:49:02) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
```

**Linux Environment** (fails)
```bash
Debian GNU/Linux 10 (buster)
Linux 4.19.0-10-amd64

Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux

Python 3.9.1 (default, Feb 3 2021, 20:42:26)
[GCC 8.3.0] on linux
```

**Stacktrace:**

```bash
./pants version

10:33:48.67 [INFO] initializing pantsd...
10:33:49.60 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'

Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 154, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 124, in prepare
    self._initialize(options_fingerprint, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 99, in _initialize
    raise e
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 85, in _initialize
    options_bootstrapper, raise_=True
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 108, in build_config_and_options
    build_config = _initialize_build_configuration(self._plugin_resolver, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 43, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 109, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 130, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._interpreter_constraints])[
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 592, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'
```



Discussion:
Sorry for the trouble!

Would you mind attaching the contents of the `/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8` directory as a zip file? cc @jsirois 
Thanks @sureshjoshi! I encountered this yesterday when trying to upgrade a repository using Circle CI, but upgrading one with Travis worked. I haven't been able to isolate why it works in some places yet, and that's helpful to know you're hitting it on Debian.

I had been waiting to announce the release because of this. We'll get out a new release asap once we figure out what's going on.
@stuhood  Attached

[pex.zip](https://github.com/pantsbuild/pants/files/6037902/pex.zip)

pants.toml - created an empty project, no special pip resolvers (that fails too though), no other fields.

```bash
[GLOBAL]
pants_version = "2.3.0rc0"
```

@Eric-Arellano 
No worries, there is no particular reason for me to use rc0 specifically. 

I'm using 2.3.x because of the venv additions in the pex tool, but every now and again, I just upgrade to the latest dev versions to keep an eye on regressions that might burn me when I have less free time  :) 

`dev3` works perfectly fine for me - so just wanted to post this issue as a flag for anyone else who might run into it if they're doing what I am too.


@sureshjoshi / @Eric-Arellano the relevant file in the zip is the top level `pex` file. In order to debug this I'll either need:
1. You to manually try to run that file using the same python 3.7 interpreter Pants picked / edit, repeat to figure out the cause of the syntax error
2. More details on the vm / docker image. The exact Linux distribution and how you installed the relevant Python 3.7 if it wasn't pre-installed. This should allow me to repro and debug.
@jsirois 

I'll play around with option 1 in the evening when I have some time, not sure how useful I'll be though.

Re 2:

Parallels installed Debian (10.7) - system python 3.7.3, pyenv installed 3.9.1 (which I guess isn't used yet, so doesn't matter)

```bash
>> hostnamectl
   Static hostname: debian-gnu-linux-10-vm
         Icon name: computer-vm
           Chassis: vm
        Machine ID: d86f5054f02e804a80c10d6e62349e7f
           Boot ID: 101a0a3078bc4becb84aaccdeed49437
    Virtualization: kvm
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-10-amd64
      Architecture: x86-64

>> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

>> cat /etc/debian_version
10.7

>> apt list --installed | grep python
dh-python/stable,now 3.20190308 all [installed,automatic]
libpython-stdlib/stable,now 2.7.16-1 amd64 [installed,automatic]
libpython2-stdlib/stable,now 2.7.16-1 amd64 [installed,automatic]
libpython2.7-minimal/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython2.7-stdlib/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython2.7/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
libpython3-dev/stable,now 3.7.3-1 amd64 [installed,automatic]
libpython3-stdlib/stable,now 3.7.3-1 amd64 [installed,automatic]
libpython3.7-dev/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7-minimal/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7-stdlib/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
libpython3.7/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python-apt-common/stable,now 1.8.4.3 all [installed,automatic]
python-crypto/stable,now 2.6.1-9+b1 amd64 [installed,automatic]
python-dnspython/stable,now 1.16.0-1 all [installed,auto-removable]
python-gpg/stable,now 1.12.0-6 amd64 [installed,automatic]
python-ldb/stable,now 2:1.5.1+really1.4.6-3 amd64 [installed,automatic]
python-minimal/stable,now 2.7.16-1 amd64 [installed]
python-pip-whl/stable,now 18.1-5 all [installed,automatic]
python-samba/stable,now 2:4.9.5+dfsg-5+deb10u1 amd64 [installed,automatic]
python-talloc/stable,now 2.1.14-2 amd64 [installed,automatic]
python-tdb/stable,now 1.3.16-2+b1 amd64 [installed,automatic]
python2-minimal/stable,now 2.7.16-1 amd64 [installed,automatic]
python2.7-minimal/stable,now 2.7.16-2+deb10u1 amd64 [installed,automatic]
python2.7/stable,now 2.7.16-2+deb10u1 amd64 [installed]
python2/stable,now 2.7.16-1 amd64 [installed,automatic]
python3-apt/stable,now 1.8.4.3 amd64 [installed,automatic]
python3-asn1crypto/stable,now 0.24.0-1 all [installed,automatic]
python3-bcrypt/stable,now 3.1.6-1 amd64 [installed,auto-removable]
python3-brlapi/stable,now 5.6-10+deb10u1 amd64 [installed,automatic]
python3-cached-property/stable,now 1.5.1-3 all [installed,auto-removable]
python3-cairo/stable,now 1.16.2-1+b1 amd64 [installed,automatic]
python3-certifi/stable,now 2018.8.24-1 all [installed,automatic]
python3-cffi-backend/stable,now 1.12.2-1 amd64 [installed,automatic]
python3-chardet/stable,now 3.0.4-3 all [installed,automatic]
python3-crypto/stable,now 2.6.1-9+b1 amd64 [installed,automatic]
python3-cryptography/stable,now 2.6.1-3+deb10u2 amd64 [installed,automatic]
python3-cups/stable,now 1.9.73-2+b1 amd64 [installed,automatic]
python3-cupshelpers/stable,now 1.5.11-4 all [installed,automatic]
python3-dbus/stable,now 1.2.8-3 amd64 [installed,automatic]
python3-debconf/stable,now 1.5.71 all [installed,automatic]
python3-debian/stable,now 0.1.35 all [installed,automatic]
python3-debianbts/stable,now 2.8.2 all [installed,automatic]
python3-dev/stable,now 3.7.3-1 amd64 [installed,automatic]
python3-distro-info/stable,now 0.21 all [installed,automatic]
python3-distro/stable,now 1.3.0-1 all [installed,automatic]
python3-distutils/stable,now 3.7.3-1 all [installed,automatic]
python3-docker/stable,now 3.4.1-4 all [installed,auto-removable]
python3-dockerpty/stable,now 0.4.1-1 all [installed,auto-removable]
python3-dockerpycreds/stable,now 0.3.0-1 all [installed,auto-removable]
python3-docopt/stable,now 0.6.2-2 all [installed,auto-removable]
python3-entrypoints/stable,now 0.3-1 all [installed,automatic]
python3-gi-cairo/stable,now 3.30.4-1 amd64 [installed,automatic]
python3-gi/stable,now 3.30.4-1 amd64 [installed,automatic]
python3-gst-1.0/stable,now 1.14.4-1+b1 amd64 [installed,automatic]
python3-httplib2/stable,now 0.11.3-2 all [installed,automatic]
python3-idna/stable,now 2.6-1 all [installed,automatic]
python3-jinja2/stable,now 2.10-2 all [installed,auto-removable]
python3-jmespath/stable,now 0.9.4-1 all [installed,auto-removable]
python3-jsonschema/stable,now 2.6.0-4 all [installed,auto-removable]
python3-kerberos/stable,now 1.1.14-2 amd64 [installed,auto-removable]
python3-keyring/stable,now 17.1.1-1 all [installed,automatic]
python3-keyrings.alt/stable,now 3.1.1-1 all [installed,automatic]
python3-lib2to3/stable,now 3.7.3-1 all [installed,automatic]
python3-libcloud/stable,now 2.4.0-1 all [installed,auto-removable]
python3-lockfile/stable,now 1:0.12.2-2 all [installed,auto-removable]
python3-louis/stable,now 3.8.0-2 all [installed,automatic]
python3-mako/stable,now 1.0.7+ds1-1 all [installed,automatic]
python3-markupsafe/stable,now 1.1.0-1 amd64 [installed,automatic]
python3-minimal/stable,now 3.7.3-1 amd64 [installed,automatic]
python3-nacl/stable,now 1.3.0-2 amd64 [installed,auto-removable]
python3-netaddr/stable,now 0.7.19-1 all [installed,auto-removable]
python3-ntlm-auth/stable,now 1.1.0-1 all [installed,auto-removable]
python3-paramiko/stable,now 2.4.2-0.1 all [installed,auto-removable]
python3-pip/stable,now 18.1-5 all [installed]
python3-pkg-resources/stable,now 40.8.0-1 all [installed,automatic]
python3-pyasn1/stable,now 0.4.2-3 all [installed,auto-removable]
python3-pyatspi/stable,now 2.30.0+dfsg-3 all [installed,automatic]
python3-pycurl/stable,now 7.43.0.2-0.1 amd64 [installed,automatic]
python3-pysimplesoap/stable,now 1.16.2-1 all [installed,automatic]
python3-reportbug/stable,now 7.5.3~deb10u1 all [installed]
python3-requests-kerberos/stable,now 0.11.0-2 all [installed,auto-removable]
python3-requests-ntlm/stable,now 1.1.0-1 all [installed,auto-removable]
python3-requests/stable,now 2.21.0-1 all [installed,automatic]
python3-secretstorage/stable,now 2.3.1-2 all [installed,automatic]
python3-selinux/stable,now 2.8-1+b1 amd64 [installed,auto-removable]
python3-setuptools/stable,now 40.8.0-1 all [installed,automatic]
python3-simplejson/stable,now 3.16.0-1 amd64 [installed,auto-removable]
python3-six/stable,now 1.12.0-1 all [installed,automatic]
python3-smbc/stable,now 1.0.15.6-1+b2 amd64 [installed,automatic]
python3-software-properties/stable,now 0.96.20.2-2 all [installed,automatic]
python3-speechd/stable,now 0.9.0-5+deb10u1 all [installed,automatic]
python3-texttable/stable,now 1.6.0-1 all [installed,auto-removable]
python3-uno/stable,now 1:6.1.5-3+deb10u6 amd64 [installed,automatic]
python3-urllib3/stable,now 1.24.1-1 all [installed,automatic]
python3-venv/stable,now 3.7.3-1 amd64 [installed]
python3-websocket/stable,now 0.53.0-1 all [installed,auto-removable]
python3-wheel/stable,now 0.32.3-2 all [installed,automatic]
python3-winrm/stable,now 0.3.0-2 all [installed,auto-removable]
python3-xdg/stable,now 0.25-5 all [installed,automatic]
python3-xmltodict/stable,now 0.11.0-2 all [installed,auto-removable]
python3-yaml/stable,now 3.13-2 amd64 [installed,auto-removable]
python3.7-dev/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7-minimal/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7-venv/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3.7/stable,now 3.7.3-2+deb10u2 amd64 [installed,automatic]
python3/stable,now 3.7.3-1 amd64 [installed]
python/stable,now 2.7.16-1 amd64 [installed]
unit-python3.7/stable,now 1.21.0-1~buster amd64 [installed,upgradable to: 1.22.0-1~buster]


```

I don't repro:
```
$ docker run --rm -it -v $HOME/Downloads:/downloads debian:10.7 bash -c '
apt -qq update && apt -qq install --yes python3.7-dev python3.7-venv unzip
unzip -qd triage /downloads/pex.zip
triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python --version
triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex -c "import os, site; print(os.linesep.join(site.getsitepackages()))"
'
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
The following additional packages will be installed:
  bzip2 ca-certificates file libc-dev-bin libc6-dev libexpat1 libexpat1-dev libmagic-mgc libmagic1 libmpdec2 libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib libreadline7 libsqlite3-0 libssl1.1 linux-libc-dev manpages manpages-dev mime-support openssl
  python-pip-whl python3.7 python3.7-minimal readline-common xz-utils
Suggested packages:
  bzip2-doc glibc-doc man-browser python3.7-doc binutils binfmt-support readline-doc zip
The following NEW packages will be installed:
  bzip2 ca-certificates file libc-dev-bin libc6-dev libexpat1 libexpat1-dev libmagic-mgc libmagic1 libmpdec2 libpython3.7 libpython3.7-dev libpython3.7-minimal libpython3.7-stdlib libreadline7 libsqlite3-0 libssl1.1 linux-libc-dev manpages manpages-dev mime-support openssl
  python-pip-whl python3.7 python3.7-dev python3.7-minimal python3.7-venv readline-common unzip xz-utils
0 upgraded, 30 newly installed, 0 to remove and 7 not upgraded.
Need to get 68.9 MB of archives.
After this operation, 161 MB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libssl1.1:amd64.
(Reading database ... 6677 files and directories currently installed.)
Preparing to unpack .../00-libssl1.1_1.1.1d-0+deb10u5_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1d-0+deb10u5) ...
Selecting previously unselected package libpython3.7-minimal:amd64.
Preparing to unpack .../01-libpython3.7-minimal_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-minimal:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../02-libexpat1_2.2.6-2+deb10u1_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.6-2+deb10u1) ...
Selecting previously unselected package python3.7-minimal.
Preparing to unpack .../03-python3.7-minimal_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-minimal (3.7.3-2+deb10u2) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../04-readline-common_7.0-5_all.deb ...
Unpacking readline-common (7.0-5) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../05-bzip2_1.0.6-9.2~deb10u1_amd64.deb ...
Unpacking bzip2 (1.0.6-9.2~deb10u1) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../06-libmagic-mgc_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic-mgc (1:5.35-4+deb10u2) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../07-libmagic1_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.35-4+deb10u2) ...
Selecting previously unselected package file.
Preparing to unpack .../08-file_1%3a5.35-4+deb10u2_amd64.deb ...
Unpacking file (1:5.35-4+deb10u2) ...
Selecting previously unselected package manpages.
Preparing to unpack .../09-manpages_4.16-2_all.deb ...
Unpacking manpages (4.16-2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../10-mime-support_3.62_all.deb ...
Unpacking mime-support (3.62) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../11-xz-utils_5.2.4-1_amd64.deb ...
Unpacking xz-utils (5.2.4-1) ...
Selecting previously unselected package openssl.
Preparing to unpack .../12-openssl_1.1.1d-0+deb10u5_amd64.deb ...
Unpacking openssl (1.1.1d-0+deb10u5) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../13-ca-certificates_20200601~deb10u2_all.deb ...
Unpacking ca-certificates (20200601~deb10u2) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../14-libc-dev-bin_2.28-10_amd64.deb ...
Unpacking libc-dev-bin (2.28-10) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../15-linux-libc-dev_4.19.171-2_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.19.171-2) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../16-libc6-dev_2.28-10_amd64.deb ...
Unpacking libc6-dev:amd64 (2.28-10) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../17-libexpat1-dev_2.2.6-2+deb10u1_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.6-2+deb10u1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../18-libmpdec2_2.4.2-2_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-2) ...
Selecting previously unselected package libreadline7:amd64.
Preparing to unpack .../19-libreadline7_7.0-5_amd64.deb ...
Unpacking libreadline7:amd64 (7.0-5) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../20-libsqlite3-0_3.27.2-3+deb10u1_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Selecting previously unselected package libpython3.7-stdlib:amd64.
Preparing to unpack .../21-libpython3.7-stdlib_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-stdlib:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libpython3.7:amd64.
Preparing to unpack .../22-libpython3.7_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package libpython3.7-dev:amd64.
Preparing to unpack .../23-libpython3.7-dev_3.7.3-2+deb10u2_amd64.deb ...
Unpacking libpython3.7-dev:amd64 (3.7.3-2+deb10u2) ...
Selecting previously unselected package manpages-dev.
Preparing to unpack .../24-manpages-dev_4.16-2_all.deb ...
Unpacking manpages-dev (4.16-2) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../25-python-pip-whl_18.1-5_all.deb ...
Unpacking python-pip-whl (18.1-5) ...
Selecting previously unselected package python3.7.
Preparing to unpack .../26-python3.7_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7 (3.7.3-2+deb10u2) ...
Selecting previously unselected package python3.7-dev.
Preparing to unpack .../27-python3.7-dev_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-dev (3.7.3-2+deb10u2) ...
Selecting previously unselected package python3.7-venv.
Preparing to unpack .../28-python3.7-venv_3.7.3-2+deb10u2_amd64.deb ...
Unpacking python3.7-venv (3.7.3-2+deb10u2) ...
Selecting previously unselected package unzip.
Preparing to unpack .../29-unzip_6.0-23+deb10u2_amd64.deb ...
Unpacking unzip (6.0-23+deb10u2) ...
Setting up libexpat1:amd64 (2.2.6-2+deb10u1) ...
Setting up mime-support (3.62) ...
Setting up libmagic-mgc (1:5.35-4+deb10u2) ...
Setting up manpages (4.16-2) ...
Setting up libssl1.1:amd64 (1.1.1d-0+deb10u5) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.
28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up unzip (6.0-23+deb10u2) ...
Setting up libsqlite3-0:amd64 (3.27.2-3+deb10u1) ...
Setting up libmagic1:amd64 (1:5.35-4+deb10u2) ...
Setting up linux-libc-dev:amd64 (4.19.171-2) ...
Setting up file (1:5.35-4+deb10u2) ...
Setting up bzip2 (1.0.6-9.2~deb10u1) ...
Setting up libpython3.7-minimal:amd64 (3.7.3-2+deb10u2) ...
Setting up xz-utils (5.2.4-1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up python3.7-minimal (3.7.3-2+deb10u2) ...
Setting up libmpdec2:amd64 (2.4.2-2) ...
Setting up libc-dev-bin (2.28-10) ...
Setting up openssl (1.1.1d-0+deb10u5) ...
Setting up readline-common (7.0-5) ...
Setting up libreadline7:amd64 (7.0-5) ...
Setting up manpages-dev (4.16-2) ...
Setting up libpython3.7-stdlib:amd64 (3.7.3-2+deb10u2) ...
Setting up libpython3.7:amd64 (3.7.3-2+deb10u2) ...
Setting up ca-certificates (20200601~deb10u2) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.
28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Updating certificates in /etc/ssl/certs...
137 added, 0 removed; done.
Setting up libc6-dev:amd64 (2.28-10) ...
Setting up python-pip-whl (18.1-5) ...
Setting up python3.7 (3.7.3-2+deb10u2) ...
Setting up python3.7-venv (3.7.3-2+deb10u2) ...
Setting up libexpat1-dev:amd64 (2.2.6-2+deb10u1) ...
Setting up libpython3.7-dev:amd64 (3.7.3-2+deb10u2) ...
Setting up python3.7-dev (3.7.3-2+deb10u2) ...
Processing triggers for libc-bin (2.28-10) ...
Processing triggers for ca-certificates (20200601~deb10u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Python 3.7.3
Re-execing from /triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/site-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/local/lib/python3.7/dist-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3/dist-packages
/triage/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/dist-packages
```

Is there anything odd with the parallels setup @sureshjoshi ? In particular any mounts that leak out into macOS land or something like that?
So strange. 

My VM is isolated, and works fine on `2.3.0.dev1` and `2.3.0.dev3` as mentioned above, specifically `rc0` giving me a hassle. I've tried wiping all the caches I can think of as well. 

I'll try creating a fresh VM and seeing what happens there in the eve - but as far as I can imagine, not sure why the other two dev versions would work.
Ah, so in the never ending - read closer dummy department - I read closer and got less dumb: "syntax error near unexpected token `('" - That's bash speaking and not Python AFAICT. Ruminating on that...
As to the other two dev versions @sureshjoshi , plugin resolution - where this is failing - is new between the last dev release and rc0.
Ah, I was wondering if it was actually something between pex 2.1.30 and 2.1.31 - because I vaguely recall messing with something the other day where I was changing pex versions (in a docker container somewhere, unrelated to this problem) and seeing something similar. 

But, sounds like we're far outside my wheelhouse. I'm available to run tests on my environ as needed
@jsirois Weird question... Could we be exceeding the shebang line limit?

It's 155 characters in my example, I put a symlink to my home dir, used that instead, and running ./pex now works

Symlink:
```bash
ln -s /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 ~/pppython
```

In `__main__.py`:

From:
```bash
#!/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 -sE
```

To:
```bash
#! /home/parallels/pppython -sE
```

`./pants version` succeeds
I'm ignorant of shebang length limits. I do know ecrpytfs encrypted directories have ~140 character limits for ... path components IIRC. But do you have that setup for home dirs even? And then, even if so, I'm pretty sure that's a path component (file name) length limit, not a total path length limit.

I'll poke around though to learn about any shebang specific restrictions...
Yeah - no dice on straight up debian 10.7 with no encrypted home dir:
```
$ docker run --rm -it -v $HOME/Downloads:/downloads debian:10.7 bash -c '
apt -qq update && apt -qq install --yes python3.7-dev python3.7-venv unzip
mkdir -p /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943
unzip -qd /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943 /downloads/pex.zip
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python --version
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex -c "import os, site; print(os.linesep.join(site.getsitepackages()))"
'
...
Python 3.7.3
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/site-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/local/lib/python3.7/dist-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3/dist-packages
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/lib/python3.7/dist-packages
```
FWIW: This shebang thing seems to be a known Python issue after looking for it

https://www.in-ulm.de/~mascheck/various/shebang/
https://github.com/pypa/pip/issues/1773
https://github.com/pypa/virtualenv/issues/596
https://github.com/pypa/virtualenv/issues/997

It also looks like the latest pex re-wrote a lot of code around this functionality with the venvs `__main__.py` file. Not sure if it's related - since the shebang at the top is the same, but there seems to be code to re-exec if python isn't in use: https://github.com/pantsbuild/pex/compare/v2.1.30...v2.1.31#diff-44e8fd52cd3a63a2d3f2fc516418c2aa55e8ee96f6eb82b2186765a8b853922f

Is there some way that code be added to create a pex_root symlink to the appropriate venv? That could chomp the length in half probably. 

As I'm not familiar with the codebase, I'm not really sure why the venvs folders are so nested (some pex hashes I guess along the way) - so I don't know if there are expected to be multiple venvs in the same folders which would necessitate some clever symlink naming in my example (which could just be a pain to do and maintain).   

What really sucks is that this appears to be an OS-specific version, and if the Linux distro in use increases the specified length, this is basically a non-issue.

Ah yes. Thanks. Not a known Python issue - just a known shebang issue regardless of interpreter. I'll fork a Pex issue, fix over there and upgrade Pants, etc.

Thanks again for digging with me @sureshjoshi .
And to stomp out the last variable: This explain _exactly_ why my docker repro didn't repro since that uses the host kernel which is my Arch Linux kernel:
```
$ grep BINPRM_BUF_SIZE /usr/include/linux/binfmts.h 
#define BINPRM_BUF_SIZE 256
```
https://github.com/pantsbuild/pex/issues/1252

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,7 +37,7 @@ class PexBinary(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.31"
+    default_version = "v2.1.32"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
 
     @classproperty
@@ -47,8 +47,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "e829fc2e14b501e6f158b536dad3b93cc00affa6e41b71c78c363d9f0af31ce4",
-                    "3069767",
+                    "9cea86ae4338943ab347a5e2b3a71e3c27b9835cdd5c115c52d1a328fdb61160",
+                    "3071734",
                 )
             )
             for plat in ["darwin", "linux"]


Test patch: