Repository: pex-tool/pex

Issue:
pex vendoring does not (always) isolate itself from a colliding setuptools install in site-packages
Having issues creating pex files in an image based on `databricksruntime/python-virtualenv`.

Dockerfile:
```
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update \
  && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential

# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
```

Command run in container:

`pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3`


Portion of failure StackTrace:

```
Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-kh23prp9/setup.py", line 59, in <module>
      'Programming Language :: Python :: 3.8',
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/__init__.py", line 169, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/wheel/wheel/bdist_wheel.py", line 228, in run
      self.run_command('install')
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install.py", line 65, in run
      return orig.install.run(self)
    File "/usr/lib/python3.7/distutils/command/install.py", line 601, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install_scripts.py", line 22, in run
      import setuptools.command.easy_install as ei  # vendor:skip
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/easy_install.py", line 69, in <module>
      from setuptools.sandbox import run_setup  # vendor:skip
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/sandbox.py", line 24, in <module>
      import pkg_resources.py31compat  # vendor:skip
  ModuleNotFoundError: No module named 'pkg_resources.py31compat'
  ----------------------------------------
  ERROR: Failed building wheel for boto3
```


Discussion:
As requested: @stuhood 
Thanks! I repro the issue, with or without `setuptools` explicitly installed alongside `pex`. Investigating.
According to https://github.com/pypa/setuptools/issues/1167, this is likely a collision between multiple `setuptools` installs, one of which is vendored inside pex, and the other of which comes from your sitepackages due to `--system-site-packages`.

So I suspect that this might be bug in the vendoring that `pex` does that is allowing an external copy of `setuptools` to collide with the vendored copy.

Two workarounds:
1. Remove the `--system-site-packages` flag and rely on explicitly pip installing the things you need in the virtualenv
2. run pex _from a_ pex, which will allow pex to isolate itself rather than relying on virtualenv to do so:
```
$ pex pex -o pex.pex -m pex
$ ./pex.pex --interpreter-constraint='CPython>=3.7' boto3
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
```
This backtrace is in a Pex execution of Pip. There, Pex dogfoods itself to run a Pip PEX in isolation - with the exception of setuptools and wheel whose vendored versions are deliberately exposed so Pip can execute `setup.py`s as needed:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L53

So that's where to start spelunking.
I'll dig on this a bit.
Getting back to the conversation above, I wanted to correct the record on:
> 2. run pex from a pex, which will allow pex to isolate itself rather than relying on virtualenv to do so:

Pex isolates itself whether run from a pex file or not. It does this by:
1. Shading all its dependencies, so it imports pex.third_party.setuptools and not setuptools and so cannot pick this up from the system.
2. Wraps its internal executions of python interpreter subprocesses in PEXes to provide isolation to those subprocesses.

My answer above about Pip PEX dogfood concerns point 2.
FWIW, I tested that both 1) and 2) from my answer resolved the issue in the repro. I might have rushed things though.
Aha, ok. That's not making sense but good to know.

That said, I don't repro:
```
$ cat <<EOF | docker build -
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential

# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
EOF
Sending build context to Docker daemon  3.072kB
Step 1/9 : FROM databricksruntime/python-virtualenv:latest
 ---> 6d14d1c4da8d
Step 2/9 : RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
 ---> Running in 085a999e997e
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
...
Processing triggers for dbus (1.10.6-1ubuntu3.3) ...
Removing intermediate container 085a999e997e
 ---> b39aead9ebae
Step 3/9 : RUN add-apt-repository ppa:deadsnakes/ppa
 ---> Running in a985ceb3f4b6
 This PPA contains more recent Python versions packaged for Ubuntu.
...
gpg: key 6A755776: public key "Launchpad PPA for deadsnakes" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
Removing intermediate container a985ceb3f4b6
 ---> 654f182d6ef2
Step 4/9 : RUN apt update
 ---> Running in 25d3cbe0c6d1
...
76 packages can be upgraded. Run 'apt list --upgradable' to see them.
Removing intermediate container 25d3cbe0c6d1
 ---> 31e7a456c83b
Step 5/9 : RUN apt install --yes python3.7 python3.7-dev
 ---> Running in e7601428f9c3
...
Setting up python3.7-dev (3.7.9-1+xenial1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Removing intermediate container e7601428f9c3
 ---> 2a03e6c57939
Step 6/9 : RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
 ---> Running in 8bfcdf409659
Using base prefix '/usr'
New python executable in /databricks/python3/bin/python3.7
...
Running virtualenv with interpreter /usr/bin/python3.7
Removing intermediate container 8bfcdf409659
 ---> 2e2296a1ba42
Step 7/9 : RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
 ---> Running in bee547a895cb
Collecting pex
  Downloading pex-2.1.16-py2.py3-none-any.whl (2.3 MB)
...
Successfully installed pex-2.1.16 setuptools-45.2.0
Removing intermediate container bee547a895cb
 ---> 320f252b829d
Step 8/9 : ENV PATH="/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin:/home/jsirois/.bash.d/bin:/databricks/python3/bin"
 ---> Running in 8887cb7aa912
Removing intermediate container 8887cb7aa912
 ---> ea3611e65fa8
Step 9/9 : RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
 ---> Running in e9ff82a8f6ff
Removing intermediate container e9ff82a8f6ff
 ---> 22d0b02808bd
Successfully built 22d0b02808bd
```

And the pex seems ok:
```
$ docker run --rm -it -ePEX_INTERPRETER=1 22d0b02808bd ./boto3.pex
Python 3.7.9 (default, Aug 18 2020, 06:24:24) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import boto3
>>> 
now exiting InteractiveConsole...
```

@jperkelens or @stuhood does the original bug report still repro for you? Probably most importantly, does it repro after appropriate `docker image prune`ing (fresh image builds)?
@jsirois : Yea, I repro with the exact commands from the description via:
```bash
docker image prune
docker run \
  -v "$(pwd):/workdir" \
  -w '/workdir' \
  --rm \
  -it \
  "databricksruntime/python-virtualenv:latest" \
  /bin/bash

# ... run each of the commands from the description
```

The docker-image-prune reports zero bytes pruned though, so I might be on a different copy of the `databricksruntime/python-virtualenv:latest` tag...? My docker foo is weak.

----

EDIT: @jsirois : but also, FWIW, @jperkelens was able to resolve his issue by removing the `--system-site-packages` flag, so he is no longer blocked on this.
If docker output says ` ---> Using cache`, the step is cached:
```
Step 2/9 : RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
 ---> Using cache
 ---> b39aead9ebae
```
> a different copy of the `databricksruntime/python-virtualenv:latest` tag

They have not changed the image for 13 months, so that's unlikely it, but the image incluides a `RUN apt update` which can pick up new packages. So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".

> but also, FWIW, @jperkelens was able to resolve his issue by removing the --system-site-packages flag, so he is no longer blocked on this.

Yeah - thanks. I'd still like to squash the issue if possible.

> So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".

It wasn't... all of the steps ran. Probably (at least) because I was actually copy pasting them into a live shell one by one, rather than running them via a Dockerfile.
Hrm. Not being able to repro is ... maddening. If either of you have time could you run this on your image substituting your image id for `22d0b02808bd`?:
```
$ docker run --rm -it 22d0b02808bd apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,upgradable to: 2.20151103ubuntu1.2]
libpython3-stdlib/xenial,now 3.5.1-3 amd64 [installed,automatic]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-stdlib/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
python-apt-common/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 all [installed,automatic]
python-pip-whl/xenial-updates,now 8.1.1-2ubuntu0.4 all [installed,automatic]
python3/xenial,now 3.5.1-3 amd64 [installed]
python3-apt/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 amd64 [installed,automatic]
python3-dbus/xenial,now 1.2.0-3 amd64 [installed,automatic]
python3-gi/xenial,now 3.20.0-0ubuntu1 amd64 [installed,automatic]
python3-minimal/xenial,now 3.5.1-3 amd64 [installed,automatic]
python3-pkg-resources/xenial,now 20.7.0-1 all [installed,automatic]
python3-pycurl/xenial,now 7.43.0-1ubuntu1 amd64 [installed,automatic]
python3-software-properties/xenial-updates,xenial-security,now 0.96.20.10 all [installed,automatic]
python3-virtualenv/xenial-updates,now 15.0.1+ds-3ubuntu1 all [installed,automatic]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-distutils/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-lib2to3/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
```
```
$ docker image ls databricksruntime/python-virtualenv:latest
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
databricksruntime/python-virtualenv   latest              6d14d1c4da8d        18 months ago       778MB
```

As mentioned above, I have not been using a `Dockerfile`: rather, using `docker run` followed by pasting each command, so the `apt-list` bit is maybe not interesting:
```
$ docker run --rm -it 6d14d1c4da8d apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,local]
libpython3-stdlib/now 3.5.1-3 amd64 [installed,local]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python-pip-whl/now 8.1.1-2ubuntu0.4 all [installed,local]
python3/now 3.5.1-3 amd64 [installed,local]
python3-minimal/now 3.5.1-3 amd64 [installed,local]
python3-pkg-resources/now 20.7.0-1 all [installed,local]
python3-virtualenv/now 15.0.1+ds-3ubuntu1 all [installed,local]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
```
Thanks - the means by which the command are run in the image - by hand or by Dockefile RUN, are immaterial, so the listing is useful. But now I see you ran this against the base image and not the commands you subsequently ran - gotcha. Yeah, not so useful. Thanks - I'll try to debug this by other means.
OK, so focusing on `--system-site-packages` (thanks @stuhood), virtualenv implements this via a custom `site.py`. That `site.py` calculates the underlying system site packages when the venv interpreter starts up and adds all those paths to `sys.path` via direct `sys.path` manipulation. As such, there is no way for the Pex runtime to detect these `sys.path`s should be scrubbed without adding a hunk of code that knows an awful lot about virtualenv's implementation details. In short, this means Pex cannot trust `sys.path`. It currently does when running Pip PEXed internally:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L54

I can instead construct PYTHONPATH from the paths of the vendored setuptools and wheel directly, the main difference being this will pre-pend those paths to the sys.path of spawned Pip subprocesses. Right now they come in the order they are in sys.path which may put `--system-site-packages` before the pexed up setuptools and wheel packages.
Noting the abive fix does not solve the general hermeticity problem here, just the specific problem for the Pip PEX used at PEX buildtime.

For example:
```
root@4185a16f2609:/# pex -v setuptools
pex: Building pex :: Resolving distributions (['setuptools']) :: Resolving for:
...
pex: Activating PEX virtual environment from /tmp/tmp4xggexvl: 28.6ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /tmp/tmp4xggexvl
pex:   * /databricks/python3/lib/python37.zip
pex:     /databricks/python3/lib/python3.7
pex:     /databricks/python3/lib/python3.7/lib-dynload
pex:     /usr/lib/python3.7
pex:     /usr/local/lib/python3.7/dist-packages
pex:     /usr/lib/python3/dist-packages
pex:     /tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl
pex:     /tmp/tmp4xggexvl/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Python 3.7.9 (default, Aug 18 2020, 06:24:24) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
```
Here the `--system-site-packages` added by virtualenv to `sys.path` directly in `site.py`and unscrubbed by Pex are:
+ `pex:     /usr/local/lib/python3.7/dist-packages`
+ `pex:     /usr/lib/python3/dist-packages`
Besides not being scrubbed, these come before `/tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl` in the sys.path.

In this particular builtime case, we're burned by `pkg_resources` (Note `pkg_resources` should come from `setuptools` but there is no `setuptools` package in this listing! - odd but not germain):
```
root@4185a16f2609:/# ls -1 /usr/lib/python3/dist-packages
__pycache__
_dbus_bindings.cpython-35m-x86_64-linux-gnu.so
_dbus_glib_bindings.cpython-35m-x86_64-linux-gnu.so
apt
apt_inst.cpython-35m-x86_64-linux-gnu.so
apt_pkg.cpython-35m-x86_64-linux-gnu.so
aptsources
curl
dbus
debconf.py
gi
lsb_release.py
pkg_resources
pycurl-7.43.0.egg-info
pycurl.cpython-35m-x86_64-linux-gnu.so
pygobject-3.20.0.egg-info
pygtkcompat
python_apt-1.1.0.b1_ubuntu0.16.04.9.egg-info
softwareproperties
unattended_upgrades-0.1.egg-info
virtualenv-15.0.1.egg-info
virtualenv.py
```
It looks like one way to solve this is to invert how we calculate which `sys.path` entries to scrub. Instead of calculating site-libraries, user-site-libraries and extras to scrub:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pex.py#L309-L315
We could calculate just the core libraries and scrub everything else via:
```
python -sSE -c 'import sys; print("\n".join(sys.path))'
```
That solves the case above and seems as robust as you can possibly be since the answer is delegated to the python interpreter in question.
Checking back in here. Although I still cannot repro the failure you get using the commands above I can repro the leak by inspection of sys.path in a PEX built in the databaricks image. I've done some work over in #1128 that makes Pex both aware of virtual environments and equips it with the capability to bust out of a virtual environment. I'll use that to fix this. The approach above using ~ `python -sSE -c 'import sys; print("\n".join(sys.path))'` was not robust across all interpreter installations (those that include bits of the stdlib in ~site-packages :/).

Resolving patch:
diff --git a/pex/interpreter_constraints.py b/pex/interpreter_constraints.py
--- a/pex/interpreter_constraints.py
+++ b/pex/interpreter_constraints.py
@@ -5,8 +5,6 @@
 
 from __future__ import absolute_import
 
-import os
-
 from pex.common import die
 from pex.interpreter import PythonIdentity, PythonInterpreter
 from pex.typing import TYPE_CHECKING
@@ -37,17 +35,27 @@ def __init__(
         constraints,  # type: Iterable[str]
         candidates,  # type: Iterable[PythonInterpreter]
         failures,  # type: Iterable[InterpreterIdentificationError]
+        preamble=None,  # type: Optional[str]
     ):
         # type: (...) -> None
         """
         :param constraints: The constraints that could not be satisfied.
         :param candidates: The python interpreters that were compared against the constraints.
         :param failures: Descriptions of the python interpreters that were unidentifiable.
+        :param preamble: An optional preamble for the exception message.
         """
         self.constraints = tuple(constraints)
         self.candidates = tuple(candidates)
         self.failures = tuple(failures)
-        super(UnsatisfiableInterpreterConstraintsError, self).__init__(self.create_message())
+        super(UnsatisfiableInterpreterConstraintsError, self).__init__(
+            self.create_message(preamble=preamble)
+        )
+
+    def with_preamble(self, preamble):
+        # type: (str) -> UnsatisfiableInterpreterConstraintsError
+        return UnsatisfiableInterpreterConstraintsError(
+            self.constraints, self.candidates, self.failures, preamble=preamble
+        )
 
     def create_message(self, preamble=None):
         # type: (Optional[str]) -> str
diff --git a/pex/pex_bootstrapper.py b/pex/pex_bootstrapper.py
--- a/pex/pex_bootstrapper.py
+++ b/pex/pex_bootstrapper.py
@@ -8,6 +8,7 @@
 
 from pex import pex_warnings
 from pex.common import die
+from pex.inherit_path import InheritPath
 from pex.interpreter import PythonInterpreter
 from pex.interpreter_constraints import UnsatisfiableInterpreterConstraintsError
 from pex.orderedset import OrderedSet
@@ -157,13 +158,13 @@ def _valid_interpreter(interp_or_error):
 def _select_path_interpreter(
     path=None,  # type: Optional[str]
     valid_basenames=None,  # type: Optional[Tuple[str, ...]]
-    compatibility_constraints=None,  # type: Optional[Iterable[str]]
+    interpreter_constraints=None,  # type: Optional[Iterable[str]]
 ):
     # type: (...) -> Optional[PythonInterpreter]
     candidate_interpreters_iter = iter_compatible_interpreters(
         path=path,
         valid_basenames=valid_basenames,
-        interpreter_constraints=compatibility_constraints,
+        interpreter_constraints=interpreter_constraints,
     )
     current_interpreter = PythonInterpreter.get()  # type: PythonInterpreter
     candidate_interpreters = []
@@ -182,46 +183,10 @@ def _select_path_interpreter(
     return PythonInterpreter.latest_release_of_min_compatible_version(candidate_interpreters)
 
 
-def maybe_reexec_pex(compatibility_constraints=None):
-    # type: (Optional[Iterable[str]]) -> Union[None, NoReturn]
-    """Handle environment overrides for the Python interpreter to use when executing this pex.
-
-    This function supports interpreter filtering based on interpreter constraints stored in PEX-INFO
-    metadata. If PEX_PYTHON is set it attempts to obtain the binary location of the interpreter
-    specified by PEX_PYTHON. If PEX_PYTHON_PATH is set, it attempts to search the path for a matching
-    interpreter in accordance with the interpreter constraints. If both variables are present, this
-    function gives precedence to PEX_PYTHON_PATH and errors out if no compatible interpreters can be
-    found on said path.
-
-    If neither variable is set, we fall back to plain PEX execution using PATH searching or the
-    currently executing interpreter. If compatibility constraints are used, we match those constraints
-    against these interpreters.
-
-    :param compatibility_constraints: optional list of requirements-style strings that constrain the
-                                      Python interpreter to re-exec this pex with.
-    """
-
+def find_compatible_interpreter(interpreter_constraints=None):
+    # type: (Optional[Iterable[str]]) -> PythonInterpreter
     current_interpreter = PythonInterpreter.get()
-    target = None  # type: Optional[PythonInterpreter]
-
-    # NB: Used only for tests.
-    if "_PEX_EXEC_CHAIN" in os.environ:
-        flag_or_chain = os.environ.pop("_PEX_EXEC_CHAIN")
-        pex_exec_chain = [] if flag_or_chain == "1" else flag_or_chain.split(os.pathsep)
-        pex_exec_chain.append(current_interpreter.binary)
-        os.environ["_PEX_EXEC_CHAIN"] = os.pathsep.join(pex_exec_chain)
-
-    current_interpreter_blessed_env_var = "_PEX_SHOULD_EXIT_BOOTSTRAP_REEXEC"
-    if os.environ.pop(current_interpreter_blessed_env_var, None):
-        # We've already been here and selected an interpreter. Continue to execution.
-        return None
-
-    from . import pex
-
-    pythonpath = pex.PEX.stash_pythonpath()
-    if pythonpath is not None:
-        TRACER.log("Stashed PYTHONPATH of {}".format(pythonpath), V=2)
-
+    target = current_interpreter  # type: Optional[PythonInterpreter]
     with TRACER.timed("Selecting runtime interpreter", V=3):
         if ENV.PEX_PYTHON and not ENV.PEX_PYTHON_PATH:
             # preserve PEX_PYTHON re-exec for backwards compatibility
@@ -229,7 +194,7 @@ def maybe_reexec_pex(compatibility_constraints=None):
             # https://github.com/pantsbuild/pex/issues/431
             TRACER.log(
                 "Using PEX_PYTHON={} constrained by {}".format(
-                    ENV.PEX_PYTHON, compatibility_constraints
+                    ENV.PEX_PYTHON, interpreter_constraints
                 ),
                 V=3,
             )
@@ -237,78 +202,124 @@ def maybe_reexec_pex(compatibility_constraints=None):
                 if os.path.isabs(ENV.PEX_PYTHON):
                     target = _select_path_interpreter(
                         path=ENV.PEX_PYTHON,
-                        compatibility_constraints=compatibility_constraints,
+                        interpreter_constraints=interpreter_constraints,
                     )
                 else:
                     target = _select_path_interpreter(
                         valid_basenames=(os.path.basename(ENV.PEX_PYTHON),),
-                        compatibility_constraints=compatibility_constraints,
+                        interpreter_constraints=interpreter_constraints,
                     )
             except UnsatisfiableInterpreterConstraintsError as e:
-                die(
-                    e.create_message(
-                        "Failed to find a compatible PEX_PYTHON={pex_python}.".format(
-                            pex_python=ENV.PEX_PYTHON
-                        )
+                raise e.with_preamble(
+                    "Failed to find a compatible PEX_PYTHON={pex_python}.".format(
+                        pex_python=ENV.PEX_PYTHON
                     )
                 )
-        elif ENV.PEX_PYTHON_PATH or compatibility_constraints:
+        elif ENV.PEX_PYTHON_PATH or interpreter_constraints:
             TRACER.log(
                 "Using {path} constrained by {constraints}".format(
                     path="PEX_PYTHON_PATH={}".format(ENV.PEX_PYTHON_PATH)
                     if ENV.PEX_PYTHON_PATH
                     else "$PATH",
-                    constraints=compatibility_constraints,
+                    constraints=interpreter_constraints,
                 ),
                 V=3,
             )
             try:
                 target = _select_path_interpreter(
-                    path=ENV.PEX_PYTHON_PATH, compatibility_constraints=compatibility_constraints
+                    path=ENV.PEX_PYTHON_PATH, interpreter_constraints=interpreter_constraints
                 )
             except UnsatisfiableInterpreterConstraintsError as e:
-                die(
-                    e.create_message(
-                        "Failed to find compatible interpreter on path {path}.".format(
-                            path=ENV.PEX_PYTHON_PATH or os.getenv("PATH")
-                        )
+                raise e.with_preamble(
+                    "Failed to find compatible interpreter on path {path}.".format(
+                        path=ENV.PEX_PYTHON_PATH or os.getenv("PATH")
                     )
                 )
-        elif pythonpath is None:
-            TRACER.log(
-                "Using the current interpreter {} since no constraints have been specified and "
-                "PYTHONPATH is not set.".format(sys.executable),
-                V=3,
-            )
-            return None
-        else:
-            target = current_interpreter
-
-    if not target:
-        # N.B.: This can only happen when PEX_PYTHON_PATH is set and compatibility_constraints is
-        # not set, but we handle all constraints generally for sanity sake.
-        constraints = []
-        if ENV.PEX_PYTHON:
-            constraints.append("PEX_PYTHON={}".format(ENV.PEX_PYTHON))
-        if ENV.PEX_PYTHON_PATH:
-            constraints.append("PEX_PYTHON_PATH={}".format(ENV.PEX_PYTHON_PATH))
-        if compatibility_constraints:
-            constraints.extend(
-                "--interpreter-constraint={}".format(compatibility_constraint)
-                for compatibility_constraint in compatibility_constraints
+
+        if target is None:
+            # N.B.: This can only happen when PEX_PYTHON_PATH is set and interpreter_constraints
+            # is empty, but we handle all constraints generally for sanity sake.
+            constraints = []
+            if ENV.PEX_PYTHON:
+                constraints.append("PEX_PYTHON={}".format(ENV.PEX_PYTHON))
+            if ENV.PEX_PYTHON_PATH:
+                constraints.append("PEX_PYTHON_PATH={}".format(ENV.PEX_PYTHON_PATH))
+            if interpreter_constraints:
+                constraints.append(
+                    "Version matches {}".format(" or ".join(interpreter_constraints))
+                )
+            raise UnsatisfiableInterpreterConstraintsError(
+                constraints=constraints,
+                candidates=[current_interpreter],
+                failures=[],
+                preamble="Could not find a compatible interpreter.",
             )
+        return target
 
-        die(
-            "Failed to find an appropriate Python interpreter.\n"
-            "\n"
-            "Although the current interpreter is {python}, the following constraints exclude it:\n"
-            "  {constraints}".format(python=sys.executable, constraints="\n  ".join(constraints))
+
+def maybe_reexec_pex(interpreter_constraints=None):
+    # type: (Optional[Iterable[str]]) -> Union[None, NoReturn]
+    """Handle environment overrides for the Python interpreter to use when executing this pex.
+
+    This function supports interpreter filtering based on interpreter constraints stored in PEX-INFO
+    metadata. If PEX_PYTHON is set it attempts to obtain the binary location of the interpreter
+    specified by PEX_PYTHON. If PEX_PYTHON_PATH is set, it attempts to search the path for a
+    matching interpreter in accordance with the interpreter constraints. If both variables are
+    present, this function gives precedence to PEX_PYTHON_PATH and errors out if no compatible
+    interpreters can be found on said path.
+
+    If neither variable is set, we fall back to plain PEX execution using PATH searching or the
+    currently executing interpreter. If compatibility constraints are used, we match those
+    constraints against these interpreters.
+
+    :param interpreter_constraints: Optional list of requirements-style strings that constrain the
+                                    Python interpreter to re-exec this pex with.
+    """
+
+    current_interpreter = PythonInterpreter.get()
+
+    # NB: Used only for tests.
+    if "_PEX_EXEC_CHAIN" in os.environ:
+        flag_or_chain = os.environ.pop("_PEX_EXEC_CHAIN")
+        pex_exec_chain = [] if flag_or_chain == "1" else flag_or_chain.split(os.pathsep)
+        pex_exec_chain.append(current_interpreter.binary)
+        os.environ["_PEX_EXEC_CHAIN"] = os.pathsep.join(pex_exec_chain)
+
+    current_interpreter_blessed_env_var = "_PEX_SHOULD_EXIT_BOOTSTRAP_REEXEC"
+    if os.environ.pop(current_interpreter_blessed_env_var, None):
+        # We've already been here and selected an interpreter. Continue to execution.
+        return None
+
+    try:
+        target = find_compatible_interpreter(
+            interpreter_constraints=interpreter_constraints,
         )
+    except UnsatisfiableInterpreterConstraintsError as e:
+        die(str(e))
 
     os.environ.pop("PEX_PYTHON", None)
     os.environ.pop("PEX_PYTHON_PATH", None)
 
-    if pythonpath is None and target == current_interpreter:
+    if ENV.PEX_INHERIT_PATH == InheritPath.FALSE:
+        # Now that we've found a compatible Python interpreter, make sure we resolve out of any
+        # virtual environments it may be contained in since virtual environments created with
+        # `--system-site-packages` foil PEX attempts to scrub the sys.path.
+        resolved = target.resolve_base_interpreter()
+        if resolved != target:
+            TRACER.log(
+                "Resolved base interpreter of {} from virtual environment at {}".format(
+                    resolved, target.prefix
+                ),
+                V=3,
+            )
+        target = resolved
+
+    from . import pex
+
+    pythonpath = pex.PEX.stash_pythonpath()
+    if pythonpath is not None:
+        TRACER.log("Stashed PYTHONPATH of {}".format(pythonpath), V=2)
+    elif target == current_interpreter:
         TRACER.log(
             "Using the current interpreter {} since it matches constraints and "
             "PYTHONPATH is not set.".format(sys.executable)
@@ -323,13 +334,13 @@ def maybe_reexec_pex(compatibility_constraints=None):
         "sys.executable={python!r}, "
         "PEX_PYTHON={pex_python!r}, "
         "PEX_PYTHON_PATH={pex_python_path!r}, "
-        "COMPATIBILITY_CONSTRAINTS={compatibility_constraints!r}"
+        "interpreter_constraints={interpreter_constraints!r}"
         "{pythonpath}".format(
             cmdline=" ".join(cmdline),
             python=sys.executable,
             pex_python=ENV.PEX_PYTHON,
             pex_python_path=ENV.PEX_PYTHON_PATH,
-            compatibility_constraints=compatibility_constraints,
+            interpreter_constraints=interpreter_constraints,
             pythonpath=', (stashed) PYTHONPATH="{}"'.format(pythonpath)
             if pythonpath is not None
             else "",


Test patch:
diff --git a/pex/testing.py b/pex/testing.py
--- a/pex/testing.py
+++ b/pex/testing.py
@@ -479,17 +479,23 @@ def run_pyenv(args):
     return python, pip, run_pyenv
 
 
-def ensure_python_venv(version, latest_pip=True):
+def ensure_python_venv(version, latest_pip=True, system_site_packages=False):
     python, pip, _ = ensure_python_distribution(version)
     venv = safe_mkdtemp()
     if version in _ALL_PY3_VERSIONS:
-        subprocess.check_call([python, "-m", "venv", venv])
+        args = [python, "-m", "venv", venv]
+        if system_site_packages:
+            args.append("--system-site-packages")
+        subprocess.check_call(args=args)
     else:
-        subprocess.check_call([pip, "install", "virtualenv==16.7.10"])
-        subprocess.check_call([python, "-m", "virtualenv", venv])
+        subprocess.check_call(args=[pip, "install", "virtualenv==16.7.10"])
+        args = [python, "-m", "virtualenv", venv, "-q"]
+        if system_site_packages:
+            args.append("--system-site-packages")
+        subprocess.check_call(args=args)
     python, pip = tuple(os.path.join(venv, "bin", exe) for exe in ("python", "pip"))
     if latest_pip:
-        subprocess.check_call([pip, "install", "-U", "pip"])
+        subprocess.check_call(args=[pip, "install", "-U", "pip"])
     return python, pip
 
 
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1,6 +1,8 @@
 # Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
+from __future__ import absolute_import, print_function
+
 import errno
 import filecmp
 import functools
@@ -33,15 +35,10 @@
 from pex.executor import Executor
 from pex.interpreter import PythonInterpreter
 from pex.network_configuration import NetworkConfiguration
+from pex.orderedset import OrderedSet
 from pex.pex_info import PexInfo
 from pex.pip import get_pip
-from pex.requirements import (
-    LogicalLine,
-    ReqInfo,
-    URLFetcher,
-    parse_requirement_file,
-    parse_requirements,
-)
+from pex.requirements import LogicalLine, ReqInfo, URLFetcher, parse_requirement_file
 from pex.testing import (
     IS_PYPY,
     NOT_CPYTHON27,
@@ -75,6 +72,7 @@
         Iterable,
         Iterator,
         List,
+        MutableSet,
         Optional,
         Tuple,
     )
@@ -639,16 +637,16 @@ def test_plain_pex_exec_no_ppp_no_pp_no_constraints():
     # type: () -> None
     with temporary_dir() as td:
         pex_out_path = os.path.join(td, "pex.pex")
-        env = make_env(
-            PEX_IGNORE_RCFILES="1", PATH=os.path.dirname(os.path.realpath(sys.executable))
-        )
+        env = make_env(PEX_IGNORE_RCFILES="1")
         res = run_pex_command(["--disable-cache", "-o", pex_out_path], env=env)
         res.assert_success()
 
-        stdin_payload = b"import os, sys; print(os.path.realpath(sys.executable)); sys.exit(0)"
+        stdin_payload = b"import os, sys; print(sys.executable); sys.exit(0)"
         stdout, rc = run_simple_pex(pex_out_path, stdin=stdin_payload, env=env)
         assert rc == 0
-        assert os.path.realpath(sys.executable).encode() in stdout
+        assert (
+            PythonInterpreter.get().resolve_base_interpreter().binary.encode() in stdout
+        ), "Expected the current interpreter to be used when no constraints were supplied."
 
 
 def test_pex_exec_with_pex_python_path_only():
@@ -1907,9 +1905,10 @@ def add_to_path(entry):
             PYTHONPATH=os.pathsep.join(pythonpath) if pythonpath else None,
         )
 
+        initial_interpreter = PythonInterpreter.get()
         output = subprocess.check_output(
             [
-                sys.executable,
+                initial_interpreter.binary,
                 test_pex,
                 "-c",
                 "import json, os; print(json.dumps(os.environ.copy()))",
@@ -1922,10 +1921,25 @@ def add_to_path(entry):
         assert "PEX_PYTHON_PATH" not in final_env
         assert "_PEX_SHOULD_EXIT_BOOTSTRAP_REEXEC" not in final_env
 
-        expected_exec_chain = [
-            PythonInterpreter.from_binary(i).binary for i in [sys.executable] + (exec_chain or [])
-        ]
-        assert expected_exec_chain == final_env["_PEX_EXEC_CHAIN"].split(os.pathsep)
+        expected_exec_interpreters = [initial_interpreter]
+        if exec_chain:
+            expected_exec_interpreters.extend(PythonInterpreter.from_binary(b) for b in exec_chain)
+        final_interpreter = expected_exec_interpreters[-1]
+        if final_interpreter.is_venv:
+            # If the last interpreter in the chain is in a virtual environment, it should be fully
+            # resolved and re-exec'd against in order to escape the virtual environment since we're
+            # not setting PEX_INHERIT_PATH in these tests.
+            resolved = final_interpreter.resolve_base_interpreter()
+            if exec_chain:
+                # There is already an expected reason to re-exec; so no extra exec step is needed.
+                expected_exec_interpreters[-1] = resolved
+            else:
+                # The expected exec chain is just the initial_interpreter, but it turned out to be a
+                # venv which forces a re-exec.
+                expected_exec_interpreters.append(resolved)
+        expected_exec_chain = [i.binary for i in expected_exec_interpreters]
+        actual_exec_chain = final_env["_PEX_EXEC_CHAIN"].split(os.pathsep)
+        assert expected_exec_chain == actual_exec_chain
 
 
 def test_pex_no_reexec_no_constraints():
@@ -1940,17 +1954,15 @@ def test_pex_reexec_no_constraints_pythonpath_present():
 
 def test_pex_no_reexec_constraints_match_current():
     # type: () -> None
-    current_version = ".".join(str(component) for component in sys.version_info[0:3])
-    _assert_exec_chain(interpreter_constraints=["=={}".format(current_version)])
+    _assert_exec_chain(interpreter_constraints=[PythonInterpreter.get().identity.requirement])
 
 
 def test_pex_reexec_constraints_match_current_pythonpath_present():
     # type: () -> None
-    current_version = ".".join(str(component) for component in sys.version_info[0:3])
     _assert_exec_chain(
         exec_chain=[sys.executable],
         pythonpath=["."],
-        interpreter_constraints=["=={}".format(current_version)],
+        interpreter_constraints=[PythonInterpreter.get().identity.requirement],
     )
 
 
@@ -2546,3 +2558,76 @@ def line(
             ReqInfo.create(line=line("translate>=3.2.1", 6), project_name="translate"),
             ReqInfo.create(line=line("protobuf>=3.11.3", 7), project_name="protobuf"),
         ] == list(reqs)
+
+
+@pytest.mark.parametrize(
+    "py_version",
+    [
+        pytest.param(PY27, id="virtualenv-16.7.10"),
+        pytest.param(PY36, id="pyvenv"),
+    ],
+)
+def test_issues_1031(py_version):
+    # type: (str) -> None
+    system_site_packages_venv, _ = ensure_python_venv(
+        py_version, latest_pip=False, system_site_packages=True
+    )
+    standard_venv, _ = ensure_python_venv(py_version, latest_pip=False, system_site_packages=False)
+
+    print_sys_path_code = "import os, sys; print('\\n'.join(map(os.path.realpath, sys.path)))"
+
+    def get_sys_path(python):
+        # type: (str) -> MutableSet[str]
+        _, stdout, _ = PythonInterpreter.from_binary(python).execute(
+            args=["-c", print_sys_path_code]
+        )
+        return OrderedSet(stdout.strip().splitlines())
+
+    system_site_packages_venv_sys_path = get_sys_path(system_site_packages_venv)
+    standard_venv_sys_path = get_sys_path(standard_venv)
+
+    def venv_dir(python):
+        # type: (str) -> str
+        bin_dir = os.path.dirname(python)
+        venv_dir = os.path.dirname(bin_dir)
+        return os.path.realpath(venv_dir)
+
+    system_site_packages = {
+        p
+        for p in (system_site_packages_venv_sys_path - standard_venv_sys_path)
+        if not p.startswith((venv_dir(system_site_packages_venv), venv_dir(standard_venv)))
+    }
+    assert len(system_site_packages) == 1, (
+        "system_site_packages_venv_sys_path:\n"
+        "\t{}\n"
+        "standard_venv_sys_path:\n"
+        "\t{}\n"
+        "difference:\n"
+        "\t{}".format(
+            "\n\t".join(system_site_packages_venv_sys_path),
+            "\n\t".join(standard_venv_sys_path),
+            "\n\t".join(system_site_packages),
+        )
+    )
+    system_site_packages_path = system_site_packages.pop()
+
+    def get_system_site_packages_pex_sys_path(**env):
+        # type: (**Any) -> MutableSet[str]
+        output, returncode = run_simple_pex_test(
+            body=print_sys_path_code,
+            interpreter=PythonInterpreter.from_binary(system_site_packages_venv),
+            env=make_env(**env),
+        )
+        assert returncode == 0
+        return OrderedSet(output.decode("utf-8").strip().splitlines())
+
+    assert system_site_packages_path not in get_system_site_packages_pex_sys_path()
+    assert system_site_packages_path not in get_system_site_packages_pex_sys_path(
+        PEX_INHERIT_PATH="false"
+    )
+    assert system_site_packages_path in get_system_site_packages_pex_sys_path(
+        PEX_INHERIT_PATH="prefer"
+    )
+    assert system_site_packages_path in get_system_site_packages_pex_sys_path(
+        PEX_INHERIT_PATH="fallback"
+    )