Repository: pex-tool/pex

Issue:
pex vendoring does not (always) isolate itself from a colliding setuptools install in site-packages
Having issues creating pex files in an image based on `databricksruntime/python-virtualenv`.

Dockerfile:
```
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update \
  && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential

# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
```

Command run in container:

`pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3`


Portion of failure StackTrace:

```
Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-kh23prp9/setup.py", line 59, in <module>
      'Programming Language :: Python :: 3.8',
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/__init__.py", line 169, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/wheel/wheel/bdist_wheel.py", line 228, in run
      self.run_command('install')
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install.py", line 65, in run
      return orig.install.run(self)
    File "/usr/lib/python3.7/distutils/command/install.py", line 601, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install_scripts.py", line 22, in run
      import setuptools.command.easy_install as ei  # vendor:skip
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/easy_install.py", line 69, in <module>
      from setuptools.sandbox import run_setup  # vendor:skip
    File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/sandbox.py", line 24, in <module>
      import pkg_resources.py31compat  # vendor:skip
  ModuleNotFoundError: No module named 'pkg_resources.py31compat'
  ----------------------------------------
  ERROR: Failed building wheel for boto3
```


Discussion:
As requested: @stuhood 
Thanks! I repro the issue, with or without `setuptools` explicitly installed alongside `pex`. Investigating.
According to https://github.com/pypa/setuptools/issues/1167, this is likely a collision between multiple `setuptools` installs, one of which is vendored inside pex, and the other of which comes from your sitepackages due to `--system-site-packages`.

So I suspect that this might be bug in the vendoring that `pex` does that is allowing an external copy of `setuptools` to collide with the vendored copy.

Two workarounds:
1. Remove the `--system-site-packages` flag and rely on explicitly pip installing the things you need in the virtualenv
2. run pex _from a_ pex, which will allow pex to isolate itself rather than relying on virtualenv to do so:
```
$ pex pex -o pex.pex -m pex
$ ./pex.pex --interpreter-constraint='CPython>=3.7' boto3
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
```
This backtrace is in a Pex execution of Pip. There, Pex dogfoods itself to run a Pip PEX in isolation - with the exception of setuptools and wheel whose vendored versions are deliberately exposed so Pip can execute `setup.py`s as needed:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L53

So that's where to start spelunking.
I'll dig on this a bit.
Getting back to the conversation above, I wanted to correct the record on:
> 2. run pex from a pex, which will allow pex to isolate itself rather than relying on virtualenv to do so:

Pex isolates itself whether run from a pex file or not. It does this by:
1. Shading all its dependencies, so it imports pex.third_party.setuptools and not setuptools and so cannot pick this up from the system.
2. Wraps its internal executions of python interpreter subprocesses in PEXes to provide isolation to those subprocesses.

My answer above about Pip PEX dogfood concerns point 2.
FWIW, I tested that both 1) and 2) from my answer resolved the issue in the repro. I might have rushed things though.
Aha, ok. That's not making sense but good to know.

That said, I don't repro:
```
$ cat <<EOF | docker build -
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential

# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
EOF
Sending build context to Docker daemon  3.072kB
Step 1/9 : FROM databricksruntime/python-virtualenv:latest
 ---> 6d14d1c4da8d
Step 2/9 : RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
 ---> Running in 085a999e997e
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
...
Processing triggers for dbus (1.10.6-1ubuntu3.3) ...
Removing intermediate container 085a999e997e
 ---> b39aead9ebae
Step 3/9 : RUN add-apt-repository ppa:deadsnakes/ppa
 ---> Running in a985ceb3f4b6
 This PPA contains more recent Python versions packaged for Ubuntu.
...
gpg: key 6A755776: public key "Launchpad PPA for deadsnakes" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
Removing intermediate container a985ceb3f4b6
 ---> 654f182d6ef2
Step 4/9 : RUN apt update
 ---> Running in 25d3cbe0c6d1
...
76 packages can be upgraded. Run 'apt list --upgradable' to see them.
Removing intermediate container 25d3cbe0c6d1
 ---> 31e7a456c83b
Step 5/9 : RUN apt install --yes python3.7 python3.7-dev
 ---> Running in e7601428f9c3
...
Setting up python3.7-dev (3.7.9-1+xenial1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Removing intermediate container e7601428f9c3
 ---> 2a03e6c57939
Step 6/9 : RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
 ---> Running in 8bfcdf409659
Using base prefix '/usr'
New python executable in /databricks/python3/bin/python3.7
...
Running virtualenv with interpreter /usr/bin/python3.7
Removing intermediate container 8bfcdf409659
 ---> 2e2296a1ba42
Step 7/9 : RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
 ---> Running in bee547a895cb
Collecting pex
  Downloading pex-2.1.16-py2.py3-none-any.whl (2.3 MB)
...
Successfully installed pex-2.1.16 setuptools-45.2.0
Removing intermediate container bee547a895cb
 ---> 320f252b829d
Step 8/9 : ENV PATH="/home/jsirois/.rvm/gems/ruby-2.6.0/bin:/home/jsirois/.rvm/gems/ruby-2.6.0@global/bin:/home/jsirois/.rvm/rubies/ruby-2.6.0/bin:/home/jsirois/.cargo/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/home/jsirois/.nvm/versions/node/v8.11.3/bin:/home/jsirois/.poetry/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/jsirois/.bash.d/bin:/home/jsirois/go/bin:/home/jsirois/.gem/ruby/2.5.0/bin:/home/jsirois/.rvm/bin:/home/jsirois/.bash.d/bin:/databricks/python3/bin"
 ---> Running in 8887cb7aa912
Removing intermediate container 8887cb7aa912
 ---> ea3611e65fa8
Step 9/9 : RUN pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
 ---> Running in e9ff82a8f6ff
Removing intermediate container e9ff82a8f6ff
 ---> 22d0b02808bd
Successfully built 22d0b02808bd
```

And the pex seems ok:
```
$ docker run --rm -it -ePEX_INTERPRETER=1 22d0b02808bd ./boto3.pex
Python 3.7.9 (default, Aug 18 2020, 06:24:24) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import boto3
>>> 
now exiting InteractiveConsole...
```

@jperkelens or @stuhood does the original bug report still repro for you? Probably most importantly, does it repro after appropriate `docker image prune`ing (fresh image builds)?
@jsirois : Yea, I repro with the exact commands from the description via:
```bash
docker image prune
docker run \
  -v "$(pwd):/workdir" \
  -w '/workdir' \
  --rm \
  -it \
  "databricksruntime/python-virtualenv:latest" \
  /bin/bash

# ... run each of the commands from the description
```

The docker-image-prune reports zero bytes pruned though, so I might be on a different copy of the `databricksruntime/python-virtualenv:latest` tag...? My docker foo is weak.

----

EDIT: @jsirois : but also, FWIW, @jperkelens was able to resolve his issue by removing the `--system-site-packages` flag, so he is no longer blocked on this.
If docker output says ` ---> Using cache`, the step is cached:
```
Step 2/9 : RUN apt-get update   && apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
 ---> Using cache
 ---> b39aead9ebae
```
> a different copy of the `databricksruntime/python-virtualenv:latest` tag

They have not changed the image for 13 months, so that's unlikely it, but the image incluides a `RUN apt update` which can pick up new packages. So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".

> but also, FWIW, @jperkelens was able to resolve his issue by removing the --system-site-packages flag, so he is no longer blocked on this.

Yeah - thanks. I'd still like to squash the issue if possible.

> So if that update step is cached, you'd pick up older packages than me. It could be that an upgraded package has the "fix".

It wasn't... all of the steps ran. Probably (at least) because I was actually copy pasting them into a live shell one by one, rather than running them via a Dockerfile.
Hrm. Not being able to repro is ... maddening. If either of you have time could you run this on your image substituting your image id for `22d0b02808bd`?:
```
$ docker run --rm -it 22d0b02808bd apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,upgradable to: 2.20151103ubuntu1.2]
libpython3-stdlib/xenial,now 3.5.1-3 amd64 [installed,automatic]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
libpython3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
libpython3.7-stdlib/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
python-apt-common/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 all [installed,automatic]
python-pip-whl/xenial-updates,now 8.1.1-2ubuntu0.4 all [installed,automatic]
python3/xenial,now 3.5.1-3 amd64 [installed]
python3-apt/xenial-updates,now 1.1.0~beta1ubuntu0.16.04.9 amd64 [installed,automatic]
python3-dbus/xenial,now 1.2.0-3 amd64 [installed,automatic]
python3-gi/xenial,now 3.20.0-0ubuntu1 amd64 [installed,automatic]
python3-minimal/xenial,now 3.5.1-3 amd64 [installed,automatic]
python3-pkg-resources/xenial,now 20.7.0-1 all [installed,automatic]
python3-pycurl/xenial,now 7.43.0-1ubuntu1 amd64 [installed,automatic]
python3-software-properties/xenial-updates,xenial-security,now 0.96.20.10 all [installed,automatic]
python3-virtualenv/xenial-updates,now 15.0.1+ds-3ubuntu1 all [installed,automatic]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,upgradable to: 3.5.2-2ubuntu0~16.04.11]
python3.7/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-dev/xenial,now 3.7.9-1+xenial1 amd64 [installed]
python3.7-distutils/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-lib2to3/xenial,now 3.7.9-1+xenial1 all [installed,automatic]
python3.7-minimal/xenial,now 3.7.9-1+xenial1 amd64 [installed,automatic]
```
```
$ docker image ls databricksruntime/python-virtualenv:latest
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
databricksruntime/python-virtualenv   latest              6d14d1c4da8d        18 months ago       778MB
```

As mentioned above, I have not been using a `Dockerfile`: rather, using `docker run` followed by pasting each command, so the `apt-list` bit is maybe not interesting:
```
$ docker run --rm -it 6d14d1c4da8d apt list --installed | grep python
dh-python/now 2.20151103ubuntu1.1 all [installed,local]
libpython3-stdlib/now 3.5.1-3 amd64 [installed,local]
libpython3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
libpython3.5-stdlib/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python-pip-whl/now 8.1.1-2ubuntu0.4 all [installed,local]
python3/now 3.5.1-3 amd64 [installed,local]
python3-minimal/now 3.5.1-3 amd64 [installed,local]
python3-pkg-resources/now 20.7.0-1 all [installed,local]
python3-virtualenv/now 15.0.1+ds-3ubuntu1 all [installed,local]
python3.5/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
python3.5-minimal/now 3.5.2-2ubuntu0~16.04.5 amd64 [installed,local]
```
Thanks - the means by which the command are run in the image - by hand or by Dockefile RUN, are immaterial, so the listing is useful. But now I see you ran this against the base image and not the commands you subsequently ran - gotcha. Yeah, not so useful. Thanks - I'll try to debug this by other means.
OK, so focusing on `--system-site-packages` (thanks @stuhood), virtualenv implements this via a custom `site.py`. That `site.py` calculates the underlying system site packages when the venv interpreter starts up and adds all those paths to `sys.path` via direct `sys.path` manipulation. As such, there is no way for the Pex runtime to detect these `sys.path`s should be scrubbed without adding a hunk of code that knows an awful lot about virtualenv's implementation details. In short, this means Pex cannot trust `sys.path`. It currently does when running Pip PEXed internally:
https://github.com/pantsbuild/pex/blob/2783d304daf43dc1d59753c7e5fc2599b16acd01/pex/pip.py#L35-L54

I can instead construct PYTHONPATH from the paths of the vendored setuptools and wheel directly, the main difference being this will pre-pend those paths to the sys.path of spawned Pip subprocesses. Right now they come in the order they are in sys.path which may put `--system-site-packages` before the pexed up setuptools and wheel packages.
Noting the abive fix does not solve the general hermeticity problem here, just the specific problem for the Pip PEX used at PEX buildtime.

For example:
```
root@4185a16f2609:/# pex -v setuptools
pex: Building pex :: Resolving distributions (['setuptools']) :: Resolving for:
...
pex: Activating PEX virtual environment from /tmp/tmp4xggexvl: 28.6ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /tmp/tmp4xggexvl
pex:   * /databricks/python3/lib/python37.zip
pex:     /databricks/python3/lib/python3.7
pex:     /databricks/python3/lib/python3.7/lib-dynload
pex:     /usr/lib/python3.7
pex:     /usr/local/lib/python3.7/dist-packages
pex:     /usr/lib/python3/dist-packages
pex:     /tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl
pex:     /tmp/tmp4xggexvl/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Python 3.7.9 (default, Aug 18 2020, 06:24:24) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
```
Here the `--system-site-packages` added by virtualenv to `sys.path` directly in `site.py`and unscrubbed by Pex are:
+ `pex:     /usr/local/lib/python3.7/dist-packages`
+ `pex:     /usr/lib/python3/dist-packages`
Besides not being scrubbed, these come before `/tmp/tmp4xggexvl/.deps/setuptools-50.3.0-py3-none-any.whl` in the sys.path.

In this particular builtime case, we're burned by `pkg_resources` (Note `pkg_resources` should come from `setuptools` but there is no `setuptools` package in this listing! - odd but not germain):
```
root@4185a16f2609:/# ls -1 /usr/lib/python3/dist-packages
__pycache__
_dbus_bindings.cpython-35m-x86_64-linux-gnu.so
_dbus_glib_bindings.cpython-35m-x86_64-linux-gnu.so
apt
apt_inst.cpython-35m-x86_64-linux-gnu.so
apt_pkg.cpython-35m-x86_64-linux-gnu.so
aptsources
curl
dbus
debconf.py
gi
lsb_release.py
pkg_resources
pycurl-7.43.0.egg-info
pycurl.cpython-35m-x86_64-linux-gnu.so
pygobject-3.20.0.egg-info
pygtkcompat
python_apt-1.1.0.b1_ubuntu0.16.04.9.egg-info
softwareproperties
unattended_upgrades-0.1.egg-info
virtualenv-15.0.1.egg-info
virtualenv.py
```
It looks like one way to solve this is to invert how we calculate which `sys.path` entries to scrub. Instead of calculating site-libraries, user-site-libraries and extras to scrub:
https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pex.py#L309-L315
We could calculate just the core libraries and scrub everything else via:
```
python -sSE -c 'import sys; print("\n".join(sys.path))'
```
That solves the case above and seems as robust as you can possibly be since the answer is delegated to the python interpreter in question.
Checking back in here. Although I still cannot repro the failure you get using the commands above I can repro the leak by inspection of sys.path in a PEX built in the databaricks image. I've done some work over in #1128 that makes Pex both aware of virtual environments and equips it with the capability to bust out of a virtual environment. I'll use that to fix this. The approach above using ~ `python -sSE -c 'import sys; print("\n".join(sys.path))'` was not robust across all interpreter installations (those that include bits of the stdlib in ~site-packages :/).

Resolving patch:
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -16,7 +16,7 @@
 from textwrap import dedent
 
 from pex import third_party
-from pex.common import safe_rmtree
+from pex.common import is_exe, safe_rmtree
 from pex.compatibility import string
 from pex.executor import Executor
 from pex.jobs import ErrorHandler, Job, Retain, SpawnedJob, execute_parallel
@@ -34,6 +34,7 @@
         Dict,
         Iterable,
         Iterator,
+        List,
         MutableMapping,
         Optional,
         Sequence,
@@ -95,6 +96,14 @@ def get(cls, binary=None):
         preferred_tag = supported_tags[0]
         return cls(
             binary=binary or sys.executable,
+            prefix=sys.prefix,
+            base_prefix=(
+                # Old virtualenv (16 series and lower) sets `sys.real_prefix` in all cases.
+                getattr(sys, "real_prefix", None)
+                # Both pyvenv and virtualenv 20+ set `sys.base_prefix` as per
+                # https://www.python.org/dev/peps/pep-0405/.
+                or getattr(sys, "base_prefix", sys.prefix)
+            ),
             python_tag=preferred_tag.interpreter,
             abi_tag=preferred_tag.abi,
             platform_tag=preferred_tag.platform,
@@ -107,7 +116,7 @@ def get(cls, binary=None):
     def decode(cls, encoded):
         TRACER.log("creating PythonIdentity from encoded: %s" % encoded, V=9)
         values = json.loads(encoded)
-        if len(values) != 7:
+        if len(values) != 9:
             raise cls.InvalidError("Invalid interpreter identity: %s" % encoded)
 
         supported_tags = values.pop("supported_tags")
@@ -126,13 +135,25 @@ def _find_interpreter_name(cls, python_tag):
         raise ValueError("Unknown interpreter: {}".format(python_tag))
 
     def __init__(
-        self, binary, python_tag, abi_tag, platform_tag, version, supported_tags, env_markers
+        self,
+        binary,  # type: str
+        prefix,  # type: str
+        base_prefix,  # type: str
+        python_tag,  # type: str
+        abi_tag,  # type: str
+        platform_tag,  # type: str
+        version,  # type: Iterable[int]
+        supported_tags,  # type: Iterable[tags.Tag]
+        env_markers,  # type: Dict[str, str]
     ):
+        # type: (...) -> None
         # N.B.: We keep this mapping to support historical values for `distribution` and `requirement`
         # properties.
         self._interpreter_name = self._find_interpreter_name(python_tag)
 
         self._binary = binary
+        self._prefix = prefix
+        self._base_prefix = base_prefix
         self._python_tag = python_tag
         self._abi_tag = abi_tag
         self._platform_tag = platform_tag
@@ -143,6 +164,8 @@ def __init__(
     def encode(self):
         values = dict(
             binary=self._binary,
+            prefix=self._prefix,
+            base_prefix=self._base_prefix,
             python_tag=self._python_tag,
             abi_tag=self._abi_tag,
             platform_tag=self._platform_tag,
@@ -158,6 +181,16 @@ def encode(self):
     def binary(self):
         return self._binary
 
+    @property
+    def prefix(self):
+        # type: () -> str
+        return self._prefix
+
+    @property
+    def base_prefix(self):
+        # type: () -> str
+        return self._base_prefix
+
     @property
     def python_tag(self):
         return self._python_tag
@@ -308,7 +341,7 @@ class PythonInterpreter(object):
     _PYTHON_INTERPRETER_BY_NORMALIZED_PATH = {}  # type: Dict
 
     @staticmethod
-    def _read_pyvenv_home(path):
+    def _get_pyvenv_cfg(path):
         # type: (str) -> Optional[str]
         # See: https://www.python.org/dev/peps/pep-0405/#specification
         pyvenv_cfg_path = os.path.join(path, "pyvenv.cfg")
@@ -317,11 +350,11 @@ def _read_pyvenv_home(path):
                 for line in fp:
                     name, _, value = line.partition("=")
                     if name.strip() == "home":
-                        return value.strip()
+                        return pyvenv_cfg_path
         return None
 
     @classmethod
-    def _find_pyvenv_home(cls, maybe_venv_python_binary):
+    def _find_pyvenv_cfg(cls, maybe_venv_python_binary):
         # type: (str) -> Optional[str]
         # A pyvenv is identified by a pyvenv.cfg file with a home key in one of the two following
         # directory layouts:
@@ -340,11 +373,11 @@ def _find_pyvenv_home(cls, maybe_venv_python_binary):
         #
         # See: # See: https://www.python.org/dev/peps/pep-0405/#specification
         maybe_venv_bin_dir = os.path.dirname(maybe_venv_python_binary)
-        home_dir = cls._read_pyvenv_home(maybe_venv_bin_dir)
-        if not home_dir:
+        pyvenv_cfg = cls._get_pyvenv_cfg(maybe_venv_bin_dir)
+        if not pyvenv_cfg:
             maybe_venv_dir = os.path.dirname(maybe_venv_bin_dir)
-            home_dir = cls._read_pyvenv_home(maybe_venv_dir)
-        return home_dir
+            pyvenv_cfg = cls._get_pyvenv_cfg(maybe_venv_dir)
+        return pyvenv_cfg
 
     @classmethod
     def _resolve_pyvenv_canonical_python_binary(
@@ -357,8 +390,8 @@ def _resolve_pyvenv_canonical_python_binary(
         if not os.path.islink(maybe_venv_python_binary):
             return None
 
-        home_dir = cls._find_pyvenv_home(maybe_venv_python_binary)
-        if os.path.dirname(real_binary) != home_dir:
+        pyvenv_cfg = cls._find_pyvenv_cfg(maybe_venv_python_binary)
+        if pyvenv_cfg is None:
             return None
 
         while os.path.islink(maybe_venv_python_binary):
@@ -798,8 +831,109 @@ def __init__(self, identity):
 
     @property
     def binary(self):
+        # type: () -> str
         return self._binary
 
+    @property
+    def is_venv(self):
+        # type: () -> bool
+        """Return `True` if this interpreter is homed in a virtual environment."""
+        return self._identity.prefix != self._identity.base_prefix
+
+    @property
+    def prefix(self):
+        # type: () -> str
+        """Return the `sys.prefix` of this interpreter.
+
+        For virtual environments, this will be the virtual environment directory itself.
+        """
+        return self._identity.prefix
+
+    class BaseInterpreterResolutionError(Exception):
+        """Indicates the base interpreter for a virtual environment could not be resolved."""
+
+    def resolve_base_interpreter(self):
+        # type: () -> PythonInterpreter
+        """Finds the base system interpreter used to create a virtual environment.
+
+        If this interpreter is not homed in a virtual environment, returns itself.
+        """
+        if not self.is_venv:
+            return self
+
+        # In the case of PyPy, the <base_prefix> dir might contain one of the following:
+        #
+        # 1. On a system with PyPy 2.7 series and one PyPy 3.x series
+        # bin/
+        #   pypy
+        #   pypy3
+        #
+        # 2. On a system with PyPy 2.7 series and more than one PyPy 3.x series
+        # bin/
+        #   pypy
+        #   pypy3
+        #   pypy3.6
+        #   pypy3.7
+        #
+        # In both cases, bin/pypy is a 2.7 series interpreter. In case 2 bin/pypy3 could be either
+        # PyPy 3.6 series or PyPy 3.7 series. In order to ensure we pick the correct base executable
+        # of a PyPy virtual environment, we always try to resolve the most specific basename first
+        # to the least specific basename last and we also verify that, if the basename resolves, it
+        # resolves to an equivalent interpreter. We employ the same strategy for CPython, but only
+        # for uniformity in the algorithm. It appears to always be the case for CPython that
+        # python<major>.<minor> is present in any given <prefix>/bin/ directory; so the algorithm
+        # gets a hit on 1st try for CPython binaries incurring ~no extra overhead.
+
+        version = self._identity.version
+        abi_tag = self._identity.abi_tag
+
+        prefix = "pypy" if self._identity.interpreter == "PyPy" else "python"
+        suffixes = ("{}.{}".format(version[0], version[1]), str(version[0]), "")
+        candidate_binaries = tuple("{}{}".format(prefix, suffix) for suffix in suffixes)
+
+        def iter_base_candidate_binary_paths(interpreter):
+            # type: (PythonInterpreter) -> Iterator[str]
+            bin_dir = os.path.join(interpreter._identity.base_prefix, "bin")
+            for candidate_binary in candidate_binaries:
+                candidate_binary_path = os.path.join(bin_dir, candidate_binary)
+                if is_exe(candidate_binary_path):
+                    yield candidate_binary_path
+
+        def is_same_interpreter(interpreter):
+            # type: (PythonInterpreter) -> bool
+            identity = interpreter._identity
+            return identity.version == version and identity.abi_tag == abi_tag
+
+        resolution_path = []  # type: List[str]
+        base_interpreter = self
+        while base_interpreter.is_venv:
+            resolved = None  # type: Optional[PythonInterpreter]
+            for candidate_path in iter_base_candidate_binary_paths(base_interpreter):
+                resolved_interpreter = self.from_binary(candidate_path)
+                if is_same_interpreter(resolved_interpreter):
+                    resolved = resolved_interpreter
+                    break
+            if resolved is None:
+                message = [
+                    "Failed to resolve the base interpreter for the virtual environment at "
+                    "{venv_dir}.".format(venv_dir=self._identity.prefix)
+                ]
+                if resolution_path:
+                    message.append(
+                        "Resolved through {path}".format(
+                            path=" -> ".join(binary for binary in resolution_path)
+                        )
+                    )
+                message.append(
+                    "Search of base_prefix {} found no equivalent interpreter for {}".format(
+                        base_interpreter._identity.base_prefix, base_interpreter._binary
+                    )
+                )
+                raise self.BaseInterpreterResolutionError("\n".join(message))
+            base_interpreter = resolved_interpreter
+            resolution_path.append(base_interpreter.binary)
+        return base_interpreter
+
     @property
     def identity(self):
         # type: () -> PythonIdentity


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1922,7 +1922,9 @@ def add_to_path(entry):
         assert "PEX_PYTHON_PATH" not in final_env
         assert "_PEX_SHOULD_EXIT_BOOTSTRAP_REEXEC" not in final_env
 
-        expected_exec_chain = [os.path.realpath(i) for i in [sys.executable] + (exec_chain or [])]
+        expected_exec_chain = [
+            PythonInterpreter.from_binary(i).binary for i in [sys.executable] + (exec_chain or [])
+        ]
         assert expected_exec_chain == final_env["_PEX_EXEC_CHAIN"].split(os.pathsep)
 
 
@@ -1933,7 +1935,7 @@ def test_pex_no_reexec_no_constraints():
 
 def test_pex_reexec_no_constraints_pythonpath_present():
     # type: () -> None
-    _assert_exec_chain(exec_chain=[os.path.realpath(sys.executable)], pythonpath=["."])
+    _assert_exec_chain(exec_chain=[sys.executable], pythonpath=["."])
 
 
 def test_pex_no_reexec_constraints_match_current():
@@ -1946,7 +1948,7 @@ def test_pex_reexec_constraints_match_current_pythonpath_present():
     # type: () -> None
     current_version = ".".join(str(component) for component in sys.version_info[0:3])
     _assert_exec_chain(
-        exec_chain=[os.path.realpath(sys.executable)],
+        exec_chain=[sys.executable],
         pythonpath=["."],
         interpreter_constraints=["=={}".format(current_version)],
     )
diff --git a/tests/test_interpreter.py b/tests/test_interpreter.py
--- a/tests/test_interpreter.py
+++ b/tests/test_interpreter.py
@@ -9,7 +9,7 @@
 import pytest
 
 from pex import interpreter
-from pex.common import temporary_dir, touch
+from pex.common import safe_mkdtemp, temporary_dir, touch
 from pex.compatibility import PY3
 from pex.executor import Executor
 from pex.interpreter import PythonInterpreter
@@ -17,6 +17,7 @@
     PY27,
     PY35,
     PY36,
+    PY_VER,
     ensure_python_distribution,
     ensure_python_interpreter,
     environment_as,
@@ -30,7 +31,7 @@
     from unittest.mock import Mock, patch  # type: ignore[misc,no-redef,import]
 
 if TYPE_CHECKING:
-    from typing import Iterator, Tuple, Union, Any
+    from typing import Iterator, Tuple, Union, Any, List
 
     InterpreterIdentificationError = Tuple[str, str]
     InterpreterOrError = Union[PythonInterpreter, InterpreterIdentificationError]
@@ -255,7 +256,7 @@ def assert_chosen(expected_version, other_version):
     assert_chosen(expected_version="3.6.1", other_version="3.6.0")
 
 
-def test_pyvenv(tmpdir):
+def test_detect_pyvenv(tmpdir):
     # type: (Any) -> None
     venv = str(tmpdir)
     py35 = ensure_python_interpreter(PY35)
@@ -285,3 +286,48 @@ def test_pyvenv(tmpdir):
     assert len(pythons) >= 2, "Expected at least two virtualenv python binaries, found: {}".format(
         pythons
     )
+
+
+def check_resolve_venv(real_interpreter):
+    # type: (PythonInterpreter) -> None
+    tmpdir = safe_mkdtemp()
+
+    def create_venv(
+        interpreter,  # type: PythonInterpreter
+        rel_path,  # type: str
+    ):
+        # type: (...) -> List[str]
+        venv_dir = os.path.join(tmpdir, rel_path)
+        interpreter.execute(["-m", "venv", venv_dir])
+        return glob.glob(os.path.join(venv_dir, "bin", "python*"))
+
+    assert not real_interpreter.is_venv
+    assert real_interpreter is real_interpreter.resolve_base_interpreter()
+
+    for index, python in enumerate(create_venv(real_interpreter, "first-level")):
+        venv_interpreter = PythonInterpreter.from_binary(python)
+        assert venv_interpreter.is_venv
+        assert venv_interpreter != real_interpreter.binary
+        assert real_interpreter == venv_interpreter.resolve_base_interpreter()
+
+        for nested_python in create_venv(venv_interpreter, "second-level{}".format(index)):
+            nested_venv_interpreter = PythonInterpreter.from_binary(nested_python)
+            assert nested_venv_interpreter.is_venv
+            assert nested_venv_interpreter != venv_interpreter
+            assert nested_venv_interpreter != real_interpreter
+            assert real_interpreter == nested_venv_interpreter.resolve_base_interpreter()
+
+
+def test_resolve_venv():
+    # type: () -> None
+    real_interpreter = PythonInterpreter.from_binary(ensure_python_interpreter(PY35))
+    check_resolve_venv(real_interpreter)
+
+
+@pytest.mark.skipif(
+    PY_VER < (3, 0), reason="Test requires the venv module which is not present in Python 2."
+)
+def test_resolve_venv_ambient():
+    # type: () -> None
+    ambient_real_interpreter = PythonInterpreter.get().resolve_base_interpreter()
+    check_resolve_venv(ambient_real_interpreter)
diff --git a/tests/test_pex_bootstrapper.py b/tests/test_pex_bootstrapper.py
--- a/tests/test_pex_bootstrapper.py
+++ b/tests/test_pex_bootstrapper.py
@@ -30,7 +30,7 @@ def find_interpreters(
     constraints=None,  # type: Optional[Iterable[str]]
     preferred_interpreter=None,  # type: Optional[PythonInterpreter]
 ):
-    # type: (...) -> List[AnyStr]
+    # type: (...) -> List[str]
     return [
         interp.binary
         for interp in iter_compatible_interpreters(
@@ -163,8 +163,9 @@ def test_find_compatible_interpreters_with_valid_basenames_and_constraints():
 def test_find_compatible_interpreters_bias_current():
     # type: () -> None
     py36 = ensure_python_interpreter(PY36)
-    assert [os.path.realpath(sys.executable), py36] == find_interpreters([py36, sys.executable])
-    assert [os.path.realpath(sys.executable), py36] == find_interpreters([sys.executable, py36])
+    current_interpreter = PythonInterpreter.get()
+    assert [current_interpreter.binary, py36] == find_interpreters([py36, sys.executable])
+    assert [current_interpreter.binary, py36] == find_interpreters([sys.executable, py36])
 
 
 def test_find_compatible_interpreters_siblings_of_current_issues_1109():