Repository: pex-tool/pex

Issue:
Pex 1.4.6 builds a failing pex
pex 1.4.6 will allow you to build a pex against the [responses](https://github.com/getsentry/responses) library, but blows up when you try to actually import it.

This works fine outside of a pex environment and with pex 1.4.5.

```
 ~/sandbox $ . venv/bin/activate
(venv)  ~/sandbox $ python -V
Python 2.7.15
(venv)  ~/sandbox $ pex --version
pex 1.4.6
(venv)  ~/sandbox $ pex responses
Python 2.7.15 (default, May  1 2018, 16:44:08)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.deps/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpReh1br/.bootstrap/pkg_resources/__init__.py", line 77, in <module>
    from . import py31compat
ImportError: cannot import name py31compat
```


Discussion:
I don't repro. I'm pretty sure this is going to hinge on a setuptools version. If you could provide more details about your environment I'd be grateful.

My test:
```
$ venv -2 -c pex1.4.6
New python executable in /home/jsirois/.venv/pex1.4.6/bin/python2
Also creating executable in /home/jsirois/.venv/pex1.4.6/bin/python
Installing setuptools, pip, wheel...done.
/home/jsirois/.venv/pex1.4.6
$ venv_activate pex1.4.6
(pex1.4.6) $ python --version
Python 2.7.15
(pex1.4.6) $ pip install pex
Collecting pex
  Using cached https://files.pythonhosted.org/packages/57/1f/892b5e810df3435fd3ef18f78c9cdffb5df80cb4c5bf2b2bd8a1d3c8b590/pex-1.4.6-py2.py3-none-any.whl
Requirement already satisfied: wheel<0.32,>=0.26.0 in /home/jsirois/.venv/pex1.4.6/lib/python2.7/site-packages (from pex) (0.31.1)
Requirement already satisfied: setuptools!=34.*,!=35.*,<41,>=20.3 in /home/jsirois/.venv/pex1.4.6/lib/python2.7/site-packages (from pex) (40.4.1)
Installing collected packages: pex
Successfully installed pex-1.4.6
(pex1.4.6) $ pip list
Package    Version
---------- -------
pex        1.4.6  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 
(pex1.4.6) $ which pex
/home/jsirois/.venv/pex1.4.6/bin/pex
(pex1.4.6) $ pex --version
pex 1.4.6
(pex1.4.6) $ pex responses
Python 3.7.0 (default, Sep 15 2018, 19:13:07) 
[GCC 8.2.1 20180831] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
>>> 
now exiting InteractiveConsole...
$ venv_activate pex1.4.6
(pex1.4.6) $ pex --python=python2.7 responses
Python 2.7.15 (default, Jun 27 2018, 13:05:28) 
[GCC 8.1.1 20180531] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
>>> 
$ venv_activate pex1.4.6
```
This started happening for me too just now, with pex 1.4.6 (was not an issue before).

I'm on OSX 10.13.6. This happens with system python (not with more modern installations). I use pyenv, but package with system python to be able to deliver my apps with just whatever system python is.

I've installed `pip` (a long time ago, and not much else), the list below shows what's on my "system python" currently (notable `setuptools==18.5` and `pip==9.0.1`)

```
~: /usr/local/bin/pip freeze --all
altgraph==0.10.2
bdist-mpkg==0.5.0
bonjour-py==0.3
macholib==1.5.1
matplotlib==1.3.1
modulegraph==0.10.4
numpy==1.8.0rc1
pip==9.0.1
pluggy==0.6.0
py==1.5.2
py2app==0.7.3
pyobjc-core==2.5.1
pyobjc-framework-Accounts==2.5.1
pyobjc-framework-AddressBook==2.5.1
pyobjc-framework-AppleScriptKit==2.5.1
pyobjc-framework-AppleScriptObjC==2.5.1
pyobjc-framework-Automator==2.5.1
pyobjc-framework-CFNetwork==2.5.1
pyobjc-framework-Cocoa==2.5.1
pyobjc-framework-Collaboration==2.5.1
pyobjc-framework-CoreData==2.5.1
pyobjc-framework-CoreLocation==2.5.1
pyobjc-framework-CoreText==2.5.1
pyobjc-framework-DictionaryServices==2.5.1
pyobjc-framework-EventKit==2.5.1
pyobjc-framework-ExceptionHandling==2.5.1
pyobjc-framework-FSEvents==2.5.1
pyobjc-framework-InputMethodKit==2.5.1
pyobjc-framework-InstallerPlugins==2.5.1
pyobjc-framework-InstantMessage==2.5.1
pyobjc-framework-LatentSemanticMapping==2.5.1
pyobjc-framework-LaunchServices==2.5.1
pyobjc-framework-Message==2.5.1
pyobjc-framework-OpenDirectory==2.5.1
pyobjc-framework-PreferencePanes==2.5.1
pyobjc-framework-PubSub==2.5.1
pyobjc-framework-QTKit==2.5.1
pyobjc-framework-Quartz==2.5.1
pyobjc-framework-ScreenSaver==2.5.1
pyobjc-framework-ScriptingBridge==2.5.1
pyobjc-framework-SearchKit==2.5.1
pyobjc-framework-ServiceManagement==2.5.1
pyobjc-framework-Social==2.5.1
pyobjc-framework-SyncServices==2.5.1
pyobjc-framework-SystemConfiguration==2.5.1
pyobjc-framework-WebKit==2.5.1
pyOpenSSL==0.13.1
pyparsing==2.0.1
python-dateutil==1.5
pytz==2013.7
scipy==0.13.0b1
setuptools==18.5
six==1.4.1
vboxapi==1.0
virtualenv==15.1.0
xattr==0.6.4
zope.interface==4.1.1
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
```
Pex explicitly requires `setuptools>=20.3` and your output shows `18.5`:
https://github.com/pantsbuild/pex/blob/64d2cd30b7e8e72a2bb60918237ad3f7d65395ff/pex/version.py#L8

Are you sure pip installation of pex doesn't issue a warning about this? Can you try installing in a fresh virtualenv or else upgrading your setuptools to 20.3 or greater?
I package with a recent version of python, in this case 2.7.15 which has setuptools 39.0.1
The issue manifests at runtime: ie, when the pex gets executed with the OSX system python 2.7.10

My cli (https://pypi.org/project/pickley/) is universal, shebang is `#!/usr/bin/env python`.
At runtime, `pickley --version` fails, pex unpacks itself etc, then crashes trying to determine version.

To clarify: pex itself isn't crashing, but the package it created doesn't work anymore with the OSX system python, I get a similar error `ImportError: cannot import name py31compat` (in my case, when `click` or my own code tries to determine version). 

I'm looking into an alternative way of determining version. It's just new version of pex triggered this situation.
Setup details. Please let me know if you need anything else

```
~/sandbox $ virtualenv --version
16.0.0
~/sandbox $ virtualenv venv
New python executable in /Users/Nakis/sandbox/venv/bin/python2.7
Also creating executable in /Users/Nakis/sandbox/venv/bin/python
Installing setuptools, pip, wheel...done.
 ~/sandbox $ . venv/bin/activate
(venv)  ~/sandbox $ pip install responses pex
...snip...
(venv)  ~/sandbox $ pip freeze --all
certifi==2018.8.24
chardet==3.0.4
cookies==2.2.1
funcsigs==1.0.2
idna==2.7
mock==2.0.0
pbr==4.2.0
pex==1.4.6
pip==18.0
requests==2.19.1
responses==0.9.0
setuptools==40.4.1
six==1.11.0
urllib3==1.23
wheel==0.31.1
(venv)  ~/sandbox $ pex responses
Python 2.7.15 (default, May  1 2018, 16:44:08)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.deps/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File "/private/var/folders/mk/bz4w0f3505190h3n4x21_jlc0000gn/T/tmpun0c31/.bootstrap/pkg_resources/__init__.py", line 77, in <module>
    from . import py31compat
ImportError: cannot import name py31compat
>>> quit()
(venv)  ~/sandbox $ pip uninstall pex
Uninstalling pex-1.4.6:
  Would remove:
    /Users/Nakis/sandbox/venv/bin/pex
    /Users/Nakis/sandbox/venv/lib/python2.7/site-packages/pex-1.4.6.dist-info/*
    /Users/Nakis/sandbox/venv/lib/python2.7/site-packages/pex/*
Proceed (y/n)? y
  Successfully uninstalled pex-1.4.6
(venv)  ~/sandbox $ pip install pex==1.4.5
...snip...
Installing collected packages: setuptools, pex
  Found existing installation: setuptools 40.4.1
    Uninstalling setuptools-40.4.1:
      Successfully uninstalled setuptools-40.4.1
Successfully installed pex-1.4.5 setuptools-33.1.1
(venv)  ~/sandbox $
(venv)  ~/sandbox $ pex responses
Python 2.7.15 (default, May  1 2018, 16:44:08)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
>>>
```
The 1.4.6 release is definitely broken when used with Python 2.7.
This is the commit that causes the issue: #542

I created virtualenv with Python 2.7.13 and used pex-1.4.6.
```
(py27) $ pip list
Package    Version
---------- -------
pex        1.4.6  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 

(py27) $ pex responses
Python 2.7.13 ...
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.deps/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File "/private/var/folders/wz/zvcwh0gs3xqdvjp1_p6xs1p8000_lt/T/tmpC7plPj/.bootstrap/pkg_resources/__init__.py", line 77, in <module>
    from . import py31compat
ImportError: cannot import name py31compat
>>> ^D
```

Then I created a branch on the clone of my pex fork, bumped the version to 1.4.7, and did
```
$ git revert 8f9c8dd
[test-breakage fb7f49f] Revert "Cleanup bootstrap dependencies before handoff. (#542)"
 4 files changed, 37 insertions(+), 144 deletions(-)

$ python setup.py sdist
running sdist
running egg_info
...
creating dist
Creating tar archive
removing 'pex-1.4.7' (and everything under it)
```

Then the import works:
```
(py27) $ pip install -U .../pex/dist/pex-1.4.7.tar.gz 
Processing .../pex/dist/pex-1.4.7.tar.gz
...
Building wheels for collected packages: pex
  Running setup.py bdist_wheel for pex ... done
...
Successfully built pex
Installing collected packages: pex
  Found existing installation: pex 1.4.6
    Uninstalling pex-1.4.6:
      Successfully uninstalled pex-1.4.6
Successfully installed pex-1.4.7

(py27) $ pip list
Package    Version
---------- -------
pex        1.4.7  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 

(py27) $ pex responses
Python 2.7.13 ...
(InteractiveConsole)
>>> import responses
>>> ^D
```

With Python 3.7 everything works with or without revert of #542.
```
(py37) $ pip list
Package    Version
---------- -------
pex        1.4.6  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 

(py37) $ pex responses
Python 3.7.0 ...
(InteractiveConsole)
>>> import responses
>>> ^D
now exiting InteractiveConsole...

(py37) $ pip install -U .../pex/dist/pex-1.4.7.tar.gz 
Processing .../pex/dist/pex-1.4.7.tar.gz
...
Building wheels for collected packages: pex
  Running setup.py bdist_wheel for pex ... done
...
Successfully built pex
Installing collected packages: pex
  Found existing installation: pex 1.4.6
    Uninstalling pex-1.4.6:
      Successfully uninstalled pex-1.4.6
Successfully installed pex-1.4.7

(py37) $ pip list
Package    Version
---------- -------
pex        1.4.7  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 

(py37) $ pex responses
Python 3.7.0 ...
(InteractiveConsole)
>>> import responses
>>> ^D
now exiting InteractiveConsole...
```

Notice that #542 was created in December 2017 and that *setuptools* had a lot of bug fixes and releases itself. Perhaps those changes were not needed any more? You will know better as maintainers the exact use case where this change might be necessary.

Would you consider reverting this change for users of Python 2.7 or at least looking why/how it causes issues and forward-fixing it so that we can have a new working release of pex that supports modern setuptools for the reasons I described previously in #544?
In particular, the issues appear to be with py27 on osx. I'll take some time today to 1st tweak CI to add OSX shard(s) and get a repro with more details. Then fix forward or revert and finally release asap.
Thanks for your help and patience.
@jsirois I am able to reproduce this on Linux consistently too.

The only reason I worked on macOS last night is because I had the checkout of pex repo there and could quickly make an sdist of "1.4.7" to confirm that revert fixes the issue. In my initial testing last week I always used Linux. 

Here are more details in case they can help with your investigation into the root cause.
The Python interpreter I used was not a system vended Python, but a clean compiled Python with no external packages added to its site-packages, so we're not affected by stale vended versions needed by yum, etc.

I used _virtualenv-16.0.0_ to build a virtual environment not the _venv_ module from Python 3 standard library. This is, again, an intentional choice because _venv_ still doesn't have all the features of _virtualenv_.
```
$ curl -LO https://pypi.org/packages/source/v/virtualenv/virtualenv-16.0.0.tar.gz
$ tar xzvf virtualenv-16.0.0.tar.gz
$ /path/to/bin/python virtualenv-16.0.0/virtualenv.py --no-download ~/.venv/py27
$ . ~/.venv/py27/bin/activate
(py27) $ pip install setuptools==40.4.1
(py27) $ pip install -U pip
(py27) $ pip install pex==1.4.6
(py27) $ pip list
Package    Version
---------- -------
pex        1.4.6  
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 
```
With this virtualenv ready, the issue is reproducible on Linux too (I left the lines in the output that identify the interpreter on Linux and the `/tmp` directory is seen in traceback instead of `/private/tmp` from macOS):
```
(py27) $ pex responses
Python 2.7.13 (default, Apr 19 2017, 02:44:33) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/tmp/tmpm8vjVH/.deps/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/tmp/tmpm8vjVH/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/tmp/tmpm8vjVH/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/tmp/tmpm8vjVH/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/tmp/tmpm8vjVH/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File "/tmp/tmpm8vjVH/.bootstrap/pkg_resources/__init__.py", line 77, in <module>
    from . import py31compat
ImportError: cannot import name py31compat
>>> Ctrl-D

(py27) $ pip install -U pex-1.4.7.tar.gz   # the same sdist I made after revert
Processing ./pex-1.4.7.tar.gz
...
Building wheels for collected packages: pex
  Running setup.py bdist_wheel for pex ... done
...
Successfully built pex
Installing collected packages: pex
  Found existing installation: pex 1.4.6
    Uninstalling pex-1.4.6:
      Successfully uninstalled pex-1.4.6
Successfully installed pex-1.4.7

(py27) $ pex responses
Python 2.7.13 (default, Apr 19 2017, 02:44:33) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
>>> Ctrl-D
```
The revert fixes the issue on Linux too.
Hrm. @zvezdan can you, instead of `pex responses` do one of these and provide full output?:
1. `PEX_VERBOSE=9 pex responses`
2. `PEX_VERBOSE=9 pex responses -o responses.pex && PEX_VERBOSE=9 responses.pex`

If so, #2 would be most useful, giving clear distinction between build-time info and run-time info.


@jsirois Here's the output for 1:
```
(py27) $ PEX_VERBOSE=9 pex responses
pex: Please build pex with the subprocess32 module for more reliable requirement installation and interpreter execution.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f83cdcf3290>
Python 2.7.13 (default, Apr 19 2017, 02:44:33)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/tmp/tmp3rr5AM/.deps/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/tmp/tmp3rr5AM/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/tmp/tmp3rr5AM/.deps/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/tmp/tmp3rr5AM/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/tmp/tmp3rr5AM/.deps/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File "/tmp/tmp3rr5AM/.bootstrap/pkg_resources/__init__.py", line 77, in <module>
    from . import py31compat
ImportError: cannot import name py31compat
>>> Ctrl-D
```
and the output for 2 with scrubbed paths:
```
(py27) $ PEX_VERBOSE=9 pex responses -o responses.pex && PEX_VERBOSE=9 ./responses.pex
pex: Please build pex with the subprocess32 module for more reliable requirement installation and interpreter execution.
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f7ac86ce290>
pex: Please build pex with the subprocess32 module for more reliable requirement installation and interpreter execution.
pex: Selecting runtime interpreter based on pexrc: 0.1ms
pex: Found site-library: /home/.../.venv/py27/lib/python2.7/site-packages
pex: Not a tainted path element: /home/.../src/tmp/pex-test/responses.pex/.bootstrap
pex: Not a tainted path element: /home/.../src/tmp/pex-test/responses.pex
pex: Not a tainted path element: /home/.../.venv/py27/lib/python27.zip
pex: Not a tainted path element: /home/.../.venv/py27/lib/python2.7
pex: Not a tainted path element: /home/.../.venv/py27/lib/python2.7/plat-linux2
pex: Not a tainted path element: /home/.../.venv/py27/lib/python2.7/lib-tk
pex: Not a tainted path element: /home/.../.venv/py27/lib/python2.7/lib-old
pex: Not a tainted path element: /home/.../.venv/py27/lib/python2.7/lib-dynload
pex: Not a tainted path element: /path/to/.../lib/python2.7
pex: Not a tainted path element: /path/to/.../lib/python2.7/plat-linux2
pex: Not a tainted path element: /path/to/.../lib/python2.7/lib-tk
pex: Tainted path element: /home/.../.venv/py27/lib/python2.7/site-packages
pex: Scrubbing from user site: /home/.../.local/lib/python2.7/site-packages
pex: Scrubbing from site-packages: /home/.../.venv/py27/lib/python2.7/site-packages
pex: Scrubbing from path_importer_cache: /home/.../.local/lib/python2.7/site-packages
pex: Scrubbing from path_importer_cache: /home/.../.venv/py27/lib/python2.7/site-packages
pex: E: tags for 'linux-x86_64' x PythonInterpreter('/home/.../.venv/py27/bin/python', PythonIdentity('CPython', 2, 7, 13), {('python', '2.7.13'): '/path/to/....13/lib/python2.7/lib-dynload', ('pex', '1.4.6'): '/home/.../.venv/py27/lib/python2.7/site-packages', ('wheel', '0.31.1'): '/home/.../.venv/py27/lib/python2.7/site-packages', ('pip', '18.0'): '/home/.../.venv/py27/lib/python2.7/site-packages', ('wsgiref', '0.1.2'): '/path/to/....13/lib/python2.7', ('setuptools', '40.4.1'): '/home/.../.venv/py27/lib/python2.7/site-packages'}) -> [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Searching dependency cache: ./responses.pex/.deps :: pex: Activating PEX virtual environment from ./responses.pex :: Resolving cookies==2.2.1                             pex: Activating PEX virtual environment from ./responses.pex: 93.0ms
pex:   Searching dependency cache: ./responses.pex/.deps: 13.1ms
pex:     Adding pbr 4.2.0: 0.1ms
pex:     Adding idna 2.7: 0.1ms
pex:     Adding cookies 2.2.1: 0.1ms
pex:     Adding six 1.11.0: 0.1ms
pex:     Adding funcsigs 1.0.2: 0.1ms
pex:     Adding certifi 2018.8.24: 0.1ms
pex:     Adding responses 0.9.0: 0.1ms
pex:     Adding chardet 3.0.4: 0.1ms
pex:     Adding urllib3 1.23: 0.1ms
pex:     Adding mock 2.0.0: 0.1ms
pex:     Adding requests 2.19.1: 0.1ms
pex:   Resolving cookies==2.2.1: 0.7ms
pex:   Resolving funcsigs==1.0.2: 8.0ms
pex:   Resolving responses==0.9.0: 57.0ms
pex:   Resolving pbr==4.2.0: 0.3ms
pex:   Resolving six==1.11.0: 0.3ms
pex:   Resolving urllib3==1.23: 0.3ms
pex:   Resolving chardet==3.0.4: 0.3ms
pex:   Resolving requests==2.19.1: 1.0ms
pex:   Resolving certifi==2018.8.24: 0.3ms
pex:   Resolving idna==2.7: 0.3ms
pex:   Resolving mock==2.0.0: 0.7ms
pex:   Activating pbr 4.2.0: 0.4ms
pex:     Adding sitedir: 0.2ms
pex:   Activating urllib3 1.23: 0.4ms
pex:     Adding sitedir: 0.2ms
pex:   Activating mock 2.0.0: 0.4ms
pex:     Adding sitedir: 0.2ms
pex:   Activating six 1.11.0: 0.3ms
pex:     Adding sitedir: 0.2ms
pex:   Activating idna 2.7: 0.4ms
pex:     Adding sitedir: 0.2ms
pex:   Activating cookies 2.2.1: 0.4ms
pex:     Adding sitedir: 0.3ms
pex:   Activating funcsigs 1.0.2: 0.4ms
pex:     Adding sitedir: 0.3ms
pex:   Activating chardet 3.0.4: 0.4ms
pex:     Adding sitedir: 0.3ms
pex:   Activating certifi 2018.8.24: 0.4ms
pex:     Adding sitedir: 0.3ms
pex:   Activating responses 0.9.0: 0.5ms
pex:     Adding sitedir: 0.3ms
pex:   Activating requests 2.19.1: 0.5ms
pex:     Adding sitedir: 0.3ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: Un-importing third party bootstrap dependency pkg_resources from /home/.../src/tmp/pex-test/responses.pex/.bootstrap
pex: PYTHONPATH contains:
pex:     /home/.../src/tmp/pex-test/responses.pex
pex:   * /home/.../.venv/py27/lib/python27.zip
pex:     /home/.../.venv/py27/lib/python2.7
pex:   * /home/.../.venv/py27/lib/python2.7/plat-linux2
pex:   * /home/.../.venv/py27/lib/python2.7/lib-tk
pex:   * /home/.../.venv/py27/lib/python2.7/lib-old
pex:     /home/.../.venv/py27/lib/python2.7/lib-dynload
pex:     /path/to/.../lib/python2.7
pex:     /path/to/.../lib/python2.7/plat-linux2
pex:     /path/to/.../lib/python2.7/lib-tk
pex:     /home/.../.pex/install/pbr-4.2.0-py2.py3-none-any.whl.6be96516238e4e5b1baee293889642c4e4c3eac2/pbr-4.2.0-py2.py3-none-any.whl
pex:     /home/.../.pex/install/urllib3-1.23-py2.py3-none-any.whl.3630f56fd6bcc8cfd3c568a65834f5b56ebc405e/urllib3-1.23-py2.py3-none-any.whl
pex:     /home/.../.pex/install/mock-2.0.0-py2.py3-none-any.whl.9c729aadd91e3f45efeaa1ada526a02a56176e27/mock-2.0.0-py2.py3-none-any.whl
pex:     /home/.../.pex/install/six-1.11.0-py2.py3-none-any.whl.5fad94645bceb01109c56b5d9bf64de201140300/six-1.11.0-py2.py3-none-any.whl
pex:     /home/.../.pex/install/idna-2.7-py2.py3-none-any.whl.96a4a6d75966bb0aae4ce9bdd177fc4886b37a07/idna-2.7-py2.py3-none-any.whl
pex:     /home/.../.pex/install/cookies-2.2.1-py2.py3-none-any.whl.6b0c79d8a7243c5e05ef8286485f4ec02ce81152/cookies-2.2.1-py2.py3-none-any.whl
pex:     /home/.../.pex/install/funcsigs-1.0.2-py2.py3-none-any.whl.1abdb57bb4c560603d658d202032bc95bdfa8f8a/funcsigs-1.0.2-py2.py3-none-any.whl
pex:     /home/.../.pex/install/chardet-3.0.4-py2.py3-none-any.whl.c4332e34e38b781695ce775973fe40663558a897/chardet-3.0.4-py2.py3-none-any.whl
pex:     /home/.../.pex/install/certifi-2018.8.24-py2.py3-none-any.whl.d24ebf6f0b213fc082815d0f3a816ab6c64b2d1f/certifi-2018.8.24-py2.py3-none-any.whl
pex:     /home/.../.pex/install/responses-0.9.0-py2.py3-none-any.whl.a5224fde5182386549956aca3dd40fed5c9a198b/responses-0.9.0-py2.py3-none-any.whl
pex:     /home/.../.pex/install/requests-2.19.1-py2.py3-none-any.whl.9c19d1f49965483c8e82326b964ec1050dff6851/requests-2.19.1-py2.py3-none-any.whl
pex:   * /home/.../src/tmp/pex-test/responses.pex/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Python 2.7.13 (default, Apr 19 2017, 02:44:33)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import responses
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/.../.pex/install/responses-0.9.0-py2.py3-none-any.whl.a5224fde5182386549956aca3dd40fed5c9a198b/responses-0.9.0-py2.py3-none-any.whl/responses.py", line 41, in <module>
    import mock as std_mock
  File "/home/.../.pex/install/mock-2.0.0-py2.py3-none-any.whl.9c729aadd91e3f45efeaa1ada526a02a56176e27/mock-2.0.0-py2.py3-none-any.whl/mock/__init__.py", line 2, in <module>
    import mock.mock as _mock
  File "/home/.../.pex/install/mock-2.0.0-py2.py3-none-any.whl.9c729aadd91e3f45efeaa1ada526a02a56176e27/mock-2.0.0-py2.py3-none-any.whl/mock/mock.py", line 71, in <module>
    _v = VersionInfo('mock').semantic_version()
  File "/home/.../.pex/install/pbr-4.2.0-py2.py3-none-any.whl.6be96516238e4e5b1baee293889642c4e4c3eac2/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/home/.../.pex/install/pbr-4.2.0-py2.py3-none-any.whl.6be96516238e4e5b1baee293889642c4e4c3eac2/pbr-4.2.0-py2.py3-none-any.whl/pbr/version.py", line 439, in _get_version_from_pkg_resources
    import pkg_resources
  File ".bootstrap/pkg_resources/__init__.py", line 77, in <module>
ImportError: cannot import name py31compat
>>> Ctrl-D
```
Alright - thanks for the help. I have things figured out, but with a little smash all things, here: https://github.com/pantsbuild/pex/pull/553

I'll pick this back up in the morning to whittle down the minimal crisp changes needed and provide a full, satisfying explanation.

Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -419,6 +419,9 @@ def demote_bootstrap(cls):
         TRACER.log('Un-importing third party bootstrap dependency %s from %s'
                    % (mod, bootstrap_path))
         sys.modules.pop(mod)
+        submod_prefix = mod + '.'
+        for submod in [m for m in sys.modules.keys() if m.startswith(submod_prefix)]:
+          sys.modules.pop(submod)
     sys.path.pop(bootstrap_path_index)
     sys.path.append(bootstrap_path)
 


Test patch:
diff --git a/pex/testing.py b/pex/testing.py
--- a/pex/testing.py
+++ b/pex/testing.py
@@ -180,7 +180,7 @@ def make_bdist(name='my_project', version='0.0.0', installer_impl=EggInstaller,
 """
 
 
-def write_simple_pex(td, exe_contents, dists=None, sources=None, coverage=False):
+def write_simple_pex(td, exe_contents, dists=None, sources=None, coverage=False, interpreter=None):
   """Write a pex file that contains an executable entry point
 
   :param td: temporary directory path
@@ -189,6 +189,7 @@ def write_simple_pex(td, exe_contents, dists=None, sources=None, coverage=False)
   :param dists: distributions to include, typically sdists or bdists
   :param sources: sources to include, as a list of pairs (env_filename, contents)
   :param coverage: include coverage header
+  :param interpreter: a custom interpreter to use to build the pex
   """
   dists = dists or []
   sources = sources or []
@@ -198,7 +199,9 @@ def write_simple_pex(td, exe_contents, dists=None, sources=None, coverage=False)
   with open(os.path.join(td, 'exe.py'), 'w') as fp:
     fp.write(exe_contents)
 
-  pb = PEXBuilder(path=td, preamble=COVERAGE_PREAMBLE if coverage else None)
+  pb = PEXBuilder(path=td,
+                  preamble=COVERAGE_PREAMBLE if coverage else None,
+                  interpreter=interpreter)
 
   for dist in dists:
     pb.add_dist_location(dist.location)
diff --git a/tests/test_pex.py b/tests/test_pex.py
--- a/tests/test_pex.py
+++ b/tests/test_pex.py
@@ -14,6 +14,7 @@
 from pex.bin.pex import get_interpreter
 from pex.compatibility import PY2, WINDOWS, nested, to_bytes
 from pex.installer import EggInstaller, WheelInstaller
+from pex.interpreter import PythonInterpreter
 from pex.pex import PEX
 from pex.pex_builder import PEXBuilder
 from pex.pex_info import PexInfo
@@ -373,3 +374,29 @@ def test_pex_run_custom_setuptools_useable():
       )
       rc = PEX(pex.path()).run()
       assert rc == 0
+
+
+def test_pex_run_conflicting_custom_setuptools_useable():
+  # Here we use an older setuptools to build the pex which has a newer setuptools requirement.
+  # These setuptools dists have different pkg_resources APIs:
+  # $ diff \
+  #   <(zipinfo -1 setuptools-20.3.1-py2.py3-none-any.whl | grep pkg_resources/ | sort) \
+  #   <(zipinfo -1 setuptools-40.4.3-py2.py3-none-any.whl | grep pkg_resources/ | sort)
+  # 2a3,4
+  # > pkg_resources/py31compat.py
+  # > pkg_resources/_vendor/appdirs.py
+  with temporary_dir() as resolve_cache:
+    dists = resolve(['setuptools==20.3.1'], cache=resolve_cache)
+    interpreter = PythonInterpreter.from_binary(sys.executable,
+                                                path_extras=[dist.location for dist in dists],
+                                                include_site_extras=False)
+    dists = resolve(['setuptools==40.4.3'], cache=resolve_cache)
+    with temporary_dir() as temp_dir:
+      pex = write_simple_pex(
+        temp_dir,
+        'from pkg_resources import appdirs, py31compat',
+        dists=dists,
+        interpreter=interpreter
+      )
+      rc = PEX(pex.path()).run()
+      assert rc == 0