Repository: pex-tool/pex

Issue:
import path leak for `six` with OSX default python
when running a pex that bundles `six` with the OSX system python (`/usr/bin/python`), the import of `six` references the system path (an ancient version of `six`) rather than the one provided with the pex. this seems to relate to OSX's bespoke usage of the "Extras" path in their installation, which pex does not classify as user-site or site-packages during the sys.path scrubbing.

repro:

```
[illuminati source (master)]$ PATH=/usr/bin ~/bin/pex six -o /tmp/throwaway.pex
[illuminati source (master)]$ PATH=/usr/bin /tmp/throwaway.pex
Python 2.7.10 (default, Jul 30 2016, 18:31:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys, six
>>> print(sys.executable, six.__file__)
('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc')
>>> from pprint import pprint
>>> pprint(sys.path)
['/tmp/throwaway.pex/.bootstrap',
 '/tmp/throwaway.pex',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 u'/Users/kwilson/.pex/install/six-1.10.0-py2.py3-none-any.whl.a99dfb27e60da3957f6667853b91ea52e173da0a/six-1.10.0-py2.py3-none-any.whl']
```



Discussion:
related: https://github.com/pypa/pip/issues/2468

how do you intend to address this? the first thing that comes to my mind is just scrubbing 'Extras' from sys.path, though that could get messy...

haven't dug into it but on the surface it seems like it could be as simple as just scrubbing `os.path.join(sys.prefix, 'Extras')` if present, I think.

let me know if you have a better idea.

I think that's pretty reasonable, considering the scrubbing that pex already does.

Is there any interim solution to this for affected users?
this is limited to the OSX default interpreter, so one possible solution would be to use an alternate python interpreter (e.g. one from python.org, homebrew or built from sources).
I'm running into this issue even when using a virtualenv with either a python.org interpreter or homebrew interpreter. Is there by chance any way to tell Pants which interpreter to use with PEX? I can't find anything in the documentation, but I'm new to working with this so I might be missing something.

Or is there some way maybe to specify additional paths to scrub? I could see that being fairly useful if it exists. We have a couple devs running into this issue and it'd be great if we didn't have to wipe and revert back to El Capitan to get our local builds working.
can you demonstrate the issue with a python.org or homebrew interpreter? I don't seem to reproduce that on my end and afaict this is very specific to modern OSX /usr/bin/python.

for posterity, here's a quick code-based workaround that was discussed with another user:

```
$ /usr/bin/python ~/bin/pex six
Python 2.7.10 (default, Jul 30 2016, 19:40:32) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> sys.path = [p for p in sys.path if 'Extras/lib/python' not in p]
>>> import six; six.__file__
'/private/var/folders/3t/xkwqrkld4xxgklk2s4n41jb80000gn/T/tmpEXfjuN/.deps/six-1.10.0-py2.py3-none-any.whl/six.py'
>>>
```
```
mateo-2:pants mateo$ ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<bootstraps>
                     tests/python/pants_test/util/test_contextutil.py:16: in <module>
                         import mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/__init__.py:2: in <module>
                         import mock.mock as _mock
                     .pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl/mock/mock.py:68: in <module>
                         from six import wraps
                     E   ImportError: cannot import name wraps
                     ============ 1 error in 0.06 seconds =============

mateo-2:~ mateo$ /usr/local/bin/python2.7
Python 2.7.13 (default, Jan 17 2017, 19:17:41)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

```

I was able to repro as you see above - this was triggered when setting up Pants on a new MacBookPro. It was using the homebrew python and had a reboot mixed in, on a fresh clone of Pants at today's HEAD.

The TLDR is that, on a new MacBookPro at least, your set of scrubbed sites is perhaps incomplete.
```
mateo-2:pants mateo$ cat /Library/Python/2.7/site-packages/Extras.pth
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: Not a tainted path element: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
<snip>
mateo-2:~ mateo$ /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Python 2.7.10 (default, Jul 30 2016, 19:40:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> from six import wraps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name wraps
```

But I don't see why Pex is constructing the path the way it does? It seems like the `.deps` should be above the randomly found site-packages dirs on disk:
``` 
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
<snip>
pex: PYTHONPATH contains:
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.bootstrap
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4
                     pex:     /Users/mateo/dev/pants/src/python
                     pex:     /usr/local/lib/python2.7/site-packages
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
                     pex:   * /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
                     pex:     /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/linecache2-1.0.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_cov-1.8.1-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest-2.6.4-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/cov_core-1.15.0-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/funcsigs-1.0.2-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/six-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pytest_timeout-0.5-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/py-1.4.32-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/traceback2-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/unittest2-1.1.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/argparse-1.4.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/pbr-1.10.0-py2.py3-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/coverage-3.7.1-cp27-cp27m-macosx_10_12_intel.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/ansicolors-1.0.2-py2-none-any.whl
                     pex:     /Users/mateo/dev/pants/.pants.d/python-setup/chroots/0b95cab868da3fb1387cfe97b2e86b22b627b4b4/.deps/mock-1.3.0-py2.py3-none-any.whl
```

This passes for me - unless I uninstall the six module from pip - then it falls back to the System 1.4.1.

Workaround for the afflicted is to install homebrew python, install `six` or another problem module, and then override the global PYTHONPATH, a la
```
export PYTHONPATH="/usr/local/lib/python2.7/site-packages/"
```

Now - if you do that and then uninstall the global six...
```
mateo-2:pants mateo$ pip uninstall six
Uninstalling six-1.10.0:
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/DESCRIPTION.rst
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/INSTALLER
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/METADATA
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/RECORD
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/WHEEL
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/metadata.json
  /usr/local/lib/python2.7/site-packages/six-1.10.0.dist-info/top_level.txt
  /usr/local/lib/python2.7/site-packages/six.py
  /usr/local/lib/python2.7/site-packages/six.pyc
Proceed (y/n)? y
  Successfully uninstalled six-1.10.0
mateo-2:pants mateo$ PEX_VERBOSE=4 ./pants test --no-test-pytest-timeouts tests/python/pants_test/util:contextutil
Traceback (most recent call last):
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_exe.py", line 18, in <module>
    from pants.bin.pants_runner import PantsRunner  # isort:skip
  File "/Users/mateo/dev/pants/src/python/pants/bin/pants_runner.py", line 12, in <module>
    from pants.bin.remote_pants_runner import RemotePantsRunner
  File "/Users/mateo/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 12, in <module>
    from pants.java.nailgun_client import NailgunClient
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_client.py", line 14, in <module>
    from pants.java.nailgun_io import NailgunStreamReader
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_io.py", line 16, in <module>
    from pants.java.nailgun_protocol import ChunkType, NailgunProtocol
  File "/Users/mateo/dev/pants/src/python/pants/java/nailgun_protocol.py", line 10, in <module>
    import six
ImportError: No module named six
mateo-2:pants mateo$
```
Perhaps this is all expected.

I had a couple false starts to this report where I thought I was seeing non-determinism in the sys.path patching - it seemed to do surprising things to the order, and _perhaps_ reverse the ordering at times? It is completely possible that I was just getting fooled by the cache hits.

But I can provide more of the verbose output or sys.paths as needed. 
Any updates on this? I'm hitting it too now. I have a brew python installed, but Pants selects the system one for whatever reason, and the ancient version of six there is leaking into the chroot.
It was probably buried in the breadth of my reply but you can hardcode the
PYTHONPATH to point to just the homebrew one and at least proceed.

On Sun, Jan 29, 2017, 3:11 AM Benjy Weinberger <notifications@github.com>
wrote:

> Any updates on this? I'm hitting it too now. I have a brew python
> installed, but Pants selects the system one for whatever reason, and the
> ancient version of six there is leaking into the chroot.
>
> —
> You are receiving this because you commented.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/pantsbuild/pex/issues/302#issuecomment-275900288>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABqcAadjDjPg7MalJ4lszwoXbo9uctJFks5rXEm0gaJpZM4KJWTZ>
> .
>

grabbing this one due to user impact.

Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -21,6 +21,7 @@
 from .orderedset import OrderedSet
 from .pex_info import PexInfo
 from .tracer import TRACER
+from .util import iter_pth_paths
 from .variables import ENV
 
 
@@ -88,18 +89,38 @@ def _activate(self):
   @classmethod
   def _extras_paths(cls):
     standard_lib = sysconfig.get_python_lib(standard_lib=True)
+
     try:
       makefile = sysconfig.parse_makefile(sysconfig.get_makefile_filename())
     except (AttributeError, IOError):
       # This is not available by default in PyPy's distutils.sysconfig or it simply is
       # no longer available on the system (IOError ENOENT)
       makefile = {}
+
     extras_paths = filter(None, makefile.get('EXTRASPATH', '').split(':'))
     for path in extras_paths:
       yield os.path.join(standard_lib, path)
 
-  @classmethod
-  def _get_site_packages(cls):
+    # Handle .pth injected paths as extras.
+    sitedirs = cls._get_site_packages()
+    for pth_path in cls._scan_pth_files(sitedirs):
+      TRACER.log('Found .pth file: %s' % pth_path, V=3)
+      for extras_path in iter_pth_paths(pth_path):
+        yield extras_path
+
+  @staticmethod
+  def _scan_pth_files(dir_paths):
+    """Given an iterable of directory paths, yield paths to all .pth files within."""
+    for dir_path in dir_paths:
+      if not os.path.exists(dir_path):
+        continue
+
+      pth_filenames = (f for f in os.listdir(dir_path) if f.endswith('.pth'))
+      for pth_filename in pth_filenames:
+        yield os.path.join(dir_path, pth_filename)
+
+  @staticmethod
+  def _get_site_packages():
     try:
       from site import getsitepackages
       return set(getsitepackages())
diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -9,11 +9,13 @@
 import tempfile
 import uuid
 from hashlib import sha1
+from site import makepath
 from threading import Lock
 
 from pkg_resources import find_distributions, resource_isdir, resource_listdir, resource_string
 
 from .common import rename_if_empty, safe_mkdir, safe_mkdtemp, safe_open
+from .compatibility import exec_function
 from .finders import register_finders
 
 
@@ -213,3 +215,33 @@ def named_temporary_file(*args, **kwargs):
       yield fp
   finally:
     os.remove(fp.name)
+
+
+def iter_pth_paths(filename):
+  """Given a .pth file, extract and yield all inner paths without honoring imports. This shadows
+  python's site.py behavior, which is invoked at interpreter startup."""
+  try:
+    f = open(filename, 'rU')  # noqa
+  except IOError:
+    return
+
+  dirname = os.path.dirname(filename)
+  known_paths = set()
+
+  with f:
+    for line in f:
+      line = line.rstrip()
+      if not line or line.startswith('#'):
+        continue
+      elif line.startswith(('import ', 'import\t')):
+        try:
+          exec_function(line)
+          continue
+        except Exception:
+          # Defer error handling to the higher level site.py logic invoked at startup.
+          return
+      else:
+        extras_dir, extras_dir_case_insensitive = makepath(dirname, line)
+        if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):
+          yield extras_dir
+          known_paths.add(extras_dir_case_insensitive)


Test patch:
diff --git a/tests/test_util.py b/tests/test_util.py
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -11,11 +11,11 @@
 from twitter.common.contextutil import temporary_dir
 
 from pex.common import safe_mkdir
-from pex.compatibility import nested
+from pex.compatibility import nested, to_bytes
 from pex.installer import EggInstaller, WheelInstaller
 from pex.pex_builder import PEXBuilder
 from pex.testing import make_bdist, run_simple_pex, temporary_content, write_zipfile
-from pex.util import CacheHelper, DistributionHelper, named_temporary_file
+from pex.util import CacheHelper, DistributionHelper, iter_pth_paths, named_temporary_file
 
 try:
   from unittest import mock
@@ -174,3 +174,35 @@ def test_distributionhelper_egg_assert():
     'setuptools'
   )
   assert len(d.resource_listdir('/')) > 3
+
+
+@mock.patch('os.path.exists', autospec=True, spec_set=True)
+def test_iter_pth_paths(mock_exists):
+  # Ensure path checking always returns True for dummy paths.
+  mock_exists.return_value = True
+
+  with temporary_dir() as tmpdir:
+    in_tmp = lambda f: os.path.join(tmpdir, f)
+
+    PTH_TEST_MAPPING = {
+      # A mapping of .pth file content -> expected paths.
+      '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python\n': [
+        '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python'
+      ],
+      'relative_path\nrelative_path2\n\nrelative_path3': [
+        in_tmp('relative_path'),
+        in_tmp('relative_path2'),
+        in_tmp('relative_path3')
+      ],
+      'duplicate_path\nduplicate_path': [in_tmp('duplicate_path')],
+      'randompath\nimport nosuchmodule\n': [in_tmp('randompath')],
+      'import nosuchmodule\nfoo': [],
+      'import nosuchmodule\n': [],
+      'import bad)syntax\n': [],
+    }
+
+    for i, pth_content in enumerate(PTH_TEST_MAPPING):
+      pth_tmp_path = os.path.abspath(os.path.join(tmpdir, 'test%s.pth' % i))
+      with open(pth_tmp_path, 'wb') as f:
+        f.write(to_bytes(pth_content))
+      assert sorted(PTH_TEST_MAPPING[pth_content]) == sorted(list(iter_pth_paths(pth_tmp_path)))