Repository: pex-tool/pex

Issue:
/usr/lib64/python2.7 from system python gets into sys.path at runtime
(I filed this before but realised I didn't have all the info)

This is with rhel7. I've also tried on a clean vm.

Building a pex with poetry, which requires subprocess32:
```
$ pex -c poetry poetry==0.12.17 --interpreter-constraint ">=2.7,<2.8" -o poetry
$ ./poetry show
                                          
[AttributeError]           
'module' object has no attribute 'run'

$ PEX_INTERPRETER=1 ./poetry
Python 2.7.5 (default, Sep 12 2018, 05:31:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess32
>>> subprocess32.__file__
'/usr/lib64/python2.7/site-packages/subprocess32.pyc'
>>> import sys
>>> sys.path
['/home/daniel.j.murray/git/mas/poetry', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2' ... <snip>
```
(The error is because I have a super old subprocess32 installed as part of the OS and it doesn't seem to have a run() method)

This only happens when running the pex, if I use pex (which is in a virtual environment) to start an interpreter the paths work ok.

If anyone could assist it would be greatly appreciated :)


Discussion:
I don't have my hands on an rhel7 installation, but I could not repro using centos7, which should be pretty dang close. What pex version are you using?

My repro attempt went like so using pex latest (1.6.8):
```
$ cat <<EOF | docker build --network host -t pantsbuild/pex/issues/745 -
FROM centos:7

RUN yum -y update && yum -y install gcc python-devel
EOF
...
$ docker run --rm -it --network host pantsbuild/pex/issues/745 bash -i
[root@gill /]# curl -sSL https://github.com/pantsbuild/pex/releases/download/v1.6.8/pex -O
[root@gill /]# chmod +x pex
[root@gill /]# PEX_VERBOSE=9 ./pex -vvvvvvvvv -c poetry poetry==0.12.17 --interpreter-constraint ">=2.7,<2.8" -o poetry
...
[root@gill /]# PEX_VERBOSE=9 ./poetry show
...
                                                                   
[RuntimeError]                                      
Poetry could not find a pyproject.toml file in / or its parents  
                                                                   
show [--no-dev] [-t|--tree] [-l|--latest] [-o|--outdated] [-a|--all] [--] [<package>]
```

Maybe you could re-run the `pex -c ...` command and `./poetry show` command exactly as in my example above setting (PEX_VERBOSE=9 and using -vvvvvvvvv) and include full output if this doesn't work for you using pex latest. That debug info will reveal any system python library paths that are not scrubbed by PEX.
Thanks for the help!

There is one thing that I didn't realise, which is the released version of poetry doesn't use subprocess32 (yet), I had a package from a private repo. It's still possible to demonstrate the issue though through looking at the paths:

```
$ docker run --rm -it centos:7
# https_proxy=172.17.0.1:3128 curl -k -sSL https://github.com/pantsbuild/pex/releases/download/v1.6.8/pex -O
# chmod a+x pex
# ./pex
Python 2.7.5 (default, Oct 30 2018, 23:45:53) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> sys.path
['/tmp/tmpGlErjN', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/tmp/tmpGlErjN/.bootstrap']
```

and if I install subprocess32 I can import it:
```
# yum install -y python2-subprocess32
<snip>
# ./pex
Python 2.7.5 (default, Oct 30 2018, 23:45:53) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess32
>>> subprocess32.__file__
'/usr/lib64/python2.7/site-packages/subprocess32.pyc'
```

With the pex for poetry created I can get to the system subprocess32:
```
# PEX_INTERPRETER=1 ./poetry
Python 2.7.5 (default, Jun 20 2019, 20:27:34) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess32
>>> subprocess32.__file__
'/usr/lib64/python2.7/site-packages/subprocess32.pyc'
>>>
```

I've attached the log files for building the poetry pex, here are what looks like to me the relevant bits:

```
pex: Using $PATH constrained by [u'==2.7.*', u'>=3.4,<4']
pex: Constraints on interpreters: [u'==2.7.*', u'>=3.4,<4'], Matching Interpreter: /usr/bin/python2.7
pex: Selecting runtime interpreter: 6.4ms
pex: Using the current interpreter /usr/bin/python since it matches constraints.
pex: Found site-library: /usr/lib/python2.7/site-packages
pex: Found site-library: /usr/lib/site-python
pex: Found site-library: /usr/lib64/python2.7/site-packages
pex: Not a tainted path element: /pex/.bootstrap
pex: Not a tainted path element: /pex
pex: Not a tainted path element: /usr/lib64/python27.zip
pex: Not a tainted path element: /usr/lib64/python2.7
pex: Not a tainted path element: /usr/lib64/python2.7/plat-linux2
pex: Not a tainted path element: /usr/lib64/python2.7/lib-tk
pex: Not a tainted path element: /usr/lib64/python2.7/lib-old
pex: Not a tainted path element: /usr/lib64/python2.7/lib-dynload
pex: Tainted path element: /usr/lib64/python2.7/site-packages
pex: Tainted path element: /usr/lib/python2.7/site-packages
pex: Scrubbing from user site: /root/.local/lib/python2.7/site-packages
pex: Scrubbing from site-packages: /usr/lib64/python2.7/site-packages
pex: Scrubbing from site-packages: /usr/lib/python2.7/site-packages
pex: Scrubbing from path_importer_cache: /usr/lib64/python2.7/site-packages
pex: Scrubbing from path_importer_cache: /usr/lib/python2.7/site-packages
pex: Scrubbing from path_importer_cache: /root/.local/lib/python2.7/site-packages
```
Full log:
[pex_build_poetry.log](https://github.com/pantsbuild/pex/files/3395362/pex_build_poetry.log)

And a log of just running pex as an interpreter to look at the path:
[pex_run.log](https://github.com/pantsbuild/pex/files/3395361/pex_run.log)

Thank you so much for your help, and please let me know if there's anything I can do to help.
OK, this appears to be a bug scrubbing libraries pex itself uses. This is not a problem for the stdlib, setuptools or wheel (pex vendors the latter two). It _is_ a problem for any pex extras installed on the standard sys.path, namely:
+ subprocess32
+ requests
+ cachecontrol (CacheControl)

A quick hack in the docker container we're sharing shows this fixes:
```diff
[root@80cb58667c2d /]# diff -u pex.original/.deps/pex-1.6.8-py2.py3-none-any.whl/pex/pex.py pex.pex/.deps/pex-1.6.8-py2.py3-none-any.whl/pex/pex.py
--- pex.original/.deps/pex-1.6.8-py2.py3-none-any.whl/pex/pex.py	2019-06-26 20:59:18.000000000 +0000
+++ pex.pex/.deps/pex-1.6.8-py2.py3-none-any.whl/pex/pex.py	2019-08-07 02:10:43.693349188 +0000
@@ -173,7 +173,11 @@
     for module_name, module in modules.items():
       # builtins can stay
       if not hasattr(module, '__path__'):
-        new_modules[module_name] = module
+        path = getattr(module, '__file__', '')
+        if not any(path.startswith(s) for s in site_libs):
+          new_modules[module_name] = module
+        else:
+          print('>>> scrubbed {}: {}'.format(module_name, module))
         continue
 
       # Unexpected objects, e.g. PEP 420 namespace packages, should just be dropped.
```

Thanks very much for your detailed repro info - it was immensely useful.

I'll try to put together a PR.



Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -174,7 +174,13 @@ def minimum_sys_modules(cls, site_libs, modules=None):
     new_modules = {}
 
     for module_name, module in modules.items():
-      # builtins can stay
+      # Tainted modules should be dropped.
+      module_file = getattr(module, '__file__', os.devnull)
+      if cls._tainted_path(module_file, site_libs):
+        TRACER.log('Dropping %s' % (module_name,), V=3)
+        continue
+
+      # Untainted non-packages (builtin modules) need no further special handling and can stay.
       if not hasattr(module, '__path__'):
         new_modules[module_name] = module
         continue
@@ -184,13 +190,13 @@ def minimum_sys_modules(cls, site_libs, modules=None):
         TRACER.log('Dropping %s' % (module_name,), V=3)
         continue
 
-      # Pop off site-impacting __path__ elements in-place.
+      # Drop tainted package paths.
       for k in reversed(range(len(module.__path__))):
         if cls._tainted_path(module.__path__[k], site_libs):
           TRACER.log('Scrubbing %s.__path__: %s' % (module_name, module.__path__[k]), V=3)
           module.__path__.pop(k)
 
-      # It still contains path elements not in site packages, so it can stay in sys.modules
+      # The package still contains untainted path elements, so it can stay.
       if module.__path__:
         new_modules[module_name] = module
 


Test patch:
diff --git a/pex/testing.py b/pex/testing.py
--- a/pex/testing.py
+++ b/pex/testing.py
@@ -270,14 +270,14 @@ def assert_failure(self):
     assert self.return_code != 0
 
 
-def run_pex_command(args, env=None):
+def run_pex_command(args, env=None, python=sys.executable):
   """Simulate running pex command for integration testing.
 
   This is different from run_simple_pex in that it calls the pex command rather
   than running a generated pex.  This is useful for testing end to end runs
   with specific command line arguments or env options.
   """
-  cmd = [sys.executable, '-mpex', '-vvvvv'] + list(args)
+  cmd = [python, '-mpex', '-vvvvv'] + list(args)
   process = Executor.open_process(cmd=cmd, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
   output, error = process.communicate()
   return IntegResults(output.decode('utf-8'), error.decode('utf-8'), process.returncode)
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -14,7 +14,7 @@
 
 import pytest
 
-from pex.common import safe_copy, safe_sleep
+from pex.common import safe_copy, safe_open, safe_sleep
 from pex.compatibility import WINDOWS, nested, to_bytes
 from pex.installer import EggInstaller
 from pex.pex_info import PexInfo
@@ -29,6 +29,7 @@
     PY27,
     PY35,
     PY36,
+    ensure_python_distribution,
     ensure_python_interpreter,
     get_dep_dist_names_from_pex,
     make_sdist,
@@ -1600,3 +1601,47 @@ def test_pex_reexec_constraints_dont_match_current_pex_python():
   _assert_exec_chain(exec_chain=[interpreter],
                      pex_python=interpreter,
                      interpreter_constraints=['=={}'.format(version)])
+
+
+@pytest.mark.skipif(IS_PYPY,
+                    reason='Our pyenv interpreter setup fails under pypy: '
+                           'https://github.com/pantsbuild/pex/issues/477')
+def test_issues_745_extras_isolation():
+  # Here we ensure one of our extras, `subprocess32`, is properly isolated in the transition from
+  # pex bootstrapping where it is imported by `pex.executor` to execution of user code.
+  python, pip = ensure_python_distribution(PY27)
+  subprocess.check_call([pip, 'install', 'subprocess32'])
+  with temporary_dir() as td:
+    src_dir = os.path.join(td, 'src')
+    with safe_open(os.path.join(src_dir, 'test_issues_745.py'), 'w') as fp:
+      fp.write(dedent("""
+        import subprocess32
+
+        print(subprocess32.__file__)
+      """))
+
+    pex_file = os.path.join(td, 'test.pex')
+
+    run_pex_command(['--sources-directory={}'.format(src_dir),
+                     '--entry-point=test_issues_745',
+                     '-o', pex_file],
+                    python=python)
+
+    # The pex runtime should scrub subprocess32 since it comes from site-packages and so we should
+    # not have access to it.
+    with pytest.raises(subprocess.CalledProcessError):
+      subprocess.check_call([python, pex_file])
+
+    # But if the pex has a declared dependency on subprocess32 we should be able to find the
+    # subprocess32 bundled into the pex.
+    pex_root = os.path.realpath(os.path.join(td, 'pex_root'))
+    run_pex_command(['subprocess32',
+                     '--sources-directory={}'.format(src_dir),
+                     '--entry-point=test_issues_745',
+                     '-o', pex_file],
+                    python=python)
+
+    output = subprocess.check_output([python, pex_file], env=make_env(PEX_ROOT=pex_root))
+
+    subprocess32_location = os.path.realpath(output.decode('utf-8').strip())
+    assert subprocess32_location.startswith(pex_root)
diff --git a/tests/test_pex.py b/tests/test_pex.py
--- a/tests/test_pex.py
+++ b/tests/test_pex.py
@@ -118,31 +118,42 @@ def raise_on_exit():
 
 
 def test_minimum_sys_modules():
+  # tainted modules evict
+  tainted_module = ModuleType('tainted_module')
+  tainted_module.__file__ = 'bad_path'
+  modules = {'tainted_module': tainted_module}
+  new_modules = PEX.minimum_sys_modules(site_libs=[], modules=modules)
+  assert new_modules == modules
+  new_modules = PEX.minimum_sys_modules(site_libs=['bad_path'], modules=modules)
+  assert new_modules == {}
+
   # builtins stay
   builtin_module = ModuleType('my_builtin')
-  modules = {'my_builtin': builtin_module}
-  new_modules = PEX.minimum_sys_modules([], modules)
+  stdlib_module = ModuleType('my_stdlib')
+  stdlib_module.__file__ = 'good_path'
+  modules = {'my_builtin': builtin_module, 'my_stdlib': stdlib_module}
+  new_modules = PEX.minimum_sys_modules(site_libs=[], modules=modules)
   assert new_modules == modules
-  new_modules = PEX.minimum_sys_modules(['bad_path'], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=['bad_path'], modules=modules)
   assert new_modules == modules
 
-  # tainted evict
+  # tainted packages evict
   tainted_module = ModuleType('tainted_module')
   tainted_module.__path__ = ['bad_path']
   modules = {'tainted_module': tainted_module}
-  new_modules = PEX.minimum_sys_modules([], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=[], modules=modules)
   assert new_modules == modules
-  new_modules = PEX.minimum_sys_modules(['bad_path'], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=['bad_path'], modules=modules)
   assert new_modules == {}
   assert tainted_module.__path__ == []
 
-  # tainted cleaned
+  # tainted packages cleaned
   tainted_module = ModuleType('tainted_module')
   tainted_module.__path__ = ['bad_path', 'good_path']
   modules = {'tainted_module': tainted_module}
-  new_modules = PEX.minimum_sys_modules([], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=[], modules=modules)
   assert new_modules == modules
-  new_modules = PEX.minimum_sys_modules(['bad_path'], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=['bad_path'], modules=modules)
   assert new_modules == modules
   assert tainted_module.__path__ == ['good_path']
 
@@ -159,7 +170,7 @@ class FakeModule(object):
   tainted_module = FakeModule()
   tainted_module.__path__ = bad_path   # Not a list as expected
   modules = {'tainted_module': tainted_module}
-  new_modules = PEX.minimum_sys_modules(['bad_path'], modules)
+  new_modules = PEX.minimum_sys_modules(site_libs=['bad_path'], modules=modules)
   assert new_modules == {}