Repository: pex-tool/pex

Issue:
Path scrubbing not working for python 3.8 package installed with homebrew
Hi there, I got an issue with a pex file embedding a dependency (protobuf) under OSX. I guess it could be related to the protobuf package and how the package is installed with brew but maybe also in the way the scrubbing is done by PEX.

If I install `protobuf` through `pip3.8 install protobuf` the scrubbing works fine because `protobuf` is inside the path `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//google/protobuf/`.

But if you install the `protobuf` brew package through `brew install protobuf`, it is installed under `/usr/local/Cellar/protobuf/` and the python package is registered inside `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages//homebrew-protobuf.pth` with the following line:
```
import site; site.addsitedir('/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages')
```

I put some logs to print `module.__path__` and `site_libs` before `cls._tainted_path(...)` is [called](https://github.com/pantsbuild/pex/blob/master/pex/pex.py#L209). Here is the difference between python3.7 and 3.8.

Python 3.7:
```
Module path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google']
Site_libs : {'/usr/local/lib/python3.7/site-packages', '/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages'}
pex: Scrubbing google.__path__: /usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google
```

Python 3.8:
```
Module path: ['/Users/XXX/repack.pex/google', '/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google']
Site_libs : {'/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages', '/usr/local/lib/python3.8/site-packages'}

```

Notice there is no "Scrubbing" for python 3.8.  It is due to the change in where python package is located. Instead of being inside the python 3.8 directory, it is put as a standard `brew` package. The import is indirectly managed inside the `homebrew-protofbu.pth` file (see above for the path and content)

The `cls._tainted_path(...)` is then not able to to identify that the path matches in python 3.8 since they are completely different.

python 3.7 match
Module path : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**/google
Site_libs : **/usr/local/Cellar/python@3.7/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages**

python 3.8 no match
Module path : /usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages/google
Site_libs : /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages


Any solution ?


Discussion:
```
python3.8 -m site
sys.path = [
    '/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python38.zip',
    '/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8',
    '/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload',
    '/usr/local/lib/python3.8/site-packages',
    '/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages',
    '/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages',
]
```

It seems the issue is maybe related to how the site-package is retrieved here https://github.com/pantsbuild/pex/blob/master/pex/pex.py#L146 ? 

I don't understand all the magic under the hood, let me know if you have a solution to my problem.
Just responding to your second comment: https://github.com/pantsbuild/pex/issues/1025#issuecomment-683922218
python3.8 -m site prints out the sys.path, which includes `/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages` but not because that entry is a "site package". IOW, you should find a similar difference using the following:
```
$ python3.8 -m site
sys.path = [
    '/home/jsirois',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/home/jsirois/.local/lib/python3.8/site-packages',
    '/usr/lib/python3.8/site-packages',
]
USER_BASE: '/home/jsirois/.local' (exists)
USER_SITE: '/home/jsirois/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True
```

Vs:
```
$ python3.8 -c 'import site; print(site.getsitepackages())'
['/usr/lib/python3.8/site-packages']
```

@R1kk3r can you run your PEX file like so?: `PEX_VERBOSE=9 /my/pex/file ...`

I want to see if this log line triggers for `/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/homebrew-protobuf.pth`:
https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/pex.py#L130

I'm guessing you will see the log line. As to why things are not working, well, its the contents of the `.pth` file:
```
import site; site.addsitedir('/usr/local/Cellar/protobuf/3.12.4/libexec/lib/python3.8/site-packages')
```

Our `.pth` handling code does not properly anticipate this form of manipulation. I think we may just need to iteratively allow for the definition of site-packages to be changed by each .pth file. As things stand we calculate site-packages once up front, and then iterate through .pth files expecting site-packages will be unaltered:
https://github.com/pantsbuild/pex/blob/b51b340bea8a585a246997071a815d008986d6e3/pex/pex.py#L309-L315
Note to self, the relevant docs: https://docs.python.org/3/library/site.html
Actually, it looks like `site.addsitedir` just adds the directory to `sys.path`, It does not alter what `site.getsitepackages()` returns. I think then the solution is to snapshot `sys.path` prior to executing `.pth` files and then consider any new paths added to `sys.path` post-execution of all such files as taineted.

Resolving patch:
diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -5,6 +5,7 @@
 
 import contextlib
 import os
+import sys
 import tempfile
 from hashlib import sha1
 from site import makepath
@@ -198,21 +199,42 @@ def iter_pth_paths(filename):
     known_paths = set()
 
     with f:
-        for line in f:
+        for i, line in enumerate(f, start=1):
             line = line.rstrip()
             if not line or line.startswith("#"):
                 continue
             elif line.startswith(("import ", "import\t")):
+                # One important side effect of executing import lines can be alteration of the
+                # sys.path directly or indirectly as a programmatic way to add sys.path entries
+                # in contrast to the standard .pth mechanism of including fixed paths as
+                # individual lines in the file. Here we capture all such programmatic attempts
+                # to expand the sys.path and report the additions.
+                original_sys_path = sys.path[:]
                 try:
+                    # N.B.: Setting sys.path to empty is ok since all the .pth files we find and
+                    # execute have already been found and executed by our ambient sys.executable
+                    # when it started up before running this PEX file. As such, all symbols imported
+                    # by the .pth files then will still be available now as cached in sys.modules.
+                    sys.path = []
                     exec_function(line, globals_map={})
-                    continue
-                except Exception:
-                    # NB: import lines are routinely abused with extra code appended using `;` so the class of
-                    # exceptions that might be raised in broader than ImportError. As such we cacth broadly
-                    # here.
+                    for path in sys.path:
+                        yield path
+                except Exception as e:
+                    # NB: import lines are routinely abused with extra code appended using `;` so
+                    # the class of exceptions that might be raised in broader than ImportError. As
+                    # such we catch broadly here.
+                    TRACER.log(
+                        "Error executing line {linenumber} of {pth_file} with content:\n"
+                        "{content}\n"
+                        "Error was:\n"
+                        "{error}".format(linenumber=i, pth_file=filename, content=line, error=e),
+                        V=9,
+                    )
 
                     # Defer error handling to the higher level site.py logic invoked at startup.
                     return
+                finally:
+                    sys.path = original_sys_path
             else:
                 extras_dir, extras_dir_case_insensitive = makepath(dirname, line)
                 if extras_dir_case_insensitive not in known_paths and os.path.exists(extras_dir):


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1,6 +1,7 @@
 # Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
+import errno
 import filecmp
 import functools
 import glob
@@ -11,6 +12,7 @@
 import shutil
 import subprocess
 import sys
+import uuid
 from contextlib import contextmanager
 from textwrap import dedent
 from zipfile import ZipFile
@@ -20,6 +22,7 @@
 from pex import pex_builder
 from pex.common import safe_copy, safe_mkdir, safe_open, safe_rmtree, safe_sleep, temporary_dir
 from pex.compatibility import WINDOWS, nested, to_bytes
+from pex.executor import Executor
 from pex.interpreter import PythonInterpreter
 from pex.pex_info import PexInfo
 from pex.pip import get_pip
@@ -1917,6 +1920,61 @@ def test_issues_745_extras_isolation():
         assert subprocess32_location.startswith(pex_root)
 
 
+@pytest.yield_fixture
+def issues_1025_pth():
+    def safe_rm(path):
+        try:
+            os.unlink(path)
+        except OSError as e:
+            if e.errno != errno.ENOENT:
+                raise
+
+    cleanups = []
+
+    def write_pth(pth_path, sitedir):
+        cleanups.append(lambda: safe_rm(pth_path))
+        with open(pth_path, "w") as fp:
+            fp.write("import site; site.addsitedir({!r})\n".format(sitedir))
+
+    try:
+        yield write_pth
+    finally:
+        for cleanup in cleanups:
+            cleanup()
+
+
+def test_issues_1025_extras_isolation(issues_1025_pth):
+    python, pip = ensure_python_distribution(PY36)
+    interpreter = PythonInterpreter.from_binary(python)
+    _, stdout, _ = interpreter.execute(args=["-c", "import site; print(site.getsitepackages()[0])"])
+    with temporary_dir() as tmpdir:
+        sitedir = os.path.join(tmpdir, "sitedir")
+        Executor.execute(cmd=[pip, "install", "--target", sitedir, "ansicolors==1.1.8"])
+
+        pth_path = os.path.join(stdout.strip(), "issues_1025.{}.pth".format(uuid.uuid4().hex))
+        issues_1025_pth(pth_path, sitedir)
+
+        pex_file = os.path.join(tmpdir, "isolated.pex")
+        results = run_pex_command(args=["-o", pex_file], python=python)
+        results.assert_success()
+
+        output, returncode = run_simple_pex(
+            pex_file,
+            args=["-c", "import colors"],
+            interpreter=interpreter,
+            env=make_env(PEX_VERBOSE="9"),
+        )
+        assert returncode != 0, output
+
+        output, returncode = run_simple_pex(
+            pex_file,
+            args=["-c", "import colors"],
+            interpreter=interpreter,
+            env=make_env(PEX_VERBOSE="9", PEX_INHERIT_PATH="fallback"),
+        )
+        assert returncode == 0, output
+
+
 def test_trusted_host_handling():
     python = ensure_python_interpreter(PY27)
     # Since we explicitly ask Pex to find links at http://www.antlr3.org/download/Python, it should