Repository: pex-tool/pex

Issue:
`--inherit-path {prefer,fallback}` does not work but `PEX_INHERIT_PATH={prefer,fallback}` does.
As demonstrated below.

1. Create a venv with `ansicolors` installed.
```
$ python -mvenv ansicolors.venv
$ ansicolors.venv/bin/pip -q install -U pip
$ ansicolors.venv/bin/pip -q install ansicolors
```

2. Create an empty PEX with no user code or 3rdparty dependencies, but with inherit_path configured as "prefer".
```
$ pex --inherit-path=prefer -o empty.pex
$ pex-tools empty.pex info | jq .inherit_path
"prefer"
```

3. Try to run it as configured and pick up the venv ansicolors:
```
$ PEX_VERBOSE=1 ansicolors.venv/bin/python empty.pex -c 'import colors; print(colors.__file__)'
pex: Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex: 0.1ms
pex: Executing installed PEX for /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex at /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5
pex:   Testing /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/bin/python can resolve PEX at /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5: 0.1ms
pex: Re-executing: cmdline='/usr/bin/python3.10 /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex -c import colors; print(colors.__file__)', sys.executable='/home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/bin/python', PEX_PYTHON=None, PEX_PYTHON_PATH=None, interpreter_constraints=[]
pex: Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex: 0.1ms
pex: Executing installed PEX for /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex at /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5
pex: Found site-library: /usr/lib/python3.10/site-packages
pex: Tainted path element: /usr/lib/python3.10/site-packages
pex: New sys.path: ['/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap', '/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/lib/python3.10/site-packages']
pex: Activating PEX virtual environment from /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5: 0.1ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:   * 
pex:     /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5
pex:   * /usr/lib/python310.zip
pex:     /usr/lib/python3.10
pex:     /usr/lib/python3.10/lib-dynload
pex:     /usr/lib/python3.10/site-packages
pex:     /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex_bootstrapper.py", line 601, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 528, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 435, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 466, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_interpreter()
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 617, in execute_interpreter
    return self.execute_content("-c <cmd>", content, argv0="-c")
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 713, in execute_content
    return cls.execute_ast(name, program, argv0=argv0)
  File "/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap/pex/pex.py", line 731, in execute_ast
    exec_function(program, globals_map)
  File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/compatibility.py", line 94, in exec_function
    exec (ast, globals_map, locals_map)
  File "-c <cmd>", line 1, in <module>
ModuleNotFoundError: No module named 'colors'
```

4. Instead run it with `PEX_INHERIT_PATH=prefer`:
```
$ PEX_VERBOSE=1 PEX_INHERIT_PATH=prefer ansicolors.venv/bin/python empty.pex -c 'import colors; print(colors.__file__)'
pex: Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex: 0.1ms
pex: Executing installed PEX for /home/jsirois/dev/pantsbuild/jsirois-pex/empty.pex at /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5
pex:   Testing /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/bin/python can resolve PEX at /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5: 0.1ms
pex: Using the current interpreter /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/bin/python since it matches constraints and PYTHONPATH is not set.
pex: Found site-library: /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/lib/python3.10/site-packages
pex: Tainted path element: /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/lib/python3.10/site-packages
pex: New sys.path: ['/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap', '/home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/lib/python3.10/site-packages']
pex: Activating PEX virtual environment from /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5: 0.0ms
pex: No entry point specified, dropping into interpreter
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:   * 
pex:     /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5
pex:   * /usr/lib/python310.zip
pex:     /usr/lib/python3.10
pex:     /usr/lib/python3.10/lib-dynload
pex:     /home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/lib/python3.10/site-packages
pex:     /home/jsirois/.pex/unzipped_pexes/8a70498a7d95a6ecedde810dd5f69e9ad7407dd5/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
/home/jsirois/dev/pantsbuild/jsirois-pex/ansicolors.venv/lib/python3.10/site-packages/colors/__init__.py
```


Discussion:
Ok, the reading of the setting is just plain broken afaict and has been for a long time:
https://github.com/pantsbuild/pex/blob/4aecc86263ccf8d27630ef761335591ab5100759/pex/pex.py#L387-L394
Ah, no. That code is actually fine. The problem is here and fully explains the env-var / PEX-INFO venv-busting behavior discrepancy:
https://github.com/pantsbuild/pex/blob/bb68814b476c96ae25576620b685122695429bad/pex/pex_bootstrapper.py#L386-L398

Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -240,13 +240,25 @@ def minimum_sys_modules(cls, site_libs, modules=None):
         irreversible operation.
         """
 
+        is_venv = PythonInterpreter.get().is_venv
         modules = modules or sys.modules
         new_modules = {}
 
         for module_name, module in modules.items():
             # Tainted modules should be dropped.
             module_file = getattr(module, "__file__", None)
-            if module_file and cls._tainted_path(module_file, site_libs):
+            if (
+                # The `_virtualenv` module is a known special case provided by the virtualenv
+                # project. It should not be un-imported or else the virtualenv importer it installs
+                # for performing needed patches to the `distutils` stdlib breaks.
+                #
+                # See:
+                # + https://github.com/pantsbuild/pex/issues/992
+                # + https://github.com/pypa/virtualenv/pull/1688
+                (not is_venv or module_name != "_virtualenv")
+                and module_file
+                and cls._tainted_path(module_file, site_libs)
+            ):
                 TRACER.log("Dropping %s" % (module_name,), V=3)
                 continue
 
diff --git a/pex/pex_bootstrapper.py b/pex/pex_bootstrapper.py
--- a/pex/pex_bootstrapper.py
+++ b/pex/pex_bootstrapper.py
@@ -383,7 +383,7 @@ def maybe_reexec_pex(interpreter_test):
     os.environ.pop("PEX_PYTHON", None)
     os.environ.pop("PEX_PYTHON_PATH", None)
 
-    if ENV.PEX_INHERIT_PATH == InheritPath.FALSE:
+    if interpreter_test.pex_info.inherit_path == InheritPath.FALSE:
         # Now that we've found a compatible Python interpreter, make sure we resolve out of any
         # virtual environments it may be contained in since virtual environments created with
         # `--system-site-packages` foil PEX attempts to scrub the sys.path.


Test patch:
diff --git a/tests/integration/test_issue_1031.py b/tests/integration/test_issue_1031.py
--- a/tests/integration/test_issue_1031.py
+++ b/tests/integration/test_issue_1031.py
@@ -7,7 +7,7 @@
 
 from pex.interpreter import PythonInterpreter
 from pex.orderedset import OrderedSet
-from pex.testing import PY27, PY310, ensure_python_venv, make_env, run_simple_pex_test
+from pex.testing import PY27, PY310, ensure_python_venv, make_env, run_pex_command
 from pex.typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
@@ -21,9 +21,12 @@
         pytest.param(PY310, id="pyvenv"),
     ],
 )
-def test_setuptools_isolation_with_system_site_packages(py_version):
-    # type: (str) -> None
-    system_site_packages_venv, _ = ensure_python_venv(
+def test_setuptools_isolation_with_system_site_packages(
+    tmpdir,  # type: Any
+    py_version,  # type: str
+):
+    # type: (...) -> None
+    system_site_packages_venv_python, _ = ensure_python_venv(
         py_version, latest_pip=False, system_site_packages=True
     )
     standard_venv, _ = ensure_python_venv(py_version, latest_pip=False, system_site_packages=False)
@@ -37,7 +40,7 @@ def get_sys_path(python):
         )
         return OrderedSet(stdout.strip().splitlines())
 
-    system_site_packages_venv_sys_path = get_sys_path(system_site_packages_venv)
+    system_site_packages_venv_sys_path = get_sys_path(system_site_packages_venv_python)
     standard_venv_sys_path = get_sys_path(standard_venv)
 
     def venv_dir(python):
@@ -49,7 +52,7 @@ def venv_dir(python):
     system_site_packages = {
         p
         for p in (system_site_packages_venv_sys_path - standard_venv_sys_path)
-        if not p.startswith((venv_dir(system_site_packages_venv), venv_dir(standard_venv)))
+        if not p.startswith((venv_dir(system_site_packages_venv_python), venv_dir(standard_venv)))
     }
     assert len(system_site_packages) == 1, (
         "system_site_packages_venv_sys_path:\n"
@@ -65,23 +68,38 @@ def venv_dir(python):
     )
     system_site_packages_path = system_site_packages.pop()
 
-    def get_system_site_packages_pex_sys_path(**env):
-        # type: (**Any) -> MutableSet[str]
-        output, returncode = run_simple_pex_test(
-            body=print_sys_path_code,
-            interpreter=PythonInterpreter.from_binary(system_site_packages_venv),
+    def get_system_site_packages_pex_sys_path(
+        *args,  # type: str
+        **env  # type: str
+    ):
+        # type: (...) -> MutableSet[str]
+        result = run_pex_command(
+            args=args + ("--", "-c", print_sys_path_code),
+            python=system_site_packages_venv_python,
             env=make_env(**env),
         )
-        assert returncode == 0
-        return OrderedSet(output.decode("utf-8").strip().splitlines())
+        result.assert_success()
+        return OrderedSet(result.output.strip().splitlines())
 
     assert system_site_packages_path not in get_system_site_packages_pex_sys_path()
+
+    assert system_site_packages_path not in get_system_site_packages_pex_sys_path(
+        "--inherit-path=false"
+    )
     assert system_site_packages_path not in get_system_site_packages_pex_sys_path(
         PEX_INHERIT_PATH="false"
     )
+
+    assert system_site_packages_path in get_system_site_packages_pex_sys_path(
+        "--inherit-path=prefer"
+    )
     assert system_site_packages_path in get_system_site_packages_pex_sys_path(
         PEX_INHERIT_PATH="prefer"
     )
+
+    assert system_site_packages_path in get_system_site_packages_pex_sys_path(
+        "--inherit-path=fallback"
+    )
     assert system_site_packages_path in get_system_site_packages_pex_sys_path(
         PEX_INHERIT_PATH="fallback"
     )
diff --git a/tests/integration/test_issue_1870.py b/tests/integration/test_issue_1870.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_1870.py
@@ -0,0 +1,69 @@
+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+import os
+import subprocess
+
+import colors
+import pytest
+
+from pex.inherit_path import InheritPath
+from pex.testing import PY_VER, make_env, run_pex_command
+from pex.typing import TYPE_CHECKING
+from pex.venv.virtualenv import Virtualenv
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+@pytest.mark.parametrize(
+    "inherit_path",
+    [pytest.param(inherit_path, id=inherit_path.value) for inherit_path in InheritPath.values()],
+)
+def test_inherit_path_pex_info(
+    tmpdir,  # type: Any
+    inherit_path,  # type: InheritPath.Value
+):
+    # type: (...) -> None
+
+    venv_dir = os.path.join(str(tmpdir), "venv")
+    run_pex_command(
+        args=["ansicolors==1.1.8", "--include-tools", "--", "venv", venv_dir],
+        env=make_env(PEX_TOOLS=1),
+    ).assert_success()
+    venv_python = Virtualenv(venv_dir).interpreter.binary
+
+    def assert_inherit_path(
+        pex,  # type: str
+        **env  # type: Any
+    ):
+        # type: (...) -> None
+
+        expect_success = inherit_path is not InheritPath.FALSE
+        process = subprocess.Popen(
+            args=[venv_python, pex, "-c", "import colors; print(colors.yellow('Babel Fish'))"],
+            env=make_env(**env),
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+        stdout, stderr = process.communicate()
+        if expect_success:
+            assert process.returncode == 0, stderr
+            assert colors.yellow("Babel Fish") == stdout.decode("utf-8").strip()
+        else:
+            assert process.returncode != 0
+            assert (
+                "ImportError: No module named colors"
+                if PY_VER == (2, 7)
+                else "ModuleNotFoundError: No module named 'colors'"
+            ) in stderr.decode("utf-8")
+
+    empty_pex_build_time = os.path.join(str(tmpdir), "empty-build-time.pex")
+    run_pex_command(
+        args=["--inherit-path={value}".format(value=inherit_path.value), "-o", empty_pex_build_time]
+    ).assert_success()
+    assert_inherit_path(empty_pex_build_time)
+
+    empty_pex_run_time = os.path.join(str(tmpdir), "empty-run-time.pex")
+    run_pex_command(args=["-o", empty_pex_run_time]).assert_success()
+    assert_inherit_path(empty_pex_run_time, PEX_INHERIT_PATH=inherit_path)
diff --git a/tests/test_pex.py b/tests/test_pex.py
--- a/tests/test_pex.py
+++ b/tests/test_pex.py
@@ -316,26 +316,29 @@ def pex_info(inherit_path):
     def assert_isolation(self, inherit_path, expected_output):
         # type: (Union[str, bool], str) -> None
         env = dict(PYTHONPATH=self.pythonpath)
-        with named_temporary_file() as fake_stdout:
-            with temporary_dir() as temp_dir:
-                pex_builder = write_simple_pex(
-                    temp_dir,
-                    pex_info=self.pex_info(inherit_path),
-                    dists=self.dists,
-                    exe_contents=self.exe,
-                )
-
-                # Test the PEX.run API.
-                rc = PEX(pex_builder.path()).run(stdout=fake_stdout, env=env)
-                assert rc == 0
+        with temporary_dir() as temp_dir:
+            pex_builder = write_simple_pex(
+                temp_dir,
+                pex_info=self.pex_info(inherit_path),
+                dists=self.dists,
+                exe_contents=self.exe,
+            )
 
-                fake_stdout.seek(0)
-                assert expected_output == fake_stdout.read().decode("utf-8")
+            # Test the PEX.run API.
+            process = PEX(pex_builder.path()).run(
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                env=env,
+                blocking=False,
+            )
+            stdout, stderr = process.communicate()
+            assert process.returncode == 0, stderr.decode("utf-8")
+            assert expected_output == stdout.decode("utf-8")
 
-                # Test direct PEX execution.
-                assert expected_output == subprocess.check_output(
-                    [sys.executable, pex_builder.path()], env=env
-                ).decode("utf-8")
+            # Test direct PEX execution.
+            assert expected_output == subprocess.check_output(
+                [sys.executable, pex_builder.path()], env=env
+            ).decode("utf-8")
 
 
 @pytest.fixture(scope="module")