Repository: pex-tool/pex

Issue:
The `--seed` mode for a `--venv` PEX is unsafe.
Currently, `--seed`ing of a `--venv` PEX operates on the PEXBuilder chroot and not the final PEX zip:
https://github.com/pantsbuild/pex/blob/37c9ff0fa815ac1f03a73d3f2e33fb20352c8843/pex/bin/pex.py#L1072-L1085
https://github.com/pantsbuild/pex/blob/37c9ff0fa815ac1f03a73d3f2e33fb20352c8843/pex/bin/pex.py#L1124-L1127

Since the PEXBuilder in question is setup to use symlinks:
https://github.com/pantsbuild/pex/blob/37c9ff0fa815ac1f03a73d3f2e33fb20352c8843/pex/bin/pex.py#L869-L875

That means any loose sources and resources added to the PEXBuilder are copied straight from their symlinks:
https://github.com/pantsbuild/pex/blob/37c9ff0fa815ac1f03a73d3f2e33fb20352c8843/pex/tools/commands/venv.py#L103-L109
https://github.com/pantsbuild/pex/blob/37c9ff0fa815ac1f03a73d3f2e33fb20352c8843/pex/tools/commands/venv.py#L56-L64

On linux, when the symlink is on the same filesystem as the PEX_ROOT, os.link is used to make the copy and os.link on a symlink makes a new symlink. If the target of the symlink is later deleted; then the symlink in the seeded venv is left dangling.




Discussion:
Pants can hit this issue easily since it builds PEXes in ephemeral `/tmp` dirs by default. The default PEX_ROOT is under `~` and common Mac and Linux setups have `/tmp` and `~` on different filesystems, masking the issue. In CI, though, Pants hits a case where `/tmp` and `~` are on the same filesystem and runs into these dangling symlinks in seeded venvs in tests.

Resolving patch:
diff --git a/pex/tools/commands/venv.py b/pex/tools/commands/venv.py
--- a/pex/tools/commands/venv.py
+++ b/pex/tools/commands/venv.py
@@ -53,7 +53,10 @@ def _copytree(
             dst_entry = os.path.join(dst, os.path.relpath(src_entry, src))
             yield src_entry, dst_entry
             try:
-                if link:
+                # We only try to link regular files since linking a symlink on Linux can produce
+                # another symlink, which leaves open the possibility the src_entry target could
+                # later go missing leaving the dst_entry dangling.
+                if link and not os.path.islink(src_entry):
                     try:
                         os.link(src_entry, dst_entry)
                         continue


Test patch:
diff --git a/tests/tools/commands/test_venv.py b/tests/tools/commands/test_venv.py
--- a/tests/tools/commands/test_venv.py
+++ b/tests/tools/commands/test_venv.py
@@ -5,7 +5,9 @@
 
 import multiprocessing
 import os
+import shutil
 import subprocess
+import sys
 import tempfile
 from subprocess import CalledProcessError
 from textwrap import dedent
@@ -14,6 +16,7 @@
 
 from pex.common import safe_open, temporary_dir, touch
 from pex.executor import Executor
+from pex.pex_builder import CopyMode, PEXBuilder
 from pex.testing import run_pex_command
 from pex.tools.commands.virtualenv import Virtualenv
 from pex.typing import TYPE_CHECKING, cast
@@ -341,3 +344,25 @@ def bar():
     subprocess.check_call(args=[pex_file, "venv", venv], env=make_env(PEX_TOOLS=True))
     output = subprocess.check_output(args=[os.path.join(venv, "pex")])
     assert "hello" == output.decode("utf-8").strip()
+
+
+def test_venv_symlinked_source_issues_1239(tmpdir):
+    # type: (Any) -> None
+    src = os.path.join(str(tmpdir), "src")
+    main = os.path.join(src, "main.py")
+    with safe_open(main, "w") as fp:
+        fp.write("import sys; sys.exit(42)")
+
+    pex_builder = PEXBuilder(copy_mode=CopyMode.SYMLINK)
+    pex_builder.set_executable(main)
+    pex_file = os.path.join(str(tmpdir), "a.pex")
+    pex_builder.build(pex_file, bytecode_compile=False)
+    assert 42 == subprocess.Popen(args=[pex_file]).wait()
+
+    venv = os.path.join(str(tmpdir), "a.venv")
+    subprocess.check_call(
+        args=[sys.executable, "-m", "pex.tools", pex_builder.path(), "venv", venv]
+    )
+    venv_pex = os.path.join(venv, "pex")
+    shutil.rmtree(src)
+    assert 42 == subprocess.Popen(args=[venv_pex]).wait()