Repository: linkedin/shiv

Issue:
The change to sys.path potentially overrides standard library modules.
The change in #48 makes sure that shiv packed content (packages) has the precedence over the vended Python distribution packages or things installed into system Python site-packages. Fair enough. We want to run what we packed into the shiv.

However, it does have a potentially buggy behavior. Example:
* an old deprecated 3rd party package such as *uuid* or *argparse* is pulled in transitively
* these packages are unmaintained and did not get any changes (uuid since Python 2.5, for example)
* the standard library modules have been maintained and added new APIs (uuid did)
* since shiv's site-packages are stored on the sys.path **before** the standard library path, an obsolete 3rd party backward-compatibility package (but not **forward** compatible) will override the corresponding standard library module
* if any other package uses the new APIs from the affected package (e.g., new uuid APIs), the application will break because the old 3rd party package does not have these functions/methods.

I believe this requires changes to what was done in #48 to insert *shiv* site-packages *before* any other site-packages paths, but *after* the standard library path.

Comments:

Patch:
diff --git a/src/shiv/bootstrap/__init__.py b/src/shiv/bootstrap/__init__.py
index 82169bf..02488ba 100644
--- a/src/shiv/bootstrap/__init__.py
+++ b/src/shiv/bootstrap/__init__.py
@@ -83,6 +83,12 @@ def extract_site_packages(archive, target_path):
     shutil.move(str(target_path_tmp), str(target_path))
 
 
+def _first_sitedir_index():
+    for index, part in enumerate(sys.path):
+        if Path(part).stem == 'site-packages':
+            return index
+
+
 def bootstrap():
     """Actually bootstrap our shiv environment."""
 
@@ -99,18 +105,18 @@ def bootstrap():
     if not site_packages.exists() or env.force_extract:
         extract_site_packages(archive, site_packages.parent)
 
-    preserved = sys.path[1:]
+    # get sys.path's length
+    length = len(sys.path)
 
-    # truncate the sys.path so our package will be at the start,
-    # and take precedence over anything else (eg: dist-packages)
-    sys.path = sys.path[0:1]
+    # Find the first instance of an existing site-packages on sys.path
+    index = _first_sitedir_index() or length
 
     # append site-packages using the stdlib blessed way of extending path
     # so as to handle .pth files correctly
     site.addsitedir(site_packages)
 
-    # restore the previous sys.path entries after our package
-    sys.path.extend(preserved)
+    # reorder to place our site-packages before any others found
+    sys.path = sys.path[:index] + sys.path[length:] + sys.path[index:length]
 
     # do entry point import and call
     if env.entry_point is not None and env.interpreter is None:


Test patch:
diff --git a/test/test_bootstrap.py b/test/test_bootstrap.py
index 5ece54f..1f77034 100644
--- a/test/test_bootstrap.py
+++ b/test/test_bootstrap.py
@@ -12,7 +12,7 @@ import pytest
 
 from unittest import mock
 
-from shiv.bootstrap import import_string, current_zipfile, cache_path
+from shiv.bootstrap import import_string, current_zipfile, cache_path, _first_sitedir_index
 from shiv.bootstrap.environment import Environment
 
 
@@ -61,6 +61,13 @@ class TestBootstrap:
 
         assert cache_path(mock_zip, Path.cwd(), uuid) == Path.cwd() / f"test_{uuid}"
 
+    def test_first_sitedir_index(self):
+        with mock.patch.object(sys, 'path', ['site-packages', 'dir', 'dir', 'dir']):
+            assert _first_sitedir_index() == 0
+
+        with mock.patch.object(sys, 'path', []):
+            assert _first_sitedir_index() is None
+
 
 class TestEnvironment:
     def test_overrides(self):