Repository: pypa/virtualenv

Issue:
20b1: problems with editable installs
I've decided to try running `tox` on [all the Python projects I maintain](https://projects.gedmin.as/) with virtualenv 20.0.0b1, to check for regressions in the new beta.  Three of the projects failed, all on Python 2.7:

- gtimelog (python: No module named gtimelog)
- irclog2html (ImportError: No module named zope.testrunner)
- zodbbrowser (ImportError: No module named zope.testrunner)

Here's how to reproduce the gtimelog issue:

- git clone https://github.com/gtimelog/gtimelog
- virtualenv -p python2.7 /tmp/venv
- /tmp/venv/bin/pip install -e gtimelog --no-deps
- /tmp/venv/bin/python -c 'import gtimelog'

I expect: no output, the import should succeed.

I get

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named gtimelog
```

The `--no-deps` to pip install is there to make the test simpler and faster; I get the same failure if I do a full `pip install -e gtimelog`.

I'll create a separate bug for the irclog2html/zodbbrowser issue (they're the same issue): #1518.


I'm on Ubuntu 19.10.  I installed virtualenv 20.0.0b1 with `pipx install --pip-args=--pre` using my system Python 3.7.  `pipx runpip virtualenv list` produces

```
Package            Version 
------------------ --------
appdirs            1.4.3   
filelock           3.0.12  
importlib-metadata 1.5.0   
pip                20.0.2  
pkg-resources      0.0.0   
setuptools         45.1.0  
six                1.14.0  
virtualenv         20.0.0b1
wheel              0.34.1  
zipp               2.1.0   
```


Discussion:
One difference between virtualenv-created-with-20b1 and virtualenv-created-with-15 appears to be in sys.path: the working virtualenv has ~/src/gtimelog/src in it, the non-working one doesn't.

They both have identical easy-install.pth files in site-packages, and they both have identical gtimelog.egg-link files.

Could it be that the core problem with both this and #1518 is that `*.pth` files are not processed in the virtualenv's site-packages directory in virtualenvs created with 20b1 on Python 2.7?
@mgedmin we no longer ship our own site.py but rather use the host ``site.py``, so in theory, it well could be that Ubuntu decided to not honour pth files in their ``site.py`` which causes the issue. This also explains why I can't reproduce this on macOS.

Resolving patch:
diff --git a/src/virtualenv/create/via_global_ref/builtin/python2/site.py b/src/virtualenv/create/via_global_ref/builtin/python2/site.py
--- a/src/virtualenv/create/via_global_ref/builtin/python2/site.py
+++ b/src/virtualenv/create/via_global_ref/builtin/python2/site.py
@@ -22,11 +22,24 @@ def main():
 
 def load_host_site():
     """trigger reload of site.py - now it will use the standard library instance that will take care of init"""
-    # the standard library will be the first element starting with the real prefix, not zip, must be present
-    custom_site_package_path = __file__
+    # we have a duality here, we generate the platform and pure library path based on what distutils.install specifies
+    # because this is what pip will be using; the host site.py though may contain it's own pattern for where the
+    # platform and pure library paths should exist
+
+    # notably on Ubuntu there's a patch for getsitepackages to point to
+    # - prefix + local/lib/pythonx.y/dist-packages
+    # - prefix + lib/pythonx.y/dist-packages
+    # while distutils.install.cmd still points both of these to
+    # - prefix + lib/python2.7/site-packages
+
+    # to facilitate when the two match, or not we first reload the site.py, now triggering the import of host site.py,
+    # as this will ensure that initialization code within host site.py runs
+
+    here = __file__  # the distutils.install patterns will be injected relative to this site.py, save it here
     reload(sys.modules["site"])  # noqa
 
-    # ensure that our expected site packages is on the sys.path
+    # and then if the distutils site packages are not on the sys.path we add them via add_site_dir; note we must add
+    # them by invoking add_site_dir to trigger the processing of pth files
     import os
 
     site_packages = r"""
@@ -34,10 +47,11 @@ def load_host_site():
     """
     import json
 
+    add_site_dir = sys.modules["site"].addsitedir
     for path in json.loads(site_packages):
-        full_path = os.path.abspath(os.path.join(custom_site_package_path, path.encode("utf-8")))
+        full_path = os.path.abspath(os.path.join(here, path.encode("utf-8")))
         if full_path not in sys.path:
-            sys.path.append(full_path)
+            add_site_dir(full_path)
 
 
 def read_pyvenv():


Test patch: