Repository: pypa/virtualenv

Issue:
16.1.0 with --python creates symlink to distutils directory 
PR #1227 seems to have created a regression in 16.1.0 as compared to 16.0.0 and earlier, at least on my dev systems where I use `make altinstall` to keep a bunch of different builds of Python for compatibility testing purposes. Now, ``lib/pythonX.Y/distutils`` in the virtualenv is created as a symlink to the distutils directory for my Python installation if `--python=pythonX.Y` is supplied, and then the subsequent ``install_distutils()`` call overwrites ``distutils/__init__.py`` in that Python's stdlib (following the symlink) with virtualenv's modified version. This of course results in all manner of later havoc as other callers into distutils end up getting virtualenv's init for it rather than the stdlib implementation.

Prior to the PR (or with it reverted), files in the virtualenv libdir are still symlinked as expected but the distutils directory is (correctly) not a symlink. I apologize for not having a better suggestion at this point than a revert, though hopefully someone with a deeper knowledge of the linkfarm bits will see a simple solution. As it stands, this basically renders tox unusable for me in conjunction with non-system-packaged Python interpreters.

Python built as follows:

    export TAG=3.7.2
    git clean -dfx
    git checkout v$TAG
    ./configure --enable-optimizations --prefix /home/fungi/lib/cpython/$TAG
    make profile-opt
    make altinstall
    ln -sf ../lib/cpython/$TAG/bin/python3.7 ~/bin/

Virtualenv installed with:

    python3.7 -m venv ~/lib/virtualenv
    ~/lib/virtualenv/bin/pip install virtualenv
    ln -sf ../lib/virtualenv/bin/virtualenv ~/bin/

Behavior reproduced with:

    virtualenv --python=python3.7 foo

With this, ``foo/lib/python3.7/distutils`` is created as a symlink to ``~/lib/cpython/3.7.2/lib/python3.7/distutils`` so then virtualenv overwrites the stdlib-supplied ``~/lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py`` with the contents from the https://github.com/pypa/virtualenv/blob/master/virtualenv_embedded/distutils-init.py file.


Discussion:
Just for reference, the claim is that https://github.com/pypa/virtualenv/pull/1208 (a MacOS fix) conflicts with (Linux?). Can you detail the exact error you're getting? For reference, I have MacOS with custom compiled pythons (system-wide installation though) which works just fine. I'm not clear here what's the issue is here and I would be reluctant to do any changes before we get to the bottom of this (even though as such we're probably missing the next release with this).
> Just for reference, the claim is that #1208 (a MacOS fix) conflicts with (Linux?).

Correct. While #1208 may have been trying to solve a MacOS-specific problem, it most definitely does *not* alter only MacOS-specific code paths within ``virtualenv.py`` and so has altered behavior in some other environments in detrimental ways.

> Can you detail the exact error you're getting?

I'm not sure what you're asking for. While this is erroneous behavior, it doesn't necessarily emit any error message (though as pypa/pipenv#3395 also indicates, this can result in permission errors if the ``lib/pythonX.Y/distutils`` at the dereferenced symlink target or its content is not writable by the user).

I can however provide comparative example output from 16.0.0 and 16.1.0 for the same environment to show you what non-error messages are emitted:

    fungi@dhole:~$ virtualenv --version
    16.0.0
    fungi@dhole:~$ md5sum lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    ce44c86dc92e40d9c3d59dfbacd7029c  lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    fungi@dhole:~$ rm -rf foo
    fungi@dhole:~$ virtualenv --python python3.7 foo
    Running virtualenv with interpreter /home/fungi/bin/python3.7
    Using base prefix '/home/fungi/bin/../lib/cpython/3.7.2'
    /home/fungi/lib/virtualenv/lib/python3.7/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses  import imp
    New python executable in /home/fungi/foo/bin/python3.7
    Also creating executable in /home/fungi/foo/bin/python
    Installing setuptools, pip, wheel...done.
    fungi@dhole:~$ ls -dl foo/lib/python3.7/distutils
    drwxr-xr-x 3 fungi fungi 4096 Dec 31 14:49 foo/lib/python3.7/distutils
    fungi@dhole:~$ md5sum lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    ce44c86dc92e40d9c3d59dfbacd7029c  lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py

    fungi@dhole:~$ virtualenv --version
    16.1.0
    fungi@dhole:~$ md5sum lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    ce44c86dc92e40d9c3d59dfbacd7029c  lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    fungi@dhole:~$ rm -rf foo
    fungi@dhole:~$ virtualenv --python python3.7 foo
    Running virtualenv with interpreter /home/fungi/bin/python3.7
    Using base prefix '/home/fungi/bin/../lib/cpython/3.7.2'
    New python executable in /home/fungi/foo/bin/python3.7
    Also creating executable in /home/fungi/foo/bin/python
    Overwriting /home/fungi/foo/lib/python3.7/distutils/__init__.py with new content
    Installing setuptools, pip, wheel...
    done.
    fungi@dhole:~$ ls -dl foo/lib/python3.7/distutils
    lrwxrwxrwx 1 fungi fungi 53 Dec 31 14:51 foo/lib/python3.7/distutils -> /home/fungi/lib/cpython/3.7.2/lib/python3.7/distutils
    fungi@dhole:~$ md5sum lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py
    552b973cb01606c0cc2818b686e42eb0  lib/cpython/3.7.2/lib/python3.7/distutils/__init__.py

In particular, the ``Overwriting /home/fungi/foo/lib/python3.7/distutils/__init__.py with new content`` message is the obvious difference in output for 16.1.0, and as you can see the checksum of the Python 3.7.2 stdlib's distutils init module indicates it has indeed been overwritten.
Recently I've met this issue too and found that symlink from virtual environment's ``distutils`` to original Python's ``distutils`` is created by the code copying files when ``sys.prefix`` does not match ``sys.exec_prefix`` in case when they do not match literally but actually are the same paths.

Related part of code:
```
    if sys.exec_prefix != prefix and not IS_PYPY:
        if IS_WIN:
            exec_dir = join(sys.exec_prefix, "lib")
        elif IS_JYTHON:
            exec_dir = join(sys.exec_prefix, "Lib")
        else:
            exec_dir = join(sys.exec_prefix, "lib", PY_VERSION)
        if os.path.isdir(exec_dir):
            for fn in os.listdir(exec_dir):
                copyfile(join(exec_dir, fn), join(lib_dir, fn), symlink)
```
Here ``sys.exec_prefix`` is compared to prefix got from ``sys.prefix`` or ``sys.base_prefix``. Change in #1208 transformed that prefix to absolute path so if that prefix contained relative parts they were eliminated however no such transformation is applied to ``sys.exec_prefix`` and thus ``prefix`` and ``sys.exec_prefix`` may become literally unequal if they were literally equal orignally and point to the same path.

For example when I run ``python`` and check prefixes I get this:
```
>>> import sys
>>> sys.exec_prefix
'/usr/lib/python-exec/python2.7/../../..'
>>> sys.prefix
'/usr/lib/python-exec/python2.7/../../..'
```
``virtualenv`` uses ``abspath(sys.prefix)`` for ``prefix`` which is equal to ``/usr`` and later it compares this prefix with original ``sys.exec_prefix``  and thus prefixes which were equal originally happen to be not equal.

Resolving patch:
diff --git a/virtualenv.py b/virtualenv.py
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -1192,6 +1192,15 @@ def copy_required_modules(dst_prefix, symlink):
                     copyfile(py_file, dst_filename[:-1], symlink)
 
 
+def copy_required_files(src_dir, lib_dir, symlink):
+    if not os.path.isdir(src_dir):
+        return
+    for fn in os.listdir(src_dir):
+        bn = os.path.splitext(fn)[0]
+        if fn != "site-packages" and bn in REQUIRED_FILES:
+            copyfile(join(src_dir, fn), join(lib_dir, fn), symlink)
+
+
 def copy_tcltk(src, dest, symlink):
     """ copy tcl/tk libraries on Windows (issue #93) """
     for lib_version in "8.5", "8.6":
@@ -1249,12 +1258,7 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear, sy
     try:
         # copy required files...
         for stdlib_dir in stdlib_dirs:
-            if not os.path.isdir(stdlib_dir):
-                continue
-            for fn in os.listdir(stdlib_dir):
-                bn = os.path.splitext(fn)[0]
-                if fn != "site-packages" and bn in REQUIRED_FILES:
-                    copyfile(join(stdlib_dir, fn), join(lib_dir, fn), symlink)
+            copy_required_files(stdlib_dir, lib_dir, symlink)
         # ...and modules
         copy_required_modules(home_dir, symlink)
     finally:
@@ -1302,16 +1306,14 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear, sy
             copyfile(platform_include_dir, platform_include_dest, symlink)
 
     # pypy never uses exec_prefix, just ignore it
-    if sys.exec_prefix != prefix and not IS_PYPY:
+    if os.path.realpath(sys.exec_prefix) != os.path.realpath(prefix) and not IS_PYPY:
         if IS_WIN:
             exec_dir = join(sys.exec_prefix, "lib")
         elif IS_JYTHON:
             exec_dir = join(sys.exec_prefix, "Lib")
         else:
             exec_dir = join(sys.exec_prefix, "lib", PY_VERSION)
-        if os.path.isdir(exec_dir):
-            for fn in os.listdir(exec_dir):
-                copyfile(join(exec_dir, fn), join(lib_dir, fn), symlink)
+        copy_required_files(exec_dir, lib_dir, symlink)
 
     if IS_JYTHON:
         # Jython has either jython-dev.jar and javalib/ dir, or just


Test patch:
diff --git a/tests/test_virtualenv.py b/tests/test_virtualenv.py
--- a/tests/test_virtualenv.py
+++ b/tests/test_virtualenv.py
@@ -518,3 +518,37 @@ def test_pypy_pre_import(tmp_path):
         subprocess.check_output(cmd, universal_newlines=True, stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError as exception:
         assert not exception.returncode, exception.output
+
+
+@pytest.mark.skipif(not hasattr(os, "symlink"), reason="requires working symlink implementation")
+def test_create_environment_with_exec_prefix_pointing_to_prefix(tmpdir):
+    """Create virtual environment for Python with ``sys.exec_prefix`` pointing
+    to ``sys.prefix`` or ``sys.base_prefix`` or ``sys.real_prefix`` under a
+    different name
+    """
+    venvdir = str(tmpdir / "venv")
+    python_dir = tmpdir / "python"
+    python_dir.mkdir()
+    path_key = str("PATH")
+    old_path = os.environ[path_key]
+    if hasattr(sys, "real_prefix"):
+        os.environ[path_key] = os.pathsep.join(
+            p for p in os.environ[path_key].split(os.pathsep) if not p.startswith(sys.prefix)
+        )
+    python = virtualenv.resolve_interpreter(os.path.basename(sys.executable))
+    try:
+        subprocess.check_call([sys.executable, "-m", "virtualenv", "-p", python, venvdir])
+        home_dir, lib_dir, inc_dir, bin_dir = virtualenv.path_locations(venvdir)
+        assert not os.path.islink(os.path.join(lib_dir, "distutils"))
+    finally:
+        os.environ[path_key] = old_path
+
+
+@pytest.mark.skipif(not hasattr(sys, "real_prefix"), reason="requires running from inside virtualenv")
+def test_create_environment_from_virtual_environment(tmpdir):
+    """Create virtual environment using Python from another virtual environment
+    """
+    venvdir = str(tmpdir / "venv")
+    home_dir, lib_dir, inc_dir, bin_dir = virtualenv.path_locations(venvdir)
+    virtualenv.create_environment(venvdir)
+    assert not os.path.islink(os.path.join(lib_dir, "distutils"))