Repository: pypa/virtualenv

Issue:
virtualenv -p python3.4 fails with ImportError: No module named 'copy_reg' if you have pies2overrides installed
I've created a Python 2.7 virtualenv for various tools I want to have around and added its `bin` directory to my PATH.  One of the tools I've installed is `virtualenv` itself (possibly a newer version than the one I used to create my ~/.venv).

When I try to use the latest upstream virtualenv (1.11.6) to create a Python 3 virtualenv with

```
~/.venv/bin/virtualenv -p /usr/bin/python3.4 /tmp/py34
```

I get the following error:

```
Running virtualenv with interpreter /usr/bin/python3.4
Traceback (most recent call last):
  File "/home/mg/.venv/local/lib/python2.7/site-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/home/mg/.venv/lib/python2.7/site-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/home/mg/.venv/lib/python2.7/site-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/home/mg/.venv/local/lib/python2.7/site-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/home/mg/.venv/lib/python2.7/site-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'
```

It seems wrong to see Python 3.4 and Python 2.7 stdlib paths intermixed in the traceback.

I'm on Ubuntu 14.10 and Ubuntu's `python-virtualenv` package works fine with Python 3.x (or 2.x, but not pypy 2.3.1)



Discussion:
I cannot reproduce this problem if I `pip install --user virtualenv` and use `~/.local/bin/virtualenv -p python3.4`.

I cannot reproduce this in a freshly created virtualenv either:

```
/usr/bin/virtualenv -p python2.7 /tmp/sandbox
/tmp/sandbox/bin/pip install virtualenv
/tmp/sandbox/bin/virtualenv -p python3.4 /tmp/py34
```

works fine.

I've tried to strace the failing virtualenv script and saw that the first thing it does is re-exec itself with the specified Python interpreter:

```
execve("/usr/bin/python3.4", ["/usr/bin/python3.4", "/home/mg/.venv/local/lib/python2.7/site-packages/virtualenv.py", "-p", "python3.4", ...], [...]) = 0
```

This puts `/home/mg/.venv/local/lib/python2.7/site-packages/` into sys.path!  Now a freshly-created virtualenv hasn't got anything especially dangerous in there, but my ~/.venv has got some interesting files:

```
-rw-rw-r--  1 mg mg     62 Dec  1 15:25 _thread.py
-rw-rw-r--  1 mg mg    241 Dec  1 15:25 _thread.pyc
-rw-rw-r--  1 mg mg     66 Dec  1 15:25 builtins.py
-rw-rw-r--  1 mg mg    246 Dec  1 15:25 builtins.pyc
-rw-rw-r--  1 mg mg     67 Dec  1 15:25 configparser.py
-rw-rw-r--  1 mg mg    251 Dec  1 15:25 configparser.pyc
-rw-rw-r--  1 mg mg     63 Dec  1 15:25 copyreg.py
-rw-rw-r--  1 mg mg    242 Dec  1 15:25 copyreg.pyc
-rw-rw-r--  1 mg mg     60 Dec  1 15:25 queue.py
-rw-rw-r--  1 mg mg    237 Dec  1 15:25 queue.pyc
-rw-rw-r--  1 mg mg     59 Dec  1 15:25 reprlib.py
-rw-rw-r--  1 mg mg    238 Dec  1 15:25 reprlib.pyc
-rw-rw-r--  1 mg mg     67 Dec  1 15:25 socketserver.py
-rw-rw-r--  1 mg mg    251 Dec  1 15:25 socketserver.pyc
```

How do you spell dpkg -S with pip?  Anyway, if I sort it all by date, I think I can blame `pip install isort` that pulled in `pies` that pulled in `pies2overrides` with all this crap.

I'm not closing this issue because I think running pythonX with a site-packages directory of pythonY included in sys.path is a bad idea.

I'm trying to come up with ways to run a script without adding that script's dirname into sys.path, and I can't think of anything better than

```
python3.4 -c 'import sys; del sys.argv[0]; s = sys.argv[0]; exec(open(s).read(), {"__file__": s, "__name__": "__main__"})' ~/.venv/lib/python2.7/site-packages/virtualenv.py -p python3.4 /tmp/py34
```

(I've tested this command and it works, even if I've got `pies2overrides` installed.)

This might be a duplicate of #625 which was fixed in the `develop` branch, in commit dde220efe38b99020cd61e3c3a88c5d44922c544.

For completeness: dde220e did not fix this issue for me, in fact it broke virtualenv even further, see #673.  PR #672 replaces dde220e with a different patch that fixes both this issue and #673.


Resolving patch:
diff --git a/virtualenv.py b/virtualenv.py
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -5,7 +5,8 @@
 __version__ = "12.0.dev1"
 virtualenv_version = __version__  # legacy
 
-import base64
+# NB: avoid placing additional imports here, before sys.path is fixed!
+
 import sys
 import os
 
@@ -29,6 +30,7 @@
 if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
     del sys.path[0]
 
+import base64
 import codecs
 import optparse
 import re
@@ -1106,45 +1108,33 @@ def change_prefix(filename, dst_prefix):
 
 def copy_required_modules(dst_prefix, symlink):
     import imp
-    # If we are running under -p, we need to remove the current
-    # directory from sys.path temporarily here, so that we
-    # definitely get the modules from the site directory of
-    # the interpreter we are running under, not the one
-    # virtualenv.py is installed under (which might lead to py2/py3
-    # incompatibility issues)
-    _prev_sys_path = sys.path
-    if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
-        sys.path = sys.path[1:]
-    try:
-        for modname in REQUIRED_MODULES:
-            if modname in sys.builtin_module_names:
-                logger.info("Ignoring built-in bootstrap module: %s" % modname)
-                continue
-            try:
-                f, filename, _ = imp.find_module(modname)
-            except ImportError:
-                logger.info("Cannot import bootstrap module: %s" % modname)
+    for modname in REQUIRED_MODULES:
+        if modname in sys.builtin_module_names:
+            logger.info("Ignoring built-in bootstrap module: %s" % modname)
+            continue
+        try:
+            f, filename, _ = imp.find_module(modname)
+        except ImportError:
+            logger.info("Cannot import bootstrap module: %s" % modname)
+        else:
+            if f is not None:
+                f.close()
+            # special-case custom readline.so on OS X, but not for pypy:
+            if modname == 'readline' and sys.platform == 'darwin' and not (
+                    is_pypy or filename.endswith(join('lib-dynload', 'readline.so'))):
+                dst_filename = join(dst_prefix, 'lib', 'python%s' % sys.version[:3], 'readline.so')
+            elif modname == 'readline' and sys.platform == 'win32':
+                # special-case for Windows, where readline is not a
+                # standard module, though it may have been installed in
+                # site-packages by a third-party package
+                pass
             else:
-                if f is not None:
-                    f.close()
-                # special-case custom readline.so on OS X, but not for pypy:
-                if modname == 'readline' and sys.platform == 'darwin' and not (
-                        is_pypy or filename.endswith(join('lib-dynload', 'readline.so'))):
-                    dst_filename = join(dst_prefix, 'lib', 'python%s' % sys.version[:3], 'readline.so')
-                elif modname == 'readline' and sys.platform == 'win32':
-                    # special-case for Windows, where readline is not a
-                    # standard module, though it may have been installed in
-                    # site-packages by a third-party package
-                    pass
-                else:
-                    dst_filename = change_prefix(filename, dst_prefix)
-                copyfile(filename, dst_filename, symlink)
-                if filename.endswith('.pyc'):
-                    pyfile = filename[:-1]
-                    if os.path.exists(pyfile):
-                        copyfile(pyfile, dst_filename[:-1], symlink)
-    finally:
-        sys.path = _prev_sys_path
+                dst_filename = change_prefix(filename, dst_prefix)
+            copyfile(filename, dst_filename, symlink)
+            if filename.endswith('.pyc'):
+                pyfile = filename[:-1]
+                if os.path.exists(pyfile):
+                    copyfile(pyfile, dst_filename[:-1], symlink)
 
 
 def subst_path(prefix_path, prefix, home_dir):


Test patch: