Repository: pypa/virtualenv

Issue:
Stopped working on python3
Stopped working for Python-3.3.0 a day ago. Looks like a problem with embedded 'distribute'. Older versions of virtualenv work fine. Traceback:

```
somename@myhost:~/workspace/somename$ /home/somename/opt/Python-3.3.0/bin/python3 ./virtualenv.py --no-site-packages env_somename
Using base prefix '/home/somename/opt/Python-3.3.0'
New python executable in env_somename/bin/python3
Also creating executable in env_somename/bin/python
Installing distribute......
  Complete output from command /home/somename/works...somename/bin/python3 -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute:
  Usage: -c [options]

-c: error: no such option: --always-copy
----------------------------------------
...Installing distribute...done.
Traceback (most recent call last):
  File "./virtualenv.py", line 2557, in <module>
    main()
  File "./virtualenv.py", line 961, in main
    never_download=options.never_download)
  File "./virtualenv.py", line 1068, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "./virtualenv.py", line 613, in install_distribute
    search_dirs=search_dirs, never_download=never_download)
  File "./virtualenv.py", line 580, in _install_req
    cwd=cwd)
  File "./virtualenv.py", line 1039, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/somename/works...somename/bin/python3 -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute failed with error code 2
```

Stopped working on python3
Stopped working for Python-3.3.0 a day ago. Looks like a problem with embedded 'distribute'. Older versions of virtualenv work fine. Traceback:

```
somename@myhost:~/workspace/somename$ /home/somename/opt/Python-3.3.0/bin/python3 ./virtualenv.py --no-site-packages env_somename
Using base prefix '/home/somename/opt/Python-3.3.0'
New python executable in env_somename/bin/python3
Also creating executable in env_somename/bin/python
Installing distribute......
  Complete output from command /home/somename/works...somename/bin/python3 -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute:
  Usage: -c [options]

-c: error: no such option: --always-copy
----------------------------------------
...Installing distribute...done.
Traceback (most recent call last):
  File "./virtualenv.py", line 2557, in <module>
    main()
  File "./virtualenv.py", line 961, in main
    never_download=options.never_download)
  File "./virtualenv.py", line 1068, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "./virtualenv.py", line 613, in install_distribute
    search_dirs=search_dirs, never_download=never_download)
  File "./virtualenv.py", line 580, in _install_req
    cwd=cwd)
  File "./virtualenv.py", line 1039, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/somename/works...somename/bin/python3 -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute failed with error code 2
```



Discussion:
Does develop branch work for you?

No. Both master and develop branches fail the same way.

Actually virtualenv.py file is the same in master and develop branches (md5s match). 

@adamiak yeah, master and develop are the same since we just released 1.8.3 merging develop into master.

I can't reproduce this error though, on Mac OS at least. While I am able to reproduce #359.

Also broken for me on a fresh installation of Debian Squeeze with Python 2.6.6:

```
jheino@localhost:~$ curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  111k  100  111k    0     0  83572      0  0:00:01  0:00:01 --:--:-- 96610
jheino@localhost:~$ md5sum virtualenv.py
9c074fa84daef13f13fbedeef056a9d5  virtualenv.py
jheino@localhost:~$ python virtualenv.py --distribute ~
New python executable in /home/jheino/bin/python
Installing distribute......
  Complete output from command /home/jheino/bin/python -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute:
  Usage: -c [options]

-c: error: no such option: --always-copy
----------------------------------------
...Installing distribute...done.
Traceback (most recent call last):
  File "virtualenv.py", line 2557, in <module>
    main()
  File "virtualenv.py", line 961, in main
    never_download=options.never_download)
  File "virtualenv.py", line 1068, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv.py", line 613, in install_distribute
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv.py", line 580, in _install_req
    cwd=cwd)
  File "virtualenv.py", line 1039, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jheino/bin/python -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute failed with error code 2
jheino@localhost:~$ python -V
Python 2.6.6
```

@g2p this is possibly due to 0c780772 

Yes it's related, I missed some code paths when removing unexpected arguments that the new distribute [doesn't ignore](https://bitbucket.org/tarek/distribute/changeset/1c01b2620acb553fc101e3b5c8d0540d204d79a4/).
Whether to pass --always-copy should be decided when bootstrap_script is set earlier in the function, and I don't think there's a need to make it conditional on finding a local tgz.
I'll try to extend the coverage of the functional tests to make sure all branches are tested.

Does develop branch work for you?

No. Both master and develop branches fail the same way.

Actually virtualenv.py file is the same in master and develop branches (md5s match). 

@adamiak yeah, master and develop are the same since we just released 1.8.3 merging develop into master.

I can't reproduce this error though, on Mac OS at least. While I am able to reproduce #359.

Also broken for me on a fresh installation of Debian Squeeze with Python 2.6.6:

```
jheino@localhost:~$ curl -O https://raw.github.com/pypa/virtualenv/master/virtualenv.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  111k  100  111k    0     0  83572      0  0:00:01  0:00:01 --:--:-- 96610
jheino@localhost:~$ md5sum virtualenv.py
9c074fa84daef13f13fbedeef056a9d5  virtualenv.py
jheino@localhost:~$ python virtualenv.py --distribute ~
New python executable in /home/jheino/bin/python
Installing distribute......
  Complete output from command /home/jheino/bin/python -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute:
  Usage: -c [options]

-c: error: no such option: --always-copy
----------------------------------------
...Installing distribute...done.
Traceback (most recent call last):
  File "virtualenv.py", line 2557, in <module>
    main()
  File "virtualenv.py", line 961, in main
    never_download=options.never_download)
  File "virtualenv.py", line 1068, in create_environment
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv.py", line 613, in install_distribute
    search_dirs=search_dirs, never_download=never_download)
  File "virtualenv.py", line 580, in _install_req
    cwd=cwd)
  File "virtualenv.py", line 1039, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jheino/bin/python -c "#!python
\"\"\"Bootstra...   sys.exit(main())
" --always-copy -U distribute failed with error code 2
jheino@localhost:~$ python -V
Python 2.6.6
```

@g2p this is possibly due to 0c780772 

Yes it's related, I missed some code paths when removing unexpected arguments that the new distribute [doesn't ignore](https://bitbucket.org/tarek/distribute/changeset/1c01b2620acb553fc101e3b5c8d0540d204d79a4/).
Whether to pass --always-copy should be decided when bootstrap_script is set earlier in the function, and I don't think there's a need to make it conditional on finding a local tgz.
I'll try to extend the coverage of the functional tests to make sure all branches are tested.


Resolving patch:
diff --git a/virtualenv.py b/virtualenv.py
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -505,6 +505,9 @@ def _install_req(py_executable, unzip=False, distribute=False,
             bootstrap_script = DISTRIBUTE_FROM_EGG_PY
         else:
             # Fall back to sdist
+            # NB: egg_path is not None iff tgz_path is None
+            # iff bootstrap_script is a generic setup script accepting
+            # the standard arguments.
             egg_path = None
             tgz_path = 'distribute-*.tar.gz'
             found, tgz_path = _find_file(tgz_path, search_dirs)
@@ -534,28 +537,28 @@ def _install_req(py_executable, unzip=False, distribute=False,
             env['PYTHONPATH'] = egg_path + os.path.pathsep + os.environ['PYTHONPATH']
         else:
             env['PYTHONPATH'] = egg_path
-    else:
+    elif tgz_path is not None and os.path.exists(tgz_path):
         # Found a tgz source dist, let's chdir
-        if tgz_path is not None and os.path.exists(tgz_path):
-            logger.info('Using existing %s egg: %s' % (project_name, tgz_path))
-            os.chdir(os.path.dirname(tgz_path))
-            # in this case, we want to be sure that PYTHONPATH is unset (not
-            # just empty, really unset), else CPython tries to import the
-            # site.py that it's in virtualenv_support
-            remove_from_env.append('PYTHONPATH')
-        else:
-            if never_download:
-                logger.fatal("Can't find any local distributions of %s to install "
-                             "and --never-download is set.  Either re-run virtualenv "
-                             "without the --never-download option, or place a %s "
-                             "distribution (%s) in one of these "
-                             "locations: %r" % (project_name, project_name,
-                                                egg_path or tgz_path,
-                                                search_dirs))
-                sys.exit(1)
-
-            logger.info('No %s egg found; downloading' % project_name)
-            cmd.extend(['--always-copy', '-U', project_name])
+        logger.info('Using existing %s egg: %s' % (project_name, tgz_path))
+        os.chdir(os.path.dirname(tgz_path))
+        # in this case, we want to be sure that PYTHONPATH is unset (not
+        # just empty, really unset), else CPython tries to import the
+        # site.py that it's in virtualenv_support
+        remove_from_env.append('PYTHONPATH')
+    elif never_download:
+        logger.fatal("Can't find any local distributions of %s to install "
+                     "and --never-download is set.  Either re-run virtualenv "
+                     "without the --never-download option, or place a %s "
+                     "distribution (%s) in one of these "
+                     "locations: %r" % (project_name, project_name,
+                                        egg_path or tgz_path,
+                                        search_dirs))
+        sys.exit(1)
+    elif egg_path:
+        logger.info('No %s egg found; downloading' % project_name)
+        cmd.extend(['--always-copy', '-U', project_name])
+    else:
+        logger.info('No %s tgz found; downloading' % project_name)
     logger.start_progress('Installing %s...' % project_name)
     logger.indent += 2
     cwd = None


Test patch: