Repository: pre-commit/pre-commit

Issue:
venv tests break virtualenv's `pip` when run from a `-mvirtualenv` virtualenv
Here's a reproduction, not exactly sure what's happening here:

```
$ tox -e py36 -r --notest
GLOB sdist-make: /home/asottile/workspace/pre-commit/setup.py
py36 create: /home/asottile/workspace/pre-commit/.tox/py36
py36 installdeps: -rrequirements-dev.txt
py36 inst: /home/asottile/workspace/pre-commit/.tox/dist/pre_commit-1.10.0.zip
py36 installed: You are using pip version 9.0.1, however version 10.0.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,aspy.yaml==1.1.1,atomicwrites==1.1.5,attrs==18.1.0,cached-property==1.4.2,cfgv==1.0.0,coverage==4.5.1,flake8==3.5.0,identify==1.0.18,mccabe==0.6.1,mock==2.0.0,more-itertools==4.2.0,nodeenv==1.3.0,pbr==4.0.3,pluggy==0.6.0,-e git+git@github.com:pre-commit/pre-commit@97fb49a533de9a378d20f0a41e79df118362e534#egg=pre_commit,py==1.5.3,pycodestyle==2.3.1,pyflakes==1.6.0,pytest==3.6.0,pytest-env==0.6.2,PyYAML==3.12,six==1.11.0,toml==0.9.4,virtualenv==16.0.0
___________________________________ summary ____________________________________
  py36: skipped tests
  congratulations :)

$ head -1 .tox/py36/bin/pip
#!/home/asottile/workspace/pre-commit/.tox/py36/bin/python3.6
$ .tox/py36/bin/pytest tests -k venv
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.6.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/asottile/workspace/pre-commit, inifile: tox.ini
plugins: env-0.6.2
collected 500 items / 492 deselected                                           

tests/repository_test.py ..                                              [ 25%]
tests/commands/install_uninstall_test.py .                               [ 37%]
tests/languages/all_test.py .....                                        [100%]

=================== 8 passed, 492 deselected in 4.12 seconds ===================
$ head -1 .tox/py36/bin/pip
#!/home/asottile/workspace/pre-commit/.tox/py36/bin/python3.6
$ tox -e py36 -- tests -k venv
GLOB sdist-make: /home/asottile/workspace/pre-commit/setup.py
py36 inst-nodeps: /home/asottile/workspace/pre-commit/.tox/dist/pre_commit-1.10.0.zip
py36 installed: You are using pip version 9.0.1, however version 10.0.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,aspy.yaml==1.1.1,atomicwrites==1.1.5,attrs==18.1.0,cached-property==1.4.2,cfgv==1.0.0,coverage==4.5.1,flake8==3.5.0,identify==1.0.18,mccabe==0.6.1,mock==2.0.0,more-itertools==4.2.0,nodeenv==1.3.0,pbr==4.0.3,pluggy==0.6.0,pre-commit==1.10.0,py==1.5.3,pycodestyle==2.3.1,pyflakes==1.6.0,pytest==3.6.0,pytest-env==0.6.2,PyYAML==3.12,six==1.11.0,toml==0.9.4,virtualenv==16.0.0
py36 runtests: PYTHONHASHSEED='93802395'
py36 runtests: commands[0] | coverage erase
py36 runtests: commands[1] | coverage run -m pytest tests -k venv
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.6.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/asottile/workspace/pre-commit, inifile: tox.ini
plugins: env-0.6.2
collected 500 items / 492 deselected                                           

tests/repository_test.py ..                                              [ 25%]
tests/commands/install_uninstall_test.py .                               [ 37%]
tests/languages/all_test.py .....                                        [100%]

=================== 8 passed, 492 deselected in 4.32 seconds ===================
py36 runtests: commands[2] | coverage report --fail-under 99
Name                                              Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------------------------------------------
...
17 files skipped due to complete coverage.
ERROR: InvocationError: '/home/asottile/workspace/pre-commit/.tox/py36/bin/coverage report --fail-under 99'
___________________________________ summary ____________________________________
ERROR:   py36: commands failed

$ head -1 .tox/py36/bin/pip
#!/tmp/pytest-of-asottile/pytest-3/test_python_venv0/0/.pre-commit/repo5xcuq11q/py_venv-python3.6/bin/python3.6
```


Discussion:
Ok, I can factor `tox` out of the situation:

```
$ . .tox/py36/bin/activate
(py36) $ pytest tests -k venv
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.6.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/asottile/workspace/pre-commit, inifile: tox.ini
plugins: env-0.6.2
collected 500 items / 492 deselected                                           

tests/repository_test.py ..                                              [ 25%]
tests/commands/install_uninstall_test.py .                               [ 37%]
tests/languages/all_test.py .....                                        [100%]

=================== 8 passed, 492 deselected in 4.19 seconds ===================
(py36) $ head -1 .tox/py36/bin/pip
#!/tmp/pytest-of-asottile/pytest-4/test_python_venv0/0/.pre-commit/reposmsza1fi/py_venv-python3.6/bin/python3.6
```
@ojii any ideas here? (`-mvenv` :spider:s)

This seems to be the minimal reproduction:

```
#!/usr/bin/env bash
set -euxo pipefail

tox -e py36 -r --notest
PATH=$PWD/.tox/py36/bin:$PATH
.tox/py36/bin/pytest tests/repository_test.py -k test_python_venv
head -1 .tox/py36/bin/pip
```
It seems making a `-mvenv` virtualenv from a `-mvirtualenv` virtualenv just doesn't work great:

```
$ virtualenv venv -ppython3.6
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /tmp/t/venv/bin/python3.6
Also creating executable in /tmp/t/venv/bin/python
Installing setuptools, pip, wheel...done.
$ . venv/bin/activate
(venv) $ python3.6 -m venv venv3
(venv) $ head -1 $(which pip)
#!/tmp/t/venv/bin/python3.6
(venv) $ ls venv
bin  include  lib  pip-selfcheck.json
(venv) $ head -n1 venv*/bin/pip
#!/tmp/t/venv/bin/python3.6
(venv) $ ls venv
venv/  venv3/ 
(venv) $ ls venv3/bin/
activate  activate.csh  activate.fish  python  python3  python3.6
(venv) $ ls -al venv3/bin/python3.6 
lrwxrwxrwx 1 asottile asottile 25 May 28 11:36 venv3/bin/python3.6 -> /tmp/t/venv/bin/python3.6
```
- pypa/virtualenv issue: https://github.com/pypa/virtualenv/issues/1095
- bpo issue: https://bugs.python.org/issue30811

Resolving patch:
diff --git a/pre_commit/languages/python_venv.py b/pre_commit/languages/python_venv.py
--- a/pre_commit/languages/python_venv.py
+++ b/pre_commit/languages/python_venv.py
@@ -1,14 +1,46 @@
 from __future__ import unicode_literals
 
+import os.path
+
 from pre_commit.languages import python
+from pre_commit.util import CalledProcessError
 from pre_commit.util import cmd_output
 
 
 ENVIRONMENT_DIR = 'py_venv'
 
 
+def orig_py_exe(exe):  # pragma: no cover (platform specific)
+    """A -mvenv virtualenv made from a -mvirtualenv virtualenv installs
+    packages to the incorrect location.  Attempt to find the _original_ exe
+    and invoke `-mvenv` from there.
+
+    See:
+    - https://github.com/pre-commit/pre-commit/issues/755
+    - https://github.com/pypa/virtualenv/issues/1095
+    - https://bugs.python.org/issue30811
+    """
+    try:
+        prefix_script = 'import sys; print(sys.real_prefix)'
+        _, prefix, _ = cmd_output(exe, '-c', prefix_script)
+        prefix = prefix.strip()
+    except CalledProcessError:
+        # not created from -mvirtualenv
+        return exe
+
+    if os.name == 'nt':
+        expected = os.path.join(prefix, 'python.exe')
+    else:
+        expected = os.path.join(prefix, 'bin', os.path.basename(exe))
+
+    if os.path.exists(expected):
+        return expected
+    else:
+        return exe
+
+
 def make_venv(envdir, python):
-    cmd_output(python, '-mvenv', envdir, cwd='/')
+    cmd_output(orig_py_exe(python), '-mvenv', envdir, cwd='/')
 
 
 get_default_version = python.get_default_version


Test patch: