Repository: pypa/pip

Issue:
pip 10 is unable to install from non-editable VCS URLs
* Pip version: 10.0.0
* Python version: 3.5.4
* Operating system: Linux

### Description:

I'm trying to install a package from Git.

### What I've run:

```bash
$ pip3 install git+ssh://git@example.com/mypackage.git#egg=mypackage
```

```
Building wheels for collected packages: mypackage
Exception:
Traceback (most recent call last):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 416, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for mypackage from git+ssh://git@example.com/mypackage.git#egg=mypackage in /home/p/local/lib/python3.5/site-packages
```

If I uninstall the package and then install, it works. But it continues to fail when trying install again.


Discussion:
I saw this same error on Python 2 and found that using pip 9 worked as expected. I wonder if this is related to the deprecation and removal of the egg switch? Removing that also seemed to solve the issue.
I'm experiencing this issue on Travis-CI with Python 3.5.2 and ``pip==10.0.0``. ~~I do not experience it with Python 3.6 (local) or 2.7 (both local and on travis).~~

```pytb
Traceback (most recent call last):
...
  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for cnx-query-grammar from git+https://github.com/Connexions/cnx-query-grammar.git#egg=cnx-query-grammar in /home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages (from -r requirements/./test.txt (line 9))
```
Exact same issue, on `Ubuntu 14.04.2 LTS`, pip 10.0.0 installed with get-pip.py on python 3.4.0 specifically on my git installs.
Also using the #egg= parameter. I found this thread: https://github.com/pypa/pip/issues/1749 
Oddly removing the whole virtualenv and reinstalling (including the get-pip bit) worked.
... but it failed again on the second deploy on top of an existing virtualenv (which should have been a no-op since all the packages are installed already but I guess it redownloads the git packages each time).
Initial installation works, but attempts to reinstall are the problem space. So far I'm seeing a difference in the ``InstallRequirement``'s ``build_env`` value, where the initial install produced a `NoOpBuildEnvironment` object and a reinstall has a `BuildEnvironment`.
Also experiencing this issue with pip 10.0.0
This breaks our ops/build workflows. 😢 
In case others are also debugging this. I've narrowed down the two paths, first install and reinstall, to the difference in the results coming out of the [resolver's ``_get_abstract_dist_for``](https://github.com/pypa/pip/blob/master/src/pip/_internal/resolve.py#L257). Prior to that call both paths have ``<InstallRequirement>.source_dir == None``. After this call, only the first install path has a ``source_dir`` set to something other than ``None``. I'm guessing this is because it needs to go off and fetch the source during the first-install, but not on the reinstall.

... I could be off in the weeds though. It is my first time looking at this code. 😕 
My workaround: `pip install pip==9.0.3`
O... so the requirements file I'm working on isn't using `-e` prefixes for vcs installs. Adding `-e` to the beginning of the line does make this work. And in @RazerM 's case: ``pip3 install -e git+ssh://git@example.com/mypackage.git#egg=mypackage``

So part of this issue is user error??? The other part of it is that we should get back something to the effect of ``Requirement already satisfied: ...`` and instead we get a Traceback.


If it helps, mine also doesn't use `-e`, though I thought VCS package URLs needn't be editable.
[Correct, they needn't be:](https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support)

> VCS projects can be installed in editable mode (using the `--editable` option) or not.
> 
>  * For editable installs, the clone location by default is "\<venv path>/src/SomeProject" in virtual environments, and "\<cwd>/src/SomeProject" for global installs.  The `--src` option can be used to modify this location.
> * For non-editable installs, the project is built locally in a temp dir and then normally. Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an `--upgrade` flag. VCS requirements pin the package version (specified in the `setup.py` file) of the target commit, not necessarily the commit itself.
> * The `pip freeze` subcommand will record the VCS requirement specifier (referencing a specific commit) if and only if the install is done using the editable option.
FYI we are running into this in pipenv (our testing infrastructure caught this), here's what I can say on the topic:

* This only impacts non-editable VCS installs (i.e. not passing `-e`
* We explicitly pass a `PIP_SRC_DIR` during testing and it seems to make no difference
* All of our tests run in isolated environments

As far as the distinction between editable and non-editable dependencies, from a resolution standpoint we can resolve dependency graphs for editable dependencies but can't for non-editable dependencies. Still-- we test to ensure that they can be installed, which now fails.

As a temporary fix we are moving our `pip install` calls to `python -m <vendored pip9> install` for compatibility.
I am unable to reproduce this. Could someone provide steps to reproduce? If it's with in a docker container, that'd be awesome.
Just run this twice with pip 10 in a clean venv, it will fail on the second try:

pip install git+https://github.com/requests/requests.git#egg=requests
I can't reproduce:

```
/tmp $ python -m venv env
/tmp $ source env/bin/activate
(env) /tmp $ pip install -U pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 854kB/s
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-10.0.0
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /private/var/folders/px/dlfyqxyx3mx0t_yx3dvbb9nc0000gn/T/pip-install-32abmlma/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 6.9MB/s
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 9.3MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 10.9MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 10.3MB/s
Installing collected packages: chardet, idna, urllib3, certifi, requests
  Running setup.py install for requests ... done
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python3.6/site-packages (2.18.4)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python3.6/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python3.6/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.6/site-packages (from requests) (2018.4.16)
```
@di @pradyunsg first spotted: https://travis-ci.org/pypa/pipenv/jobs/367322542#L715 

log:
```console
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install --no-cache-dir pip
Requirement already satisfied: pip in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (10.0.0)
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Collecting six from git+https://github.com/benjaminp/six.git#egg=six
  Cloning https://github.com/benjaminp/six.git to /tmp/pip-install-2ff500pr/six
Building wheels for collected packages: six
  Running setup.py bdist_wheel for six ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-ebgi9ij6/wheels/a9/36/0a/8173089e25b3c1df51c0bb21d21ec8dc5d7ee9d0ec9b6b51e1
Successfully built six
Installing collected packages: six
Successfully installed six-1.11.0
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Requirement already satisfied: six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (1.11.0)
Building wheels for collected packages: six
Exception:
Traceback (most recent call last):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages
```
And a Dockerfile which reproduces:

```dockerfile
FROM python:3.6-alpine3.7

ENV PYTHONUNBUFFERED=1
ENV LC_ALL=en_US.UTF-8

RUN apk update \
    && apk add git

WORKDIR /app/


RUN set -ex \
    && pip install --upgrade pip \
    && pip install git+https://github.com/benjaminp/six.git#egg=six \
    && pip install git+https://github.com/benjaminp/six.git#egg=six

CMD ["/bin/bash"]
```
We can reproduce on our build machines, try this:

```
docker run -it circleci/python:2.7.13
cd ~
virtualenv env
. env/bin/activate
pip install -U pip
pip install git+https://github.com/requests/requests.git#egg=requests
pip install git+https://github.com/requests/requests.git#egg=requests
```

Here's my output:

```
ericely$ docker run -it circleci/python:2.7.13
$ cd ~
$ virtualenv env
New python executable in /home/circleci/env/bin/python
Installing setuptools, pip, wheel...
done.
$ $ . env/bin/activate
(env) $ pip install -U pip
Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages (10.0.0)
(env) $ pip list
Package    Version
---------- -------
pip        10.0.0 
setuptools 39.0.1 
wheel      0.31.0 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /tmp/pip-install-I8n5LX/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 3.9MB/s 
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 7.9MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 8.0MB/s 
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 4.3MB/s 
Building wheels for collected packages: requests
  Running setup.py bdist_wheel for requests ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-F7Eu9c/wheels/8d/84/fa/1ded919cc5b46b899830a8f8d18817f59abde627aa07362b23
Successfully built requests
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) $ 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages (2.18.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python2.7/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python2.7/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python2.7/site-packages (from requests) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python2.7/site-packages (from requests) (3.0.4)
Building wheels for collected packages: requests
Exception:
Traceback (most recent call last):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages
(env) $ 
```
@pfmoore This is an important core use-case which is broken. I'm looking into a fix right now.
Also experiencing this issue
@pradyunsg Cool, keep me posted - if there's anything I can help with let me know.

Resolving patch:
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -717,11 +717,11 @@ def build(self, requirements, session, autobuilding=False):
                     )
             elif autobuilding and req.editable:
                 pass
+            elif autobuilding and not req.source_dir:
+                pass
             elif autobuilding and req.link and not req.link.is_artifact:
                 # VCS checkout. Build wheel just for this run.
                 buildset.append((req, True))
-            elif autobuilding and not req.source_dir:
-                pass
             else:
                 ephem_cache = False
                 if autobuilding:


Test patch: