Repository: pypa/pip

Issue:
Pip fails to install package using git and short commit hash 
* Pip version: 10.0.0b1
* Python version: 3.5.2
* Operating system: Ubuntu 16.04

### Description:

Installing a package from a git repository specifying a commit hash using 7 characters.
`pip` shows an error and do not install the package.
I expect `pip` to install the package using the specified commit.
 
### What I've run:

```
pip install git+git://git.myproject.org/MyProject@0abcdef
Collecting git+git://git.myproject.org/MyProject@0abcdef
  Cloning git://git.myproject.org/MyProject (to revision 0abcdef) to /tmp/pip-req-build-xxxxxxxx
  Did not find branch or tag '0abcdef', assuming revision or ref.
fatal: Couldn't find remote ref 0abcdef
Command "git fetch -q git+git://git.myproject.org/MyProject 0abcdef" failed with error code 128 in /tmp/pip-req-build-xxxxxxxxx
```

The issue seems to be related to this PR https://github.com/pypa/pip/pull/4674


Discussion:
Can you confirm if this also happens with pip 9.0.3 please?
@pfmoore This does not happen with 9.0.x:

```
$ pip --version
pip 9.0.3 from /Users/di/.pyenv/versions/3.6.3/lib/python3.6/site-packages (python 3.6)

$ pip install git+git://github.com/pypa/sampleproject@941f486
Collecting git+git://github.com/pypa/sampleproject@941f486
  Cloning git://github.com/pypa/sampleproject (to 941f486) to /private/var/folders/px/dlfyqxyx3mx0t_yx3dvbb9nc0000gn/T/pip-0xce9_7h-build
  Could not find a tag or branch '941f486', assuming commit.
Requirement already satisfied: peppercorn in /Users/di/.local/lib/python3.6/site-packages (from sampleproject==1.2.0)
Installing collected packages: sampleproject
  Running setup.py install for sampleproject ... done
Successfully installed sampleproject-1.2.0
```

(cc @cjerdonek)
I'm taking a look.
I did some investigation, and things are breaking when `pip` calls `git fetch <url> <short_sha>`, e.g.

    $ git fetch -q git://github.com/pypa/sampleproject d5dcff4
    fatal: Couldn't find remote ref d5dcff4

which occurs at this line: https://github.com/pypa/pip/blob/0007825733aa862e76a66e5d1cfee121c29192ff/src/pip/_internal/vcs/git.py#L194

This line / additional call to `git fetch` was added in PR #4429.

Before PR #4429, things work okay:

    $ pyenv shell pip-test1
    (pip-test1) $ pip install git+git://github.com/pypa/pip@4ee868553c473fd1f08179534763ad342ba07db4
    Collecting git+git://github.com/pypa/pip@4ee868553c473fd1f08179534763ad342ba07db4
      Cloning git://github.com/pypa/pip (to 4ee868553c473fd1f08179534763ad342ba07db4) to /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-y6x076v5-build
      Could not find a tag or branch '4ee868553c473fd1f08179534763ad342ba07db4', assuming commit.
    Installing collected packages: pip
      Found existing installation: pip 9.0.3
        Uninstalling pip-9.0.3:
          Successfully uninstalled pip-9.0.3
      Running setup.py install for pip ... done
    Successfully installed pip-10.0.0.dev0
    (pip-test1) $ pip install git+git://github.com/pypa/sampleproject@d5dcff4
    Collecting git+git://github.com/pypa/sampleproject@d5dcff4
      Cloning git://github.com/pypa/sampleproject (to d5dcff4) to /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-7ric9dvy-build
      Could not find a tag or branch 'd5dcff4', assuming commit.
    Collecting peppercorn (from sampleproject==1.2.0)
      Using cached peppercorn-0.5.tar.gz
    Installing collected packages: peppercorn, sampleproject
      Running setup.py install for peppercorn ... done
      Running setup.py install for sampleproject ... done
    Successfully installed peppercorn-0.5 sampleproject-1.2.0
 
 After PR #4429 (two `pip` commits later), the same invocation fails:
 
    (pip-test1) $ pyenv shell pip-test2
    (pip-test2) $ pip install git+git://github.com/pypa/pip@253d2a673e3b6df2f2e9d0df921d49aad861987f
    Collecting git+git://github.com/pypa/pip@253d2a673e3b6df2f2e9d0df921d49aad861987f
      Cloning git://github.com/pypa/pip (to 253d2a673e3b6df2f2e9d0df921d49aad861987f) to /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-e2v9hgmz-build
      Could not find a tag or branch '253d2a673e3b6df2f2e9d0df921d49aad861987f', assuming commit.
    Installing collected packages: pip
      Found existing installation: pip 9.0.3
        Uninstalling pip-9.0.3:
          Successfully uninstalled pip-9.0.3
      Running setup.py install for pip ... done
    Successfully installed pip-10.0.0.dev0
    (pip-test2) $ pip install git+git://github.com/pypa/sampleproject@d5dcff4
    Collecting git+git://github.com/pypa/sampleproject@d5dcff4
      Cloning git://github.com/pypa/sampleproject (to d5dcff4) to /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-4ngm43_i-build
      Could not find a tag or branch 'd5dcff4', assuming commit or ref
    fatal: Couldn't find remote ref d5dcff4
    Command "git fetch -q git://github.com/pypa/sampleproject d5dcff4" failed with error code 128 in /private/var/folders/q9/j0_5hxt88v5592006s6dd3n80000gs/T/pip-4ngm43_i-build

Note that in the above it was necessary to use a `sampleproject` "short sha" _other than the most recent commit_ since if the most recent commit is used, then `pip` doesn't call `git fetch` (since HEAD already matches the requested revision).

It seems like this issue could be fixed by adjusting the call to `git fetch`, e.g. by having it fetch the latest of everything, or perhaps by keeping the call the same but being okay with it erroring out.

Thanks for debugging @cjerdonek.

Seems like it would be wasteful to fetch the latest of everything. 

If we let the call error, I'm not sure what `FETCH_HEAD` will be in the command that follows it. We might be able to use `git rev-parse <ref>` instead which will give a full SHA for both short SHAs and refs, e.g.:

```
$ git rev-parse d5dcff4
d5dcff4b49a230ea27fb143fce123688138686a3
$ git rev-parse refs/heads/master
aafe3d8d8b9b27b9b467aec33dc56d344f6c5428
```

Unfortunately `rev-parse` will error if the ref needs to be fetched, so either way I think we will have to catch an error and retry.

@cjerdonek thoughts?
Fixing `FETCH_HEAD` in the case of an error would be easy enough if that's an issue. The main reason I can think of for fetching no matter what is to surface an ambiguous ref that would otherwise go undetected if the fetch didn't take place. I haven't thought through / investigated whether this case can actually arise, but it seems like you couldn't go wrong by trying to fetch the ref each time.
One more idea that I don't know if it will work in general is to use e.g. a regular expression to detect if you're in the ref case. And fetch only in that case. But I don't know if you can cleanly distinguish the two cases (e.g. if there are strings that "look" like a ref but could nevertheless still succeed in a subsequent call to `git checkout` without performing a fetch).
PS - in the case of always fetching and if fetch fails, why wouldn't it work just to ignore `FETCH_HEAD` and checkout `rev_options` instead? That's what the code did prior to PR #4429 when no fetch was taking place. Perhaps it would even work to use `rev_options` also in the success case (i.e. always use `rev_options`), but maybe there's a reason that wouldn't work. 
What I've found is that function [`looks_like_hash`](https://github.com/pypa/pip/blob/0007825733aa862e76a66e5d1cfee121c29192ff/src/pip/_internal/vcs/git.py#L27) returns `False` for a short commit hash.
It’s supposed to (and should continue to do so IMO). I believe there are
even tests of this if I remember right, since I wrote that function. My
last comment in reference to a regular expression was about the possibility
of detecting the “ref” case, which is the issue here. The looks-like-hash
function only comes into play for whether to display a warning and doesn’t
otherwise affect the code path.

On Mon, Apr 2, 2018 at 2:31 AM Alejandro Varas <notifications@github.com>
wrote:

> What I've found is that function looks_like_hash
> <https://github.com/pypa/pip/blob/0007825733aa862e76a66e5d1cfee121c29192ff/src/pip/_internal/vcs/git.py#L27>
> returns False for a short commit hash.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pypa/pip/issues/5140#issuecomment-377894130>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAVt7lbrQn7fCvGYa9kQMRI2PR-PLOXPks5tke_bgaJpZM4TC5o4>
> .
>

Here is how I would suggest fixing the immediate issue:

```python
rev = rev_options.rev
if rev.startswith('refs/'):
    cmd_args = ['fetch', '-q', url] + rev_options.to_args()
    self.run_command(cmd_args, cwd=dest)
    rev = 'FETCH_HEAD'

self.run_command(
    ['checkout', '-q', rev],
    cwd=dest,
)
```

This way it's clear that the extra fetch only takes place for the Git ref case it was added to support (in PR #4429).

[Added:] Other benefits of this approach are that (1) it will surface the "ambiguous ref" case I mentioned above, if this case can actually arise, and (2) it avoids an extra fetch in the common, non-ref case.


Resolving patch:
diff --git a/src/pip/_internal/vcs/git.py b/src/pip/_internal/vcs/git.py
--- a/src/pip/_internal/vcs/git.py
+++ b/src/pip/_internal/vcs/git.py
@@ -191,12 +191,16 @@ def obtain(self, dest):
                 # Only do a checkout if the current commit id doesn't match
                 # the requested revision.
                 if not self.is_commit_id_equal(dest, rev_options.rev):
-                    cmd_args = ['fetch', '-q', url] + rev_options.to_args()
-                    self.run_command(cmd_args, cwd=dest)
-                    self.run_command(
-                        ['checkout', '-q', 'FETCH_HEAD'],
-                        cwd=dest,
-                    )
+                    rev = rev_options.rev
+                    # Only fetch the revision if it's a ref
+                    if rev.startswith('refs/'):
+                        self.run_command(
+                            ['fetch', '-q', url] + rev_options.to_args(),
+                            cwd=dest,
+                        )
+                        # Change the revision to the SHA of the ref we fetched
+                        rev = 'FETCH_HEAD'
+                    self.run_command(['checkout', '-q', rev], cwd=dest)
 
             #: repo may contain submodules
             self.update_submodules(dest)


Test patch:
diff --git a/tests/functional/test_install_vcs.py b/tests/functional/test_install_vcs.py
--- a/tests/functional/test_install_vcs.py
+++ b/tests/functional/test_install_vcs.py
@@ -115,6 +115,27 @@ def test_git_with_sha1_revisions(script):
     assert '0.1' in version.stdout, version.stdout
 
 
+@pytest.mark.network
+def test_git_with_short_sha1_revisions(script):
+    """
+    Git backend should be able to install from SHA1 revisions
+    """
+    version_pkg_path = _create_test_package(script)
+    _change_test_package_version(script, version_pkg_path)
+    sha1 = script.run(
+        'git', 'rev-parse', 'HEAD~1',
+        cwd=version_pkg_path,
+    ).stdout.strip()[:7]
+    script.pip(
+        'install', '-e',
+        '%s@%s#egg=version_pkg' %
+        ('git+file://' + version_pkg_path.abspath.replace('\\', '/'), sha1),
+        expect_stderr=True
+    )
+    version = script.run('version_pkg')
+    assert '0.1' in version.stdout, version.stdout
+
+
 @pytest.mark.network
 def test_git_with_branch_name_as_revision(script):
     """