Repository: pypa/pip

Issue:
[RFC] caching VCS requirements that are considered immutables
**What's the problem this feature will solve?**

Pip allows installing from VCS URL references such as git+https://github.com/org/repo@{ref}#egg=name.

When {ref} points to an immutable revision (ie a commit hash / sha in case of git),
it would help performance to cache the wheel that was built from that reference,
and subsequently reuse it.

**Describe the solution you'd like**

I'm willing to implement the PR myself.

My plan is to add a `is_immutable(cls, revision/link)` to the `VersionControl` class.
In the case of git it would check revision looks like a sha ('^[a-fA-F0-9]{40}$').
Then exploit it in caching mechanisms.

Potential issues:

- something that looks like a sha might also be a branch name which would therefore not be immutable: I believe this is not an issue in practice because nobody would name a branch
with exactly 40 characters in a way that git could confuse with a a sha
- ~~in case of binary wheels (ie containing C code), the VCS URL may not contain enough information to ensure a reproducible build, as some build parameters may be environmental; in such cases the cache may trigger returning a wheel that otherwise would have to be rebuilt (eg for a different architecture): I'm not seeing any practical issue here either; if there would be a risk we can consider controlling the feature with a flag (`such as --(no)-vcs-cache`)~~ [update] pip's wheel cache takes care of this as wheel names contain platform specific tags

**Alternative Solutions**

I currently have a pip wrapper that does something close, but I thought it would be useful to others, so I propose to implement it in pip itself.

Before attempting an implementation, I wanted to check if such a feature is of interest to pip maintainers, or if any of the potential issues would be considered blocking.

**Additional context**

N/A



Discussion:
This sounds like a good idea to me. /cc @cjerdonek since he's most familiar with the VCS code.
> something that looks like a sha might also be a branch name which would therefore not be immutable: I believe this is not an issue in practice because nobody would name a branch
with exactly 40 characters in a way that git could confuse with a a sha

This is actually the thing I'm concerned about with this (and the reason I hadn't responded to this issue yet). I feel uncomfortable about the idea of changing pip's behavior based on what the revision "looks like." This kind of approach seems brittle to me, and yes, would break edge cases where someone has or might want to name a branch that happens to have the form of a sha. I can see that capability being useful as a work-around or in testing situations, or maybe for reasons we don't know. My instinct is that pip should really have a more structured way to specify "this revision is a SHA" as opposed to relying on whether the string "looks like" something. But I haven't really given thought to what that should be since it could likely involve a new format / notation of URL, etc.
Would you then be open to having an option that enables the feature? Or an option to disable it if we think it could still be on by default (in which case `--no-cache-dir` would maybe do the job).

I'm certainly open to having a way to specify "this revision is a SHA," but I'm not sure yet what that should be. Do you have any ideas? Having a global option that means "interpret revisions that look like SHA's as commit hashes" seems less attractive to me because it would be an option that is basically a work-around for not being able to specify revisions in a structured way (and pip already has a lot of specialized options). I do wonder if this (caching) could be leveraged with the PEP that we're working on -- because at that point, we _would_ have the information we need in a structured way (in the JSON file). So maybe working on this after the PEP would make more sense? What do you think?
I've no immediate idea on how to change the pip requirements format to express that a rev is a commit id. Especially to do it in a way that would not break tools out there that interpret it.

Another way to address this concern would be to decide to cache or not *after* building the wheel ([here](https://github.com/pypa/pip/blob/8720d322cbb4d5ac59695ce693a70dc45b7502fe/src/pip/_internal/wheel.py#L1099)?), when we have a local checkout that we can inspect to know if the rev is a commit or a branch.

The only remaining edge case would then be when a branch has the same name as an existing commit id. That, if possible at all, is exotic enough to be ignored.

This approach indeed relates closely to the PEP for #609, since to implement it we need to extract the commit id to store it in the direct url origin metadata as `resolved_commit_id`.

So yes, we can keep this in mind while implementing #609 and come back to it just after.

The ball is with you to get the PEP rolling :)

Also, I think there's also the possibility of doing something where the revision is cached, but only after pip inspects it to see if the revision corresponds to a branch, tag, or SHA, etc. So the difference in perspective (e.g. from your PR I looked at) would be: when a revision is encountered, we don't ask "does this revision correspond to a commit hash because it looks like a SHA," but rather "is this revision cached" and use its presence in the cache to know if it corresponded to a commit hash. And seeing if it looks like a hash would basically be a check / optimization to know whether it has the possibility of being in the cache.
> Another way to address this concern would be to decide to cache or not after building the wheel (here?), when we have a local checkout that we can inspect to know if the rev is a commit or a branch.

Yes, what I just typed up is I think the same as what you're suggesting, though the way I would phrase it is "decide to cache after cloning the URL and inspecting the refs and SHA, etc. to see if the revision was a commit hash." pip already has this logic (except possibly for the last step of checking whether the revision matches the SHA). I do think we should keep pip's behavior of preferring the branch name over the SHA if there is a collision (since that is what Git does). So it would only be cached if the revision is a SHA that doesn't match a branch or tag name, etc.
By the way, when this (vcs caching) and #609 (vcs freeze) are implemented, it will make `--editable` a much less important feature for many.
#609 is probably more inline with something that we should look into adding into Pipfile, if it isn't already included.
> The ball is with you to get the PEP rolling :)

From what I recall of the discussion, @cjerdonek had offered to sponsor this PEP (I'm not 100% sure, because the sponsor details haven't been added to the PEP yet) so he'll be better able to help you navigate the process, but I think that, as PEP author, the ball is really with you :-)

From a quick skim to remind myself of where the discussion is at, I think that the debate on Discourse has died down. I'm not aware of any actions people are waiting on, so I think what you should probably do now is summarise the position, and list out what *you* think the next steps are, and how you think they should be progressed (remembering that this is all being done on volunteer time, so it's important to allow for people simply not having the free time to work on this).

In my experience, PEPs do need a lot of work to "manage" the process, as otherwise they tend to die through lack of interest, rather than any inherent flaw in the proposal.
@pfmoore He was actually giving me a friendly nudge because he's waiting on an email from me. :) I'm now co-authoring the PEP with him, and he was awaiting a reply from me confirming it's ready for submission. The list should see an update within the next couple days or so once I get to replying to his email..
Oh excellent - sorry for butting in in that case :-) The problems of things happening in a mixture of public and private forums.

Sorry @sbidoul for the unwarranted assumption on my part.

Resolving patch:
diff --git a/src/pip/_internal/vcs/git.py b/src/pip/_internal/vcs/git.py
--- a/src/pip/_internal/vcs/git.py
+++ b/src/pip/_internal/vcs/git.py
@@ -12,7 +12,7 @@
 from pip._vendor.six.moves.urllib import request as urllib_request
 
 from pip._internal.exceptions import BadCommand
-from pip._internal.utils.misc import display_path
+from pip._internal.utils.misc import display_path, hide_url
 from pip._internal.utils.subprocess import make_command
 from pip._internal.utils.temp_dir import TempDirectory
 from pip._internal.utils.typing import MYPY_CHECK_RUNNING
@@ -59,6 +59,23 @@ class Git(VersionControl):
     def get_base_rev_args(rev):
         return [rev]
 
+    def is_immutable_rev_checkout(self, url, dest):
+        # type: (str, str) -> bool
+        _, rev_options = self.get_url_rev_options(hide_url(url))
+        if not rev_options.rev:
+            return False
+        if not self.is_commit_id_equal(dest, rev_options.rev):
+            # the current commit is different from rev,
+            # which means rev was something else than a commit hash
+            return False
+        # return False in the rare case rev is both a commit hash
+        # and a tag or a branch; we don't want to cache in that case
+        # because that branch/tag could point to something else in the future
+        is_tag_or_branch = bool(
+            self.get_revision_sha(dest, rev_options.rev)[0]
+        )
+        return not is_tag_or_branch
+
     def get_git_version(self):
         VERSION_PFX = 'git version '
         version = self.run_command(['version'], show_stdout=False)
diff --git a/src/pip/_internal/vcs/versioncontrol.py b/src/pip/_internal/vcs/versioncontrol.py
--- a/src/pip/_internal/vcs/versioncontrol.py
+++ b/src/pip/_internal/vcs/versioncontrol.py
@@ -329,6 +329,20 @@ def get_base_rev_args(rev):
         """
         raise NotImplementedError
 
+    def is_immutable_rev_checkout(self, url, dest):
+        # type: (str, str) -> bool
+        """
+        Return true if the commit hash checked out at dest matches
+        the revision in url.
+
+        Always return False, if the VCS does not support immutable commit
+        hashes.
+
+        This method does not check if there are local uncommitted changes
+        in dest after checkout, as pip currently has no use case for that.
+        """
+        return False
+
     @classmethod
     def make_rev_options(cls, rev=None, extra_args=None):
         # type: (Optional[str], Optional[CommandArgs]) -> RevOptions
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -52,6 +52,7 @@
 from pip._internal.utils.ui import open_spinner
 from pip._internal.utils.unpacking import unpack_file
 from pip._internal.utils.urls import path_to_url
+from pip._internal.vcs import vcs
 
 if MYPY_CHECK_RUNNING:
     from typing import (
@@ -838,7 +839,15 @@ def should_cache(
         return False
 
     if req.link and req.link.is_vcs:
-        # VCS checkout. Build wheel just for this run.
+        # VCS checkout. Build wheel just for this run
+        # unless it points to an immutable commit hash in which
+        # case it can be cached.
+        assert not req.editable
+        assert req.source_dir
+        vcs_backend = vcs.get_backend_for_scheme(req.link.scheme)
+        assert vcs_backend
+        if vcs_backend.is_immutable_rev_checkout(req.link.url, req.source_dir):
+            return True
         return False
 
     link = req.link


Test patch:
diff --git a/tests/functional/test_install_vcs_git.py b/tests/functional/test_install_vcs_git.py
--- a/tests/functional/test_install_vcs_git.py
+++ b/tests/functional/test_install_vcs_git.py
@@ -67,7 +67,7 @@ def _github_checkout(url_path, temp_dir, rev=None, egg=None, scheme=None):
     return local_url
 
 
-def _make_version_pkg_url(path, rev=None):
+def _make_version_pkg_url(path, rev=None, name="version_pkg"):
     """
     Return a "git+file://" URL to the version_pkg test package.
 
@@ -78,7 +78,7 @@ def _make_version_pkg_url(path, rev=None):
     """
     file_url = _test_path_to_file_url(path)
     url_rev = '' if rev is None else '@{}'.format(rev)
-    url = 'git+{}{}#egg=version_pkg'.format(file_url, url_rev)
+    url = 'git+{}{}#egg={}'.format(file_url, url_rev, name)
 
     return url
 
@@ -476,3 +476,40 @@ def test_check_submodule_addition(script):
         script.venv / 'src/version-pkg/testpkg/static/testfile2'
         in update_result.files_created
     )
+
+
+def test_install_git_branch_not_cached(script, with_wheel):
+    """
+    Installing git urls with a branch revision does not cause wheel caching.
+    """
+    PKG = "gitbranchnotcached"
+    repo_dir = _create_test_package(script, name=PKG)
+    url = _make_version_pkg_url(repo_dir, rev="master", name=PKG)
+    result = script.pip("install", url, "--only-binary=:all:")
+    assert "Successfully built {}".format(PKG) in result.stdout, result.stdout
+    script.pip("uninstall", "-y", PKG)
+    # build occurs on the second install too because it is not cached
+    result = script.pip("install", url)
+    assert (
+        "Successfully built {}".format(PKG) in result.stdout
+    ), result.stdout
+
+
+def test_install_git_sha_cached(script, with_wheel):
+    """
+    Installing git urls with a sha revision does cause wheel caching.
+    """
+    PKG = "gitshacached"
+    repo_dir = _create_test_package(script, name=PKG)
+    commit = script.run(
+        'git', 'rev-parse', 'HEAD', cwd=repo_dir
+    ).stdout.strip()
+    url = _make_version_pkg_url(repo_dir, rev=commit, name=PKG)
+    result = script.pip("install", url)
+    assert "Successfully built {}".format(PKG) in result.stdout, result.stdout
+    script.pip("uninstall", "-y", PKG)
+    # build does not occur on the second install because it is cached
+    result = script.pip("install", url)
+    assert (
+        "Successfully built {}".format(PKG) not in result.stdout
+    ), result.stdout
diff --git a/tests/functional/test_vcs_git.py b/tests/functional/test_vcs_git.py
--- a/tests/functional/test_vcs_git.py
+++ b/tests/functional/test_vcs_git.py
@@ -221,3 +221,20 @@ def test_is_commit_id_equal(script):
     assert not Git.is_commit_id_equal(version_pkg_path, 'abc123')
     # Also check passing a None value.
     assert not Git.is_commit_id_equal(version_pkg_path, None)
+
+
+def test_is_immutable_rev_checkout(script):
+    version_pkg_path = _create_test_package(script)
+    commit = script.run(
+        'git', 'rev-parse', 'HEAD',
+        cwd=version_pkg_path
+    ).stdout.strip()
+    assert Git().is_immutable_rev_checkout(
+        "git+https://g.c/o/r@" + commit, version_pkg_path
+    )
+    assert not Git().is_immutable_rev_checkout(
+        "git+https://g.c/o/r", version_pkg_path
+    )
+    assert not Git().is_immutable_rev_checkout(
+        "git+https://g.c/o/r@master", version_pkg_path
+    )
diff --git a/tests/unit/test_wheel.py b/tests/unit/test_wheel.py
--- a/tests/unit/test_wheel.py
+++ b/tests/unit/test_wheel.py
@@ -20,7 +20,7 @@
     MissingCallableSuffix,
     _raise_for_invalid_entrypoint,
 )
-from tests.lib import DATA_DIR, assert_paths_equal
+from tests.lib import DATA_DIR, _create_test_package, assert_paths_equal
 
 
 class ReqMock:
@@ -166,6 +166,21 @@ def check_binary_allowed(req):
     assert should_cache is expected
 
 
+def test_should_cache_git_sha(script, tmpdir):
+    repo_path = _create_test_package(script, name="mypkg")
+    commit = script.run(
+        "git", "rev-parse", "HEAD", cwd=repo_path
+    ).stdout.strip()
+    # a link referencing a sha should be cached
+    url = "git+https://g.c/o/r@" + commit + "#egg=mypkg"
+    req = ReqMock(link=Link(url), source_dir=repo_path)
+    assert wheel.should_cache(req, check_binary_allowed=lambda r: True)
+    # a link not referencing a sha should not be cached
+    url = "git+https://g.c/o/r@master#egg=mypkg"
+    req = ReqMock(link=Link(url), source_dir=repo_path)
+    assert not wheel.should_cache(req, check_binary_allowed=lambda r: True)
+
+
 def test_format_command_result__INFO(caplog):
     caplog.set_level(logging.INFO)
     actual = wheel.format_command_result(