Repository: pypa/pip

Issue:
Determine if sdist name is "unique" enough for wheel cache key
Right now #2618 uses the sdist name as a cache key (though it uses a 1:M relationship to handle sdists which can output multiple different Wheels). We need to figure out if that is sufficient or if we should use the entire URL like the download cache used to use. A pro is that this will allow sharing a wheel cache between multiple indexes, but a con is that there's possible cache corruption between multiple indexes if you don't generate a local version for modifications and the like.



Discussion:
How will we determine this?

I meant to ping @pypa/pip-developers to see what other people thought, if we could trust that two different repositories (like PyPI and devpi, or two different devpis) that had the same _public_ version (not local version) for the same project (`pip-6.1.0.tar.gz` for example) would be the same thing. If we don't think it's OK to trust that, do we think that people using multiple repositories with the "same" version but different contents are just getting into undefined behavior territory and thus "here be dragons" or do we think that we need to forgo the benefits of sharing cached wheels amongst multiple repositories.

To answer your specific question, there's not a good way to determine this besides folks to try and guess and maybe solicit some anecdotal evidence. Mostly I added this issue to remember to do that.

I think its here-be-dragons, because there's no defined guarantee about which repo pip will choose from when two repos have the same version of X.

Well, you can do `--index-url` so that you, in a particular invocation of pip, override the default PyPI index so that you only have the one index. In other words, the two indexes don't have to exist in the same invocation since the cache directory is (obviously) shared between them. It's really the effect between two separate unrelated invocations that worries me.

Before auto wheels you could do `pip install --index-url https://example.com/ foo==1.0` and then in a different environment do `pip install foo==1.0` where the contents of foo 1.0 are different and it would work with no adverse effects (though it'd get confused if you tried to install one into an environment that was already installed, but you can work around that with `--upgrade --force-reinstall`. After auto wheels whichever one of those you do _first_ is going to be the one you get everytime no matter what you do for `--index`.

If I recall even when the old download cache used _just_ the URL, there was complaints about it not picking up updated copies of the package, and it was modified so that it also took into account the `#md5=fragment` of the URL.

I personally think it's a bit silly to have the same version mean different things, I'm mostly worried about breaking someones workflow for a (minor) optimization when guarding against it isn't very hard. I'll probably go ahead and work something up myself for this one, feel free to ignore it for now.

Ok, but if I can help let me know - I hate having release-blocker's on a project.


Resolving patch:
diff --git a/pip/wheel.py b/pip/wheel.py
--- a/pip/wheel.py
+++ b/pip/wheel.py
@@ -62,8 +62,9 @@ def cached_wheel(self, link, package_name):
             self._cache_dir, link, self._format_control, package_name)
 
 
-def _cache_for_filename(cache_dir, sdistfilename):
-    """Return a directory to store cached wheels in for sdistfilename.
+def _cache_for_link(cache_dir, link):
+    """
+    Return a directory to store cached wheels in for link.
 
     Because there are M wheels for any one sdist, we provide a directory
     to cache them in, and then consult that directory when looking up
@@ -76,10 +77,30 @@ def _cache_for_filename(cache_dir, sdistfilename):
     the same wheel even if the source has been edited.
 
     :param cache_dir: The cache_dir being used by pip.
-    :param sdistfilename: The filename of the sdist for which this will cache
-        wheels.
+    :param link: The link of the sdist for which this will cache wheels.
     """
-    return os.path.join(cache_dir, 'wheels', sdistfilename)
+
+    # We want to generate an url to use as our cache key, we don't want to just
+    # re-use the URL because it might have other items in the fragment and we
+    # don't care about those.
+    key_parts = [link.url_without_fragment]
+    if link.hash_name is not None and link.hash is not None:
+        key_parts.append("=".join([link.hash_name, link.hash]))
+    key_url = "#".join(key_parts)
+
+    # Encode our key url with sha224, we'll use this because it has similar
+    # security properties to sha256, but with a shorter total output (and thus
+    # less secure). However the differences don't make a lot of difference for
+    # our use case here.
+    hashed = hashlib.sha224(key_url.encode()).hexdigest()
+
+    # We want to nest the directories some to prevent having a ton of top level
+    # directories where we might run out of sub directories on some FS.
+    parts = [hashed[:2], hashed[2:4], hashed[4:6], hashed[6:]]
+
+    # Inside of the base location for cached wheels, expand our parts and join
+    # them all together.
+    return os.path.join(cache_dir, "wheels", *parts)
 
 
 def cached_wheel(cache_dir, link, format_control, package_name):
@@ -95,7 +116,7 @@ def cached_wheel(cache_dir, link, format_control, package_name):
     formats = pip.index.fmt_ctl_formats(format_control, canonical_name)
     if "binary" not in formats:
         return link
-    root = _cache_for_filename(cache_dir, link.filename)
+    root = _cache_for_link(cache_dir, link)
     try:
         wheel_names = os.listdir(root)
     except OSError as e:
@@ -724,8 +745,7 @@ def build(self, autobuilding=False):
             build_success, build_failure = [], []
             for req in buildset:
                 if autobuilding:
-                    output_dir = _cache_for_filename(
-                        self._cache_root, req.link.filename)
+                    output_dir = _cache_for_link(self._cache_root, req.link)
                     ensure_dir(output_dir)
                 else:
                     output_dir = self._wheel_dir


Test patch: