Repository: pypa/setuptools

Issue:
pkg_resources.Distribution.hashcmp is slow
As it removes md5 fragment on the fly, `pkg_resources.Distribution.hashcmp` is slow:

    @property
    def hashcmp(self):
        return (
            self.parsed_version,
            self.precedence,
            self.key,
            _remove_md5_fragment(self.location),
            self.py_version or '',
            self.platform or '',
        )

Would there be side effects if `location` is changed into a property where the setter would not only store the location but also store the location without fragment ?
pkg_resources.Distribution.hashcmp is slow
As it removes md5 fragment on the fly, `pkg_resources.Distribution.hashcmp` is slow:

    @property
    def hashcmp(self):
        return (
            self.parsed_version,
            self.precedence,
            self.key,
            _remove_md5_fragment(self.location),
            self.py_version or '',
            self.platform or '',
        )

Would there be side effects if `location` is changed into a property where the setter would not only store the location but also store the location without fragment ?


Discussion:
Can you quantify how slow is hashcmp and what the user impact is (e.g. what are the conditions under which a user would notice the slowness)?
The simple example below calls `hashcmp` (thus `_remove_md5_fragment`) nearly 1.3 million times:

    import pkg_resources
    import setuptools.package_index
    
    pki = setuptools.package_index.PackageIndex()
    req = pkg_resources.Requirement.parse('setuptools')
    pki.obtain(req)

I guess that `setuptools` is the most patholigical case.
Nevertheless, trying to obtain `django` calls `hashcmp` more than 200k times.
Thanks for that. It sounds like they way `package_index` uses it, it's very inefficient.

On my machine, that code takes about 9000ms to run. A long time, to be sure, but bearable on an incidental basis and a good motivator to avoid that code path.

I'm pretty sure `package_index` is deprecated as it's only used as part of the deprecated `easy_install`. Did you encounter this issue in real-world usage not using easy_install?
Can you quantify how slow is hashcmp and what the user impact is (e.g. what are the conditions under which a user would notice the slowness)?
The simple example below calls `hashcmp` (thus `_remove_md5_fragment`) nearly 1.3 million times:

    import pkg_resources
    import setuptools.package_index
    
    pki = setuptools.package_index.PackageIndex()
    req = pkg_resources.Requirement.parse('setuptools')
    pki.obtain(req)

I guess that `setuptools` is the most patholigical case.
Nevertheless, trying to obtain `django` calls `hashcmp` more than 200k times.
Thanks for that. It sounds like they way `package_index` uses it, it's very inefficient.

On my machine, that code takes about 9000ms to run. A long time, to be sure, but bearable on an incidental basis and a good motivator to avoid that code path.

I'm pretty sure `package_index` is deprecated as it's only used as part of the deprecated `easy_install`. Did you encounter this issue in real-world usage not using easy_install?

Resolving patch:
diff --git a/pkg_resources/__init__.py b/pkg_resources/__init__.py
--- a/pkg_resources/__init__.py
+++ b/pkg_resources/__init__.py
@@ -55,7 +55,7 @@
     FileExistsError = OSError
 
 from pkg_resources.extern import six
-from pkg_resources.extern.six.moves import urllib, map, filter
+from pkg_resources.extern.six.moves import map, filter
 
 # capture these to bypass sandboxing
 from os import utime
@@ -2546,15 +2546,6 @@ def parse_map(cls, data, dist=None):
         return maps
 
 
-def _remove_md5_fragment(location):
-    if not location:
-        return ''
-    parsed = urllib.parse.urlparse(location)
-    if parsed[-1].startswith('md5='):
-        return urllib.parse.urlunparse(parsed[:-1] + ('',))
-    return location
-
-
 def _version_from_file(lines):
     """
     Given an iterable of lines from a Metadata file, return
@@ -2611,7 +2602,7 @@ def hashcmp(self):
             self.parsed_version,
             self.precedence,
             self.key,
-            _remove_md5_fragment(self.location),
+            self.location,
             self.py_version or '',
             self.platform or '',
         )


Test patch:
diff --git a/setuptools/tests/test_packageindex.py b/setuptools/tests/test_packageindex.py
--- a/setuptools/tests/test_packageindex.py
+++ b/setuptools/tests/test_packageindex.py
@@ -9,9 +9,7 @@
 import mock
 import pytest
 
-import pkg_resources
 import setuptools.package_index
-from setuptools.tests.server import IndexServer
 from .textwrap import DALS
 
 
@@ -114,43 +112,6 @@ def test_url_ok(self):
         url = 'file:///tmp/test_package_index'
         assert index.url_ok(url, True)
 
-    def test_links_priority(self):
-        """
-        Download links from the pypi simple index should be used before
-        external download links.
-        https://bitbucket.org/tarek/distribute/issue/163
-
-        Usecase :
-        - someone uploads a package on pypi, a md5 is generated
-        - someone manually copies this link (with the md5 in the url) onto an
-          external page accessible from the package page.
-        - someone reuploads the package (with a different md5)
-        - while easy_installing, an MD5 error occurs because the external link
-          is used
-        -> Setuptools should use the link from pypi, not the external one.
-        """
-        if sys.platform.startswith('java'):
-            # Skip this test on jython because binding to :0 fails
-            return
-
-        # start an index server
-        server = IndexServer()
-        server.start()
-        index_url = server.base_url() + 'test_links_priority/simple/'
-
-        # scan a test index
-        pi = setuptools.package_index.PackageIndex(index_url)
-        requirement = pkg_resources.Requirement.parse('foobar')
-        pi.find_packages(requirement)
-        server.stop()
-
-        # the distribution has been found
-        assert 'foobar' in pi
-        # we have only one link, because links are compared without md5
-        assert len(pi['foobar']) == 1
-        # the link should be from the index
-        assert 'correct_md5' in pi['foobar'][0].location
-
     def test_parse_bdist_wininst(self):
         parse = setuptools.package_index.parse_bdist_wininst