Repository: pypa/pipenv

Issue:
Bug: 404 error while calculating hashes with 2023.8.28
### Issue description

when upgrading from pipenv 2023.7.3 to 2023.8.28. First there is very verbose logging for INFO:pipenv.patched.pip._internal.operations.prepare. In another bug.

For some repostories (like nexus) that used to return md5 instead of sha256, pipenv downloads the file and calculates the hash. 
It seems that automatic calculation of hash in case md5 is used in nexus it uses incorrect url.

INFO:pipenv.patched.pip._internal.operations.prepare:Collecting wcmatch==8.3 (from semgrep==0.78.0->-r /var/folders/l6/nmk965l50gv_90_c4xh5x0_4ctpk94/T/pipenv-jgl0rhw4-requirements/pipenv-an17je3b-constraints.txt (line 19))
INFO:pipenv.patched.pip._internal.network.download:Using cached https://nexus.company.net/repository/pypi-blessed/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl (42 kB)

Downloading file wcmatch-8.3-py3-none-any.whl to obtain hash...
HTTP error 404 while getting https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl#md5=8d2acdbf5586e2175b9f88d16e599ac8
Downloading file wcmatch-8.3.tar.gz to obtain hash...
HTTP error 404 while getting https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8
Downloading file wcmatch-8.3.tar.gz to obtain hash...
Downloading file wcmatch-8.3-py3-none-any.whl to obtain hash...

When resolving downloading hashes part of url is missing. But it works when downloading original whl or tar.gz.

[[source]]
name = "nexus"
url = "https://u:p@nexus.company.net/repository/pypi-blessed/simple"
verify_ssl = true

### Expected result

It should download from correct url and calculate the hash.

### Actual result

It fetches from wrongly build url and gets 404 from the nexus.

### Steps to replicate

It seems that this behaviour works for https://pypi.python.org/simple because /simple is at root.
However for https://u:p@nexus.company.net/repository/pypi-blessed/simple it does not.

Pipfile:
[[source]]
name = "nexus"
url = "https://u:p@nexus.company.net/repository/pypi-blessed/simple" 
verify_ssl = true

-------------------------------------------------------------------------------




Discussion:
It seems that hashing seems quite broken in pipenv currently.
Another issue with missing hashes for different naming (case sensitivity, dot vs dash)

What we have in nexus:
![image](https://github.com/pypa/pipenv/assets/3091122/eae06efa-bcdb-454d-8a20-38c50a6e037b)

and with upgraded pipenv all the hashes are removed.
![image](https://github.com/pypa/pipenv/assets/3091122/9939e608-a66d-4af1-aea4-c8ab681a027c)

Same story for pyyaml:
![image](https://github.com/pypa/pipenv/assets/3091122/62bb370f-e458-4601-a89b-b22ed6c53882)

and removed hahes:
![image](https://github.com/pypa/pipenv/assets/3091122/e151d9e0-6add-4de9-8879-ad286f835c4d)


Or dots vs dashes:
<img width="592" alt="image" src="https://github.com/pypa/pipenv/assets/3091122/d1dd13fd-3b76-47c5-8589-1d1b99a39935">

![image](https://github.com/pypa/pipenv/assets/3091122/67fdd166-186b-4994-a63e-d83af3e4ed86)

MarkupSafe is another case for broken hashes.
@matejsp What makes you say it has to do with /simple/ -- from what I can see for non pypi sources, it is trying to find the hashes and package links from:
```
pkg_url = f"{source['url']}/{ireq.name}/"
```

Does `https://u:p@nexus.company.net/repository/pypi-blessed/simple/{package_name}` exist in your case?

I notice in your logs:
`https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8` but that seems to be missing the `/repository/pypi-blessed/simple` part of the path from your original source URL.

This is high on my list to get to the bottom of, your help is appreciated.
My current best guess is there is a relative href on an index that is like `https://u:p@nexus.company.net/repository/pypi-blessed/simple/{package_name}` and that its trying to get the package url from `package_url = urljoin(source["url"], package_url)`
since your source has a path like `/repository/pypi-blessed/simple/` its getting truncated and replaced with the partial relative path on that index page.   Does this sound accurate?
Yes first url exists and where whl were always downloaded from, first line from is downloading (or using from cache):
https://nexus.company.net/repository/pypi-blessed/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl

versus (wrongly concated)

https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8

Md5 is returned when downloading the package from nexus. old uploaded ones have md5 new one have sha256 (but no way to fix old ones).

Resolving patch:
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -292,17 +292,16 @@ def get_hashes_from_pypi(self, ireq, source):
             return None
 
     def get_hashes_from_remote_index_urls(self, ireq, source):
-        pkg_url = f"{source['url']}/{ireq.name}/"
+        normalized_name = normalize_name(ireq.name)
+        url_name = normalized_name.replace(".", "-")
+        pkg_url = f"{source['url']}/{url_name}/"
         session = self.get_requests_session_for_source(source)
+
         try:
             collected_hashes = set()
-            # Grab the hashes from the new warehouse API.
             response = session.get(pkg_url, timeout=10)
-            # Create an instance of the parser
             parser = PackageIndexHTMLParser()
-            # Feed the HTML to the parser
             parser.feed(response.text)
-            # Extract hrefs
             hrefs = parser.urls
 
             version = ""
@@ -310,19 +309,45 @@ def get_hashes_from_remote_index_urls(self, ireq, source):
                 spec = next(iter(s for s in ireq.specifier), None)
                 if spec:
                     version = spec.version
+
+            # We'll check if the href looks like a version-specific page (i.e., ends with '/')
             for package_url in hrefs:
-                if version in parse.unquote(package_url):
+                parsed_url = parse.urlparse(package_url)
+                if version in parsed_url.path and parsed_url.path.endswith("/"):
+                    # This might be a version-specific page. Fetch and parse it
+                    version_url = urljoin(pkg_url, package_url)
+                    version_response = session.get(version_url, timeout=10)
+                    version_parser = PackageIndexHTMLParser()
+                    version_parser.feed(version_response.text)
+                    version_hrefs = version_parser.urls
+
+                    # Process these new hrefs as potential wheels
+                    for v_package_url in version_hrefs:
+                        url_params = parse.urlparse(v_package_url).fragment
+                        params_dict = parse.parse_qs(url_params)
+                        if params_dict.get(FAVORITE_HASH):
+                            collected_hashes.add(params_dict[FAVORITE_HASH][0])
+                        else:  # Fallback to downloading the file to obtain hash
+                            v_package_full_url = urljoin(version_url, v_package_url)
+                            link = Link(v_package_full_url)
+                            file_hash = self.get_file_hash(session, link)
+                            if file_hash:
+                                collected_hashes.add(file_hash)
+                elif version in parse.unquote(package_url):
+                    # Process the current href as a potential wheel from the main page
                     url_params = parse.urlparse(package_url).fragment
                     params_dict = parse.parse_qs(url_params)
                     if params_dict.get(FAVORITE_HASH):
                         collected_hashes.add(params_dict[FAVORITE_HASH][0])
                     else:  # Fallback to downloading the file to obtain hash
-                        package_url = urljoin(source["url"], package_url)
-                        link = Link(package_url)
+                        package_full_url = urljoin(pkg_url, package_url)
+                        link = Link(package_full_url)
                         file_hash = self.get_file_hash(session, link)
                         if file_hash:
                             collected_hashes.add(file_hash)
+
             return self.prepend_hash_types(collected_hashes, FAVORITE_HASH)
+
         except (ValueError, KeyError, ConnectionError):
             if self.s.is_verbose():
                 click.echo(
@@ -1198,8 +1223,12 @@ def add_pipfile_entry_to_pipfile(self, name, normalized_name, entry, category=No
         return newly_added, category, normalized_name
 
     def src_name_from_url(self, index_url):
-        name, _, tld_guess = urllib.parse.urlsplit(index_url).netloc.rpartition(".")
-        src_name = name.replace(".", "")
+        location = urllib.parse.urlsplit(index_url).netloc
+        if "." in location:
+            name, _, tld_guess = location.rpartition(".")
+        else:
+            name = location
+        src_name = name.replace(".", "").replace(":", "")
         try:
             self.get_source(name=src_name)
         except SourceNotFound:
@@ -1221,7 +1250,7 @@ def add_index_to_pipfile(self, index, verify_ssl=True):
             with contextlib.suppress(SourceNotFound):
                 source = self.get_source(name=index)
 
-        if source is not None:
+        if source is not None and source.get("name"):
             return source["name"]
         source = {"url": index, "verify_ssl": verify_ssl}
         source["name"] = self.src_name_from_url(index)


Test patch: