Repository: pypa/pip

Issue:
`--use-deprecated=html5lib` does not parse links, even though they're present
### Description

When using Pip 22.0 with `--use-deprecated=html5lib` with JFrog as the Index packages pip throws the error:  `ERROR: No matching distribution found for requests`

Tested with the "requests" package on Windows 10 using pip 22.0 (fails) and pip 21.3.1 (works)

### Expected behavior

` --use-deprecated=html5lib` should allow JFrog indexes to work.

### pip version

22.0

### Python version

3.10

### OS

Windows

### How to Reproduce

Install package from JFrog index using pip 22.0

### Output

```sh-session
C:\>python -m pip install -vvv requests --use-deprecated=html5lib
Using pip 22.0 from <corporate_local_path>\lib\site-packages\pip (python 3.10)
Non-user install by explicit request
Created temporary directory: <corporate_user_path>\AppData\Local\Temp\pip-ephem-wheel-cache-4a5e6ucc
Created temporary directory: <corporate_user_path>\AppData\Local\Temp\pip-req-tracker-p0zhtye3
Initialized build tracking at <corporate_user_path>\AppData\Local\Temp\pip-req-tracker-p0zhtye3
Created build tracker: <corporate_user_path>\AppData\Local\Temp\pip-req-tracker-p0zhtye3
Entered build tracker: <corporate_user_path>\AppData\Local\Temp\pip-req-tracker-p0zhtye3
Created temporary directory: <corporate_user_path>\AppData\Local\Temp\pip-install-_cnfjhxu
Looking in indexes: http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple
1 location(s) to search for versions of requests:
* http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/
Fetching project page and analyzing links: http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/
Getting page http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/
Found index url http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple
Looking up http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/ in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTP connection (1): <corporate_domain>:80
http://<corporate_domain>:80 "GET /artifactory/api/pypi/pypi-release/simple/requests/ HTTP/1.1" 200 None
Updating cache with response from http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/
Skipping link: not a file: http://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/
Given no hashes to check 0 links for project 'requests': discarding no candidates
ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
ERROR: No matching distribution found for requests
Exception information:
Traceback (most recent call last):
  File "<corporate_local_path>\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "<corporate_local_path>\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('requests')
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "<corporate_local_path>\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "<corporate_local_path>\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "<corporate_local_path>\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 350, in resolve
    raise ResolutionImpossible(e.criterion.information)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('requests'), parent=None)]
 
The above exception was the direct cause of the following exception:
 
Traceback (most recent call last):
  File "<corporate_local_path>\lib\site-packages\pip\_internal\cli\base_command.py", line 165, in exc_logging_wrapper
    status = run_func(*args)
  File "<corporate_local_path>\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "<corporate_local_path>\lib\site-packages\pip\_internal\commands\install.py", line 339, in run
    requirement_set = resolver.resolve(
  File "<corporate_local_path>\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 103, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: No matching distribution found for requests

Removed build tracker: '<corporate_user_path>\\AppData\\Local\\Temp\\pip-req-tracker-p0zhtye3'
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
When I run the same command under Pip 21.3.1 I see it runs a GET on `https://<corporate_domain>/artifactory/api/pypi/pypi-release/simple/requests/ ` and is able to parse the following response: https://gist.github.com/notatallshaw/caef03cdb0592c13fab463a9fb5223a3 with many "Found link" results.
Can you share the raw HTML document returned by Artifactory? Feel free to redact the URLs as appropriate.
> Can you share the raw HTML document returned by Artifactory? Feel free to redact the URLs as appropriate.

Already done in previous comment: https://github.com/pypa/pip/issues/10845#issuecomment-1025166105
I'm able to reproduce this, with just pip's parsing logic:

```
from pathlib import Path

from pip._internal.index.collector import HTMLPage, parse_links

content = Path("/tmp/page.html").read_bytes()
page = HTMLPage(content, "utf-8", "https://private.domain.example.com/index")

try:
    print("new", len(list(parse_links(page, use_deprecated_html5lib=True))))
except TypeError:
    print("old", len(list(parse_links(page))))
```

21.3.1

```
❯ python /tmp/foo.py
old 208
```

22.0

```
❯ python /tmp/foo.py
new 0
```


Resolving patch:
diff --git a/src/pip/_internal/index/collector.py b/src/pip/_internal/index/collector.py
--- a/src/pip/_internal/index/collector.py
+++ b/src/pip/_internal/index/collector.py
@@ -343,7 +343,8 @@ def parse_links(page: "HTMLPage", use_deprecated_html5lib: bool) -> Iterable[Lin
     Parse an HTML document, and yield its anchor elements as Link objects.
     """
     if use_deprecated_html5lib:
-        return _parse_links_html5lib(page)
+        yield from _parse_links_html5lib(page)
+        return
 
     parser = HTMLLinkParser()
     encoding = page.encoding or "utf-8"


Test patch:
diff --git a/tests/unit/test_collector.py b/tests/unit/test_collector.py
--- a/tests/unit/test_collector.py
+++ b/tests/unit/test_collector.py
@@ -539,6 +539,18 @@ def test_parse_links_caches_same_page_by_url() -> None:
     assert "pkg2" in parsed_links_3[0].url
 
 
+def test_parse_link_handles_deprecated_usage_properly() -> None:
+    html = b'<a href="/pkg1-1.0.tar.gz"><a href="/pkg1-2.0.tar.gz">'
+    url = "https://example.com/simple/"
+    page = HTMLPage(html, encoding=None, url=url)
+
+    parsed_links = list(parse_links(page, use_deprecated_html5lib=True))
+
+    assert len(parsed_links) == 2
+    assert "pkg1-1.0" in parsed_links[0].url
+    assert "pkg1-2.0" in parsed_links[1].url
+
+
 @mock.patch("pip._internal.index.collector.raise_for_status")
 def test_request_http_error(
     mock_raise_for_status: mock.Mock, caplog: pytest.LogCaptureFixture