Repository: pypa/pip

Issue:
--find-links should not warn about missing HTML5 doctype
### Description

This is *not* a duplicate of #10825 -- I've moved a number of my comments from #10825 as this is a separate issue. That issue is about enforcing PEP 503, which states that servers implementing the python simple index protocol should have an HTML5 doctype. This is not about that.

---

When using `--find-links`, this warning can appear:

```
× The package index page being used does not have a proper HTML doctype declaration.
╰─> Problematic URL: https://www.tortall.net/~robotpy/wheels/2022/roborio/

note: This is an issue with the page at the URL mentioned above.
hint: You might need to reach out to the owner of that package index, to get this fixed. See https://github.com/pypa/pip/issues/10825 for context.
```

Currently the `--find-links` documentation says:

```
  -f, --find-links <url>      If a URL or path to an html file, then parse for links
                              to archives such as sdist (.tar.gz) or wheel (.whl)
                              files. If a local path or file:// URL that's a
                              directory, then look for archives in the directory
                              listing. Links to VCS project URLs are not supported.
```

There is no HTML5 doctype requirement mentioned.

---

To me, `--find-links` serves a very different purpose than a full-up pypi-style index implementation. For environments where a full python index is too much work (or in corporate environments where working with IT is really difficult), it's very convenient to stick a bunch of files on a webserver and be able to point pip at an arbitrary directory listing and install packages from that directory. Unfortunately, the most popular webservers in the world (and even python's default http.server!) do not put an HTML5 doctype by default, because it simply does not matter if all you're doing is trying to show a directory listing so users can download a file. 

You might say, that it's currently only a warning, and it'll be a long time until we make it an error! But it's a useless warning, and the only way to fully resolve this is to go to every webserver vendor in the world and tell them that they must use an HTML5 doctype in their directory listings because pip says so. And *then* those changes need to be backported to 'stable' linux distributions like RHEL. 

In many corporate environments, developers don't get a choice of which webserver IT is using, and so this warning is just unnecessary noise and will waste hundreds of hours for developers and ops teams.

### Production-quality web servers that don't emit HTML5 doctype by default

* Apache2 (likely most popular webserver in the world) [does not use an HTML5 doctype by default](https://github.com/apache/httpd/blob/9c14928e7961060376cefbdd2274ab3c3facecd0/modules/generators/mod_autoindex.c#L170)
* nginx (likely the second most popular) [does not even have configurable autoindexing capabilities](https://github.com/nginx/nginx/blob/828fb94e1dbe1c433edd39147ba085c4622c99ed/src/http/modules/ngx_http_autoindex_module.c#L447)

Others that don't

* [python's default webserver](https://github.com/python/cpython/blob/50cf4991c49e19f917305dd7b9c71085c11edddb/Lib/http/server.py#L112)
* [Twisted web framework](https://github.com/twisted/twisted/blob/1d439dd1d9c7d302641550a925705d479ea5457f/src/twisted/web/static.py#L917)
* [golang's default webserver](https://cs.opensource.google/go/go/+/refs/tags/go1.17.7:src/net/http/fs.go;l=152;drc=refs%2Ftags%2Fgo1.17.7) doesn't even bother adding a doctype of any kind
* [WEBrick, a ruby webserver](https://github.com/ruby/webrick/blob/3515081a51b91b730267ba2b224039ecfbf8bd7b/lib/webrick/httpservlet/filehandler.rb#L496)
* [busybox httpd directory index](https://github.com/mirror/busybox/blob/24198f652f10dca5603df7c704263358ca21f5ce/networking/httpd_indexcgi.c#L276)

### Those that do

* [Lighthttpd](https://github.com/lighttpd/lighttpd1.4/blob/b8ef3f855c25f45b58cad902d7ef6ebf4b1d6d7f/src/mod_dirlisting.c#L850), since late 2016
* [Apache Tomcat](https://github.com/apache/tomcat/blob/6a667943c5da6b5d61ac6bec1d7c9de061e3217c/java/org/apache/catalina/servlets/DefaultServlet.java#L1785), since late 2019

---

I appreciate that html5lib adds a lot of work for pip maintainers. If there's a way to use http.parser and ignore the doctype (which the migration from an error to a warning indicates that it is), it seems like that would save hundreds (thousands?) of person-hours for ops teams all around the world who would need to figure out how to reconfigure their webservers because pip is being unnecessarily picky.

Thanks for your consideration.




### Expected behavior

No warning

### pip version

22.0.3

### Python version

3.10

### OS

any

### How to Reproduce

N/A

### Output

_No response_

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
This is a good point. I agree that `--find-links` should not enforce the doctype restriction.

A PR to fix this would be welcome.
Honestly, I'd be fine with dropping the doctype check entirely as well.

Resolving patch:
diff --git a/src/pip/_internal/exceptions.py b/src/pip/_internal/exceptions.py
--- a/src/pip/_internal/exceptions.py
+++ b/src/pip/_internal/exceptions.py
@@ -181,30 +181,6 @@ class UninstallationError(PipError):
     """General exception during uninstallation"""
 
 
-class BadHTMLDoctypeDeclaration(DiagnosticPipError):
-    reference = "bad-index-doctype"
-
-    def __init__(self, *, url: str) -> None:
-        super().__init__(
-            kind="warning",
-            message=(
-                "The package index page being used does not have a proper HTML "
-                "doctype declaration."
-            ),
-            context=f"Problematic URL: {escape(url)}",
-            note_stmt="This is an issue with the page at the URL mentioned above.",
-            hint_stmt=(
-                "You might need to reach out to the owner of that package index, "
-                "to get this fixed. "
-                "See https://github.com/pypa/pip/issues/10825 for context."
-            ),
-        )
-
-
-class MissingHTMLDoctypeDeclaration(BadHTMLDoctypeDeclaration):
-    reference = "missing-index-doctype"
-
-
 class MissingPyProjectBuildRequires(DiagnosticPipError):
     """Raised when pyproject.toml has `build-system`, but no `build-system.requires`."""
 
diff --git a/src/pip/_internal/index/collector.py b/src/pip/_internal/index/collector.py
--- a/src/pip/_internal/index/collector.py
+++ b/src/pip/_internal/index/collector.py
@@ -32,11 +32,7 @@
 from pip._vendor.requests import Response
 from pip._vendor.requests.exceptions import RetryError, SSLError
 
-from pip._internal.exceptions import (
-    BadHTMLDoctypeDeclaration,
-    MissingHTMLDoctypeDeclaration,
-    NetworkConnectionError,
-)
+from pip._internal.exceptions import NetworkConnectionError
 from pip._internal.models.link import Link
 from pip._internal.models.search_scope import SearchScope
 from pip._internal.network.session import PipSession
@@ -401,33 +397,12 @@ class HTMLLinkParser(HTMLParser):
 
     def __init__(self, url: str) -> None:
         super().__init__(convert_charrefs=True)
-        self._dealt_with_doctype_issues = False
 
         self.url: str = url
         self.base_url: Optional[str] = None
         self.anchors: List[Dict[str, Optional[str]]] = []
 
-    def handle_decl(self, decl: str) -> None:
-        self._dealt_with_doctype_issues = True
-        match = re.match(
-            r"""doctype\s+html\s*(?:SYSTEM\s+(["'])about:legacy-compat\1)?\s*$""",
-            decl,
-            re.IGNORECASE,
-        )
-        if match is None:
-            logger.warning(
-                "[present-diagnostic] %s",
-                BadHTMLDoctypeDeclaration(url=self.url),
-            )
-
     def handle_starttag(self, tag: str, attrs: List[Tuple[str, Optional[str]]]) -> None:
-        if not self._dealt_with_doctype_issues:
-            logger.warning(
-                "[present-diagnostic] %s",
-                MissingHTMLDoctypeDeclaration(url=self.url),
-            )
-            self._dealt_with_doctype_issues = True
-
         if tag == "base" and self.base_url is None:
             href = self.get_href(attrs)
             if href is not None:


Test patch:
diff --git a/tests/functional/test_install_index.py b/tests/functional/test_install_index.py
--- a/tests/functional/test_install_index.py
+++ b/tests/functional/test_install_index.py
@@ -1,4 +1,5 @@
 import os
+import shutil
 import textwrap
 import urllib.parse
 
@@ -24,6 +25,21 @@ def test_find_links_relative_path(script: PipTestEnvironment, data: TestData) ->
     result.did_create(initools_folder)
 
 
+def test_find_links_no_doctype(script: PipTestEnvironment, data: TestData) -> None:
+    shutil.copy(data.packages / "simple-1.0.tar.gz", script.scratch_path)
+    html = script.scratch_path.joinpath("index.html")
+    html.write_text('<a href="simple-1.0.tar.gz"></a>')
+    result = script.pip(
+        "install",
+        "simple==1.0",
+        "--no-index",
+        "--find-links",
+        script.scratch_path,
+        expect_stderr=True,
+    )
+    assert not result.stderr
+
+
 @pytest.mark.usefixtures("with_wheel")
 def test_find_links_requirements_file_relative_path(
     script: PipTestEnvironment, data: TestData
diff --git a/tests/unit/test_collector.py b/tests/unit/test_collector.py
--- a/tests/unit/test_collector.py
+++ b/tests/unit/test_collector.py
@@ -551,44 +551,6 @@ def test_parse_link_handles_deprecated_usage_properly() -> None:
     assert "pkg1-2.0" in parsed_links[1].url
 
 
-def test_parse_links_presents_warning_on_missing_doctype(
-    caplog: pytest.LogCaptureFixture,
-) -> None:
-    html = b'<a href="/pkg1-1.0.tar.gz"></a><a href="/pkg1-2.0.tar.gz"></a>'
-    url = "https://example.com/simple/"
-    page = HTMLPage(html, encoding=None, url=url, cache_link_parsing=False)
-
-    with caplog.at_level(logging.WARN):
-        parsed_links = list(parse_links(page, use_deprecated_html5lib=False))
-
-    assert len(parsed_links) == 2, parsed_links
-    assert "pkg1-1.0" in parsed_links[0].url
-    assert "pkg1-2.0" in parsed_links[1].url
-
-    assert len(caplog.records) == 1
-
-
-def test_parse_links_presents_warning_on_html4_doctype(
-    caplog: pytest.LogCaptureFixture,
-) -> None:
-    html = (
-        b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" '
-        b'"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">'
-        b'<a href="/pkg1-1.0.tar.gz"></a><a href="/pkg1-2.0.tar.gz"></a>'
-    )
-    url = "https://example.com/simple/"
-    page = HTMLPage(html, encoding=None, url=url, cache_link_parsing=False)
-
-    with caplog.at_level(logging.WARN):
-        parsed_links = list(parse_links(page, use_deprecated_html5lib=False))
-
-    assert len(parsed_links) == 2, parsed_links
-    assert "pkg1-1.0" in parsed_links[0].url
-    assert "pkg1-2.0" in parsed_links[1].url
-
-    assert len(caplog.records) == 1
-
-
 @mock.patch("pip._internal.index.collector.raise_for_status")
 def test_request_http_error(
     mock_raise_for_status: mock.Mock, caplog: pytest.LogCaptureFixture