Repository: pypa/pip

Issue:
Pip silently ignores index pages with unexpected content type
**Environment**

* pip version: 19.1.1
* Python version: 3.7.2
* OS: Ubuntu Linux 18.04

<!-- Feel free to add more information about your environment here -->

**Description**

As mentioned in #6697, if a package index page configured via `--index-url` returns an unexpected content type, then pip silently ignores it and displays a generic "No matching distribution found" error message.

**Expected behavior**

pip should display a warning in the event that the content returned by the package index is ill-formed or not understood.

**How to Reproduce**

<details>
<summary><b>repro.sh</b></summary>

```
#!/bin/sh
cd $(mktemp -d)
python -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
cat <<EOF > server.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('content-type', 'application/xhtml+xml')
        self.end_headers()


if __name__ == '__main__':
    server = ThreadingHTTPServer(('localhost', 11111), Handler)
    server.serve_forever()
EOF
python server.py >/dev/null 2>&1 &
echo ========== run 1 ==========
pip install --index-url=http://localhost:11111/ requests
echo ========== run 2 ==========
pip install --verbose --index-url=http://localhost:11111/ requests
```

</details>

**Output**

<details>
<summary>run 1</summary>

```
Looking in indexes: http://localhost:11111/
Collecting requests
  ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
ERROR: No matching distribution found for requests
```

</details>

<details>
<summary>run 2 (verbose)</summary>

```
Created temporary directory: /tmp/user/1000/pip-ephem-wheel-cache-nz_lnbof
Created temporary directory: /tmp/user/1000/pip-req-tracker-68gx3hw_
Created requirements tracker '/tmp/user/1000/pip-req-tracker-68gx3hw_'
Created temporary directory: /tmp/user/1000/pip-install-3cazkrz9
Looking in indexes: http://localhost:11111/
Collecting requests
  1 location(s) to search for versions of requests:
  * http://localhost:11111/requests/
  Getting page http://localhost:11111/requests/
  Starting new HTTP connection (1): localhost:11111
  http://localhost:11111 "GET /requests/ HTTP/1.1" 200 None
  Skipping page http://localhost:11111/requests/ because the GET request got Content-Type: application/xhtml+xml
  ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
Cleaning up...
Removed build tracker '/tmp/user/1000/pip-req-tracker-68gx3hw_'
ERROR: No matching distribution found for requests
Exception information:
Traceback (most recent call last):
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 352, in run
    resolver.resolve(requirement_set)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 282, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 198, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/tmp/user/1000/tmp.ubtUD2yfZP/.venv/lib/python3.7/site-packages/pip/_internal/index.py", line 792, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for requests
```

</details>

Specifically, the issue is that the message

>   Skipping page http://localhost:11111/requests/ because the GET request got Content-Type: application/xhtml+xml

only shows up with `--verbose`.


Discussion:
Is this PR just a change from `logger.debug` to `logger.warning`? If yes I would like to take it up
At first glance this should be useful improvement, yes. We may want to add a mention of the supported content types in the error message.
Thanks, and it looks like `text/html` is the only acceptable header as per

https://github.com/pypa/pip/blob/97f639057ec0855804c5cb53423ac74394cdefc6/src/pip/_internal/index/collector.py#L98-L106

Resolving patch:
diff --git a/src/pip/_internal/index/collector.py b/src/pip/_internal/index/collector.py
--- a/src/pip/_internal/index/collector.py
+++ b/src/pip/_internal/index/collector.py
@@ -455,8 +455,9 @@ def _get_html_page(link, session=None):
             'be checked by HEAD.', link,
         )
     except _NotHTML as exc:
-        logger.debug(
-            'Skipping page %s because the %s request got Content-Type: %s',
+        logger.warning(
+            'Skipping page %s because the %s request got Content-Type: %s.'
+            'The only supported Content-Type is text/html',
             link, exc.request_desc, exc.content_type,
         )
     except HTTPError as exc:


Test patch:
diff --git a/tests/unit/test_collector.py b/tests/unit/test_collector.py
--- a/tests/unit/test_collector.py
+++ b/tests/unit/test_collector.py
@@ -476,6 +476,36 @@ def test_get_html_page_invalid_scheme(caplog, url, vcs_scheme):
     ]
 
 
+@pytest.mark.parametrize(
+    "content_type",
+    [
+        "application/xhtml+xml",
+        "application/json",
+    ],
+)
+def test_get_html_page_invalid_content_type(caplog, content_type):
+    """`_get_html_page()` should warn if an invalid content-type is given.
+    Only text/html is allowed.
+    """
+    caplog.set_level(logging.DEBUG)
+    url = 'https://pypi.org/simple/pip'
+    link = Link(url)
+
+    session = mock.Mock(PipSession)
+    session.get.return_value = mock.Mock(**{
+        "request.method": "GET",
+        "headers": {"Content-Type": content_type},
+    })
+
+    assert _get_html_page(link, session=session) is None
+    assert ('pip._internal.index.collector',
+            logging.WARNING,
+            'Skipping page {} because the GET request got Content-Type: {}.'
+            'The only supported Content-Type is text/html'.format(
+                url, content_type)) \
+        in caplog.record_tuples
+
+
 def make_fake_html_response(url):
     """
     Create a fake requests.Response object.