Repository: pypa/pip

Issue:
Handle HTML encoding better
(Disclaimer: this is based upon me poking my head around index.py for about five minutes, so I could be speaking bullshit.)

As far as I can tell, after #945 was merged, pip tries UTF-8 falling back to ISO-8859-1 if there's no Content-Type. However, what browsers do and what the HTML spec says is far more complex than that. From what I saw, the decoded data is never touched except my html5lib — therefore, it makes more sense to let html5lib deal with all the complexity, by passing `html5lib.parse` an `encoding` argument (if there is one declared in the `Content-Type` HTTP header, None (the default) can be passed otherwise) along with the bytes that was got over the wire.



Discussion:
Oh, and if you want to pass the actual urllib file object you'll need to upgrade to html5lib 1.0b3 (uh, yeah, that broke in 0.95 -> 1.0b1). If you just want to pass the bytes objection then you're fine with 1.0b1.

So what we actually do now, is we let requests handle turning the body from bytes into unicode. This will use https://github.com/kennethreitz/requests/blob/master/requests/models.py#L714-L750.

The apparent rules appear to be:
1. If there exists a charset attached to the content-type, use that
2. Fall back to ISO-8859-1

It sounds like your suggestion would be to pass the bytes from requests into `html5lib.parse` along with any encoding information we have?

Yes, that's correct as to what requests does — it's what RFC 2616 (now obsolete) requires.It's also totally incompatible with what every web browser has done forever — and not what websites expect.

But yes — that's essentially what you want to do. It's not _quite_ the right behaviour (https://github.com/html5lib/html5lib-python/issues/120 provides a rough outline for the API breakage needed to fix it) as we always use the Content-Type encoding if it exists (in reality BOMs are given priority over it), but it's a lot closer (and more compatible) than what there is today.


Resolving patch:
diff --git a/pip/index.py b/pip/index.py
--- a/pip/index.py
+++ b/pip/index.py
@@ -2,6 +2,7 @@
 from __future__ import absolute_import
 
 import logging
+import cgi
 import sys
 import os
 import re
@@ -735,16 +736,28 @@ class HTMLPage(object):
     """Represents one page, along with its URL"""
 
     # FIXME: these regexes are horrible hacks:
-    _homepage_re = re.compile(r'<th>\s*home\s*page', re.I)
-    _download_re = re.compile(r'<th>\s*download\s+url', re.I)
+    _homepage_re = re.compile(b'<th>\\s*home\\s*page', re.I)
+    _download_re = re.compile(b'<th>\\s*download\\s+url', re.I)
     _href_re = re.compile(
-        'href=(?:"([^"]*)"|\'([^\']*)\'|([^>\\s\\n]*))',
+        b'href=(?:"([^"]*)"|\'([^\']*)\'|([^>\\s\\n]*))',
         re.I | re.S
     )
 
     def __init__(self, content, url, headers=None, trusted=None):
+        # Determine if we have any encoding information in our headers
+        encoding = None
+        if headers and "Content-Type" in headers:
+            content_type, params = cgi.parse_header(headers["Content-Type"])
+
+            if "charset" in params:
+                encoding = params['charset']
+
         self.content = content
-        self.parsed = html5lib.parse(self.content, namespaceHTMLElements=False)
+        self.parsed = html5lib.parse(
+            self.content,
+            encoding=encoding,
+            namespaceHTMLElements=False,
+        )
         self.url = url
         self.headers = headers
         self.trusted = trusted
@@ -824,7 +837,10 @@ def get_page(cls, link, req, skip_archives=True, session=None):
                 )
                 return
 
-            inst = cls(resp.text, resp.url, resp.headers, trusted=link.trusted)
+            inst = cls(
+                resp.content, resp.url, resp.headers,
+                trusted=link.trusted,
+            )
         except requests.HTTPError as exc:
             level = 2 if exc.response.status_code == 404 else 1
             cls._handle_fail(req, link, exc, url, level=level)
@@ -955,6 +971,10 @@ def scraped_rel_links(self):
             )
             if not url:
                 continue
+            try:
+                url = url.decode("ascii")
+            except UnicodeDecodeError:
+                continue
             url = self.clean_link(urllib_parse.urljoin(self.base_url, url))
             yield Link(url, self, trusted=False, _deprecated_regex=True)
 


Test patch:
diff --git a/tests/unit/test_index.py b/tests/unit/test_index.py
--- a/tests/unit/test_index.py
+++ b/tests/unit/test_index.py
@@ -10,7 +10,7 @@ def test_html_page_should_be_able_to_scrap_rel_links():
     Test scraping page looking for url in href
     """
     page = HTMLPage(
-        """
+        b"""
 <!-- The <th> elements below are a terrible terrible hack for setuptools -->
 <li>
 <strong>Home Page:</strong>