Repository: encode/httpx

Issue:
Replace chardet with plain UTF-8 detection
I'd suggest replacing `chardet` with dead-simple decoding logic:

```python
encoding = "UTF-8"
try:
    text = databytes.decode(encoding)
except UnicodeDecodeError:
    encoding = "ISO-8859-1"
    text = databytes.decode(encoding)
```

It is *fairly* safe to assume that any valid UTF-8 sequence is in fact UTF-8, and should be decoded as such.

Otherwise ISO-8859-1 (as specified in HTTP RFC) should be used instead of trying to detect languages, which is always quite unreliable. Further, ISO-8859-1 maps **all** possible byte values directly to Unicode code points 0-255, so the final decoding step in the above code will never raise UnicodeDecodeError.

Since the chardet module has been around a long time and is known as the standard go-to solution for charset detection, many modules depend on it, and this is a frequent source of dependency conflicts such as the one reported here: https://github.com/huge-success/sanic/issues/1858

A more serious problem is that chardet is notoriously unreliable. Especially its tendency to misdetect anything as Turkish is well known:

```python
>>> chardet.decode("Testiä".encode("UTF-8"))   # "Testing" in Finnish
{'encoding': 'ISO-8859-9',
 'confidence': 0.5521177026603239,
 'language': 'Turkish'}

>>> "Testiä".encode("UTF-8").decode("ISO-8859-9")  # Yields gibberish, not Turkish
'TestiÃ¤'
```




Discussion:
Hiya. Okay, this is well pitched, tho it's quite hard to get a handle onto if it'd be a net positive or net negative for us to switch away from.

We use `chardet` in cases where the charset *hasn't* been included in the Content-Type, because one of the initial design goals for `httpx` was `requests` compatiblility and that's historically been their behaviour. However I'm open to reassessing this. I expect the landscape has kinda change with the prevalence of UTF-8 having dramatically risen in that time.

* It'd be interesting to collate some issues from `requests` to see what their prior experiences are here.
* It'd be interesting to get an insight into browser behaviours wrt. fuzzily determining content types where it's unset in the headers.
* We might well do a better job by using some fuzzy pattern matching for [HTML meta tags](https://en.wikipedia.org/wiki/Charset_detection) as a fallback when headers are unset, with utf-8, and then iso-8859-1 as absolute fallbacks.

I guess the upshot of this is likely to be...

* Start by allowing the "apparent encoding" mechanism to be configurable.
* Switch chardet *out* from being the default, and instead fallback on  1. fuzzy pattern match for meta tags 2. utf-8 3. iso-8859-1 in that order.

Hrm, so it looks to me like `chardet` is what Mozilla use if both Content-Type and Meta tags fail... https://www-archive.mozilla.org/projects/intl/chardet.html

As such I think we ought to be a bit careful about the assumption in "It is fairly safe to assume that any valid UTF-8 sequence is in fact UTF-8, and should be decoded as such." - it's not *necessarily* a given that'd match up with actual real-world behaviours, or with current browser behaviours.

One thing that'd be helpful would be some examples of URLs which are resolving with undesired content-types. Having a dig into real world cases would help given a better overall orientation of cases where HTML pages are omitting encoding information.
Useful context in https://github.com/psf/requests/issues/1737
Also useful context https://github.com/aio-libs/aiohttp/issues/1811 (`aiohttp` diverges from `requests` here)
Benchmark of the OP code against chardet detect, using 100 kB of ASCII and 100 kB of ISO-8859-1 (only 'ä' repeated):
```
>>> %timeit auto_decode(ascii_bytes)
7.66 µs ± 84.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> %timeit auto_decode(latin1_bytes)
12.1 µs ± 61.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

>>> %timeit chardet.detect(ascii_bytes)
1.32 ms ± 8.15 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

>>> %timeit chardet.detect(latin1_bytes)
2.89 s ± 19.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

There is a huge difference even with ASCII, and non-ASCII characters totally devastate chardet performance (tried with UTF-8 too, with similar timing results as latin1 on both implementations).

It might be tempting to try and optimise by only using the initial few kilobytes for detection but doing so is likely to catch only HTML markup, missing the actual native characters that would appear somewhere in the middle of the document. Fortunately Python's built-in decoding code is extremely fast and breaks on first error, so any such optimisations are not needed, especially when the decoding result is also actually used.
Okay, so given that the landscape has moved on since `requests` opted with `chardet` for determining unknown character encodings, and that UTF-8 is *so much more* prevalent now, I'm feeling pretty open to us considering a change in behaviour here.

There's two practical things we'd need to consider to make that change...

1. A pull request updating `TextDecoder` to instead use UTF-8 with ISO-8859-1 fallback. It needs to support iterative decoding, so probably using `IncrementalDecoder`, and initially waiting until at least (say) 4096 bytes have been buffered before the initial decode pass?
2. We might(?) want to consider an API to allow customizing the text decoder, to allow for alternate users to switch to alternate behaviours such as `ChardetDecoder` or an `HTMLMetaTagDecoder`?
Another note on this issue: https://github.com/psf/requests/issues/4848 mentions that chardet is problematic when bundling it with an application in productive deployments. I also like the suggestion to make it pluggable with https://github.com/Ousret/charset_normalizer (MIT license) or https://github.com/PyYoshi/cChardet (which is MPL, still easier to comply than LGPL). It'd be great if you could consider making the dependency optional :)
Buffering causes issues for applications that expect partial messages to come through without delay (long polling, logging and other such hacks). Thus, maybe it would be best to just *always* use ISO-8859-1 when nothing else is explicitly specified?

Section 3.7.1 of https://tools.ietf.org/html/rfc2616 says:

> When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value.

I wonder where the autodetection is actually used because I just tried serving a UTF-8 document as `text/html` with no charset or meta tags, and Chrome, Firefox, httpx and requests all decoded it as 8859-1 or similar, not as UTF-8, although chardet detected the content as UTF-8.
The autodetection is only used for media types that are not `text/*`, and that don't have a `charset` set... https://github.com/encode/httpx/blob/master/httpx/_models.py#L777

And yes, I'm more inclined to diverge from `requests` behaviour here, since it's *anyway* applied to only a pretty narrow subset of cases, *and* the landscape wrt. utf-8 has moved on since their initial design decision here.

I found this issue as I was reporting an issue with `response.text` taking 10 seconds, which I tracked down to `chardet.detect`. For what it's worth, in my case, I'm parsing a sitemap, which is XML, so the algorithm for guessing the encoding should be something like: https://www.oreilly.com/library/view/python-cookbook/0596001673/ch12s11.html.

I've implemented this outside of `httpx` via `response.content`. Do you see supporting the XML encoding guessing or a custom guessing function fitting in here?
Not really, no, although if we could provide enough good API for allowing it as a third party option then that'd be a good route.

There is a setter for `encoding` exposed on `Response`, which I think is sufficient now that I've thought about it a little more. 

Would you be open to being able to set the `encoding` in `__init__`? I'd use it when reconstructing `Response` from an on-disk cache.
@alexchamberlain Possibly, but it'd need some careful thinking about around how to make `Response` API nicely coherent in that area. (Eg. see https://github.com/encode/httpx/issues/1091)

I probably wouldn't want us to accept any API changes on it *at the moment*.

Resolving patch:
diff --git a/httpx/_decoders.py b/httpx/_decoders.py
--- a/httpx/_decoders.py
+++ b/httpx/_decoders.py
@@ -7,8 +7,6 @@
 import typing
 import zlib
 
-import chardet
-
 try:
     import brotli
 except ImportError:  # pragma: nocover
@@ -163,62 +161,52 @@ class TextDecoder:
     """
 
     def __init__(self, encoding: typing.Optional[str] = None):
-        self.decoder: typing.Optional[codecs.IncrementalDecoder] = (
-            None if encoding is None else codecs.getincrementaldecoder(encoding)()
-        )
-        self.detector = chardet.universaldetector.UniversalDetector()
-
-        # This buffer is only needed if 'decoder' is 'None'
-        # we want to trigger errors if data is getting added to
-        # our internal buffer for some silly reason while
-        # a decoder is discovered.
-        self.buffer: typing.Optional[bytearray] = None if self.decoder else bytearray()
+        self.decoder: typing.Optional[codecs.IncrementalDecoder] = None
+        if encoding is not None:
+            self.decoder = codecs.getincrementaldecoder(encoding)(errors="strict")
 
     def decode(self, data: bytes) -> str:
-        try:
-            if self.decoder is not None:
-                text = self.decoder.decode(data)
-            else:
-                assert self.buffer is not None
-                text = ""
-                self.detector.feed(data)
-                self.buffer += data
-
-                # Should be more than enough data to process, we don't
-                # want to buffer too long as chardet will wait until
-                # detector.close() is used to give back common
-                # encodings like 'utf-8'.
-                if len(self.buffer) >= 4096:
-                    self.decoder = codecs.getincrementaldecoder(
-                        self._detector_result()
-                    )()
-                    text = self.decoder.decode(bytes(self.buffer), False)
-                    self.buffer = None
-
-            return text
-        except UnicodeDecodeError as exc:  # pragma: nocover
-            raise ValueError(str(exc))
+        """
+        If an encoding is explicitly specified, then we use that.
+        Otherwise our strategy is to attempt UTF-8, and fallback to Windows 1252.
 
-    def flush(self) -> str:
-        try:
-            if self.decoder is None:
-                # Empty string case as chardet is guaranteed to not have a guess.
-                assert self.buffer is not None
-                if len(self.buffer) == 0:
-                    return ""
-                return bytes(self.buffer).decode(self._detector_result())
-
-            return self.decoder.decode(b"", True)
-        except UnicodeDecodeError as exc:  # pragma: nocover
-            raise ValueError(str(exc))
+        Note that UTF-8 is a strict superset of ascii, and Windows 1252 is a
+        superset of the non-control characters in iso-8859-1, so we essentially
+        end up supporting any of ascii, utf-8, iso-8859-1, cp1252.
 
-    def _detector_result(self) -> str:
-        self.detector.close()
-        result = self.detector.result["encoding"]
-        if not result:  # pragma: nocover
-            raise ValueError("Unable to determine encoding of content")
+        Given that UTF-8 is now by *far* the most widely used encoding, this
+        should be a pretty robust strategy for cases where a charset has
+        not been explicitly included.
 
-        return result
+        Useful stats on the prevalence of different charsets in the wild...
+
+        * https://w3techs.com/technologies/overview/character_encoding
+        * https://w3techs.com/technologies/history_overview/character_encoding
+
+        The HTML5 spec also has some useful guidelines, suggesting defaults of
+        either UTF-8 or Windows 1252 in most cases...
+
+        * https://dev.w3.org/html5/spec-LC/Overview.html
+        """
+        if self.decoder is None:
+            # If this is the first decode pass then we need to determine which
+            # encoding to use by attempting UTF-8 and raising any decode errors.
+            attempt_utf_8 = codecs.getincrementaldecoder("utf-8")(errors="strict")
+            try:
+                attempt_utf_8.decode(data)
+            except UnicodeDecodeError:
+                # Could not decode as UTF-8. Use Windows 1252.
+                self.decoder = codecs.getincrementaldecoder("cp1252")(errors="replace")
+            else:
+                # Can decode as UTF-8. Use UTF-8 with lenient error settings.
+                self.decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
+
+        return self.decoder.decode(data)
+
+    def flush(self) -> str:
+        if self.decoder is None:
+            return ""
+        return self.decoder.decode(b"", True)
 
 
 class LineDecoder:
diff --git a/httpx/_models.py b/httpx/_models.py
--- a/httpx/_models.py
+++ b/httpx/_models.py
@@ -10,7 +10,6 @@
 from http.cookiejar import Cookie, CookieJar
 from urllib.parse import parse_qsl, quote, unquote, urlencode
 
-import chardet
 import rfc3986
 import rfc3986.exceptions
 
@@ -755,19 +754,22 @@ def text(self) -> str:
             if not content:
                 self._text = ""
             else:
-                encoding = self.encoding
-                self._text = content.decode(encoding, errors="replace")
+                decoder = TextDecoder(encoding=self.encoding)
+                self._text = "".join([decoder.decode(self.content), decoder.flush()])
         return self._text
 
     @property
-    def encoding(self) -> str:
+    def encoding(self) -> typing.Optional[str]:
+        """
+        Return the encoding, which may have been set explicitly, or may have
+        been specified by the Content-Type header.
+        """
         if not hasattr(self, "_encoding"):
             encoding = self.charset_encoding
             if encoding is None or not is_known_encoding(encoding):
-                encoding = self.apparent_encoding
-                if encoding is None or not is_known_encoding(encoding):
-                    encoding = "utf-8"
-            self._encoding = encoding
+                self._encoding = None
+            else:
+                self._encoding = encoding
         return self._encoding
 
     @encoding.setter
@@ -783,25 +785,11 @@ def charset_encoding(self) -> typing.Optional[str]:
         if content_type is None:
             return None
 
-        parsed = cgi.parse_header(content_type)
-        media_type, params = parsed[0], parsed[-1]
-        if "charset" in params:
-            return params["charset"].strip("'\"")
-
-        # RFC 2616 specifies that 'iso-8859-1' should be used as the default
-        # for 'text/*' media types, if no charset is provided.
-        # See: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1
-        if media_type.startswith("text/"):
-            return "iso-8859-1"
-
-        return None
+        _, params = cgi.parse_header(content_type)
+        if "charset" not in params:
+            return None
 
-    @property
-    def apparent_encoding(self) -> typing.Optional[str]:
-        """
-        Return the encoding, as it appears to autodetection.
-        """
-        return chardet.detect(self.content)["encoding"]
+        return params["charset"].strip("'\"")
 
     def _get_content_decoder(self) -> ContentDecoder:
         """
@@ -936,7 +924,7 @@ def iter_text(self) -> typing.Iterator[str]:
         that handles both gzip, deflate, etc but also detects the content's
         string encoding.
         """
-        decoder = TextDecoder(encoding=self.charset_encoding)
+        decoder = TextDecoder(encoding=self.encoding)
         with self._wrap_decoder_errors():
             for chunk in self.iter_bytes():
                 yield decoder.decode(chunk)
@@ -1020,7 +1008,7 @@ async def aiter_text(self) -> typing.AsyncIterator[str]:
         that handles both gzip, deflate, etc but also detects the content's
         string encoding.
         """
-        decoder = TextDecoder(encoding=self.charset_encoding)
+        decoder = TextDecoder(encoding=self.encoding)
         with self._wrap_decoder_errors():
             async for chunk in self.aiter_bytes():
                 yield decoder.decode(chunk)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -57,7 +57,6 @@ def get_packages(package):
     install_requires=[
         "certifi",
         "sniffio",
-        "chardet==3.*",
         "rfc3986[idna2008]>=1.3,<2",
         "httpcore==0.10.*",
     ],


Test patch:
diff --git a/tests/client/test_client.py b/tests/client/test_client.py
--- a/tests/client/test_client.py
+++ b/tests/client/test_client.py
@@ -15,7 +15,7 @@ def test_get(server):
     assert response.content == b"Hello, world!"
     assert response.text == "Hello, world!"
     assert response.http_version == "HTTP/1.1"
-    assert response.encoding == "iso-8859-1"
+    assert response.encoding is None
     assert response.request.url == url
     assert response.headers
     assert response.is_redirect is False
diff --git a/tests/models/test_responses.py b/tests/models/test_responses.py
--- a/tests/models/test_responses.py
+++ b/tests/models/test_responses.py
@@ -81,15 +81,15 @@ def test_response_content_type_encoding():
 
 def test_response_autodetect_encoding():
     """
-    Autodetect encoding if there is no charset info in a Content-Type header.
+    Autodetect encoding if there is no Content-Type header.
     """
-    content = "おはようございます。".encode("EUC-JP")
+    content = "おはようございます。".encode("utf-8")
     response = httpx.Response(
         200,
         content=content,
     )
     assert response.text == "おはようございます。"
-    assert response.encoding == "EUC-JP"
+    assert response.encoding is None
 
 
 def test_response_fallback_to_autodetect():
@@ -97,20 +97,20 @@ def test_response_fallback_to_autodetect():
     Fallback to autodetection if we get an invalid charset in the Content-Type header.
     """
     headers = {"Content-Type": "text-plain; charset=invalid-codec-name"}
-    content = "おはようございます。".encode("EUC-JP")
+    content = "おはようございます。".encode("utf-8")
     response = httpx.Response(
         200,
         content=content,
         headers=headers,
     )
     assert response.text == "おはようございます。"
-    assert response.encoding == "EUC-JP"
+    assert response.encoding is None
 
 
-def test_response_default_text_encoding():
+def test_response_no_charset_with_ascii_content():
     """
-    A media type of 'text/*' with no charset should default to ISO-8859-1.
-    See: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1
+    A response with ascii encoded content should decode correctly,
+    even with no charset specified.
     """
     content = b"Hello, world!"
     headers = {"Content-Type": "text/plain"}
@@ -120,20 +120,56 @@ def test_response_default_text_encoding():
         headers=headers,
     )
     assert response.status_code == 200
-    assert response.encoding == "iso-8859-1"
+    assert response.encoding is None
     assert response.text == "Hello, world!"
 
 
-def test_response_default_encoding():
+def test_response_no_charset_with_utf8_content():
     """
-    Default to utf-8 if all else fails.
+    A response with UTF-8 encoded content should decode correctly,
+    even with no charset specified.
     """
+    content = "Unicode Snowman: ☃".encode("utf-8")
+    headers = {"Content-Type": "text/plain"}
     response = httpx.Response(
         200,
-        content=b"",
+        content=content,
+        headers=headers,
     )
-    assert response.text == ""
-    assert response.encoding == "utf-8"
+    assert response.text == "Unicode Snowman: ☃"
+    assert response.encoding is None
+
+
+def test_response_no_charset_with_iso_8859_1_content():
+    """
+    A response with ISO 8859-1 encoded content should decode correctly,
+    even with no charset specified.
+    """
+    content = "Accented: Österreich".encode("iso-8859-1")
+    headers = {"Content-Type": "text/plain"}
+    response = httpx.Response(
+        200,
+        content=content,
+        headers=headers,
+    )
+    assert response.text == "Accented: Österreich"
+    assert response.encoding is None
+
+
+def test_response_no_charset_with_cp_1252_content():
+    """
+    A response with Windows 1252 encoded content should decode correctly,
+    even with no charset specified.
+    """
+    content = "Euro Currency: €".encode("cp1252")
+    headers = {"Content-Type": "text/plain"}
+    response = httpx.Response(
+        200,
+        content=content,
+        headers=headers,
+    )
+    assert response.text == "Euro Currency: €"
+    assert response.encoding is None
 
 
 def test_response_non_text_encoding():
@@ -147,7 +183,7 @@ def test_response_non_text_encoding():
         headers=headers,
     )
     assert response.text == "xyz"
-    assert response.encoding == "ascii"
+    assert response.encoding is None
 
 
 def test_response_set_explicit_encoding():
@@ -184,7 +220,7 @@ def test_read():
 
     assert response.status_code == 200
     assert response.text == "Hello, world!"
-    assert response.encoding == "ascii"
+    assert response.encoding is None
     assert response.is_closed
 
     content = response.read()
@@ -203,7 +239,7 @@ async def test_aread():
 
     assert response.status_code == 200
     assert response.text == "Hello, world!"
-    assert response.encoding == "ascii"
+    assert response.encoding is None
     assert response.is_closed
 
     content = await response.aread()
diff --git a/tests/test_decoders.py b/tests/test_decoders.py
--- a/tests/test_decoders.py
+++ b/tests/test_decoders.py
@@ -177,16 +177,8 @@ def test_decoding_errors(header_value):
     [
         ((b"Hello,", b" world!"), "ascii"),
         ((b"\xe3\x83", b"\x88\xe3\x83\xa9", b"\xe3", b"\x83\x99\xe3\x83\xab"), "utf-8"),
-        ((b"\x83g\x83\x89\x83x\x83\x8b",) * 64, "shift-jis"),
-        ((b"\x83g\x83\x89\x83x\x83\x8b",) * 600, "shift-jis"),
-        (
-            (b"\xcb\xee\xf0\xe5\xec \xe8\xef\xf1\xf3\xec \xe4\xee\xeb\xee\xf0",) * 64,
-            "MacCyrillic",
-        ),
-        (
-            (b"\xa5\xa6\xa5\xa7\xa5\xd6\xa4\xce\xb9\xf1\xba\xdd\xb2\xbd",) * 512,
-            "euc-jp",
-        ),
+        ((b"Euro character: \x88!", b""), "cp1252"),
+        ((b"Accented: \xd6sterreich", b""), "iso-8859-1"),
     ],
 )
 @pytest.mark.asyncio