Repository: john-parton/chardetng-py (language: python)

Issue:
Encodings reported by chardetng-py don't always match up to python's decoding
```python-traceback
LookupError: unknown encoding: windows-874
```

In a previous version, I just passed the entire buffer to encoding_rs and had it handle decoding entirely in rust, but that was a little confusing.

Need more robust aliases


Discussion / hints:
Actually it might be even more complicated than that. https://bugs.python.org/issue25416

I think it's possible that encoding_rs and the python codec will have different output  for legacy single byte encodings
For what it’s worth, I went through and:

1. Compared the names/aliases. `windows-874` is the only one chardetng can produce that does not map correctly in Python, so that’s already fully addressed.

2. Wrote a quick script to compare the single byte WHATWG encodings definitions (found at https://encoding.spec.whatwg.org/#legacy-single-byte-encodings or more conveniently in git at https://github.com/whatwg/encoding) with the Unicode consortium definitions (found at https://unicode.org/Public/MAPPINGS/). They are all basically identical except for a few exceptions in `KOI8-U` and `windows-1255`:

    ```
    MATCH: IBM866
    MATCH: ISO-8859-2
    MATCH: ISO-8859-4
    MATCH: ISO-8859-5
    MATCH: ISO-8859-6
    MATCH: ISO-8859-7
    MATCH: ISO-8859-8
    MATCH: ISO-8859-13
    Definitions for KOI8-U do not match!
      Mismatch at byte 174 (0xae)! WHATWG = point 1118 (0x045e) | Unicode = point 9565 (0x255d)
      Mismatch at byte 190 (0xbe)! WHATWG = point 1038 (0x040e) | Unicode = point 9580 (0x256c)
    MATCH: windows-874
    MATCH: windows-1250
    MATCH: windows-1251
    MATCH: windows-1252
    MATCH: windows-1253
    MATCH: windows-1254
    Definitions for windows-1255 do not match!
      Mismatch at byte 202 (0xca)! WHATWG = point 1466 (0x05ba) | Unicode = point <UNDEFINED>
    MATCH: windows-1256
    MATCH: windows-1257
    MATCH: windows-1258
    ```

    (The format here is `byte <decimal> (<hex>)` is the actual byte being decoded and `point <decimal> (<hex>)` is the unicode code point it should decode as.)

    Caveat: I allowed points that are control characters to be treated as undefined for this comparison, which I think is probably reasonable. There are a bunch more encodings that don’t match up otherwise (i.e. a byte is defined as some control character in WHATWG’s definition and is entirely undefined/unmapped in the Unicode Consortium’s definition, or vice versa).

These mismatches are definitely not ideal, but I think are also small enough not to be a problem (at least for the use case here: detecting the encoding of some bytes; not actually decoding/encoding them). None of these bytes show up in chardetng’s frequency data in [`src/data.rs`](https://github.com/hsivonen/chardetng/blob/master/src/data.rs) (assuming I’m reading it correctly), which I think means they aren’t really being considered anyway, so their mapping *shouldn’t* have a notable impact.

On the other hand, the `windows-1255` case does feature a byte that could cause errors when [strictly] decoding in Python, and based on chardetng’s docs, I think it wouldn’t otherwise give you an answer that does that, since it says it discards options with decoding failures/unmapped bytes. That said, I think it’s good advice to avoid ever decoding a *sniffed* encoding strictly in the first place; IIRC chardet and cChardet both occasionally give answers that are wrong enough that they don’t successfully decode the bytes and code I had using them seems to always use `errors="ignore"` or `errors="replace"` when decoding with their guesses. (This might all be worth mentioning in some addendum to the the docs, though… idunno.)
Ah, some historical details on the ones that differ between WHATWG and Unicode, if you’re curious:

- `KOI8-U` as defined by WHATWG is *actually* `KOI8-RU` (the -ru encoding was never fully standardized, and is not available in Python). Apparently Internet Explorer historically interpreted `KOI8-U` this way, and so all other browsers had to follow suit. Discussion: https://github.com/whatwg/encoding/issues/74 and https://www.w3.org/Bugs/Public/show_bug.cgi?id=17053

- In `windows-1255`, the actual implementation in Windows has historically mapped 0xCA to 0x05BA, even though all the specs and code sheets listed it as unmapped. Since WHATWG’s goals are to keep old content working (don’t break the web!), they decided to implement it following how Windows did it in practice, rather than how it was spec'd. Discussion: https://github.com/whatwg/encoding/issues/73
Thanks for looking into that. So it looks like if you use python to decode, it might not be correct. I think probably it's best to just document this and move on.

Fix patch:
diff --git a/src/lib.rs b/src/lib.rs
index c61a2a5..7f44858 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -10,10 +10,10 @@ use pyo3::prelude::*;
 // https://docs.python.org/3/library/codecs.html#standard-encodings
 // https://encoding.spec.whatwg.org/#legacy-single-byte-encodings
 fn _fix_encoding_name(encoding: &str) -> &str {
-    if encoding == "windows-874" {
-        "cp874"
-    } else {
-        encoding
+    match encoding.to_lowercase().as_str() {
+        "windows-874" => "cp874",
+        "gbk" => "gb18030",
+        _ => encoding
     }
 }
 


Test patch:
diff --git a/tests/test_decode.py b/tests/test_decode.py
index 3e05bfb..7e4ee41 100644
--- a/tests/test_decode.py
+++ b/tests/test_decode.py
@@ -1,7 +1,22 @@
 """Test cases for the chardetng_py module."""
-from chardetng_py import decode
+import pytest
+from chardetng_py import detect
 
 
 def test_decode() -> None:
     """Very basic decode litmus test."""
-    assert decode(b"Jakby r\xeaka Boga") == "Jakby rêka Boga"
+    raw = b"Jakby r\xeaka Boga"
+    assert raw.decode(detect(raw)) == "Jakby rêka Boga"
+
+
+@pytest.mark.parametrize(("text", "encoding", "detected_encoding"), [
+    ("สวัสดี, โลก", "cp874", "cp874"),
+    ("员会 ⛆", "gb18030", "gb18030"),
+])
+def test_returns_python_supported_names(
+    text: str,
+    encoding: str,
+    detected_encoding: str,
+) -> None:
+    """Ensure detection returns encoding names that are supported in Python."""
+    assert detect(text.encode(encoding)) == detected_encoding