Repository: matrix-org/python-canonicaljson

Issue:
Pretty-printed JSON is ASCII-encoded whereas it is otherwise UTF-8-encoded, causing errors
With Synapse, I run into an issue doing an initial sync with curl, because it switches to pretty-printed output:

```
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]: During handling of the above exception, another exception occurred:
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]: Traceback (most recent call last):
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:   File "/home/synapse/venv/lib/python3.8/site-packages/synapse/http/server.py", line 233, in _async_render_wrapper
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:     self._send_response(request, code, response)
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:   File "/home/synapse/venv/lib/python3.8/site-packages/synapse/http/server.py", line 289, in _send_response
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:     respond_with_json(
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:   File "/home/synapse/venv/lib/python3.8/site-packages/synapse/http/server.py", line 536, in respond_with_json
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:     json_bytes = encode_pretty_printed_json(json_object) + b"\n"
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:   File "/home/synapse/venv/lib/python3.8/site-packages/canonicaljson.py", line 96, in encode_pretty_printed_json
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]:     return _pretty_encoder.encode(json_object).encode("ascii")
Aug 24 14:14:02 sallie.librepush.net synapse[3123612]: UnicodeEncodeError: 'ascii' codec can't encode characters in position 16800-16801: ordinal not in range(128)
```

On the other hand, if I use curl's `-A` flag to change user-agent, I get the non-pretty-printed output and it succeeds.

Discussion:

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index 0378bae..76aa272 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -3,6 +3,8 @@ Version next
 * Fix producing non-standard JSON for Infinity, -Infinity, and NaN. This could
   error when encoding objects into canonical JSON that previously used to work,
   but were incompatible with JSON implementations in other languages.
+* Use UTF-8 to fix ASCII encoding errors when data containing Unicode was
+  attempted to be pretty-printed.
 
 Version 1.3.0 released 2020-08-14
 
diff --git a/canonicaljson.py b/canonicaljson.py
index aab42e1..026b351 100644
--- a/canonicaljson.py
+++ b/canonicaljson.py
@@ -89,7 +89,7 @@ def iterencode_canonical_json(json_object):
 
 def encode_pretty_printed_json(json_object):
     """
-    Encodes the JSON object dict as human readable ascii bytes.
+    Encodes the JSON object dict as human readable UTF-8 bytes.
 
     Args:
         json_object (dict): The JSON object to encode.
@@ -97,11 +97,11 @@ def encode_pretty_printed_json(json_object):
     Returns:
         bytes encoding the JSON object"""
 
-    return _pretty_encoder.encode(json_object).encode("ascii")
+    return _pretty_encoder.encode(json_object).encode("utf-8")
 
 
 def iterencode_pretty_printed_json(json_object):
-    """Encodes the JSON object dict as human readable ascii bytes.
+    """Encodes the JSON object dict as human readable UTF-8 bytes.
 
     Args:
         json_object (dict): The JSON object to encode.
@@ -110,7 +110,7 @@ def iterencode_pretty_printed_json(json_object):
         generator which yields bytes encoding the JSON object"""
 
     for chunk in _pretty_encoder.iterencode(json_object):
-        yield chunk.encode("ascii")
+        yield chunk.encode("utf-8")
 
 
 if platform.python_implementation() == "PyPy":  # pragma: no cover


Test patch:
diff --git a/test_canonicaljson.py b/test_canonicaljson.py
index 359586d..003d108 100644
--- a/test_canonicaljson.py
+++ b/test_canonicaljson.py
@@ -99,6 +99,12 @@ class TestCanonicalJson(unittest.TestCase):
         self.assertEqual(encode_pretty_printed_json({}), b"{}")
         self.assertEqual(list(iterencode_pretty_printed_json({})), [b"{}"])
 
+        # non-ascii should come out utf8-encoded.
+        self.assertEqual(
+            encode_pretty_printed_json({u"la merde amusée": u"💩"}),
+            b'{\n    "la merde amus\xc3\xa9e": "\xF0\x9F\x92\xA9"\n}',
+        )
+
     def test_frozen_dict(self):
         self.assertEqual(
             encode_canonical_json(frozendict({"a": 1})), b'{"a":1}',