Repository: matrix-org/synapse

Issue:
json-serializing a sync response can block the reactor for tens of seconds
A response to an initial /sync can easily run to hundreds of megabytes of JSON, which takes tens of seconds of CPU time to serialise from the objects. This monopolises the Twisted reactor thread, causing all sorts of other problems (including dropped replication connections for worker-mode deployments).

To make matters worse, although we have a cache which is supposed to avoid redoing a lot of the hard work for duplicate requests to initialsync (though its effectiveness is disputed, see #3880), we cache the de-serialised version, so have to re-serialise the response for each request. (One reason we have to do this is to populate the `age` field).



Discussion:
A simple solution might be to farm out the json-serialisation to a threadpool?
Actually, I have a much better idea. `JSONEncoder` implements an [`iterencode` method](https://docs.python.org/3/library/json.html#json.JSONEncoder.iterencode). Rather than try to generate all the JSON at once, we could use that to generate JSON incrementally.

One approach to this would be to implement, say, a `JsonProducer` as an implementation of Twisted's `IProducer`, which we then attach to the request instead of the `NoRangeStaticProducer` we use at the moment. Every time `resumeProducing` gets called, we just send the next chunk from the `iterencode` result instead.

A slight complication would be that we wouldn't know the content-length ahead of time, so we'd have to use chunked transfer-encoding for the response; but that's easy enough to do.
@auscompgeek this feels like the sort of thing that might interest you. Any interest in picking it up?
I would agree that using `iterencode` would be the best solution here. I can't say I know enough Twisted to work on this though.

A small concern:

> Every time `resumeProducing` gets called, we just send the next chunk from the `iterencode` result instead.

Between each value, `iterencode` would yield single-char strings, for example:
```pycon
>>> list(enc.iterencode({'a': 'b', 'c': 'd'}))                                                                  
['{', '"a"', ':', '"b"', ',', '"c"', ':', '"d"', '}']
```

Does Twisted's HTTP chunking coalesce small chunks together?
> I would agree that using `iterencode` would be the best solution here. I can't say I know enough Twisted to work on this though.

I don't think you should let that be the thing that puts you off picking this up - the amount of Twisted knowledge needed should be minimal [1]. I understand you will have other demands on your time though!
 
> Does Twisted's HTTP chunking coalesce small chunks together?

I don't think Twisted has any support for generating chunked transfer-encoding (or if it does, I haven't found it), so we'd be implementing that part ourselves.  You're right though, we'll need to coalesce small chunks, to avoid generating thousands of tiny HTTP chunks each of which get passed through openssl and send out as tiny TCP segments.

[1]: currently, to send responses, we wrap the json bytes in a `BytesIO` and then use [`NoRangeStaticProducer`](https://github.com/twisted/twisted/blob/trunk/src/twisted/web/static.py#L744), which reads from a file-like object and sends the contents out. As you can see from the source it's pretty simple and it should be easy to implement a replacement which reads from a JSON iterencoder.
> I don't think Twisted has any support for generating chunked transfer-encoding (or if it does, I haven't found it), so we'd be implementing that part ourselves.

Twisted uses chunked encoding automatically if you [don't set a content-length header](https://github.com/twisted/twisted/blob/d47dafe0717b3877d5d0017ab0feb0721cb78613/src/twisted/web/http.py#L1129-L1136).

I really thought the Twisted docs had a minimal example of hooking incremental JSON producing up to an `IProducer`, but I think I'm thinking of an old project. It should be pretty straightforward to hook up the APIs.

Resolving patch:
diff --git a/synapse/http/server.py b/synapse/http/server.py
--- a/synapse/http/server.py
+++ b/synapse/http/server.py
@@ -22,12 +22,13 @@
 import urllib
 from http import HTTPStatus
 from io import BytesIO
-from typing import Any, Callable, Dict, Tuple, Union
+from typing import Any, Callable, Dict, Iterator, List, Tuple, Union
 
 import jinja2
-from canonicaljson import encode_canonical_json, encode_pretty_printed_json
+from canonicaljson import iterencode_canonical_json, iterencode_pretty_printed_json
+from zope.interface import implementer
 
-from twisted.internet import defer
+from twisted.internet import defer, interfaces
 from twisted.python import failure
 from twisted.web import resource
 from twisted.web.server import NOT_DONE_YET, Request
@@ -499,6 +500,78 @@ class RootOptionsRedirectResource(OptionsResource, RootRedirect):
     pass
 
 
+@implementer(interfaces.IPullProducer)
+class _ByteProducer:
+    """
+    Iteratively write bytes to the request.
+    """
+
+    # The minimum number of bytes for each chunk. Note that the last chunk will
+    # usually be smaller than this.
+    min_chunk_size = 1024
+
+    def __init__(
+        self, request: Request, iterator: Iterator[bytes],
+    ):
+        self._request = request
+        self._iterator = iterator
+
+    def start(self) -> None:
+        self._request.registerProducer(self, False)
+
+    def _send_data(self, data: List[bytes]) -> None:
+        """
+        Send a list of strings as a response to the request.
+        """
+        if not data:
+            return
+        self._request.write(b"".join(data))
+
+    def resumeProducing(self) -> None:
+        # We've stopped producing in the meantime (note that this might be
+        # re-entrant after calling write).
+        if not self._request:
+            return
+
+        # Get the next chunk and write it to the request.
+        #
+        # The output of the JSON encoder is coalesced until min_chunk_size is
+        # reached. (This is because JSON encoders produce a very small output
+        # per iteration.)
+        #
+        # Note that buffer stores a list of bytes (instead of appending to
+        # bytes) to hopefully avoid many allocations.
+        buffer = []
+        buffered_bytes = 0
+        while buffered_bytes < self.min_chunk_size:
+            try:
+                data = next(self._iterator)
+                buffer.append(data)
+                buffered_bytes += len(data)
+            except StopIteration:
+                # The entire JSON object has been serialized, write any
+                # remaining data, finalize the producer and the request, and
+                # clean-up any references.
+                self._send_data(buffer)
+                self._request.unregisterProducer()
+                self._request.finish()
+                self.stopProducing()
+                return
+
+        self._send_data(buffer)
+
+    def stopProducing(self) -> None:
+        self._request = None
+
+
+def _encode_json_bytes(json_object: Any) -> Iterator[bytes]:
+    """
+    Encode an object into JSON. Returns an iterator of bytes.
+    """
+    for chunk in json_encoder.iterencode(json_object):
+        yield chunk.encode("utf-8")
+
+
 def respond_with_json(
     request: Request,
     code: int,
@@ -533,15 +606,23 @@ def respond_with_json(
         return None
 
     if pretty_print:
-        json_bytes = encode_pretty_printed_json(json_object) + b"\n"
+        encoder = iterencode_pretty_printed_json
     else:
         if canonical_json or synapse.events.USE_FROZEN_DICTS:
-            # canonicaljson already encodes to bytes
-            json_bytes = encode_canonical_json(json_object)
+            encoder = iterencode_canonical_json
         else:
-            json_bytes = json_encoder.encode(json_object).encode("utf-8")
+            encoder = _encode_json_bytes
+
+    request.setResponseCode(code)
+    request.setHeader(b"Content-Type", b"application/json")
+    request.setHeader(b"Cache-Control", b"no-cache, no-store, must-revalidate")
 
-    return respond_with_json_bytes(request, code, json_bytes, send_cors=send_cors)
+    if send_cors:
+        set_cors_headers(request)
+
+    producer = _ByteProducer(request, encoder(json_object))
+    producer.start()
+    return NOT_DONE_YET
 
 
 def respond_with_json_bytes(
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -43,7 +43,7 @@
     "jsonschema>=2.5.1",
     "frozendict>=1",
     "unpaddedbase64>=1.1.0",
-    "canonicaljson>=1.2.0",
+    "canonicaljson>=1.3.0",
     # we use the type definitions added in signedjson 1.1.
     "signedjson>=1.1.0",
     "pynacl>=1.2.1",
diff --git a/synapse/rest/key/v2/remote_key_resource.py b/synapse/rest/key/v2/remote_key_resource.py
--- a/synapse/rest/key/v2/remote_key_resource.py
+++ b/synapse/rest/key/v2/remote_key_resource.py
@@ -15,12 +15,12 @@
 import logging
 from typing import Dict, Set
 
-from canonicaljson import encode_canonical_json, json
+from canonicaljson import json
 from signedjson.sign import sign_json
 
 from synapse.api.errors import Codes, SynapseError
 from synapse.crypto.keyring import ServerKeyFetcher
-from synapse.http.server import DirectServeJsonResource, respond_with_json_bytes
+from synapse.http.server import DirectServeJsonResource, respond_with_json
 from synapse.http.servlet import parse_integer, parse_json_object_from_request
 
 logger = logging.getLogger(__name__)
@@ -223,4 +223,4 @@ async def query_keys(self, request, query, query_remote_on_cache_miss=False):
 
             results = {"server_keys": signed_keys}
 
-            respond_with_json_bytes(request, 200, encode_canonical_json(results))
+            respond_with_json(request, 200, results, canonical_json=True)


Test patch:
diff --git a/tests/test_server.py b/tests/test_server.py
--- a/tests/test_server.py
+++ b/tests/test_server.py
@@ -178,7 +178,6 @@ def _callback(request, **kwargs):
 
         self.assertEqual(channel.result["code"], b"200")
         self.assertNotIn("body", channel.result)
-        self.assertEqual(channel.headers.getRawHeaders(b"Content-Length"), [b"15"])
 
 
 class OptionsResourceTests(unittest.TestCase):