Repository: matrix-org/synapse

Issue:
initialsync with huge limit takes out an initialsynchrotron
FIrst it spins CPU for 30 minutes, wedging the reactor and gradually allocating 100GB. Then it starts a massive GC which takes at least 20 minutes.

on the rare occasions the request completes, it looks like:
```
2020-06-22 13:55:18,932 - synapse.access.http.8108 - 312 - INFO - GET-17 - 92.234.32.44 - 8108 - {@matthew:matrix.org} Processed request: 4136.887sec/-3956.877sec (953.154sec, 146.548sec) (821.336sec/13010.643sec/80567) 0B 200! "GET /_matrix/client/r0/sync?access_token=<redacted>&full_state=true&timeout=30000&filter=%7B%22room%22%3A%7B%22timeline%22%3A%7B%22limit%22%3A5000%7D%2C%22state%22%3A%7B%22lazy_load_members%22%3Atrue%7D%7D%7D HTTP/1.1" "nio" [3460952 dbevts]
```

Next steps might be to route the requests to a sacrificial synchrotron and try to debyg what it's doing?


Discussion:
That filter is:
```
{"room":{"timeline":{"limit":5000},"state":{"lazy_load_members":true}}}
```

A limit of 5000 is likely to cause problems

so next step might rather be to make sure we ignore insane `limit` settings.
Looks like the maximum value here is already configurable:

https://github.com/matrix-org/synapse/blob/e452973fd2c956e57e3f0a1059d8049b76ee0195/docs/sample_config.yaml#L104-L107
huh, thanks. It feels like there should be a default limit, but, more pressingly, that we should configure a sensible limit on matrix.org.
Next steps here are to:
* Give a better default
* Lower the matrix.org limit

Resolving patch:
diff --git a/synapse/config/server.py b/synapse/config/server.py
--- a/synapse/config/server.py
+++ b/synapse/config/server.py
@@ -207,7 +207,7 @@ def read_config(self, config, **kwargs):
         # errors when attempting to search for messages.
         self.enable_search = config.get("enable_search", True)
 
-        self.filter_timeline_limit = config.get("filter_timeline_limit", -1)
+        self.filter_timeline_limit = config.get("filter_timeline_limit", 100)
 
         # Whether we should block invites sent to users on this server
         # (other than those sent by local server admins)
@@ -693,7 +693,9 @@ def generate_config_section(
         #gc_thresholds: [700, 10, 10]
 
         # Set the limit on the returned events in the timeline in the get
-        # and sync operations. The default value is -1, means no upper limit.
+        # and sync operations. The default value is 100. -1 means no upper limit.
+        #
+        # Uncomment the following to increase the limit to 5000.
         #
         #filter_timeline_limit: 5000
 
diff --git a/synapse/rest/client/v2_alpha/_base.py b/synapse/rest/client/v2_alpha/_base.py
--- a/synapse/rest/client/v2_alpha/_base.py
+++ b/synapse/rest/client/v2_alpha/_base.py
@@ -22,6 +22,7 @@
 
 from synapse.api.errors import InteractiveAuthIncompleteError
 from synapse.api.urls import CLIENT_API_PREFIX
+from synapse.types import JsonDict
 
 logger = logging.getLogger(__name__)
 
@@ -51,7 +52,15 @@ def client_patterns(path_regex, releases=(0,), unstable=True, v1=False):
     return patterns
 
 
-def set_timeline_upper_limit(filter_json, filter_timeline_limit):
+def set_timeline_upper_limit(filter_json: JsonDict, filter_timeline_limit: int) -> None:
+    """
+    Enforces a maximum limit of a timeline query.
+
+    Params:
+        filter_json: The timeline query to modify.
+        filter_timeline_limit: The maximum limit to allow, passing -1 will
+            disable enforcing a maximum limit.
+    """
     if filter_timeline_limit < 0:
         return  # no upper limits
     timeline = filter_json.get("room", {}).get("timeline", {})


Test patch: