Repository: matrix-org/synapse

Issue:
Replication request exceeds max size when trying to join room
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

<!-- Describe here the problem that you are experiencing -->

I'm getting the following error when trying to join #irc:matrix.org:

```
2021-05-07 20:16:08,373 - synapse.http.site - 107 - WARNING - sentinel - Aborting connection from IPv4Address(type='TCP', host='127.0.0.1', port=44100) because the request exceeds maximum size
2021-05-07 20:16:08,375 - synapse.http.site - 312 - INFO - sentinel - Connection from IPv4Address(type='TCP', host='127.0.0.1', port=44100) lost before request headers were read
2021-05-07 20:16:08,376 - synapse.http.client - 440 - INFO - POST-3 - Received response to POST http://127.0.0.1:9893/_synapse/replication/remote_join/%21BAXLHOFjvDKUeLafmO%3Amatrix.org/%40deepbluev7%3Aneko.dev/qjkxpFkuVk: 502
2021-05-07 20:16:08,376 - synapse.http.server - 88 - INFO - POST-3 - <SynapseRequest at 0x7f0f95586d00 method='POST' uri='/_matrix/client/r0/join/%23irc%3Amatrix.org' clientproto='HTTP/1.0' site=8086> SynapseError: 502 - Failed to talk to 
main process
2021-05-07 20:16:08,377 - synapse.access.http.8086 - 387 - INFO - POST-3 - 127.0.0.1 - 8086 - {@deepbluev7:neko.dev} Processed request: 27.573sec/-0.000sec (0.005sec, 0.001sec) (0.002sec/0.008sec/10) 64B 502 "POST /_matrix/client/r0/join/%
23irc%3Amatrix.org HTTP/1.0" "mtxclient v0.5.1" [0 dbevts]
```

I have a few workers with an event_creator and persister as well as a master and a few federation workers. This is from the event_creator log. Increasing the maximum size in https://github.com/matrix-org/synapse/blob/3ff225175462dde8376aa584e3a47c43b1f0e790/synapse/http/site.py#L106 made the join work.

### Steps to reproduce

- Setup synapse with event_creator
- Try to join #irc:matrix.org

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**:

If not matrix.org: neko.dev

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.33.1

- **Install method**: custom ebuild
<!-- examples: package manager/git clone/pip  -->

- **Platform**: Gentoo
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
Hmm we probably shouldn't be limiting the size of replication traffic?
Or at least the limit should probably be fairly big. I have no idea, what it currently is, but a few hundred megabytes is probably reasonable?
For the record, the current limit is 1024, which sounds quite low to me.
This needs discussion on whether to drop the limit or increase it to a more sensible value.
Having the same issue now in #steamlug:matrix.org and got the report that this causes each time my bot tries to join to do a "tried to join but made no change" membership event. So it seems like it joins but synapse doesnt realize it joined
Yes, if that request fails a proper join even is sent, you synapse just fails to process the response.
> 
> 
> Yes, if that request fails a proper join even is sent, you synapse just fails to process the response.

Yeah in this case it caused hundreds of join events and most likely even explains the extrem high load I had on join replication on master 
> For the record, the current limit is 1024, which sounds quite low to me.

That's the default in the `__init__`, but Synapse seems to alway creates it with at least 200 * 65536, I believe. See #9817.


For what is worth my server (running 1.34) seemingly was affected by this issue as well, and was a show stopper in joining a very large number of federated rooms. This seems to be badly affecting Synapse workers setups with sharded event persisters.
Until something better will come from upstream I temporarily came up with this patch:
https://github.com/maranda/synapse/commit/2567207edf383e8aaccf8c65342e5722f08834d7
Which allows to set the max request size limit per worker, I did set `default_max_request_size: 65535000` in the event persister workers configuration files, which raises 5 folds the default size limit (around 64 MiB). It's not a very elegant solution but works.

I think https://github.com/matrix-org/synapse/pull/10082 might fix this?
I'm hoping it will be fixed by this, I suppose it is still possible to hit this error but should be unlikely.
It looks like it would fix it for most cases, but it will fail with 101 events, that all have the maximum event size, I suppose.
We probably _should_ reduce the batch size to account for that just to be certain. @erikjohnston to review
Yeah, looks like we cap request size to 200 events, and we're currently batching into 1000 events.

Resolving patch:
diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py
--- a/synapse/handlers/federation.py
+++ b/synapse/handlers/federation.py
@@ -3056,8 +3056,9 @@ async def persist_events_and_notify(
         """
         instance = self.config.worker.events_shard_config.get_instance(room_id)
         if instance != self._instance_name:
-            # Limit the number of events sent over federation.
-            for batch in batch_iter(event_and_contexts, 1000):
+            # Limit the number of events sent over replication. We choose 200
+            # here as that is what we default to in `max_request_body_size(..)`
+            for batch in batch_iter(event_and_contexts, 200):
                 result = await self._send_events(
                     instance_name=instance,
                     store=self.store,


Test patch: