Repository: matrix-org/synapse

Issue:
in-memory federation transaction transmission queues build up indefinitely for offline servers
when a server is offline, any events that we would have sent to it stack up in memory indefinitely.

See, for example, this graph for matrix.org:

![image](https://user-images.githubusercontent.com/1389908/87312238-30647b80-c518-11ea-848f-5d10bd9bbc1d.png)

Apart from the memory usage concerns, this means that if a server goes offline for 1000 events, then when it comes back we will try to send it all those 1000 events in turn, rather than just sending it the most recent ones in a room.


Discussion:
I suspect this was introduced way back in https://github.com/matrix-org/synapse/pull/2064.

I think we just need to clear the in-memory queues when handling a blacklisted server in `_transaction_transmission_loop`.
I'm chucking this into the todo list as it appears an obvious precursor to #2528.
This may be the reason for #7176 <del>since it was reported shortly after #2064 was merged</del> (didn't look at the year). For reference here's my federation_sender gc times graph:

![grafik](https://user-images.githubusercontent.com/16312324/87453304-d46e2580-c5f1-11ea-8c32-42e7335909e4.png)

![grafik](https://user-images.githubusercontent.com/16312324/87454315-51e66580-c5f3-11ea-820f-d3c43ebab536.png)

This graph looks like the queue build-up was happening without causing any GC leak:

![image](https://user-images.githubusercontent.com/4224639/87454298-73078000-c60c-11ea-870e-7b221566983d.png)


Resolving patch:
diff --git a/synapse/federation/sender/per_destination_queue.py b/synapse/federation/sender/per_destination_queue.py
--- a/synapse/federation/sender/per_destination_queue.py
+++ b/synapse/federation/sender/per_destination_queue.py
@@ -337,6 +337,28 @@ async def _transaction_transmission_loop(self) -> None:
                     (e.retry_last_ts + e.retry_interval) / 1000.0
                 ),
             )
+
+            if e.retry_interval > 60 * 60 * 1000:
+                # we won't retry for another hour!
+                # (this suggests a significant outage)
+                # We drop pending PDUs and EDUs because otherwise they will
+                # rack up indefinitely.
+                # Note that:
+                # - the EDUs that are being dropped here are those that we can
+                #   afford to drop (specifically, only typing notifications,
+                #   read receipts and presence updates are being dropped here)
+                # - Other EDUs such as to_device messages are queued with a
+                #   different mechanism
+                # - this is all volatile state that would be lost if the
+                #   federation sender restarted anyway
+
+                # dropping read receipts is a bit sad but should be solved
+                # through another mechanism, because this is all volatile!
+                self._pending_pdus = []
+                self._pending_edus = []
+                self._pending_edus_keyed = {}
+                self._pending_presence = {}
+                self._pending_rrs = {}
         except FederationDeniedError as e:
             logger.info(e)
         except HttpResponseException as e:


Test patch: