Repository: matrix-org/synapse

Issue:
Doing a rolling restart caused the typing worker to spin at 100% CPU until it was restarted
When performing a rolling restart of matrix.org (Updating 5cc41f1b0 .. 8e8a00829), the `typing1` worker suddenly started using a lot of CPU.
Judging by eye (on Grafana; not with these zoomed-out shots below that are more to give an idea of the usual expected trend), it looks like it started occurring just after the main process was restarted (the `typing1` worker had not restarted by this point), but equally some synchrotrons and client readers were started at a similar time (and it's hard to see exactly since the granularity of the graph is not very fine).

It seems to coincide with a high rate of `GET ReplicationGetStreamUpdates`.

<img width="930" alt="image" src="https://user-images.githubusercontent.com/38398653/149517877-1e8c7731-a685-47ea-8558-d69aa4725d6f.png">

RDATA commands (particularly for `caches` and `events`) seem to be queueing up (and that sounds bad):

<img width="1858" alt="image" src="https://user-images.githubusercontent.com/38398653/149519105-37375eed-47bb-4fdb-b46b-207598cb6d51.png">

The oscillatory 'Replication connections' graph seems strange — I don't know what it means exactly as of right now but it's something that might be worth looking into.


I restarted the worker myself (at approximately 12:44 UTC) to get it to return to normalcy.

Here is an excerpt of the schedule for the worker restarts, in case that's useful:
```
2022-01-14 12:03:00,725 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-main.service
2022-01-14 12:03:40,797 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron1.service
2022-01-14 12:04:15,073 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron2.service
2022-01-14 12:04:49,849 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@client_reader1.service
2022-01-14 12:05:23,509 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron3.service
2022-01-14 12:05:57,697 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron4.service
2022-01-14 12:06:32,614 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron5.service
2022-01-14 12:07:07,650 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@client_reader2.service
2022-01-14 12:07:43,080 - ssh.hippogriff - 42 - INFO - $ systemctl --user restart synapse-worker@synchrotron6.service
```


Discussion:
I think this is related to https://github.com/matrix-org/synapse/issues/8524
I took the tiniest stab at this but was overwhelmed with the size of the log and the fact that I wasn't really sure what I was looking for -it was a little like looking for a needle in a haystack when you don't know what a needle looks like. But I did find some patterns after the rolling restart (at 12:03) that looked like this:
```
2022-01-14 12:06:53,067 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:53,385 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:53,792 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:54,652 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:55,100 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:55,728 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607635 and 409608239
2022-01-14 12:06:56,137 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409608091 and 409608239
2022-01-14 12:06:56,556 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 408653351 and 409608209
2022-01-14 12:06:57,088 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:57,467 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
2022-01-14 12:06:57,807 - synapse.replication.tcp.handler - 554 - INFO - process-replication-data-23562530 - Fetching replication rows for 'caches' between 409607869 and 409608239
```
Specifically, you can see that the tcp handler is attempting to fetch the same rows over and over for the cache (in this case, the rows between 409607869 and 409608239, but there are many other examples of rows it seemed to get stuck on). I have no idea *why* this happened but it does explain the graphs a little (cpu getting chewed up trying to fetch the same thing while the other queues pile up). This might be a total red herring but I thought I'd mention it because the logs on this will be rotated out very shortly.  Oh and ftr this view is the log after filtering out everything but "fetching replication rows" in case that matters. 
Nothing conclusive, just suspicious and making a note.

`typing` starting at 0 doesn't seem super related, but then this code below makes it look like 'stream catchup' behaviour gets triggered if the stream goes backwards.

```python
        missing_updates = cmd.prev_token != current_token
        while missing_updates:
            logger.info(
                "Fetching replication rows for '%s' between %i and %i",
                stream_name,
                current_token,
                cmd.new_token,
            )
```
I was a little wiser than usual and took a `py-spy --gil` of the process today whilst it was spinning away.

20% of time each was going to each of these lines: 426, 427 and 434 in `get_all_typing_updates`:

https://github.com/matrix-org/synapse/blob/e24ff8ebe3d4119d377355402245947f7de61c00/synapse/handlers/typing.py#L416-L444

Of course that's just a symptom — probably of the fact that the position resets back to zero, but at least mildly interesting to see what's costly here.
Have we seen this recently? If not I'd be inclined to close unless it resurfaces.
Looks like it:

![Screenshot from 2022-05-09 15-44-29](https://user-images.githubusercontent.com/8614563/167435377-de0781b4-33b5-44c8-b2b2-45e7d75707b2.png)


Resolving patch:
diff --git a/synapse/handlers/typing.py b/synapse/handlers/typing.py
--- a/synapse/handlers/typing.py
+++ b/synapse/handlers/typing.py
@@ -52,6 +52,11 @@ class RoomMember:
 FEDERATION_PING_INTERVAL = 40 * 1000
 
 
+# How long to remember a typing notification happened in a room before
+# forgetting about it.
+FORGET_TIMEOUT = 10 * 60 * 1000
+
+
 class FollowerTypingHandler:
     """A typing handler on a different process than the writer that is updated
     via replication.
@@ -83,7 +88,10 @@ def __init__(self, hs: "HomeServer"):
         self.wheel_timer: WheelTimer[RoomMember] = WheelTimer(bucket_size=5000)
         self._latest_room_serial = 0
 
+        self._rooms_updated: Set[str] = set()
+
         self.clock.looping_call(self._handle_timeouts, 5000)
+        self.clock.looping_call(self._prune_old_typing, FORGET_TIMEOUT)
 
     def _reset(self) -> None:
         """Reset the typing handler's data caches."""
@@ -92,6 +100,8 @@ def _reset(self) -> None:
         # map room IDs to sets of users currently typing
         self._room_typing = {}
 
+        self._rooms_updated = set()
+
         self._member_last_federation_poke = {}
         self.wheel_timer = WheelTimer(bucket_size=5000)
 
@@ -178,6 +188,7 @@ def process_replication_rows(
             prev_typing = self._room_typing.get(row.room_id, set())
             now_typing = set(row.user_ids)
             self._room_typing[row.room_id] = now_typing
+            self._rooms_updated.add(row.room_id)
 
             if self.federation:
                 run_as_background_process(
@@ -209,6 +220,19 @@ async def _send_changes_in_typing_to_remotes(
     def get_current_token(self) -> int:
         return self._latest_room_serial
 
+    def _prune_old_typing(self) -> None:
+        """Prune rooms that haven't seen typing updates since last time.
+
+        This is safe to do as clients should time out old typing notifications.
+        """
+        stale_rooms = self._room_serials.keys() - self._rooms_updated
+
+        for room_id in stale_rooms:
+            self._room_serials.pop(room_id, None)
+            self._room_typing.pop(room_id, None)
+
+        self._rooms_updated = set()
+
 
 class TypingWriterHandler(FollowerTypingHandler):
     def __init__(self, hs: "HomeServer"):
@@ -388,6 +412,7 @@ def _push_update_local(self, member: RoomMember, typing: bool) -> None:
         self._typing_stream_change_cache.entity_has_changed(
             member.room_id, self._latest_room_serial
         )
+        self._rooms_updated.add(member.room_id)
 
         self.notifier.on_new_event(
             StreamKeyType.TYPING, self._latest_room_serial, rooms=[member.room_id]


Test patch: