Repository: matrix-org/synapse

Issue:
Presence worker gets stuck on 1.33.0rc1
### Description
A line like this is spammed in the logs and the worker handling presence pins the CPU and gets stuck:

```
2021-04-28 12:05:20,199 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-103- Streaming: presence_federation -> 5
```

If presence is on the master process, then the master process gets stuck.

The issue goes away after disabling presence.

### Steps to reproduce

- Have presence and workers enabled
- Update to 1.33.0rc1
- Optionally, move the presence stream to a worker

### Version information
- **Version**: 1.33.0rc1
- **Install method**: Docker


Discussion:
Happens to me too shortly after updating. I installed from source. After I restarted it twice I can't reproduce it anymore though?
FTR disabling presence is a temporary fix, if you don't want to downgrade.
Logs from the start of the spam:

```
2021-04-28 11:40:39,306 - synapse.access.http.9893 - 387 - INFO - POST-111 - 127.0.0.1 - 9893 - {None} Processed request: 0.005sec/-0.000sec (0.002sec, 0.000sec) (0.001sec/0.002sec/5) 4B 200 "POST /_synapse/replication/fed_send_edu/m.presence/nCmfebTyOS HTTP/1.1" "Synapse/1.33.0rc1" [0 dbevts]
2021-04-28 11:40:39,307 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-59 - Streaming: presence -> 110906303
2021-04-28 11:40:39,328 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-60 - Streaming: presence_federation -> 1
2021-04-28 11:40:39,329 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-61 - Streaming: presence_federation -> 2
2021-04-28 11:40:39,331 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-62 - Streaming: presence_federation -> 3
2021-04-28 11:40:39,334 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-63 - Streaming: presence_federation -> 4
2021-04-28 11:40:39,351 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-64 - Streaming: presence_federation -> 4
2021-04-28 11:40:39,366 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-64 - Streaming: presence_federation -> 4
2021-04-28 11:40:39,382 - synapse.replication.tcp.resource - 187 - INFO - replication_notifier-64 - Streaming: presence_federation -> 4
```
Can someone try this with DEBUG logging on, at least for `synapse.replication.tcp.resource` please? I'm interested in log lines like: `Getting stream: presence_federation: 2 -> 3`

Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -2026,18 +2026,40 @@ async def get_replication_rows(
             )
             return result["updates"], result["upto_token"], result["limited"]
 
+        # If the from_token is the current token then there's nothing to return
+        # and we can trivially no-op.
+        if from_token == self._next_id - 1:
+            return [], upto_token, False
+
         # We can find the correct position in the queue by noting that there is
         # exactly one entry per stream ID, and that the last entry has an ID of
         # `self._next_id - 1`, so we can count backwards from the end.
         #
+        # Since we are returning all states in the range `from_token < stream_id
+        # <= upto_token` we look for the index with a `stream_id` of `from_token
+        # + 1`.
+        #
         # Since the start of the queue is periodically truncated we need to
         # handle the case where `from_token` stream ID has already been dropped.
-        start_idx = max(from_token - self._next_id, -len(self._queue))
+        start_idx = max(from_token + 1 - self._next_id, -len(self._queue))
 
         to_send = []  # type: List[Tuple[int, Tuple[str, str]]]
         limited = False
         new_id = upto_token
         for _, stream_id, destinations, user_ids in self._queue[start_idx:]:
+            if stream_id <= from_token:
+                # Paranoia check that we are actually only sending states that
+                # are have stream_id strictly greater than from_token. We should
+                # never hit this.
+                logger.warning(
+                    "Tried returning presence federation stream ID: %d less than from_token: %d (next_id: %d, len: %d)",
+                    stream_id,
+                    from_token,
+                    self._next_id,
+                    len(self._queue),
+                )
+                continue
+
             if stream_id > upto_token:
                 break
 


Test patch:
diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py
--- a/tests/handlers/test_presence.py
+++ b/tests/handlers/test_presence.py
@@ -509,6 +509,14 @@ def test_send_and_get(self):
 
         self.assertCountEqual(rows, expected_rows)
 
+        now_token = self.queue.get_current_token(self.instance_name)
+        rows, upto_token, limited = self.get_success(
+            self.queue.get_replication_rows("master", upto_token, now_token, 10)
+        )
+        self.assertEqual(upto_token, now_token)
+        self.assertFalse(limited)
+        self.assertCountEqual(rows, [])
+
     def test_send_and_get_split(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")
@@ -538,6 +546,20 @@ def test_send_and_get_split(self):
 
         self.assertCountEqual(rows, expected_rows)
 
+        now_token = self.queue.get_current_token(self.instance_name)
+        rows, upto_token, limited = self.get_success(
+            self.queue.get_replication_rows("master", upto_token, now_token, 10)
+        )
+
+        self.assertEqual(upto_token, now_token)
+        self.assertFalse(limited)
+
+        expected_rows = [
+            (2, ("dest3", "@user3:test")),
+        ]
+
+        self.assertCountEqual(rows, expected_rows)
+
     def test_clear_queue_all(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")