Repository: matrix-org/synapse

Issue:
Sync stream is either missing echo of events or is sending them earlier than anticipated
On March 31st (roughly) there was an uptick of "message stuck as latest message" reports to the riot clients (riot-web's upstream issue is https://github.com/vector-im/riot-web/issues/10032 for this and other cases). 

The cause looks to be because Synapse is not sending events down `/sync` sometimes for events we've sent. There are also some reports of events just not being received on the other end which could be related. Most reporters were on matrix.org, and the reports seemed to stop once matrix.org rolled back.

Reproduction steps appear to be time-dependent, and possibly only useful for matrix.org-sized homeservers:
1. Send an event
2. Start a `/sync` request
3. Send another event
4. Receive event ID for first event (`/send/m.room.message` completes)
5. Receive event down `/sync`
6. Send yet another event
7. Start a `/sync request`
8. Receive event ID for second and third events
9. Receive the third event down `/sync`, but not the second (sometimes the other way around)
10. Never receive the second event through `/sync`, making it stuck on Riot

Most easily reproduced during high traffic volumes on matrix.org, though one case was also reproduced at ~22:00 UTC yesterday (but never again that night).

Theory at least from my side is that the request hits a busy synchrotron which advances the stream token past the second event to the third event, invoking amnesia. 

@erikjohnston's conclusions appear to be that synapse might be sending the event down `/sync` ahead of the `/send/m.room.message` request completing? 


Discussion:
This rageshake is my ~22:00 UTC attempt: https://github.com/matrix-org/riot-web-rageshakes/issues/2501

(I think, I might have gotten the timestamp wrong).
> @erikjohnston's conclusions appear to be that synapse might be sending the event down /sync ahead of the /send/m.room.message request completing?

this is valid behaviour though.
> this is valid behaviour though.

Yup, and it seems like Riot is okay with this. My attempt revealed *no* syncs with the event ID contained within, aside from a sync which had the second user's read receipt for the event. The event ID did not appear early or late.
well there are a boatload of errors in the logs which suggest that some replication updates aren't being processed correctly. They include things like:

```
Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 254, in handle_command
    await handler(cmd)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 650, in on_POSITION
    [stream.parse_row(update[1]) for update in updates],
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 650, in <listcomp>
    [stream.parse_row(update[1]) for update in updates],
IndexError: string index out of range
```

```
AttributeError: 'GenericWorkerSlavedStore' object has no attribute 'get_all_push_rule_updates'
```

```
Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 254, in handle_command
    await handler(cmd)
  File "/home/synapse/src/synapse/replication/tcp/protocol.py", line 668, in on_POSITION
    await self.handler.on_rdata(cmd.stream_name, rows[-1].token, rows)
AttributeError: 'DeviceListsStreamRow' object has no attribute 'token'
```
(also `PushersStreamRow`, `EventsStreamRow`, `ToDeviceStreamRow`, probably others.)

My inclination would be to fix those, then redeploy and see if the problem is magically cured.


Logging for riot-web added here: https://github.com/matrix-org/matrix-js-sdk/pull/1300 

Might help narrow it down, maybe.
For the record, we rolled back matrix.org's synapse deployment on 2nd April while we investigated these problems.

As of 11:06 UTC this morning, we have redeployed matrix.org with current develop. We've fixed the obvious exceptions being spewed by the replication connections, so I am optimistic that this problem has gone away; however, we never really got to the bottom of what was going on, so it's entirely possible there are still problems.

In other words: feedback welcome as to whether this bug seems to be fixed or not.
... and the bug is still biting, so rolled back again as of 16:34 UTC
Right; I have reproduced this and tracked it down. It happens after the replication connection disconnects. After a disconnection, we perform a catchup, potentially processing multiple events as having the same stream id. `StreamChangeCache` assumes that there will only be one entity changing on each stream id, which is no longer true; updates are therefore getting lost.


Resolving patch:
diff --git a/synapse/replication/tcp/handler.py b/synapse/replication/tcp/handler.py
--- a/synapse/replication/tcp/handler.py
+++ b/synapse/replication/tcp/handler.py
@@ -15,7 +15,18 @@
 # limitations under the License.
 
 import logging
-from typing import Any, Callable, Dict, List, Optional, Set
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    Iterable,
+    Iterator,
+    List,
+    Optional,
+    Set,
+    Tuple,
+    TypeVar,
+)
 
 from prometheus_client import Counter
 
@@ -268,11 +279,14 @@ async def on_POSITION(self, cmd: PositionCommand):
                     missing_updates,
                 ) = await stream.get_updates_since(current_token, cmd.token)
 
-                if updates:
+                # TODO: add some tests for this
+
+                # Some streams return multiple rows with the same stream IDs,
+                # which need to be processed in batches.
+
+                for token, rows in _batch_updates(updates):
                     await self.on_rdata(
-                        cmd.stream_name,
-                        current_token,
-                        [stream.parse_row(update[1]) for update in updates],
+                        cmd.stream_name, token, [stream.parse_row(row) for row in rows],
                     )
 
             # We've now caught up to position sent to us, notify handler.
@@ -404,3 +418,52 @@ def stream_update(self, stream_name: str, token: str, data: Any):
         We need to check if the client is interested in the stream or not
         """
         self.send_command(RdataCommand(stream_name, token, data))
+
+
+UpdateToken = TypeVar("UpdateToken")
+UpdateRow = TypeVar("UpdateRow")
+
+
+def _batch_updates(
+    updates: Iterable[Tuple[UpdateToken, UpdateRow]]
+) -> Iterator[Tuple[UpdateToken, List[UpdateRow]]]:
+    """Collect stream updates with the same token together
+
+    Given a series of updates returned by Stream.get_updates_since(), collects
+    the updates which share the same stream_id together.
+
+    For example:
+
+        [(1, a), (1, b), (2, c), (3, d), (3, e)]
+
+    becomes:
+
+        [
+            (1, [a, b]),
+            (2, [c]),
+            (3, [d, e]),
+        ]
+    """
+
+    update_iter = iter(updates)
+
+    first_update = next(update_iter, None)
+    if first_update is None:
+        # empty input
+        return
+
+    current_batch_token = first_update[0]
+    current_batch = [first_update[1]]
+
+    for token, row in update_iter:
+        if token != current_batch_token:
+            # different token to the previous row: flush the previous
+            # batch and start anew
+            yield current_batch_token, current_batch
+            current_batch_token = token
+            current_batch = []
+
+        current_batch.append(row)
+
+    # flush the final batch
+    yield current_batch_token, current_batch
diff --git a/synapse/util/caches/stream_change_cache.py b/synapse/util/caches/stream_change_cache.py
--- a/synapse/util/caches/stream_change_cache.py
+++ b/synapse/util/caches/stream_change_cache.py
@@ -126,6 +126,9 @@ def entity_has_changed(self, entity, stream_pos):
         """
         assert type(stream_pos) is int
 
+        # FIXME: add a sanity check here that we are not overwriting existing
+        # data in self._cache
+
         if stream_pos > self._earliest_known_stream_pos:
             old_pos = self._entity_to_key.get(entity, None)
             if old_pos is not None:


Test patch: