Repository: matrix-org/synapse

Issue:
Event got sent but not synced back down everywhere
The event in question is https://app.element.io/#/room/!RbKRuLTvHIhArQcvjr:matrix.org/$1613561441289661rpDpF:matrix.org?via=matrix.org&via=vector.modular.im&via=t2l.io

The client that sent the event (element desktop) and my dekstop hydrogen running alongside did not get the event from sync.

On element desktop the event was black (e.g. sent) but stuck to the bottom of the room, and on hydrogen I check the storage and it showed that there was no limited sync for that room around those messages, and there is no trace of that event anywhere.

When opening my phone hydrogen an hour or so later, the event was synced down to it in the correct order.

At least T3chguy and Ryan did receive the event in the correct order.


Discussion:
we have also several report of that on android
https://github.com/matrix-org/element-android-rageshakes/issues/20724
https://github.com/matrix-org/element-android-rageshakes/issues/20672
https://github.com/matrix-org/element-android-rageshakes/issues/20578

I also experienced it on other client (web)

Also this issue https://github.com/vector-im/element-android/issues/516

https://github.com/matrix-org/synapse/issues/9624
@callahad @anoadragon453  I renamed this ticket as #9624 (which was marked as a dupe) is about the recipient and not the sender not receiving the message. FYI if this changes the severity of the bug.
Just an update: This is on our radar, but we expect it to be a pretty big yak to shave, so it's currently deferred until we can wrap up some other hairy problems 🐃 
Matthew reported another instance of this today. Link and details for internal users at https://github.com/matrix-org/element-web-rageshakes/issues/5808
And another set by Matthew on Aug 10:

```
Room ID: !JjwYSGirOwIXEpJgzg:matrix.org
Event ID: $ZrcrvSAESv5PBzszSXpT5EahA2eQR-raFbFfSY314LQ
```
We should add some more logs to try and track down where the event is going missing. I'd probably do something like:

* Log to the span how many rooms have changed (i.e. what's returned by `_get_rooms_changed`)
* Add a span for each `_generate_room_entry` call, tagging it with room ID
* Log to the span the number of events returned from the DB, and whether we hit a cache, and what tokens are used in the query
* Log to the span how many events there are after filtering.
* Log to the span the number of room entries and events in the response JSON just before returning

etc. The idea being that hopefully we'll be able to see if the missing event is pulled out of the DB and if so where we drop it (if we ever do). 
And another just now...

```
Room ID: !KzalCNJxkqtytlbQvX:matrix.org
Event ID: $16310279891219482qXcHE:matrix.org
...and again...

```
Room ID: !KzalCNJxkqtytlbQvX:matrix.org
Event ID: $16311860683330067SxAox:matrix.org

...and again...

```
Room ID: !KzalCNJxkqtytlbQvX:matrix.org
Event ID: $16322575506214369TPVVU:matrix.org

...and again...

```
Room ID: !rZUxrMFEsvfBNGpmDf:matrix.org
Event ID: $1633697134169238TSzOW:matrix.org

The event in question:

```
matrix=> select topological_ordering, instance_name, stream_ordering, to_timestamp(origin_server_ts/1000) from events where event_id = '$1633697134169238TSzOW:matrix.org';
 topological_ordering |   instance_name   | stream_ordering |      to_timestamp      
----------------------+-------------------+-----------------+------------------------
                 3229 | event_persister-3 |      2372403105 | 2021-10-08 13:45:34+01
(1 row)
```

The trace for the sync that should have returned it (will disappear soon): https://jaeger.int.matrix.org/trace/901206efde7b4ac6, which happened on synchrotron-36.

![Selection_054](https://user-images.githubusercontent.com/8428120/136584253-eed5cb6d-f42f-45d3-aa33-60fe69f3f2f8.png)

![Selection_056](https://user-images.githubusercontent.com/8428120/136584460-e341a38f-953d-446e-b17c-e90ad8f2a7da.png)
![Selection_057](https://user-images.githubusercontent.com/8428120/136584549-314356f3-8b04-40a2-97cd-7e024d227c92.png)


The `/sync` gets woken up twice, the first with a since and now token of:
`RoomStreamToken(topological=None, stream=2372403104, instance_map=<frozendict {'event_persister-2': 2372403109, 'event_persister-1': 2372403109}>)`
`RoomStreamToken(topological=None, stream=2372403115, instance_map=<frozendict {}>)`
this should have picked up the sent event, as it was sent on event_persister-3 with stream ordering `2372403105`. However, it decides that no rooms have changed: `rooms_have_changed: false`. The subsequent time the sync gets woken up the token are:
`RoomStreamToken(topological=None, stream=2372403104, instance_map=<frozendict {'event_persister-2': 2372403109, 'event_persister-1': 2372403109}>)`
`RoomStreamToken(topological=None, stream=2372403130, instance_map=<frozendict {}>)`,
which also finds that no rooms have changed.

The `RoomStreamToken` and `PersistedEventPosition` work correctly with these values:

```
In [1]: from synapse.types import RoomStreamToken, PersistedEventPosition                                                                                                                                         

In [2]: from frozendict import frozendict                                                                                                                                                                         

In [3]: since = RoomStreamToken(None, 2372403104, frozendict({'event_persister-2': 2372403109, 'event_persister-1': 2372403109}))                                                                                 

In [4]: pos = PersistedEventPosition("event_persister-3", 2372403105)                                                                                                                                             

In [5]: pos.persisted_after(since)                                                                                                                                                                                
Out[5]: True
```

We see in the synchtron36 logs:

`
2021-10-08 12:45:34,307 - synapse.replication.tcp.handler - 509 - INFO - replication_command_handler-140621334838808 - Handling 'POSITION events event_persister-3 2372403104 2372403108'
`

which is worrying, as `event_persister-3` did write the event in the range `2372403104..2372403108`. The event persister should *not* have sent out a `POSITION` with that range, as it means "I did nothing in this range". 

I think there is a race between workers sending out the appropriate `RDATA` and us sending out the periodic `POSITION`.
On event persister 3:

```
2021-10-08 12:45:34,150 - synapse.replication.tcp.resource - 188 - INFO - replication_notifier-6533172 - Streaming: events -> 2372403104
...
2021-10-08 12:45:34,298 - synapse.replication.tcp.resource - 204 - INFO - replication_notifier-6533172 - Sending position: events -> 2372403108
...
2021-10-08 12:45:34,330 - synapse.access.http.18129 - 421 - INFO - POST-3580545 - ::ffff:10.103.0.1 - 18129 - {@matthew:matrix.org} Processed request: 0.183sec/0.009sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 71B 200 "POST /_synapse/replication/send_event/%241633697134169238TSzOW%3Amatrix.org/shpMywvFLw HTTP/1.1" "Synapse/1.44.0rc2 (b=matrix-org-hotfixes,ebbd37b66)" [0 dbevts]

...
2021-10-08 12:45:34,366 - synapse.replication.tcp.resource - 188 - INFO - replication_notifier-6533173 - Streaming: events -> 2372403114
```

which looks to confirm that we're spuriously sending out a `POSITION` instead of the event in question.

c.f. https://github.com/matrix-org/synapse/blob/51a5da74ccd383806378b53ee8a09e27a8829f31/synapse/replication/tcp/resource.py#L186-L215
Fetching the updates doesn't appear to be hitting any caches, and just does a query with a `WHERE ? < stream_ordering <= ?` clause, so I don't think its caching.

The query in question returns the row now:

```sql
SELECT e.stream_ordering, e.event_id, e.room_id, e.type,
                 state_key, redacts, relates_to_id, membership, rejections.reason IS NOT NULL
                 FROM events AS e
                 LEFT JOIN redactions USING (event_id)
                 LEFT JOIN state_events USING (event_id)
                 LEFT JOIN event_relations USING (event_id)
                 LEFT JOIN room_memberships USING (event_id)
                 LEFT JOIN rejections USING (event_id)
                 WHERE 2372403104 < stream_ordering AND stream_ordering <= 2372403108
                 AND instance_name = 'event_persister-3'
                 ORDER BY stream_ordering ASC
                 LIMIT 100;
```

My current best guess is that there is a bug in `MultiWriterIdGenerator` which is causing it to return a current token for the local instance *before* all previous tokens have finished persisting.
...and another by @hughns this morning (taken from his rageshake):

```
2021-10-11T10:33:35.753Z I Event sent to !ZiIcIhEEPvPrkUeheO:matrix.org with event id $16339484157934361ksrnd:matrix.org
2021-10-11T10:33:35.753Z I setting pendingEvent status to sent in !ZiIcIhEEPvPrkUeheO:matrix.org event ID ~!ZiIcIhEEPvPrkUeheO:matrix.org:m1633948414939.743 -> $16339484157934361ksrnd:matrix.org
```

...and the remote echo never comes.
Worth noting that the stream ID sent by the `POSITION` `2372403108` was written by `event_persister-2`, and gets replicated to `event_persister-3` at:

```
2021-10-08 12:45:34,265 - synapse.replication.tcp.handler - 509 - INFO - replication_command_handler-139816608471832 - Handling 'POSITION events event_persister-2 2372403107 2372403108'
```

Which is *before* the missing event is finished persisting (it's `received_ts` is `12:45:34,280`). This makes it look very much like the event stream ID generator is incorrectly returning a current token of `2372403108` even though `2372403105` hasn't been persisted yet.

For reference, the code where we update the current position of the local instance when we receive updates from remote instances is:

https://github.com/matrix-org/synapse/blob/a7d22c36dbbbdd396aeb8938b57b5fd7edb689f3/synapse/storage/util/id_generators.py#L603-L611

Resolving patch:
diff --git a/synapse/storage/util/id_generators.py b/synapse/storage/util/id_generators.py
--- a/synapse/storage/util/id_generators.py
+++ b/synapse/storage/util/id_generators.py
@@ -36,7 +36,7 @@
 )
 
 import attr
-from sortedcontainers import SortedSet
+from sortedcontainers import SortedList, SortedSet
 
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.storage.database import (
@@ -265,6 +265,15 @@ def __init__(
         # should be less than the minimum of this set (if not empty).
         self._unfinished_ids: SortedSet[int] = SortedSet()
 
+        # We also need to track when we've requested some new stream IDs but
+        # they haven't yet been added to the `_unfinished_ids` set. Every time
+        # we request a new stream ID we add the current max stream ID to the
+        # list, and remove it once we've added the newly allocated IDs to the
+        # `_unfinished_ids` set. This means that we *may* be allocated stream
+        # IDs above those in the list, and so we can't advance the local current
+        # position beyond the minimum stream ID in this list.
+        self._in_flight_fetches: SortedList[int] = SortedList()
+
         # Set of local IDs that we've processed that are larger than the current
         # position, due to there being smaller unpersisted IDs.
         self._finished_ids: Set[int] = set()
@@ -290,6 +299,9 @@ def __init__(
         )
         self._known_persisted_positions: List[int] = []
 
+        # The maximum stream ID that we have seen been allocated across any writer.
+        self._max_seen_allocated_stream_id = 1
+
         self._sequence_gen = PostgresSequenceGenerator(sequence_name)
 
         # We check that the table and sequence haven't diverged.
@@ -305,6 +317,10 @@ def __init__(
         # This goes and fills out the above state from the database.
         self._load_current_ids(db_conn, tables)
 
+        self._max_seen_allocated_stream_id = max(
+            self._current_positions.values(), default=1
+        )
+
     def _load_current_ids(
         self,
         db_conn: LoggingDatabaseConnection,
@@ -411,10 +427,32 @@ def _load_current_ids(
         cur.close()
 
     def _load_next_id_txn(self, txn: Cursor) -> int:
-        return self._sequence_gen.get_next_id_txn(txn)
+        stream_ids = self._load_next_mult_id_txn(txn, 1)
+        return stream_ids[0]
 
     def _load_next_mult_id_txn(self, txn: Cursor, n: int) -> List[int]:
-        return self._sequence_gen.get_next_mult_txn(txn, n)
+        # We need to track that we've requested some more stream IDs, and what
+        # the current max allocated stream ID is. This is to prevent a race
+        # where we've been allocated stream IDs but they have not yet been added
+        # to the `_unfinished_ids` set, allowing the current position to advance
+        # past them.
+        with self._lock:
+            current_max = self._max_seen_allocated_stream_id
+            self._in_flight_fetches.add(current_max)
+
+        try:
+            stream_ids = self._sequence_gen.get_next_mult_txn(txn, n)
+
+            with self._lock:
+                self._unfinished_ids.update(stream_ids)
+                self._max_seen_allocated_stream_id = max(
+                    self._max_seen_allocated_stream_id, self._unfinished_ids[-1]
+                )
+        finally:
+            with self._lock:
+                self._in_flight_fetches.remove(current_max)
+
+        return stream_ids
 
     def get_next(self) -> AsyncContextManager[int]:
         """
@@ -463,9 +501,6 @@ def get_next_txn(self, txn: LoggingTransaction) -> int:
 
         next_id = self._load_next_id_txn(txn)
 
-        with self._lock:
-            self._unfinished_ids.add(next_id)
-
         txn.call_after(self._mark_id_as_finished, next_id)
         txn.call_on_exception(self._mark_id_as_finished, next_id)
 
@@ -497,15 +532,27 @@ def _mark_id_as_finished(self, next_id: int) -> None:
 
             new_cur: Optional[int] = None
 
-            if self._unfinished_ids:
+            if self._unfinished_ids or self._in_flight_fetches:
                 # If there are unfinished IDs then the new position will be the
-                # largest finished ID less than the minimum unfinished ID.
+                # largest finished ID strictly less than the minimum unfinished
+                # ID.
+
+                # The minimum unfinished ID needs to take account of both
+                # `_unfinished_ids` and `_in_flight_fetches`.
+                if self._unfinished_ids and self._in_flight_fetches:
+                    # `_in_flight_fetches` stores the maximum safe stream ID, so
+                    # we add one to make it equivalent to the minimum unsafe ID.
+                    min_unfinished = min(
+                        self._unfinished_ids[0], self._in_flight_fetches[0] + 1
+                    )
+                elif self._in_flight_fetches:
+                    min_unfinished = self._in_flight_fetches[0] + 1
+                else:
+                    min_unfinished = self._unfinished_ids[0]
 
                 finished = set()
-
-                min_unfinshed = self._unfinished_ids[0]
                 for s in self._finished_ids:
-                    if s < min_unfinshed:
+                    if s < min_unfinished:
                         if new_cur is None or new_cur < s:
                             new_cur = s
                     else:
@@ -575,6 +622,10 @@ def advance(self, instance_name: str, new_id: int) -> None:
                 new_id, self._current_positions.get(instance_name, 0)
             )
 
+            self._max_seen_allocated_stream_id = max(
+                self._max_seen_allocated_stream_id, new_id
+            )
+
             self._add_persisted_position(new_id)
 
     def get_persisted_upto_position(self) -> int:
@@ -605,7 +656,11 @@ def _add_persisted_position(self, new_id: int) -> None:
         # to report a recent position when asked, rather than a potentially old
         # one (if this instance hasn't written anything for a while).
         our_current_position = self._current_positions.get(self._instance_name)
-        if our_current_position and not self._unfinished_ids:
+        if (
+            our_current_position
+            and not self._unfinished_ids
+            and not self._in_flight_fetches
+        ):
             self._current_positions[self._instance_name] = max(
                 our_current_position, new_id
             )
@@ -697,9 +752,6 @@ async def __aenter__(self) -> Union[int, List[int]]:
             db_autocommit=True,
         )
 
-        with self.id_gen._lock:
-            self.id_gen._unfinished_ids.update(self.stream_ids)
-
         if self.multiple_ids is None:
             return self.stream_ids[0] * self.id_gen._return_factor
         else:


Test patch: