Repository: matrix-org/synapse

Issue:
members API returns `M_NOT_FOUND: MatrixError: [404] Can't find event for token`
### Description

See https://github.com/vector-im/element-web/issues/24179.

The user is creating a new DM room.

Element Web creates the room, waits for a sync, then calls the members API with the latest sync token:

```
https://matrix-client.matrix.org/_matrix/client/r0/rooms/!lFfrlZXUrMAOXpzuxp%3Amatrix.org/members?not_membership=leave&at=m3595246325%7E2.3595246467%7E3.3595246468%7E36.3595246466_757284974_52668403_1826042582_1854740460_3893944_721765062_6284542923_0
```


### Steps to reproduce

Since we're using a sync token given to us by Synapse from a sync that contains the new room's events, we would expect it to point to a valid event in the room.

### Homeserver

matrix.org

### Synapse Version

1.75.0rc1 (b=matrix-org-hotfixes,04aa6a9707)

But note that this was reported on 2023-01-06T11:49 UTC

### Installation Method

I don't know

### Database

PostgreSQL

### Workers

Multiple workers

### Platform

.

### Configuration

.

### Relevant log output

```shell
Element web logs: https://github.com/matrix-org/element-web-rageshakes/issues/18940
```


### Anything else that would be useful to know?

@richvdh is already investigating.


Discussion:
Initial investigation based on an example call:

```
2023-01-11 10:03:24,057 - synapse.http.server - 108 - INFO - GET-787cd28348aa75cf-LHR - <XForwardedForRequest at 0x7f77ee504be0 method='GET' uri='/_matrix/client/r0/rooms/!uvUhHnEbAYZNVcRWbx%3Amatrix.org/members?not_membership=leave&at=m3608765164%7E1.3608765719%7E2.3608765719%7E3.3608765719_757284974_3566946_1836222772_1864166937_3907868_728498125_6331086177_0' clientproto='HTTP/1.1' site='11105'> SynapseError: 404 - Can't find event for token StreamToken(room_key=RoomStreamToken(topological=None, stream=3608765164, instance_map=frozendict.frozendict({'event_persister-1': 3608765719, 'event_persister-2': 3608765719, 'event_persister-3': 3608765719})), presence_key=757284974, typing_key=3566946, receipt_key=1836222772, account_data_key=1864166937, push_rules_key=3907868, to_device_key=728498125, device_list_key=6331086177, groups_key=0)
```

The stream token in this request is:

```
m3608765164~1.3608765719~2.3608765719~3.3608765719_757284974_3566946_1836222772_1864166937_3907868_728498125_6331086177_0
```

Within that token, the interesting part is before the first `_` (corresponding to the `RoomStreamToken` - ie, in-room events). That part tells us that all event persisters have persisted up to at least stream_id `3608765164`, while persisters 1, 2, and 3 have persisted right up to `3608765719`. (See [here](https://github.com/matrix-org/synapse/blob/v1.74.0/synapse/types/__init__.py#L630-L644) and [here](https://github.com/matrix-org/synapse/blob/v1.74.0/synapse/types/__init__.py#L449-L469) for documentation of the token format). I'll call these two numbers the "minimal" and "maximal" stream orderings.

Looking at the content of this room:

```
matrix=> select stream_ordering, instance_name, sender, type from events where room_id='!uvUhHnEbAYZNVcRWbx:matrix.org' order by stream_ordering;
 stream_ordering |   instance_name   |         sender         |           type            
-----------------+-------------------+------------------------+---------------------------
      3608765557 | event_persister-2 | @dave:matrix.org       | m.room.create
      3608765563 | event_persister-2 | @dave:matrix.org       | m.room.member
      3608765577 | event_persister-2 | @dave:matrix.org       | m.room.power_levels
      3608765578 | event_persister-2 | @dave:matrix.org       | m.room.join_rules
      3608765579 | event_persister-2 | @dave:matrix.org       | m.room.history_visibility
      3608765580 | event_persister-2 | @dave:matrix.org       | m.room.guest_access
      3608765581 | event_persister-2 | @dave:matrix.org       | m.room.encryption
      3608765585 | event_persister-2 | @dave:matrix.org       | m.room.member
      3608769619 | event_persister-2 | @dave:matrix.org       | m.room.encrypted
      3608769807 | event_persister-2 | @davetest10:matrix.org | m.room.member
      3608770356 | event_persister-2 | @davetest10:matrix.org | m.room.encrypted
      3608786832 | event_persister-2 | @dave:matrix.org       | m.room.member
      3608791211 | event_persister-2 | @davetest10:matrix.org | m.room.member
```

... we can see that the room was created on event persister 2 more recently the minimal stream ordering `3608765164`. This is the cause of the error: the [`/members` handler](https://github.com/matrix-org/synapse/blob/v1.74.0/synapse/handlers/message.py#L200) calls `get_last_event_in_room_before_stream_ordering`, which only [considers the minimal stream ordering](https://github.com/matrix-org/synapse/blob/v1.74.0/synapse/storage/databases/main/stream.py#L806).

This seems to be in contrast to the behaviour of `/sync`, which will return all events (and rooms) which happened before the *maximal* stream ordering (https://github.com/matrix-org/synapse/blob/v1.74.0/synapse/storage/databases/main/stream.py#L572).

I have to admit, the sharded event stream logic tends to be indistinguishable from magic to me, but *presumably* `get_last_event_in_room_before_stream_ordering` ought to be considering these extra events.
Thinking about this further, the whole reason for providing a token to `/members` is to avoid races between `/sync` and `/members` (in a way whose details currently escape me: there may be more info in the design docs for membership lazy-loading). Surely this is completely obviated if `/sync` and `/members` then use different logic for deciding how to interpret the token?

Incidentally: it seems possible that this might have been made more obvious by recent work to speed up room creation?

Resolving patch:
diff --git a/synapse/storage/databases/main/stream.py b/synapse/storage/databases/main/stream.py
--- a/synapse/storage/databases/main/stream.py
+++ b/synapse/storage/databases/main/stream.py
@@ -801,13 +801,66 @@ async def get_last_event_in_room_before_stream_ordering(
             before this stream ordering.
         """
 
-        last_row = await self.get_room_event_before_stream_ordering(
-            room_id=room_id,
-            stream_ordering=end_token.stream,
+        def get_last_event_in_room_before_stream_ordering_txn(
+            txn: LoggingTransaction,
+        ) -> Optional[str]:
+            # We need to handle the fact that the stream tokens can be vector
+            # clocks. We do this by getting all rows between the minimum and
+            # maximum stream ordering in the token, plus one row less than the
+            # minimum stream ordering. We then filter the results against the
+            # token and return the first row that matches.
+
+            sql = """
+                SELECT * FROM (
+                    SELECT instance_name, stream_ordering, topological_ordering, event_id
+                    FROM events
+                    LEFT JOIN rejections USING (event_id)
+                    WHERE room_id = ?
+                        AND ? < stream_ordering AND stream_ordering <= ?
+                        AND NOT outlier
+                        AND rejections.event_id IS NULL
+                    ORDER BY stream_ordering DESC
+                ) AS a
+                UNION
+                SELECT * FROM (
+                    SELECT instance_name, stream_ordering, topological_ordering, event_id
+                    FROM events
+                    LEFT JOIN rejections USING (event_id)
+                    WHERE room_id = ?
+                        AND stream_ordering <= ?
+                        AND NOT outlier
+                        AND rejections.event_id IS NULL
+                    ORDER BY stream_ordering DESC
+                    LIMIT 1
+                ) AS b
+            """
+            txn.execute(
+                sql,
+                (
+                    room_id,
+                    end_token.stream,
+                    end_token.get_max_stream_pos(),
+                    room_id,
+                    end_token.stream,
+                ),
+            )
+
+            for instance_name, stream_ordering, topological_ordering, event_id in txn:
+                if _filter_results(
+                    lower_token=None,
+                    upper_token=end_token,
+                    instance_name=instance_name,
+                    topological_ordering=topological_ordering,
+                    stream_ordering=stream_ordering,
+                ):
+                    return event_id
+
+            return None
+
+        return await self.db_pool.runInteraction(
+            "get_last_event_in_room_before_stream_ordering",
+            get_last_event_in_room_before_stream_ordering_txn,
         )
-        if last_row:
-            return last_row[2]
-        return None
 
     async def get_current_room_stream_token_for_room_id(
         self, room_id: str


Test patch: