Repository: matrix-org/synapse

Issue:
Sync can 404 with "Can't find event..." when processing some invites
Until https://github.com/matrix-org/synapse/pull/6983 landed (shipped in 1.12.0 - March 23, 2020), we weren't upserting rows to the `rooms` table when receiving a new invite. We're now doing that because we need to know the version of the room the invite is for.

In some places of the code, including when handling sync, we make the assumption that if we have an event or an invite then we have the version of the room it's been sent into. When the user has an invite to a room that's not in the `rooms` table, this causes an error to be raised ("Room X for event Y is unknown") and syncs to 404 (with "Can't find event Y").

As of last Friday there were 1600 rooms with this issue on matrix.org (`select count(*) from local_current_membership left join rooms using (room_id) where membership = 'invite' and rooms.room_id is null;`).

An ideal solution would be a background update that inserts the missing rooms in the `rooms` table, however this isn't trivial because we don't know the version of these rooms. Inserting a room with version `NULL` means Synapse will assume it's v1, which can be wrong, as non-v1 room versions were introduced way earlier (v2 was introduced in https://github.com/matrix-org/synapse/pull/4307 and shipped in 0.34.1, released on Jan 9, 2019).

We could also do a similar update just for rooms which invite predates rooms v2, but that would probably be quite low impact (89 rooms on matrix.org).

A first step towards resolution could be to ensure we don't fail in that case unless it's necessary - e.g. we don't fail the entire sync response over one bogus invite.


Discussion:
I'm fairly sure that this *used* to work correctly, so I'd be somewhat interested to know at what point it regressed. Where exactly does the `/sync` handler expect that there is a `rooms` entry?

#6902 is related in that even if your sync returns correctly, you may not be able to reject the invite.

Resolving patch:
diff --git a/synapse/events/__init__.py b/synapse/events/__init__.py
--- a/synapse/events/__init__.py
+++ b/synapse/events/__init__.py
@@ -133,6 +133,8 @@ def is_out_of_band_membership(self) -> bool:
         rejection. This is needed as those events are marked as outliers, but
         they still need to be processed as if they're new events (e.g. updating
         invite state in the database, relaying to clients, etc).
+
+        (Added in synapse 0.99.0, so may be unreliable for events received before that)
         """
         return self._dict.get("out_of_band_membership", False)
 
diff --git a/synapse/storage/databases/main/events_worker.py b/synapse/storage/databases/main/events_worker.py
--- a/synapse/storage/databases/main/events_worker.py
+++ b/synapse/storage/databases/main/events_worker.py
@@ -586,19 +586,38 @@ def _get_events_from_db(self, event_ids, allow_rejected=False):
             room_version_id = row["room_version_id"]
 
             if not room_version_id:
-                # this should only happen for out-of-band membership events
-                if not internal_metadata.get("out_of_band_membership"):
-                    logger.warning(
-                        "Room %s for event %s is unknown", d["room_id"], event_id
+                # this should only happen for out-of-band membership events which
+                # arrived before #6983 landed. For all other events, we should have
+                # an entry in the 'rooms' table.
+                #
+                # However, the 'out_of_band_membership' flag is unreliable for older
+                # invites, so just accept it for all membership events.
+                #
+                if d["type"] != EventTypes.Member:
+                    raise Exception(
+                        "Room %s for event %s is unknown" % (d["room_id"], event_id)
                     )
-                    continue
 
-                # take a wild stab at the room version based on the event format
+                # so, assuming this is an out-of-band-invite that arrived before #6983
+                # landed, we know that the room version must be v5 or earlier (because
+                # v6 hadn't been invented at that point, so invites from such rooms
+                # would have been rejected.)
+                #
+                # The main reason we need to know the room version here (other than
+                # choosing the right python Event class) is in case the event later has
+                # to be redacted - and all the room versions up to v5 used the same
+                # redaction algorithm.
+                #
+                # So, the following approximations should be adequate.
+
                 if format_version == EventFormatVersions.V1:
+                    # if it's event format v1 then it must be room v1 or v2
                     room_version = RoomVersions.V1
                 elif format_version == EventFormatVersions.V2:
+                    # if it's event format v2 then it must be room v3
                     room_version = RoomVersions.V3
                 else:
+                    # if it's event format v3 then it must be room v4 or v5
                     room_version = RoomVersions.V5
             else:
                 room_version = KNOWN_ROOM_VERSIONS.get(room_version_id)


Test patch: