Repository: matrix-org/synapse

Issue:
Rooms with unsupported versions are shown as blank rooms
In the last ~4-5 days

`!VXChyxVNCBsWeSGcaN:sw1v.org`
`!eKTrlepAvBEhUIyoQV:jki.re`
`!bNAPsppHlSHCDvMRhh:abolivier.bzh`
Sync failing with 404 M_NOT_FOUND when the user is invited to a room with an unknown version
### Description

As per the title; tried logging into my matrix.org test account and /sync responded with a 404 over and over again

![image](https://user-images.githubusercontent.com/2403652/128176765-d6cc5d65-8e07-40c5-bea8-ae011b0f5fef.png)


### Version information

- **Homeserver**: matrix.org

I believe the account was `@webdevguru_test17:matrix.org` in case that helps with log spelunking

It might be because the user was in an msc3083 unstable room the server dropped support for. 🔥 


Discussion:
(i haven't actually looked at my /sync response, but riot shows them to be entirely blank v1 rooms)
I also can't see ```!eKTrlepAvBEhUIyoQV:jki.re```
All three of those rooms use experimental room versions that have since been removed from synapse, I think its basically expected for those to go pop, though they should go pop in a better way.
Right and it just about works okay when people have cached history, but switching over to a new client (like Riot electron) has this problem.

I'm going to suggest we triage this alongside other synapse bugs rather than treat it as a crisis.
#7209 also points to some errors during initial sync for unknown room versions.
suggestion is that for now we exclude such rooms from initial /sync responses, thus hiding them on clients. (you'll have to flush your client cache, but we can live with that)

In future, if we drop production room versions (eg v1) we'll need to do better than this, but this will do as a first step.
how do I get back into these rooms, though? i can't seem to participate in them enough to follow the tombstone to the new version...
you don't. The whole point is that they were experimental room versions that are no longer supported. I guess you get someone to manually invite you to the new room.
Worth noting that deleting the room from `rooms` and the events from `events` and `events_json` clears them right up.
Currently `/purge_rooms` fails to delete these rooms, as users are in them. Might be worth fixing it so that it works?
Personally, I would at least like to be able to leave these unsupported rooms, but currently even that is not possible... 😢 

Currently, the `/leave` request gives a 400 error with `M_UNSUPPORTED_ROOM_VERSION`. The unfortunate symptom of this is that the unleavable room is permanently at the top of my room list... 😭 
> Personally, I would at least like to be able to leave these unsupported rooms, but currently even that is not possible... 😢

The technical reason behind this is that Synapse doesn't know how to resolve state in those rooms.
Can we just generate a local out-of-band leave event for the client, just as we do when rejecting an invite/rescinding a knock from a room that we're not in and cannot contact?

https://github.com/matrix-org/synapse/blob/96f6293de51c2fcf530bb6ca3705cf596c19656f/synapse/handlers/room_member.py#L1404-L1425

It just generates a leave event that's propagated to the client, but not into the room proper.
even out-of-band leaves go through state res...
I think? maybe not, actually. Might be worth a go. (but then, I still think it would be easier just to exclude unsupported rooms from `/sync`.)
> I think? maybe not, actually. Might be worth a go. (but then, I still think it would be easier just to exclude unsupported rooms from `/sync`.)

I'm fairly certain they do, we still generate an event which requires a known room version (and it also requires auth/prev events).
This error is referring to an event sent in room `!KQakPoXjBfiHdvvssg:matrix.org` which indeed has the version `org.matrix.msc3083`. The event (`$h0z7A514gZv7MmOt7y6Cck7Af2PcgDZDsmoEs8I5mC0`) is `@webdevguru_test22:matrix.org` inviting `@webdevguru_test17:matrix.org` into the room.
So what's happening here is:

* the sync handler loads the list of room memberships for this user
* it sees that the user is invited to the room
* it tries to get the invite event for this room using `get_events`, which eventually calls `_get_events_from_db`
* `_get_events_from_db` retrieves the event and checks the room version against known ones
* the room version isn't known so it ignores it
* `_get_events_from_db` returns no event, so `get_events` raises a 404 error

The check on the room version is here: https://github.com/matrix-org/synapse/blob/bdfde6dca11a9468372b3c9b327ad3327cbdbe4a/synapse/storage/databases/main/events_worker.py#L781-L801
Bumping to S-Critical as I'm assuming it blocks development work for @t3chguy (please let us know if it doesn't).

Bricking unstable rooms is one thing, but bricking accounts that have interacted with unstable rooms is quite another, and something I'm keen to see resolved prior to 1.40 going out.
It isn't blocking my development work, it is one of my ~30 test accounts, just happens to be the one my FF instance defaulted to and confused me why sync was returning 404.

Though if someone created such an unstable room (which Element Android release edition allowed you to do via UI) on their primary account then it'd be S-Critical for them
Just to define the impact I believe this will only cause an issue if you have an outstanding invite to an unknown room version, not users who have joined an unknown room version (see #7083).

Just to ensure we're all discussing the same thing:
* "unstable" is a room version that Synapse knows about but is not stable, according to the spec.
* "unknown" or "unsupported" is a room version which Synapse does not know about (and does not know how to interact with).
* "experimental" is generally a synonym for "unstable" but if the spec ever made a stable room version unstable than they would differ ("experimental" would then mean unspecced, e.g. from an MSC or something).

Resolving patch:
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -1,5 +1,4 @@
-# Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2018, 2019 New Vector Ltd
+# Copyright 2015-2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -31,6 +30,7 @@
 
 from synapse.api.constants import AccountDataTypes, EventTypes, Membership
 from synapse.api.filtering import FilterCollection
+from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import EventBase
 from synapse.logging.context import current_context
 from synapse.logging.opentracing import SynapseTags, log_kv, set_tag, start_active_span
@@ -1843,6 +1843,9 @@ async def _get_all_rooms(
         knocked = []
 
         for event in room_list:
+            if event.room_version_id not in KNOWN_ROOM_VERSIONS:
+                continue
+
             if event.membership == Membership.JOIN:
                 room_entries.append(
                     RoomSyncResultBuilder(
diff --git a/synapse/storage/databases/main/roommember.py b/synapse/storage/databases/main/roommember.py
--- a/synapse/storage/databases/main/roommember.py
+++ b/synapse/storage/databases/main/roommember.py
@@ -386,9 +386,10 @@ def _get_rooms_for_local_user_where_membership_is_txn(
         )
 
         sql = """
-            SELECT room_id, e.sender, c.membership, event_id, e.stream_ordering
+            SELECT room_id, e.sender, c.membership, event_id, e.stream_ordering, r.room_version
             FROM local_current_membership AS c
             INNER JOIN events AS e USING (room_id, event_id)
+            INNER JOIN rooms AS r USING (room_id)
             WHERE
                 user_id = ?
                 AND %s
@@ -397,7 +398,7 @@ def _get_rooms_for_local_user_where_membership_is_txn(
         )
 
         txn.execute(sql, (user_id, *args))
-        results = [RoomsForUser(**r) for r in self.db_pool.cursor_to_dict(txn)]
+        results = [RoomsForUser(*r) for r in txn]
 
         return results
 
@@ -447,7 +448,8 @@ async def get_rooms_for_user_with_stream_ordering(
 
         Returns:
             Returns the rooms the user is in currently, along with the stream
-            ordering of the most recent join for that user and room.
+            ordering of the most recent join for that user and room, along with
+            the room version of the room.
         """
         return await self.db_pool.runInteraction(
             "get_rooms_for_user_with_stream_ordering",
diff --git a/synapse/storage/roommember.py b/synapse/storage/roommember.py
--- a/synapse/storage/roommember.py
+++ b/synapse/storage/roommember.py
@@ -30,6 +30,7 @@ class RoomsForUser:
     membership: str
     event_id: str
     stream_ordering: int
+    room_version_id: str
 
 
 @attr.s(slots=True, frozen=True, weakref_slot=True, auto_attribs=True)


Test patch:
diff --git a/tests/handlers/test_sync.py b/tests/handlers/test_sync.py
--- a/tests/handlers/test_sync.py
+++ b/tests/handlers/test_sync.py
@@ -12,9 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from typing import Optional
+
+from synapse.api.constants import EventTypes, JoinRules
 from synapse.api.errors import Codes, ResourceLimitError
 from synapse.api.filtering import DEFAULT_FILTER_COLLECTION
+from synapse.api.room_versions import RoomVersions
 from synapse.handlers.sync import SyncConfig
+from synapse.rest import admin
+from synapse.rest.client import knock, login, room
+from synapse.server import HomeServer
 from synapse.types import UserID, create_requester
 
 import tests.unittest
@@ -24,8 +31,14 @@
 class SyncTestCase(tests.unittest.HomeserverTestCase):
     """Tests Sync Handler."""
 
-    def prepare(self, reactor, clock, hs):
-        self.hs = hs
+    servlets = [
+        admin.register_servlets,
+        knock.register_servlets,
+        login.register_servlets,
+        room.register_servlets,
+    ]
+
+    def prepare(self, reactor, clock, hs: HomeServer):
         self.sync_handler = self.hs.get_sync_handler()
         self.store = self.hs.get_datastore()
 
@@ -68,12 +81,124 @@ def test_wait_for_sync_for_user_auth_blocking(self):
         )
         self.assertEquals(e.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)
 
+    def test_unknown_room_version(self):
+        """
+        A room with an unknown room version should not break sync (and should be excluded).
+        """
+        inviter = self.register_user("creator", "pass", admin=True)
+        inviter_tok = self.login("@creator:test", "pass")
+
+        user = self.register_user("user", "pass")
+        tok = self.login("user", "pass")
+
+        # Do an initial sync on a different device.
+        requester = create_requester(user)
+        initial_result = self.get_success(
+            self.sync_handler.wait_for_sync_for_user(
+                requester, sync_config=generate_sync_config(user, device_id="dev")
+            )
+        )
+
+        # Create a room as the user.
+        joined_room = self.helper.create_room_as(user, tok=tok)
+
+        # Invite the user to the room as someone else.
+        invite_room = self.helper.create_room_as(inviter, tok=inviter_tok)
+        self.helper.invite(invite_room, targ=user, tok=inviter_tok)
+
+        knock_room = self.helper.create_room_as(
+            inviter, room_version=RoomVersions.V7.identifier, tok=inviter_tok
+        )
+        self.helper.send_state(
+            knock_room,
+            EventTypes.JoinRules,
+            {"join_rule": JoinRules.KNOCK},
+            tok=inviter_tok,
+        )
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/r0/knock/%s" % (knock_room,),
+            b"{}",
+            tok,
+        )
+        self.assertEquals(200, channel.code, channel.result)
+
+        # The rooms should appear in the sync response.
+        result = self.get_success(
+            self.sync_handler.wait_for_sync_for_user(
+                requester, sync_config=generate_sync_config(user)
+            )
+        )
+        self.assertIn(joined_room, [r.room_id for r in result.joined])
+        self.assertIn(invite_room, [r.room_id for r in result.invited])
+        self.assertIn(knock_room, [r.room_id for r in result.knocked])
+
+        # Test a incremental sync (by providing a since_token).
+        result = self.get_success(
+            self.sync_handler.wait_for_sync_for_user(
+                requester,
+                sync_config=generate_sync_config(user, device_id="dev"),
+                since_token=initial_result.next_batch,
+            )
+        )
+        self.assertIn(joined_room, [r.room_id for r in result.joined])
+        self.assertIn(invite_room, [r.room_id for r in result.invited])
+        self.assertIn(knock_room, [r.room_id for r in result.knocked])
+
+        # Poke the database and update the room version to an unknown one.
+        for room_id in (joined_room, invite_room, knock_room):
+            self.get_success(
+                self.hs.get_datastores().main.db_pool.simple_update(
+                    "rooms",
+                    keyvalues={"room_id": room_id},
+                    updatevalues={"room_version": "unknown-room-version"},
+                    desc="updated-room-version",
+                )
+            )
+
+        # Blow away caches (supported room versions can only change due to a restart).
+        self.get_success(
+            self.store.get_rooms_for_user_with_stream_ordering.invalidate_all()
+        )
+        self.store._get_event_cache.clear()
+
+        # The rooms should be excluded from the sync response.
+        # Get a new request key.
+        result = self.get_success(
+            self.sync_handler.wait_for_sync_for_user(
+                requester, sync_config=generate_sync_config(user)
+            )
+        )
+        self.assertNotIn(joined_room, [r.room_id for r in result.joined])
+        self.assertNotIn(invite_room, [r.room_id for r in result.invited])
+        self.assertNotIn(knock_room, [r.room_id for r in result.knocked])
+
+        # The rooms should also not be in an incremental sync.
+        result = self.get_success(
+            self.sync_handler.wait_for_sync_for_user(
+                requester,
+                sync_config=generate_sync_config(user, device_id="dev"),
+                since_token=initial_result.next_batch,
+            )
+        )
+        self.assertNotIn(joined_room, [r.room_id for r in result.joined])
+        self.assertNotIn(invite_room, [r.room_id for r in result.invited])
+        self.assertNotIn(knock_room, [r.room_id for r in result.knocked])
+
+
+_request_key = 0
+
 
-def generate_sync_config(user_id: str) -> SyncConfig:
+def generate_sync_config(
+    user_id: str, device_id: Optional[str] = "device_id"
+) -> SyncConfig:
+    """Generate a sync config (with a unique request key)."""
+    global _request_key
+    _request_key += 1
     return SyncConfig(
-        user=UserID(user_id.split(":")[0][1:], user_id.split(":")[1]),
+        user=UserID.from_string(user_id),
         filter_collection=DEFAULT_FILTER_COLLECTION,
         is_guest=False,
-        request_key="request_key",
-        device_id="device_id",
+        request_key=("request_key", _request_key),
+        device_id=device_id,
     )
diff --git a/tests/replication/slave/storage/test_events.py b/tests/replication/slave/storage/test_events.py
--- a/tests/replication/slave/storage/test_events.py
+++ b/tests/replication/slave/storage/test_events.py
@@ -150,6 +150,7 @@ def test_invites(self):
                     "invite",
                     event.event_id,
                     event.internal_metadata.stream_ordering,
+                    RoomVersions.V1.identifier,
                 )
             ],
         )