Repository: matrix-org/synapse

Issue:
`delete_old_current_state_events` background update can race with room creation, causing "no create event in auth events"
As demonstrated in the logs of [this](https://buildkite.com/matrix-dot-org/sytest/builds/960#655edd00-a2b4-4b0b-bee2-142ec753a994) test run:

```
2020-07-09 09:21:24,875 - synapse.storage.background_updates - 227 - INFO - background_updates-0 - Starting update batch on background update 'delete_old_current_state_events'
2020-07-09 09:21:24,884 - synapse.storage.data_stores.main.state - 383 - INFO - background_updates-0 - Deleting current state left rooms: {'!TKNXvIHTmAsiwecNYO:localhost:8800'}
2020-07-09 09:21:24,887 - synapse.storage.background_updates - 269 - INFO - background_updates-0 - Running background update 'delete_old_current_state_events'. Processed 100 items in 10ms. (total_rate=0/ms, current_rate=0/ms, total_updated=0, batch_size=100)
2020-07-09 09:21:24,923 - synapse.handlers.message - 829 - WARNING - POST-125 - Denying new event <FrozenEventV3 event_id='$It51s0T01y0gj9QnUhghGAfoC5H762-I0Dn-q9fm59w', type='m.room.member', state_key='@anon-20200709_092116-20:localhost:8800'> because 403: No create event in auth events
2020-07-09 09:21:24,923 - synapse.http.server - 74 - INFO - POST-125 - <SynapseRequest at 0x7f9a25cb5ac8 method='POST' uri='/_matrix/client/r0/createRoom?access_token=<redacted>' clientproto='HTTP/1.1' site=8800> SynapseError: 403 - No create event in auth events
2020-07-09 09:21:24,924 - synapse.access.https.8800 - 312 - INFO - POST-125 - 127.0.0.1 - 8800 - {@anon-20200709_092116-20:localhost:8800} Processed request: 0.081sec/0.000sec (0.007sec, 0.003sec) (0.039sec/0.008sec/15) 66B 403 "POST /_matrix/client/r0/createRoom?access_token=<redacted> HTTP/1.1" "Perl + Net::Async::HTTP/0.47" [0 dbevts]
```

* `createRoom` makes the `m.room.create` event and adds it to `current_state_events`
* `delete_old_current_state_events` runs, and sees a room with no active membership events
* `createRoom` tries to add the creator's membership event, but there is no create event in the current state, leading to the observed error.


Discussion:
https://buildkite.com/matrix-dot-org/synapse/builds/9447#fcb142a2-3d35-4856-829f-2c9e404fbce4 has a similar cause.

Resolving patch:
diff --git a/synapse/storage/data_stores/main/state.py b/synapse/storage/data_stores/main/state.py
--- a/synapse/storage/data_stores/main/state.py
+++ b/synapse/storage/data_stores/main/state.py
@@ -353,6 +353,7 @@ async def _background_remove_left_rooms(self, progress, batch_size):
         last_room_id = progress.get("last_room_id", "")
 
         def _background_remove_left_rooms_txn(txn):
+            # get a batch of room ids to consider
             sql = """
                 SELECT DISTINCT room_id FROM current_state_events
                 WHERE room_id > ? ORDER BY room_id LIMIT ?
@@ -363,24 +364,68 @@ def _background_remove_left_rooms_txn(txn):
             if not room_ids:
                 return True, set()
 
+            ###########################################################################
+            #
+            # exclude rooms where we have active members
+
             sql = """
                 SELECT room_id
-                FROM current_state_events
+                FROM local_current_membership
                 WHERE
                     room_id > ? AND room_id <= ?
-                    AND type = 'm.room.member'
                     AND membership = 'join'
-                    AND state_key LIKE ?
                 GROUP BY room_id
             """
 
-            txn.execute(sql, (last_room_id, room_ids[-1], "%:" + self.server_name))
-
+            txn.execute(sql, (last_room_id, room_ids[-1]))
             joined_room_ids = {row[0] for row in txn}
+            to_delete = set(room_ids) - joined_room_ids
+
+            ###########################################################################
+            #
+            # exclude rooms which we are in the process of constructing; these otherwise
+            # qualify as "rooms with no local users", and would have their
+            # forward extremities cleaned up.
+
+            # the following query will return a list of rooms which have forward
+            # extremities that are *not* also the create event in the room - ie
+            # those that are not being created currently.
+
+            sql = """
+                SELECT DISTINCT efe.room_id
+                FROM event_forward_extremities efe
+                LEFT JOIN current_state_events cse ON
+                    cse.event_id = efe.event_id
+                    AND cse.type = 'm.room.create'
+                    AND cse.state_key = ''
+                WHERE
+                    cse.event_id IS NULL
+                    AND efe.room_id > ? AND efe.room_id <= ?
+            """
+
+            txn.execute(sql, (last_room_id, room_ids[-1]))
+
+            # build a set of those rooms within `to_delete` that do not appear in
+            # the above, leaving us with the rooms in `to_delete` that *are* being
+            # created.
+            creating_rooms = to_delete.difference(row[0] for row in txn)
+            logger.info("skipping rooms which are being created: %s", creating_rooms)
+
+            # now remove the rooms being created from the list of those to delete.
+            #
+            # (we could have just taken the intersection of `to_delete` with the result
+            # of the sql query, but it's useful to be able to log `creating_rooms`; and
+            # having done so, it's quicker to remove the (few) creating rooms from
+            # `to_delete` than it is to form the intersection with the (larger) list of
+            # not-creating-rooms)
+
+            to_delete -= creating_rooms
 
-            left_rooms = set(room_ids) - joined_room_ids
+            ###########################################################################
+            #
+            # now clear the state for the rooms
 
-            logger.info("Deleting current state left rooms: %r", left_rooms)
+            logger.info("Deleting current state left rooms: %r", to_delete)
 
             # First we get all users that we still think were joined to the
             # room. This is so that we can mark those device lists as
@@ -391,7 +436,7 @@ def _background_remove_left_rooms_txn(txn):
                 txn,
                 table="current_state_events",
                 column="room_id",
-                iterable=left_rooms,
+                iterable=to_delete,
                 keyvalues={"type": EventTypes.Member, "membership": Membership.JOIN},
                 retcols=("state_key",),
             )
@@ -403,7 +448,7 @@ def _background_remove_left_rooms_txn(txn):
                 txn,
                 table="current_state_events",
                 column="room_id",
-                iterable=left_rooms,
+                iterable=to_delete,
                 keyvalues={},
             )
 
@@ -411,7 +456,7 @@ def _background_remove_left_rooms_txn(txn):
                 txn,
                 table="event_forward_extremities",
                 column="room_id",
-                iterable=left_rooms,
+                iterable=to_delete,
                 keyvalues={},
             )
 


Test patch: