Repository: matrix-org/synapse

Issue:
Federated room cannot be joined again when left and deleted before
### Description

When a federated room (e.g. #synapse:matrix.org) is joined from self hosted federated homeserver, then left, deleted via API from self homeserver, then rejoin fails.

### Steps to reproduce

1. Join some room (e.g. #synapse:matrix.org) from self hosted homeserver (same version of Synapse 1.27.0).
2. Leave that room
3. Delete that room using API `/_synapse/admin/v1/rooms/[room_id]/delete`
4. Try to join that room again
5. It fails with following error...
```
synapse.http.server: [POST-10040] Failed handle request via 'JoinRoomAliasServlet': <XForwardedForRequest at 0x7f57646fa970 method='POST' uri='/_matrix/client/r0/join/%23synapse%3Amatrix.org' clientproto='HTTP/1.1' site='8008'>
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/synapse/http/server.py", line 252, in _async_render_wrapper
        callback_return = await self._async_render(request)
      File "/usr/lib/python3.9/site-packages/synapse/http/server.py", line 430, in _async_render
        callback_return = await raw_callback_return
      File "/usr/lib/python3.9/site-packages/synapse/rest/client/v1/room.py", line 301, in on_POST
        await self.room_member_handler.update_membership(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 333, in update_membership
        result = await self.update_membership_locked(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 549, in update_membership_locked
        remote_join_response = await self._remote_join(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 1091, in _remote_join
        event_id, stream_id = await self.federation_handler.do_invite_join(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 1400, in do_invite_join
        max_stream_id = await self._persist_auth_tree(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 2050, in _persist_auth_tree
        await self.persist_events_and_notify(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 2925, in persist_events_and_notify
        events, max_stream_token = await self.storage.persistence.persist_events(
      File "/usr/lib/python3.9/site-packages/synapse/storage/persist_events.py", line 262, in persist_events
        ret_vals = await make_deferred_yieldable(
    twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'psycopg2.errors.UniqueViolation'>: duplicate key value violates unique constraint "event_auth_chains_pkey"
    DETAIL:  Key (event_id)=($e9U026auDHIgaZPAqlblvPupACjl7jcZDblP970dJPs) already exists.

    /usr/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py:208:run
    --- <exception caught here> ---
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:172:handle_queue_loop
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:324:persisting_queue
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:532:_persist_events
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:171:_persist_events_and_state_updates
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:661:runInteraction
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:744:runWithConnection
    /usr/lib64/python3.9/site-packages/twisted/python/threadpool.py:250:inContext
    /usr/lib64/python3.9/site-packages/twisted/python/threadpool.py:266:<lambda>
    /usr/lib64/python3.9/site-packages/twisted/python/context.py:122:callWithContext
    /usr/lib64/python3.9/site-packages/twisted/python/context.py:85:callWithContext
    /usr/lib64/python3.9/site-packages/twisted/enterprise/adbapi.py:306:_runWithConnection
    /usr/lib64/python3.9/site-packages/twisted/python/compat.py:464:reraise
    /usr/lib64/python3.9/site-packages/twisted/enterprise/adbapi.py:297:_runWithConnection
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:739:inner_func
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:539:new_transaction
    /usr/lib/python3.9/site-packages/synapse/logging/utils.py:71:wrapped
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:379:_persist_events_txn
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:472:_persist_event_auth_chain_txn
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:630:_add_chain_cover_index
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:896:simple_insert_many_txn
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:274:execute_batch
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:319:_do_execute
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:274:<lambda>
    /usr/lib64/python3.9/site-packages/psycopg2/extras.py:1209:execute_batch
    ]]
```

The room should be rejoined without any problem.

### Version information

- **Homeserver**: matrix.bolesiny.net
- **Version**: 1.27.0

- **Install method**: Fedora rpm package

- **Platform**: Fedora 33



Discussion:
Related #9418
@waclaw66 This seems to be a duplicate of #9418. I'm going to close it unless you feel there's a reason that it is not.
Maybe it's the same bug causing the problem, but those are different cases, which could lead to it.
You can close it in favour of #9418, there is a reference, in case the fix wouldn't work for my problem I can reopen it.

> 
> 
> @waclaw66 This seems to be a duplicate of #9418. I'm going to close it unless you feel there's a reason that it is not.


Ah, I see that this is due to leaving while the other is due to a failure to join. 👍  Hopefully the fix is the same!
Is there any hotfix? Leaving and joining the room again works properly, but when a room is deleted via API and joined again it crashes as shown. So there supposed to be some remain of that deleted room in database which causes that crash after joining it again.
When you deleted the room you also purged it I'm guessing?

I suspect that [`purge_room_state`](https://github.com/matrix-org/synapse/blob/0b5c967813a8e5f3338b6ecd5f3742e91b8a100b/synapse/storage/databases/state/store.py#L617) needs to be updated to remove from `event_auth_chains` also?
Right, I purged the history, then I left and finally deleted it.

> 
> 
> When you deleted the room you also purged it I'm guessing?
> 
> I suspect that [`purge_room_state`](https://github.com/matrix-org/synapse/blob/0b5c967813a8e5f3338b6ecd5f3742e91b8a100b/synapse/storage/databases/state/store.py#L617) needs to be updated to remove from `event_auth_chains` also?


That's strange
`select * from state_groups_state where room_id='!ehXvUhWNASUkSLvAGP:matrix.org'`
gives me more than 300k records, but
`POST http://localhost:8008/_synapse/admin/v1/purge_history/!ehXvUhWNASUkSLvAGP:matrix.org`
with data
`{ "delete_local_events": true, "purge_up_to_ts": 1614270664000 }`
returns
`{"errcode":"M_NOT_FOUND","error":"there is no event to be purged"}`
`purge_history` is different than purging the room (it purges the events before a certain time). I was asking whether you gave `{"purge": true}` when calling `/_synapse/admin/v1/rooms/[room_id]/delete`.
Understood, I used synapse-admin to delete the room finally.
It seems, it doesn't purge...
https://github.com/Awesome-Technologies/synapse-admin/blob/684c44e47057e42fb217688e08706e9f2e6767a7/src/synapse/dataProvider.js#L69

> 
> 
> `purge_history` is different than purging the room (it purges the events before a certain time). I was asking whether you gave `{"purge": true}` when calling `/_synapse/admin/v1/rooms/[room_id]/delete`.


Oh, interesting. FWIW, in #9418, I probably purged/deleted the room at some point in the past before trying to rejoin it (and then lrunning into this issue).
The question is how to fix it. I suppose the patch from #9498 will fix it for the future. Do I need to run some sql commands to fix current state?
> The question is how to fix it. I suppose the patch from #9498 will fix it for the future. Do I need to run some sql commands to fix current state?

My goal in there would be to fix it for previously purged rooms too, if possible. If not you can hopefully run some of the commands manually.

> Oh, interesting. FWIW, in #9418, I probably purged/deleted the room at some point in the past before trying to rejoin it (and then lrunning into this issue).

Sounds like these are probably duplicates then.

Resolving patch:
diff --git a/synapse/storage/databases/main/purge_events.py b/synapse/storage/databases/main/purge_events.py
--- a/synapse/storage/databases/main/purge_events.py
+++ b/synapse/storage/databases/main/purge_events.py
@@ -28,7 +28,10 @@ class PurgeEventsStore(StateGroupWorkerStore, SQLBaseStore):
     async def purge_history(
         self, room_id: str, token: str, delete_local_events: bool
     ) -> Set[int]:
-        """Deletes room history before a certain point
+        """Deletes room history before a certain point.
+
+        Note that only a single purge can occur at once, this is guaranteed via
+        a higher level (in the PaginationHandler).
 
         Args:
             room_id:
@@ -52,7 +55,9 @@ async def purge_history(
             delete_local_events,
         )
 
-    def _purge_history_txn(self, txn, room_id, token, delete_local_events):
+    def _purge_history_txn(
+        self, txn, room_id: str, token: RoomStreamToken, delete_local_events: bool
+    ) -> Set[int]:
         # Tables that should be pruned:
         #     event_auth
         #     event_backward_extremities
@@ -103,7 +108,7 @@ def _purge_history_txn(self, txn, room_id, token, delete_local_events):
         if max_depth < token.topological:
             # We need to ensure we don't delete all the events from the database
             # otherwise we wouldn't be able to send any events (due to not
-            # having any backwards extremeties)
+            # having any backwards extremities)
             raise SynapseError(
                 400, "topological_ordering is greater than forward extremeties"
             )
@@ -154,7 +159,7 @@ def _purge_history_txn(self, txn, room_id, token, delete_local_events):
 
         logger.info("[purge] Finding new backward extremities")
 
-        # We calculate the new entries for the backward extremeties by finding
+        # We calculate the new entries for the backward extremities by finding
         # events to be purged that are pointed to by events we're not going to
         # purge.
         txn.execute(
@@ -296,7 +301,7 @@ async def purge_room(self, room_id: str) -> List[int]:
             "purge_room", self._purge_room_txn, room_id
         )
 
-    def _purge_room_txn(self, txn, room_id):
+    def _purge_room_txn(self, txn, room_id: str) -> List[int]:
         # First we fetch all the state groups that should be deleted, before
         # we delete that information.
         txn.execute(
@@ -310,6 +315,31 @@ def _purge_room_txn(self, txn, room_id):
 
         state_groups = [row[0] for row in txn]
 
+        # Get all the auth chains that are referenced by events that are to be
+        # deleted.
+        txn.execute(
+            """
+            SELECT chain_id, sequence_number FROM events
+            LEFT JOIN event_auth_chains USING (event_id)
+            WHERE room_id = ?
+            """,
+            (room_id,),
+        )
+        referenced_chain_id_tuples = list(txn)
+
+        logger.info("[purge] removing events from event_auth_chain_links")
+        txn.executemany(
+            """
+            DELETE FROM event_auth_chain_links WHERE
+            (origin_chain_id = ? AND origin_sequence_number = ?) OR
+            (target_chain_id = ? AND target_sequence_number = ?)
+            """,
+            (
+                (chain_id, seq_num, chain_id, seq_num)
+                for (chain_id, seq_num) in referenced_chain_id_tuples
+            ),
+        )
+
         # Now we delete tables which lack an index on room_id but have one on event_id
         for table in (
             "event_auth",
@@ -319,6 +349,8 @@ def _purge_room_txn(self, txn, room_id):
             "event_reference_hashes",
             "event_relations",
             "event_to_state_groups",
+            "event_auth_chains",
+            "event_auth_chain_to_calculate",
             "redactions",
             "rejections",
             "state_events",
diff --git a/synapse/storage/purge_events.py b/synapse/storage/purge_events.py
--- a/synapse/storage/purge_events.py
+++ b/synapse/storage/purge_events.py
@@ -73,9 +73,6 @@ async def _find_unreferenced_groups(self, state_groups: Set[int]) -> Set[int]:
         Returns:
             The set of state groups that can be deleted.
         """
-        # Graph of state group -> previous group
-        graph = {}
-
         # Set of events that we have found to be referenced by events
         referenced_groups = set()
 
@@ -111,8 +108,6 @@ async def _find_unreferenced_groups(self, state_groups: Set[int]) -> Set[int]:
             next_to_search |= prevs
             state_groups_seen |= prevs
 
-            graph.update(edges)
-
         to_delete = state_groups_seen - referenced_groups
 
         return to_delete


Test patch: