Repository: matrix-org/synapse

Issue:
completely broken room after purge_room/rejoin
when you purge a room (via the [delete room api](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#delete-room-api)), we do not clear the in-memory caches. Then, when you rejoin, we have a bunch of the events in the cache, so do not bother to re-persist them.

This leads to very bad brokenness, like `state_groups_state` referring to events which do not exist.

(A workaround is of course to restart synapse after purging and before rejoining)


Discussion:
Of course, if we had foreign key constraints, the failure mode would be a lot less insidious :/
Cache invalidation in this case is a bit tricky, because we don't actually know which events we're purging, and most of the caches in question aren't structured in a way that makes it easy to purge them by room id.
~~oh and thanks to #6547 you can't even re-purge your way out of the problem :(~~ now fixed
I see that one (only?) affected cache is `EventsWorkerStore._get_event_cache`, which is an instance of `AsyncLruCache`/`LruCache` with a cache type of `dict`. Could we switch this to a [`DictionaryCache`](https://github.com/matrix-org/synapse/blob/0b87eb8e0c8e2dd4a426005dce53dfdd57282475/synapse/util/caches/dictionary_cache.py) instead? This is a cache-type which allows you to efficiently delete "trees" of caches. Such trees could be constructed with the room ID as the root node, and all events as children. Then we could efficiently invalidate the cache for all events in a room while continuing to maintain quick lookup times. We currently use `DictionaryCache` to back `state_group` and `state_group_members` caches.

I'm not sure how this change will affect external cache support, though I don't think that's merged yet, so a migration would not yet be needed.

Edit: the downside is that you'd need to know the room ID when checking the cache to get an efficient lookup.
An alternative could be to maintain a separate dict cache of `room_id`->`event_id` which is populated upon pulling items from the database, and could be used to quickly invalidate a room's events from `_get_event_cache`.
I've made good headway into converting `EventWorkerStore._get_event_cache` into a new type of cache following the methodology in the above comment (running a second `room_id` -> `event_id` dict and keeping it in sync) as a backing store for LruCache. It works, in fact.

However, it is not the only event-related cache we need to deal with here. Both `EventWorkerStore._event_ref` (weak event in memory store) and `EventWorkerStore._current_event_fetches` (current ongoing event fetches) also hold references to events that may have been removed from the database.

I understand `EventWorkerStore._current_event_fetches`, and we could possibly just iterate over this list and remove events with a matching room ID (if it never grows too large?), but `EventWorkerStore._event_ref` confuses me. Why have this dict if we already have an in-memory cache for events? It has entries inserted and invalidated at the same points the `EventWorkerStore._get_event_cache` does. Does it show noticeable performance gains?

Having a second layer which keeps entries even after they've been removed from the cache just seems like a recipe for races.
The PRs that introduced `EventWorkerStore._event_ref` and `EventWorkerStore._current_event_fetches` are:

* https://github.com/matrix-org/synapse/pull/10119 (`_current_event_fetches`)
* https://github.com/matrix-org/synapse/pull/10533 (`_event_ref`)

and they help in understanding why these caches were implemented. `_current_event_fetches` helps with de-duplicating concurrent requests for the same event ID, while `_event_ref` ensures that there is only one copy of an event in memory at a given time.

We need to invalidate these by room ID as well, though this is tricky and raises some question. Unfortunately neither of these caches are a strict subset of the contents of `EventWorkerStore._get_event_cache`, so we can't simply use the results of that to know which event IDs to invalidate in the other caches. ...although it does beg the question, if `_event_ref` is a map of all events kept in memory, does that mean `_get_event_cache` is a strict subset of `_event_ref`?

If not, and thus we can't re-use the work from invalidating one cache, then that leaves the solution of having yet more room ID -> event ID maps that are maintained (one for each cache) or simply iterating over all entries, grabbing the room ID and invalidating those that match. That may not be very performant, though I haven't done any performance testing to verify. I'd like to say purging a room doesn't happen very often, but it will if we introduce purge-room-on-forget (https://github.com/matrix-org/synapse/issues/4720, the feature driving this investigation).

I'm also not sure if invalidating `_current_event_fetches` will introduce any races. We [pop the deferred](https://github.com/matrix-org/synapse/blob/d3d9ca156e323fe194b1bcb1af1628f65a2f3c1c/synapse/storage/databases/main/events_worker.py#L785) which I presume allows it to continue, but not be re-used from that point on. I assume this is mostly fine, since the request came in before the purge was initiated, so it should theoretically still return the event, even if that event is returned after it has been purged from the database.

@erikjohnston I'm interested in your thoughts specifically, since you authored much of this code.
@anoadragon453 I haven't read this thread but maybe you're interested in what the `have_seen_event` cache does using the `TreeCache`.

https://github.com/matrix-org/synapse/blob/efabf44c7652095a0e3d9d9083fc8359cdde3854/synapse/storage/databases/main/events_worker.py#L1527-L1530

You can see that we invalidate via `(room_id, event_id)` and via `(room_id, )`:

https://github.com/matrix-org/synapse/blob/2b522cceb6cb4919a3a92e99917e7199d9feaeb1/synapse/storage/databases/main/purge_events.py#L304-L306

https://github.com/matrix-org/synapse/blob/2b522cceb6cb4919a3a92e99917e7199d9feaeb1/synapse/storage/databases/main/purge_events.py#L485

---

Found while tooling around on https://github.com/matrix-org/synapse/issues/13856 and thought you might be interested.
Hey @MadLittleMods, thanks for the suggestion! Unfortunately `TreeCache` won't work for this particular use case as we often don't know the room ID when querying the `get_event` cache.

Resolving patch:
diff --git a/synapse/storage/databases/main/events_worker.py b/synapse/storage/databases/main/events_worker.py
--- a/synapse/storage/databases/main/events_worker.py
+++ b/synapse/storage/databases/main/events_worker.py
@@ -1502,21 +1502,15 @@ async def _have_seen_events_dict(
         Returns:
              a dict {event_id -> bool}
         """
-        # if the event cache contains the event, obviously we've seen it.
-
-        cache_results = {
-            event_id
-            for event_id in event_ids
-            if await self._get_event_cache.contains((event_id,))
-        }
-        results = dict.fromkeys(cache_results, True)
-        remaining = [
-            event_id for event_id in event_ids if event_id not in cache_results
-        ]
-        if not remaining:
-            return results
+        # TODO: We used to query the _get_event_cache here as a fast-path before
+        #  hitting the database. For if an event were in the cache, we've presumably
+        #  seen it before.
+        #
+        #  But this is currently an invalid assumption due to the _get_event_cache
+        #  not being invalidated when purging events from a room. The optimisation can
+        #  be re-added after https://github.com/matrix-org/synapse/issues/13476
 
-        def have_seen_events_txn(txn: LoggingTransaction) -> None:
+        def have_seen_events_txn(txn: LoggingTransaction) -> Dict[str, bool]:
             # we deliberately do *not* query the database for room_id, to make the
             # query an index-only lookup on `events_event_id_key`.
             #
@@ -1524,16 +1518,17 @@ def have_seen_events_txn(txn: LoggingTransaction) -> None:
 
             sql = "SELECT event_id FROM events AS e WHERE "
             clause, args = make_in_list_sql_clause(
-                txn.database_engine, "e.event_id", remaining
+                txn.database_engine, "e.event_id", event_ids
             )
             txn.execute(sql + clause, args)
             found_events = {eid for eid, in txn}
 
             # ... and then we can update the results for each key
-            results.update({eid: (eid in found_events) for eid in remaining})
+            return {eid: (eid in found_events) for eid in event_ids}
 
-        await self.db_pool.runInteraction("have_seen_events", have_seen_events_txn)
-        return results
+        return await self.db_pool.runInteraction(
+            "have_seen_events", have_seen_events_txn
+        )
 
     @cached(max_entries=100000, tree=True)
     async def have_seen_event(self, room_id: str, event_id: str) -> bool:


Test patch:
diff --git a/tests/storage/databases/main/test_events_worker.py b/tests/storage/databases/main/test_events_worker.py
--- a/tests/storage/databases/main/test_events_worker.py
+++ b/tests/storage/databases/main/test_events_worker.py
@@ -90,18 +90,6 @@ def test_simple(self):
             self.assertEqual(res, {self.event_ids[0]})
             self.assertEqual(ctx.get_resource_usage().db_txn_count, 0)
 
-    def test_query_via_event_cache(self):
-        # fetch an event into the event cache
-        self.get_success(self.store.get_event(self.event_ids[0]))
-
-        # looking it up should now cause no db hits
-        with LoggingContext(name="test") as ctx:
-            res = self.get_success(
-                self.store.have_seen_events(self.room_id, [self.event_ids[0]])
-            )
-            self.assertEqual(res, {self.event_ids[0]})
-            self.assertEqual(ctx.get_resource_usage().db_txn_count, 0)
-
     def test_persisting_event_invalidates_cache(self):
         """
         Test to make sure that the `have_seen_event` cache