Repository: matrix-org/synapse

Issue:
Prefill and invalidate `getEvent` cache are happening out of order when persisting transactions.
While working on learning how the `_get_event_cache` work, I spotted a strange condition of invalidation and prefill being added to a transaction to be run after the transaction was finished, but it was running the prefill, then invalidating it. Subsequently, it would just re-get the event the next time it was retrieved from the database.

In `PersistEventsStore.persist_event_txn()`:
https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/databases/main/events.py#L393-L399

Which uses [`txn.call_after()`](https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/database.py#L284) and [`txn.async_call_after()`](https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/database.py#L305) under the [hood](https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/databases/main/events_worker.py#L867).

Lower down(inside the [`_update_metadata_tables_txn()`](https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/databases/main/events.py#L1598) which then calls [`add_to_cache()`](https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/databases/main/events.py#L1697)) near the end, there is a database read of the events and then the prefill bits(which are inside `add_to_cache()`):
https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/databases/main/events.py#L1732-L1739

This uses `txn.async_call_after()` (as you can see at the bottom)

The docustring for `LoggingTransaction` has the information on what happens with `txn.call_after()` and `txn.async_call_after()`:
https://github.com/matrix-org/synapse/blob/fcc3ca37e1b404981d9a0d6f2708e14407775b97/synapse/storage/database.py#L245-L252

...which summarizes nicely as '`async_call_after()` runs before `call_after()`'.
So:
1. The invalidation of events is added with `call_after()` and `async_call_after()`
2. The prefill is added with `async_call_after()`
3. The transaction finishes
4. The async invalidation runs, purging the event from the cache
5. The prefill runs(because it's async, and those run before the non-async)
6. The non-async invalidation runs, purging the cache of the event we just loaded.
7. ...time passes(probably not much)
8. The event is reloaded from the database, because it's not in the cache.

Which, I think, means it's hitting the database for a read twice as often as necessary.

Bonus points attempt:
These were added to assist in the preparatory work for an external caching system, and for that reason I want to take that into account when looking at this. The external system will require the code to run async/await, so this will have to be maintained in any fix. I recommend an additional method to be added to `AsyncLruCache` that will explicitly handle setting a value for the external cache. However that isn't actually implemented yet(or upstreamed, whatever), so will no-op just as `get_external` does now.


Discussion:

Resolving patch:
diff --git a/synapse/storage/databases/main/events.py b/synapse/storage/databases/main/events.py
--- a/synapse/storage/databases/main/events.py
+++ b/synapse/storage/databases/main/events.py
@@ -1729,13 +1729,22 @@ def _add_to_cache(
             if not row["rejects"] and not row["redacts"]:
                 to_prefill.append(EventCacheEntry(event=event, redacted_event=None))
 
-        async def prefill() -> None:
+        async def external_prefill() -> None:
             for cache_entry in to_prefill:
-                await self.store._get_event_cache.set(
+                await self.store._get_event_cache.set_external(
                     (cache_entry.event.event_id,), cache_entry
                 )
 
-        txn.async_call_after(prefill)
+        def local_prefill() -> None:
+            for cache_entry in to_prefill:
+                self.store._get_event_cache.set_local(
+                    (cache_entry.event.event_id,), cache_entry
+                )
+
+        # The order these are called here is not as important as knowing that after the
+        # transaction is finished, the async_call_after will run before the call_after.
+        txn.async_call_after(external_prefill)
+        txn.call_after(local_prefill)
 
     def _store_redaction(self, txn: LoggingTransaction, event: EventBase) -> None:
         assert event.redacts is not None
diff --git a/synapse/storage/databases/main/events_worker.py b/synapse/storage/databases/main/events_worker.py
--- a/synapse/storage/databases/main/events_worker.py
+++ b/synapse/storage/databases/main/events_worker.py
@@ -883,7 +883,7 @@ def invalidate_get_event_cache_after_txn(
 
     async def _invalidate_async_get_event_cache(self, event_id: str) -> None:
         """
-        Invalidates an event in the asyncronous get event cache, which may be remote.
+        Invalidates an event in the asynchronous get event cache, which may be remote.
 
         Arguments:
             event_id: the event ID to invalidate
diff --git a/synapse/util/caches/lrucache.py b/synapse/util/caches/lrucache.py
--- a/synapse/util/caches/lrucache.py
+++ b/synapse/util/caches/lrucache.py
@@ -842,7 +842,13 @@ def get_local(
         return self._lru_cache.get(key, update_metrics=update_metrics)
 
     async def set(self, key: KT, value: VT) -> None:
-        self._lru_cache.set(key, value)
+        # This will add the entries in the correct order, local first external second
+        self.set_local(key, value)
+        await self.set_external(key, value)
+
+    async def set_external(self, key: KT, value: VT) -> None:
+        # This method should add an entry to any configured external cache, in this case noop.
+        pass
 
     def set_local(self, key: KT, value: VT) -> None:
         self._lru_cache.set(key, value)


Test patch:
diff --git a/tests/storage/databases/main/test_events_worker.py b/tests/storage/databases/main/test_events_worker.py
--- a/tests/storage/databases/main/test_events_worker.py
+++ b/tests/storage/databases/main/test_events_worker.py
@@ -139,6 +139,55 @@ def test_persisting_event_invalidates_cache(self) -> None:
             # That should result in a single db query to lookup
             self.assertEqual(ctx.get_resource_usage().db_txn_count, 1)
 
+    def test_persisting_event_prefills_get_event_cache(self) -> None:
+        """
+        Test to make sure that the `_get_event_cache` is prefilled after we persist an
+        event and returns the updated value.
+        """
+        event, event_context = self.get_success(
+            create_event(
+                self.hs,
+                room_id=self.room_id,
+                sender=self.user,
+                type="test_event_type",
+                content={"body": "conflabulation"},
+            )
+        )
+
+        # First, check `_get_event_cache` for the event we just made
+        # to verify it's not in the cache.
+        res = self.store._get_event_cache.get_local((event.event_id,))
+        self.assertEqual(res, None, "Event was cached when it should not have been.")
+
+        with LoggingContext(name="test") as ctx:
+            # Persist the event which should invalidate then prefill the
+            # `_get_event_cache` so we don't return stale values.
+            # Side Note: Apparently, persisting an event isn't a transaction in the
+            # sense that it is recorded in the LoggingContext
+            persistence = self.hs.get_storage_controllers().persistence
+            assert persistence is not None
+            self.get_success(
+                persistence.persist_event(
+                    event,
+                    event_context,
+                )
+            )
+
+            # Check `_get_event_cache` again and we should see the updated fact
+            # that we now have the event cached after persisting it.
+            res = self.store._get_event_cache.get_local((event.event_id,))
+            self.assertEqual(res.event, event, "Event not cached as expected.")  # type: ignore
+
+            # Try and fetch the event from the database.
+            self.get_success(self.store.get_event(event.event_id))
+
+            # Verify that the database hit was avoided.
+            self.assertEqual(
+                ctx.get_resource_usage().evt_db_fetch_count,
+                0,
+                "Database was hit, which would not happen if event was cached.",
+            )
+
     def test_invalidate_cache_by_room_id(self) -> None:
         """
         Test to make sure that all events associated with the given `(room_id,)`