Repository: matrix-org/synapse

Issue:
Split fed worker & sender config means destination retry sched doesn't invalidate when you receive traffic



Discussion:
So this is a general problem for any cache that gets invalidated on the workers, since only invalidations that happen on master get correctly propagated to the workers. However, currently there should be very few caches that get invalidated on the workers.

The destination retry cache was allowed to be invalidated on the workers on the assumption that enough federation traffic would happen on master that the cache would be correctly invalidated fairly quickly. Now that we've moved more traffic off of master this is no longer true.

@hawkowl has a number of suggestions on the way forward:

1. Remove the cache if you're a worker
2. Make the cache have a TTL
3. If you're a worker, have the cache backed by something like memcached

None of which are hugely palatable atm: 1) would unacceptably increase DB load, 2) feels icky and may work for the current situation but feels like a bit of a footgun and 3) sounds like a fair amount of effort in terms of implementation and operations.

Another alternative may be to use postgres's inbuilt `LISTEN`/`NOTIFY` pubsub mechanism to communicate cache invalidations to the master. This is a bit annoying as it'll take a bit of implementing and is asymmetric from master to worker invalidation, but has the advantage over option 3 in that it'll be less work and less ongoing operations.

@matrix-org/synapse-core Thoughts? 
For the federation mess surely we just put a TTL on the cache for now - 5 mins or whatever rather than having the federation blackholed indefinitely?
(LISTEN/NOTIFY could be fun as a future adventure tho)
Possibly, but like I said I'm disinclined to continue putting footguns in our code given that this sort of hacky solution is *exactly* what has led to this mess. As a stop gap measure it may be the best thing to do right now but I'm not comfortable sticking a band aid on this and moving on.
Related to cache backing: https://github.com/matrix-org/synapse/issues/2123
any reason not to communicate the cache invalidation from worker to master via the existing replication channels?

[fwiw if we do end up deciding to build something new, I'd much rather we use something like memcached than postgres pubsub, for many of the reasons mentioned in #2123, but really just because we shouldn't be building our own wheel]
> any reason not to communicate the cache invalidation from worker to master via the existing replication channels?

The worker -> master communication is done via HTTP, which is actually probably fine if we batch it up for `get_destination_retry_timings` though I wouldn't necessarily want to rely on it for other caches

> [fwiw if we do end up deciding to build something new, I'd much rather we use something like memcached than postgres pubsub, for many of the reasons mentioned in #2123, but really just because we shouldn't be building our own wheel]

Historically the problem is that external cache services just don't match the use cases of our caches, in particular CPU usage and latency (I believe there are places that assume that things are probably going to be quick due to things being in cache). While I agree that we shouldn't be reinventing the wheel, requiring memcached/redis/etc just for a communication layer feels a bit overkill.

OTOH, it might be useful for the different problem of reducing DB load.
After some discussion, the conclusion was that (for now) we'll just remove the cache for `get_destination_retry_timings`, since on matrix.org they're only hit ~50-100Hz

Resolving patch:
diff --git a/synapse/storage/transactions.py b/synapse/storage/transactions.py
--- a/synapse/storage/transactions.py
+++ b/synapse/storage/transactions.py
@@ -23,7 +23,6 @@
 from twisted.internet import defer
 
 from synapse.metrics.background_process_metrics import run_as_background_process
-from synapse.util.caches.descriptors import cached
 
 from ._base import SQLBaseStore, db_to_json
 
@@ -156,7 +155,6 @@ def delivered_txn(self, transaction_id, destination, code, response_dict):
         """
         pass
 
-    @cached(max_entries=10000)
     def get_destination_retry_timings(self, destination):
         """Gets the current retry timings (if any) for a given destination.
 
@@ -198,8 +196,6 @@ def set_destination_retry_timings(self, destination,
             retry_interval (int) - how long until next retry in ms
         """
 
-        # XXX: we could chose to not bother persisting this if our cache thinks
-        # this is a NOOP
         return self.runInteraction(
             "set_destination_retry_timings",
             self._set_destination_retry_timings,
@@ -212,10 +208,6 @@ def _set_destination_retry_timings(self, txn, destination,
                                        retry_last_ts, retry_interval):
         self.database_engine.lock_table(txn, "destinations")
 
-        self._invalidate_cache_and_stream(
-            txn, self.get_destination_retry_timings, (destination,)
-        )
-
         # We need to be careful here as the data may have changed from under us
         # due to a worker setting the timings.
 


Test patch: