Repository: matrix-org/synapse

Issue:
cache sync results for a while after the request completes
Both @lampholder and @Ralith have reported similar sounding problems where initial /sync takes ~5x longer to complete from Riot than it should.

@lampholder mentioned that he tried calling initial /sync directly from curl against his HS (bypassing nginx) without a clientside timeout and it took 3 mins, but when called by Riot (which times out reqs every 90s or so to avoid network glitches wedging a request) it takes ~15 mins.  Meanwhile, @Ralith (on an underpowered server with bad IO) takes 15 mins(!) to generate the sync when called directly, but >1h when called from Riot :/

I haven't seen logs from either, but my theory is that if the client times out the /sync request **after** the server has generated the response, but **before** it has sent it to the client, then the next time the client retries, the /sync will be calculated from scratch, and the failure mode may continue.

It looks like the current behaviour on develop caches sync requests via response_cache which blocks additional requests for the same sync parameters until any ongoing one completes, and then they all return with the same result: https://github.com/matrix-org/synapse/blob/b1580f50fea05ce02dc4663adb23704b33e30bf3/synapse/handlers/sync.py#L219-L225.

However, on the matrix-org-hotfixes branch @richvdh changed it to cache sync responses for an additional 2 minutes, so that any retrying clients will get the cached copy rather than starting from scratch: https://github.com/matrix-org/synapse/commit/0ca2857baab2d241a740a6ba5e8ed36d17756338.  This is similar to the old /initialSync snapshot_cache from https://github.com/matrix-org/synapse/pull/457 which never seems to have made it over to /sync.

I'm not seeing any additional /sync response cache?

If my theory is correct, it looks like this hotfix needs to be ported back into develop, and sytest fixed so that it doesn't fail if initial /sync returns stale data (perhaps we could add a null filter on or something as a cachebuster).

Another explanation for this could be if Twisted is correctly cancelling the ongoing /sync when the client goes away, thus throwing away the cached response.  The same fix would mitigate that, however.


Discussion:
ftr I did observe this behaviour on my personal homeserver a number of times. It was most evident when a client like riot-android decided it needed to resync the world - the logs showed requests coming in and stacking up, holding the lock quite effectively. After about 5-10 minutes the lock would clear and tens of requests would start getting flagged as processed - this would take 10-30 seconds. Then the client sends a plain sync again, to which the logs say there's no cached response and it tries again.

To break the loop, the client had to be lucky enough to have a request just before the homeserver spent several seconds sending responses.
> problems where initial /sync takes 10-15 mins to complete from Riot or similar

To be clear, it takes upwards of an hour for riot to sync, despite individual sync requests completing on the order of 15 minutes. Usually what happens is that Riot cancels numerous syncs but leaves a few running; eventually, all currently-running requests laboriously complete, while new ones start over from scratch. This repeats until somehow one finally catches.
right. 15m * ~5 = ~1h, so that fits this bug - and @turt2live's description of the observed behaviour precisely matches my theory.

@Ralith: can you cherrypick https://github.com/matrix-org/synapse/commit/0ca2857baab2d241a740a6ba5e8ed36d17756338 and see if it magically fixes itself?
With that commit cherry-picked, I did not observe spurious retries on an initial sync, though I may have just gotten lucky on the timing.
We should offer this as a config option, because

from @erikjohnston 

a) its a pain for sytest and b) giving people stale initial syncs makes their client think that all subsequent messages happened "now", so you end up being notified for everything in the last N minutes (both of which could potentially be fixed by caching the initial sync, but then update it with an incremental sync if necessary)

Also sucks up more mem.

for links: #8518 seems related here. We should make sure that is investigated before "fixing" this issue.
#8518 has been fixed, this issue is unblocked :D

Resolving patch:
diff --git a/synapse/config/cache.py b/synapse/config/cache.py
--- a/synapse/config/cache.py
+++ b/synapse/config/cache.py
@@ -151,6 +151,15 @@ def generate_config_section(self, **kwargs):
           # entries are never evicted based on time.
           #
           #expiry_time: 30m
+
+          # Controls how long the results of a /sync request are cached for after
+          # a successful response is returned. A higher duration can help clients with
+          # intermittent connections, at the cost of higher memory usage.
+          #
+          # By default, this is zero, which means that sync responses are not cached
+          # at all.
+          #
+          #sync_response_cache_duration: 2m
         """
 
     def read_config(self, config, **kwargs):
@@ -212,6 +221,10 @@ def read_config(self, config, **kwargs):
         else:
             self.expiry_time_msec = None
 
+        self.sync_response_cache_duration = self.parse_duration(
+            cache_config.get("sync_response_cache_duration", 0)
+        )
+
         # Resize all caches (if necessary) with the new factors we've loaded
         self.resize_all_caches()
 
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -269,14 +269,22 @@ def __init__(self, hs: "HomeServer"):
         self.presence_handler = hs.get_presence_handler()
         self.event_sources = hs.get_event_sources()
         self.clock = hs.get_clock()
-        self.response_cache: ResponseCache[SyncRequestKey] = ResponseCache(
-            hs.get_clock(), "sync"
-        )
         self.state = hs.get_state_handler()
         self.auth = hs.get_auth()
         self.storage = hs.get_storage()
         self.state_store = self.storage.state
 
+        # TODO: flush cache entries on subsequent sync request.
+        #    Once we get the next /sync request (ie, one with the same access token
+        #    that sets 'since' to 'next_batch'), we know that device won't need a
+        #    cached result any more, and we could flush the entry from the cache to save
+        #    memory.
+        self.response_cache: ResponseCache[SyncRequestKey] = ResponseCache(
+            hs.get_clock(),
+            "sync",
+            timeout_ms=hs.config.caches.sync_response_cache_duration,
+        )
+
         # ExpiringCache((User, Device)) -> LruCache(user_id => event_id)
         self.lazy_loaded_members_cache: ExpiringCache[
             Tuple[str, Optional[str]], LruCache[str, str]


Test patch: