Repository: matrix-org/synapse

Issue:
Sync requests immediately return with empty payload
### Description

Occasionally, without any specific action on my end, a normal long-polling sync loop becomes a storm of syncs, as each next request sent from Quotient returns immediately (30-40 ms) with a JSON object that has the full sync response structure (`rooms` with `invite`/`join`/`leave` inside, `account_data`, `to_device`, etc.) and reasonable-looking `next_batch` but with empty objects (apparently because no events arrived since the last sync). Usually this happens within 30 seconds to 2 minutes, and requests stay with the server in a normal way (seconds) after that.

### Steps to reproduce

I couldn't identify a particular pattern. For reference, it occured between 10:28 and 10:29 CEST with sync coming from my user (`@kitsune`), with `next_batch` at 10:29:07 looking like "s1589639296_757284957_14250253_610667103_445506552_1585308_106571744_371537760_134971"

- **Homeserver**: matrix.org


Discussion:
@KitsuneRal any idea when you first saw this? in the last couple of weeks, or longer?
We can see this happening clearly in the server logs for roughly 1m04s at the mentioned time. We're not entirely sure if this is a regression in v1.21.0 or not.

Here are the logs from the server side: https://gist.github.com/anoadragon453/4a54bbe3388cdc9f4d9c05feb1c028d2 We can see that they indeed return immediately for the same access token, even when a `30000` ms timeout is provided.

My hunch is that somehow this condition is `True`, meaning we don't wait for a new event before returning:

https://github.com/matrix-org/synapse/blob/9ca6341969b8b84c0c79a29fb914d1d8dbb3e320/synapse/handlers/sync.py#L306-L311

The parameters to the request look fine, so `timeout`, `since` or `full_state` may be changing somewhere internally.

It would be helpful to add some debug logging here to help track down what exactly is going wrong, although be aware this is a fairly hot path.
> @KitsuneRal any idea when you first saw this? in the last couple of weeks, or longer?

Yes, in something like a couple of weeks.
This seems fairly easy to reproduce with any user who sees no activity in their rooms during the sync timeout: the first /sync times out, *returning the same sync token in `next_batch` as was passed in `since`*. Subsequent requests (again using that same sync token) hit the cache and return immediately.

Look at the logs from that first /sync request:

```
2020-11-11 18:46:19,812 - synapse.access.http.8008 - 258 - DEBUG - GET-26414 - ::ffff:127.0.0.1 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s206_4657_8_25_111_5_1_48_1
2020-11-11 18:46:19,813 - synapse.rest.client.v2_alpha.sync - 119 - DEBUG - GET-26414 - /sync: user=@richvdh:fred.sw1v.org, timeout=30000, since='s206_4657_8_25_111_5_1_48_1', set_presence='online', filter_id='1', device_id='RLLEKDLCTE'
2020-11-11 18:46:19,814 - synapse.util.caches.response_cache - 154 - DEBUG - GET-26414 - [sync]: no cached result for [(@richvdh:fred.sw1v.org, 30000, 's206_4657_8_25_111_5_1_48_1', '1', False, 'RLLEKDLCTE')], calculating new one
2020-11-11 18:46:19,814 - synapse.handlers.sync - 969 - DEBUG - GET-26414 - Calculating sync response for @richvdh:fred.sw1v.org between StreamToken(room_key=RoomStreamToken(topological=None, stream=206, instance_map={}), presence_key=4657, typing_key=8, receipt_key=25, account_data_key=111, push_rules_key=5, to_device_key=1, device_list_key=48, groups_key=1) and StreamToken(room_key=RoomStreamToken(topological=None, stream=206, instance_map={}), presence_key=4657, typing_key=8, receipt_key=25, account_data_key=111, push_rules_key=5, to_device_key=1, device_list_key=48, groups_key=1)
2020-11-11 18:46:19,814 - synapse.handlers.sync - 990 - DEBUG - GET-26414 - Fetching account data
2020-11-11 18:46:19,815 - synapse.handlers.sync - 996 - DEBUG - GET-26414 - Fetching room data
2020-11-11 18:46:19,815 - synapse.handlers.sync - 1392 - DEBUG - GET-26414 - no-oping sync
2020-11-11 18:46:19,815 - synapse.handlers.sync - 1008 - DEBUG - GET-26414 - Fetching presence data
2020-11-11 18:46:19,815 - synapse.handlers.presence - 1024 - DEBUG - GET-26414 - Invoked 'get_new_events' with args: self=<synapse.handlers.presence.PresenceEventSource obj..., kwargs={'is_guest': 0}, user=@richvdh:fred.sw1v.org, from_key=4657, include_offline=True, room_ids=None, explicit_room_id=None
2020-11-11 18:46:19,815 - synapse.handlers.sync - 1013 - DEBUG - GET-26414 - Fetching to-device data
2020-11-11 18:46:19,816 - synapse.handlers.sync - 1024 - DEBUG - GET-26414 - Fetching OTK data
2020-11-11 18:46:19,816 - synapse.handlers.sync - 1036 - DEBUG - GET-26414 - Fetching group data
2020-11-11 18:46:19,816 - synapse.handlers.sync - 1047 - DEBUG - GET-26414 - Sync response calculation complete
2020-11-11 18:46:49,815 - synapse.handlers.sync - 287 - DEBUG - GET-26414 - Returning sync response for @richvdh:fred.sw1v.org
2020-11-11 18:46:49,815 - synapse.rest.client.v2_alpha.sync - 195 - DEBUG - GET-26414 - Formatting events in sync response
2020-11-11 18:46:49,816 - synapse.rest.client.v2_alpha.sync - 223 - DEBUG - GET-26414 - building sync response dict
2020-11-11 18:46:49,816 - synapse.rest.client.v2_alpha.sync - 191 - DEBUG - GET-26414 - Event formatting complete
2020-11-11 18:46:49,816 - synapse.access.http.8008 - 331 - INFO - GET-26414 - ::ffff:127.0.0.1 - 8008 - {@richvdh:fred.sw1v.org} Processed request: 30.004sec/-0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 362B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s206_4657_8_25_111_5_1_48_1 HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0" [0 dbevts]
```

It builds the empty response when the endpoint is first called, rather than waiting for the timeout. *Then* it waits for 30 seconds and returns the empty response, so even though there has been traffic in other rooms in the meantime, the sync token is not advanced.

Compare with the debug for an empty sync response from synapse 1.20:

```
2020-11-11 19:00:00,993 - synapse.access.http.8008 - 246 - DEBUG - GET-17 - ::ffff:127.0.0.1 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s206_4684_0_25_111_5_1_48_1
2020-11-11 19:00:00,993 - synapse.rest.client.v2_alpha.sync - 118 - DEBUG - GET-17 - /sync: user=@richvdh:fred.sw1v.org, timeout=30000, since='s206_4684_0_25_111_5_1_48_1', set_presence='online', filter_id='1', device_id='RLLEKDLCTE'
2020-11-11 19:00:00,994 - synapse.util.caches.response_cache - 148 - DEBUG - GET-17 - [sync]: no cached result for [(@richvdh:fred.sw1v.org, 30000, 's206_4684_0_25_111_5_1_48_1', '1', False, 'RLLEKDLCTE')], calculating new one
2020-11-11 19:00:30,996 - synapse.handlers.sync - 973 - DEBUG - GET-17 - Calculating sync response for @richvdh:fred.sw1v.org between StreamToken(room_key='s206', presence_key='4684', typing_key='0', receipt_key='25', account_data_key='111', push_rules_key='5', to_device_key='1', device_list_key='48', groups_key='1') and StreamToken(room_key='s206', presence_key=4684, typing_key=0, receipt_key=25, account_data_key=111, push_rules_key=5, to_device_key=1, device_list_key=48, groups_key=1)
2020-11-11 19:00:30,997 - synapse.handlers.sync - 994 - DEBUG - GET-17 - Fetching account data
2020-11-11 19:00:30,997 - synapse.handlers.sync - 1000 - DEBUG - GET-17 - Fetching room data
2020-11-11 19:00:30,998 - synapse.handlers.sync - 1391 - DEBUG - GET-17 - no-oping sync
2020-11-11 19:00:30,998 - synapse.handlers.sync - 1012 - DEBUG - GET-17 - Fetching presence data
2020-11-11 19:00:30,999 - synapse.handlers.sync - 1017 - DEBUG - GET-17 - Fetching to-device data
2020-11-11 19:00:30,999 - synapse.handlers.sync - 1028 - DEBUG - GET-17 - Fetching OTK data
2020-11-11 19:00:30,999 - synapse.handlers.sync - 1036 - DEBUG - GET-17 - Fetching group data
2020-11-11 19:00:31,000 - synapse.handlers.sync - 1047 - DEBUG - GET-17 - Sync response calculation complete
2020-11-11 19:00:31,000 - synapse.handlers.sync - 296 - DEBUG - GET-17 - Returning sync response for @richvdh:fred.sw1v.org
2020-11-11 19:00:31,001 - synapse.rest.client.v2_alpha.sync - 195 - DEBUG - GET-17 - Formatting events in sync response
2020-11-11 19:00:31,001 - synapse.rest.client.v2_alpha.sync - 223 - DEBUG - GET-17 - building sync response dict
2020-11-11 19:00:31,001 - synapse.rest.client.v2_alpha.sync - 191 - DEBUG - GET-17 - Event formatting complete
2020-11-11 19:00:31,002 - synapse.access.http.8008 - 311 - INFO - GET-17 - ::ffff:127.0.0.1 - 8008 - {@richvdh:fred.sw1v.org} Processed request: 30.009sec/-0.000sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 305B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s206_4684_0_25_111_5_1_48_1 HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0" [0 dbevts]
```

... it waits for 30 seconds before calling the `synapse.handlers.sync` code.

Presumably this means that the timeout at https://github.com/matrix-org/synapse/blob/v1.22.0/synapse/notifier.py#L437-L444 is not working correctly.
Note that it only happens on matrix-org-hotfixes which has a non-zero `SYNC_RESPONSE_CACHE_MS`. Presumably that implies that even before that regression was introduced, you could end up in a similar situation on an idle homeserver with non-zero `SYNC_RESPONSE_CACHE_MS`.
Preliminary observations; It looks like the `prev_key == current_key`? is it possible that due to a timeout (where nothing happened for 30 seconds) which got cached (by `prev_key`), the cache got returned which includes `next_key == prev_key`, thus creating a loop? If so, maybe we should have a way of creating a "don't cache" response in that case (if the sync result is due to timeout, and as a sanity check, `if prev_key == next_key`)?

Edit: Just read everything, yeah, it looks like that's the case.
Re-opening as the fix has been backed out, see https://github.com/matrix-org/synapse/pull/9358#issuecomment-788960683.

Resolving patch:
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -49,7 +49,7 @@
 from synapse.util.async_helpers import concurrently_execute
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.caches.lrucache import LruCache
-from synapse.util.caches.response_cache import ResponseCache
+from synapse.util.caches.response_cache import ResponseCache, ResponseCacheContext
 from synapse.util.metrics import Measure, measure_func
 from synapse.visibility import filter_events_for_client
 
@@ -83,12 +83,15 @@
 LAZY_LOADED_MEMBERS_CACHE_MAX_SIZE = 100
 
 
+SyncRequestKey = Tuple[Any, ...]
+
+
 @attr.s(slots=True, frozen=True)
 class SyncConfig:
     user = attr.ib(type=UserID)
     filter_collection = attr.ib(type=FilterCollection)
     is_guest = attr.ib(type=bool)
-    request_key = attr.ib(type=Tuple[Any, ...])
+    request_key = attr.ib(type=SyncRequestKey)
     device_id = attr.ib(type=Optional[str])
 
 
@@ -266,9 +269,9 @@ def __init__(self, hs: "HomeServer"):
         self.presence_handler = hs.get_presence_handler()
         self.event_sources = hs.get_event_sources()
         self.clock = hs.get_clock()
-        self.response_cache = ResponseCache(
+        self.response_cache: ResponseCache[SyncRequestKey] = ResponseCache(
             hs.get_clock(), "sync"
-        )  # type: ResponseCache[Tuple[Any, ...]]
+        )
         self.state = hs.get_state_handler()
         self.auth = hs.get_auth()
         self.storage = hs.get_storage()
@@ -307,6 +310,7 @@ async def wait_for_sync_for_user(
             since_token,
             timeout,
             full_state,
+            cache_context=True,
         )
         logger.debug("Returning sync response for %s", user_id)
         return res
@@ -314,9 +318,10 @@ async def wait_for_sync_for_user(
     async def _wait_for_sync_for_user(
         self,
         sync_config: SyncConfig,
-        since_token: Optional[StreamToken] = None,
-        timeout: int = 0,
-        full_state: bool = False,
+        since_token: Optional[StreamToken],
+        timeout: int,
+        full_state: bool,
+        cache_context: ResponseCacheContext[SyncRequestKey],
     ) -> SyncResult:
         if since_token is None:
             sync_type = "initial_sync"
@@ -343,13 +348,13 @@ async def _wait_for_sync_for_user(
         if timeout == 0 or since_token is None or full_state:
             # we are going to return immediately, so don't bother calling
             # notifier.wait_for_events.
-            result = await self.current_sync_for_user(
+            result: SyncResult = await self.current_sync_for_user(
                 sync_config, since_token, full_state=full_state
             )
         else:
 
-            def current_sync_callback(before_token, after_token):
-                return self.current_sync_for_user(sync_config, since_token)
+            async def current_sync_callback(before_token, after_token) -> SyncResult:
+                return await self.current_sync_for_user(sync_config, since_token)
 
             result = await self.notifier.wait_for_events(
                 sync_config.user.to_string(),
@@ -358,6 +363,17 @@ def current_sync_callback(before_token, after_token):
                 from_token=since_token,
             )
 
+            # if nothing has happened in any of the users' rooms since /sync was called,
+            # the resultant next_batch will be the same as since_token (since the result
+            # is generated when wait_for_events is first called, and not regenerated
+            # when wait_for_events times out).
+            #
+            # If that happens, we mustn't cache it, so that when the client comes back
+            # with the same cache token, we don't immediately return the same empty
+            # result, causing a tightloop. (#8518)
+            if result.next_batch == since_token:
+                cache_context.should_cache = False
+
         if result:
             if sync_config.filter_collection.lazy_load_members():
                 lazy_loaded = "true"
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -75,11 +75,9 @@
     "phonenumbers>=8.2.0",
     # we use GaugeHistogramMetric, which was added in prom-client 0.4.0.
     "prometheus_client>=0.4.0",
-    # we use attr.validators.deep_iterable, which arrived in 19.1.0 (Note:
-    # Fedora 31 only has 19.1, so if we want to upgrade we should wait until 33
-    # is out in November.)
+    # we use `order`, which arrived in attrs 19.2.0.
     # Note: 21.1.0 broke `/sync`, see #9936
-    "attrs>=19.1.0,!=21.1.0",
+    "attrs>=19.2.0,!=21.1.0",
     "netaddr>=0.7.18",
     "Jinja2>=2.9",
     "bleach>=1.4.3",
diff --git a/synapse/types.py b/synapse/types.py
--- a/synapse/types.py
+++ b/synapse/types.py
@@ -404,7 +404,7 @@ def f2(m):
     return username.decode("ascii")
 
 
-@attr.s(frozen=True, slots=True, cmp=False)
+@attr.s(frozen=True, slots=True, order=False)
 class RoomStreamToken:
     """Tokens are positions between events. The token "s1" comes after event 1.
 


Test patch:
diff --git a/tests/rest/client/v2_alpha/test_sync.py b/tests/rest/client/v2_alpha/test_sync.py
--- a/tests/rest/client/v2_alpha/test_sync.py
+++ b/tests/rest/client/v2_alpha/test_sync.py
@@ -558,3 +558,53 @@ def _check_unread_count(self, expected_count: int):
 
         # Store the next batch for the next request.
         self.next_batch = channel.json_body["next_batch"]
+
+
+class SyncCacheTestCase(unittest.HomeserverTestCase):
+    servlets = [
+        synapse.rest.admin.register_servlets,
+        login.register_servlets,
+        sync.register_servlets,
+    ]
+
+    def test_noop_sync_does_not_tightloop(self):
+        """If the sync times out, we shouldn't cache the result
+
+        Essentially a regression test for #8518.
+        """
+        self.user_id = self.register_user("kermit", "monkey")
+        self.tok = self.login("kermit", "monkey")
+
+        # we should immediately get an initial sync response
+        channel = self.make_request("GET", "/sync", access_token=self.tok)
+        self.assertEqual(channel.code, 200, channel.json_body)
+
+        # now, make an incremental sync request, with a timeout
+        next_batch = channel.json_body["next_batch"]
+        channel = self.make_request(
+            "GET",
+            f"/sync?since={next_batch}&timeout=10000",
+            access_token=self.tok,
+            await_result=False,
+        )
+        # that should block for 10 seconds
+        with self.assertRaises(TimedOutException):
+            channel.await_result(timeout_ms=9900)
+        channel.await_result(timeout_ms=200)
+        self.assertEqual(channel.code, 200, channel.json_body)
+
+        # we expect the next_batch in the result to be the same as before
+        self.assertEqual(channel.json_body["next_batch"], next_batch)
+
+        # another incremental sync should also block.
+        channel = self.make_request(
+            "GET",
+            f"/sync?since={next_batch}&timeout=10000",
+            access_token=self.tok,
+            await_result=False,
+        )
+        # that should block for 10 seconds
+        with self.assertRaises(TimedOutException):
+            channel.await_result(timeout_ms=9900)
+        channel.await_result(timeout_ms=200)
+        self.assertEqual(channel.code, 200, channel.json_body)
diff --git a/tests/server.py b/tests/server.py
--- a/tests/server.py
+++ b/tests/server.py
@@ -138,21 +138,19 @@ def isSecure(self):
     def transport(self):
         return self
 
-    def await_result(self, timeout: int = 100) -> None:
+    def await_result(self, timeout_ms: int = 1000) -> None:
         """
         Wait until the request is finished.
         """
+        end_time = self._reactor.seconds() + timeout_ms / 1000.0
         self._reactor.run()
-        x = 0
 
         while not self.is_finished():
             # If there's a producer, tell it to resume producing so we get content
             if self._producer:
                 self._producer.resumeProducing()
 
-            x += 1
-
-            if x > timeout:
+            if self._reactor.seconds() > end_time:
                 raise TimedOutException("Timed out waiting for request to finish.")
 
             self._reactor.advance(0.1)