Repository: matrix-org/synapse

Issue:
Device list updates to be sent over federation will be skipped if there are too many at one time
`get_device_updates_by_remote` always returns the next stream position as the latest device stream token, even in cases where it is limited to a maximum number of rows to return.
The effect of this is that any device updates that don't fit within one `limit`-sized batch will be skipped.

Discovered whilst looking into #11719. Due to the nature of my fix to #11719, I'll likely fix them both at the same time.




Discussion:
I wonder if this partly explains #8631?
Seems it's not quite as simple as it looked to be at the start; the plot thickens.

[device_lists_stream](https://github.com/matrix-org/synapse/blob/25f43faa70f7cc58493b636c2702ae63395779dc/synapse/storage/schema/main/delta/40/device_list_streams.sql#L38-L37) doesn't have a unique index on `stream_id`, and [test_get_device_updates_by_remote](https://github.com/matrix-org/synapse/blob/4b965c862dc66c0da5d3240add70e9b5f0aa720b/tests/storage/test_devices.py#L94) claims in a comment that it is adding two device updates with the same stream ID. (fun fact: at least on my test run, they're using sequential IDs — not the same stream ID :/)

That's all well and good, but if there are multiple rows for the same stream ID and you're paginating by stream ID (as is done currently, despite the fact that it always skips to the end) with a limit on the number of rows, that gets awkward.

Annoyingly it looks like they do overlap in practice (or have done so in the past); querying LP.net:
```sql
synapse=# select count(distinct stream_id) from device_lists_stream ;
 count  
--------
 232366
(1 row)
synapse=# select count(stream_id) from device_lists_stream ;
 count  
--------
 266670
(1 row)
```
I notice that 'recent' entries in the table on LP.net don't seem to reuse `stream_id`s, though. Maybe this is something that happened in the past but doesn't anymore :/.

```
synapse=# select stream_id from device_lists_stream order by stream_id desc limit 1;
 stream_id 
-----------
   3902416
(1 row)

synapse=# select count(stream_id) from device_lists_stream where stream_id > 902416;
 count  
--------
 176803
(1 row)

synapse=# select count(distinct stream_id) from device_lists_stream where stream_id > 902416;
 count  
--------
 176803
(1 row)
```
Rich kindly pointed me to https://github.com/matrix-org/synapse/pull/7010 — looks like the `stream_id`s were changed to not get shared between multiple rows (though historical rows are of course still there; can probably gloss over those).

Resolving patch:
diff --git a/synapse/storage/databases/main/devices.py b/synapse/storage/databases/main/devices.py
--- a/synapse/storage/databases/main/devices.py
+++ b/synapse/storage/databases/main/devices.py
@@ -270,6 +270,10 @@ async def get_device_updates_by_remote(
         # The most recent request's opentracing_context is used as the
         # context which created the Edu.
 
+        # This is the stream ID that we will return for the consumer to resume
+        # following this stream later.
+        last_processed_stream_id = from_stream_id
+
         query_map = {}
         cross_signing_keys_by_user = {}
         for user_id, device_id, update_stream_id, update_context in updates:
@@ -295,6 +299,8 @@ async def get_device_updates_by_remote(
                 if update_stream_id > previous_update_stream_id:
                     query_map[key] = (update_stream_id, update_context)
 
+            last_processed_stream_id = update_stream_id
+
         results = await self._get_device_update_edus_by_remote(
             destination, from_stream_id, query_map
         )
@@ -307,7 +313,7 @@ async def get_device_updates_by_remote(
             # FIXME: remove this when enough servers have upgraded
             results.append(("org.matrix.signing_key_update", result))
 
-        return now_stream_id, results
+        return last_processed_stream_id, results
 
     def _get_device_updates_by_remote_txn(
         self,


Test patch:
diff --git a/tests/storage/test_devices.py b/tests/storage/test_devices.py
--- a/tests/storage/test_devices.py
+++ b/tests/storage/test_devices.py
@@ -94,7 +94,7 @@ def test_count_devices_by_users(self):
     def test_get_device_updates_by_remote(self):
         device_ids = ["device_id1", "device_id2"]
 
-        # Add two device updates with a single stream_id
+        # Add two device updates with sequential `stream_id`s
         self.get_success(
             self.store.add_device_change_to_streams("user_id", device_ids, ["somehost"])
         )
@@ -107,6 +107,54 @@ def test_get_device_updates_by_remote(self):
         # Check original device_ids are contained within these updates
         self._check_devices_in_updates(device_ids, device_updates)
 
+    def test_get_device_updates_by_remote_can_limit_properly(self):
+        """
+        Tests that `get_device_updates_by_remote` returns an appropriate
+        stream_id to resume fetching from (without skipping any results).
+        """
+
+        # Add some device updates with sequential `stream_id`s
+        device_ids = [
+            "device_id1",
+            "device_id2",
+            "device_id3",
+            "device_id4",
+            "device_id5",
+        ]
+        self.get_success(
+            self.store.add_device_change_to_streams("user_id", device_ids, ["somehost"])
+        )
+
+        # Get all device updates ever meant for this remote
+        next_stream_id, device_updates = self.get_success(
+            self.store.get_device_updates_by_remote("somehost", -1, limit=3)
+        )
+
+        # Check the first three original device_ids are contained within these updates
+        self._check_devices_in_updates(device_ids[:3], device_updates)
+
+        # Get the next batch of device updates
+        next_stream_id, device_updates = self.get_success(
+            self.store.get_device_updates_by_remote("somehost", next_stream_id, limit=3)
+        )
+
+        # Check the last two original device_ids are contained within these updates
+        self._check_devices_in_updates(device_ids[3:], device_updates)
+
+        # Add some more device updates to ensure it still resumes properly
+        device_ids = ["device_id6", "device_id7"]
+        self.get_success(
+            self.store.add_device_change_to_streams("user_id", device_ids, ["somehost"])
+        )
+
+        # Get the next batch of device updates
+        next_stream_id, device_updates = self.get_success(
+            self.store.get_device_updates_by_remote("somehost", next_stream_id, limit=3)
+        )
+
+        # Check the newly-added device_ids are contained within these updates
+        self._check_devices_in_updates(device_ids, device_updates)
+
     def _check_devices_in_updates(self, expected_device_ids, device_updates):
         """Check that an specific device ids exist in a list of device update EDUs"""
         self.assertEqual(len(device_updates), len(expected_device_ids))