Repository: matrix-org/synapse

Issue:
to-device message went missing
We received a to-device message on `matrix.org`:

```
2023-03-23 13:52:18,439 - synapse.9533_debug - 890 - DEBUG - POST-3642069----- - Stored to-device messages with stream_id 821825860: ['@dave:matrix.org/CSMDXZIMRU (msgid 3eca6bb0-ebaf-4106-b3da-f66da959c621)', ... ]
```

There is no sign of this message ever being received on the client (rageshake at https://github.com/matrix-org/element-web-rageshakes/issues/20790). The server-side logging isn't completely clear, but I strongly suspect the message was never returned over `/sync`.


Discussion:
Some odd things I note:

* On all the synchrotrons, the replication data seems to jump straight from stream id 26567797 to 26567959, missing out 821825860:
  ```
  2023-03-23 13:52:19,647 - synapse.9533_debug - 492 - DEBUG - process-replication-data-26567797 - to-device messages stream id 821825789, awaking streams for [...]
  ...
  2023-03-23 13:52:20,136 - synapse.9533_debug - 492 - DEBUG - process-replication-data-26567959 - to-device messages stream id 821825870, awaking streams for [...]
  ```
  Normally there is a line for each stream id.
* On the encryption worker, I see the following:
  ```
  2023-03-23 13:52:19,452 - synapse.replication.tcp.resource - 193 - INFO - replication_notifier-1178641 - Streaming: to_device -> 821825789
  2023-03-23 13:52:19,455 - synapse.replication.tcp.resource - 245 - INFO - replication_notifier-1178641 - Sending position: to_device -> 821825860
  ```
  A batch of replication rows is sent at `13:52:19`, but it is limited due to the number of rows to be sent. Then we immediately send a `POSITION` up to stream id 821825860. This is coming from the code added in #15150.

According to https://github.com/matrix-org/synapse/blob/develop/synapse/replication/tcp/commands.py#L173, the recipient is meant to read any missing rows from the database, but I'm not convinced that is happening. I also suspect that the code added in #15150 should not be sending a POSITION in this case (is it maybe missing an `if not limited`), but I don't really understand how our replication streams are meant to work.

As @reivilibre points out, https://github.com/matrix-org/synapse/blob/develop/synapse/replication/tcp/commands.py#L164 also says of a `POSITION`

>  The tokens are chosen so that *no* rows were written by the instance between the `prev_token` and `new_token`.

... which is certainly not the case here.
After closer inspection:

> According to https://github.com/matrix-org/synapse/blob/develop/synapse/replication/tcp/commands.py#L173, the recipient is meant to read any missing rows from the database

This is correct, but it is specifically referring to the `prev_token`, which — unhelpfully — we do not log. (It does actually say it's about the `prev_token` in the next sentence, but (a) I can't read, and (b) I was led astray by the docs [here](https://github.com/matrix-org/synapse/blob/v1.80.0/synapse/replication/tcp/protocol.py#L39-L40) which say that `POSITION` only has one stream_id parameter.)

Nevertheless we have the fact that we are sending a `POSITION` with a `new_token` of 821825860, despite the fact that there are to-device messages with smaller stream ids which have not sent.

New theory is that the to-device stream updates have been limited by the `min` at https://github.com/matrix-org/synapse/blob/v1.80.0/synapse/storage/databases/main/deviceinbox.py#L620. If that happens (and it does not actually pull out 100 or more rows from the database), it will return a new token of `current_id` despite the fact we did not even look at all those rows.

(This code was introduced in #7024 and could be an explanation for many of the mysterious to-device-message-disappearances over the years; see #9533.)

Resolving patch:
diff --git a/synapse/storage/databases/main/deviceinbox.py b/synapse/storage/databases/main/deviceinbox.py
--- a/synapse/storage/databases/main/deviceinbox.py
+++ b/synapse/storage/databases/main/deviceinbox.py
@@ -617,14 +617,14 @@ def get_all_new_device_messages_txn(
             # We limit like this as we might have multiple rows per stream_id, and
             # we want to make sure we always get all entries for any stream_id
             # we return.
-            upper_pos = min(current_id, last_id + limit)
+            upto_token = min(current_id, last_id + limit)
             sql = (
                 "SELECT max(stream_id), user_id"
                 " FROM device_inbox"
                 " WHERE ? < stream_id AND stream_id <= ?"
                 " GROUP BY user_id"
             )
-            txn.execute(sql, (last_id, upper_pos))
+            txn.execute(sql, (last_id, upto_token))
             updates = [(row[0], row[1:]) for row in txn]
 
             sql = (
@@ -633,19 +633,13 @@ def get_all_new_device_messages_txn(
                 " WHERE ? < stream_id AND stream_id <= ?"
                 " GROUP BY destination"
             )
-            txn.execute(sql, (last_id, upper_pos))
+            txn.execute(sql, (last_id, upto_token))
             updates.extend((row[0], row[1:]) for row in txn)
 
             # Order by ascending stream ordering
             updates.sort()
 
-            limited = False
-            upto_token = current_id
-            if len(updates) >= limit:
-                upto_token = updates[-1][0]
-                limited = True
-
-            return updates, upto_token, limited
+            return updates, upto_token, upto_token < current_id
 
         return await self.db_pool.runInteraction(
             "get_all_new_device_messages", get_all_new_device_messages_txn


Test patch:
diff --git a/tests/replication/_base.py b/tests/replication/_base.py
--- a/tests/replication/_base.py
+++ b/tests/replication/_base.py
@@ -54,6 +54,10 @@ class BaseStreamTestCase(unittest.HomeserverTestCase):
     if not hiredis:
         skip = "Requires hiredis"
 
+    if not USE_POSTGRES_FOR_TESTS:
+        # Redis replication only takes place on Postgres
+        skip = "Requires Postgres"
+
     def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
         # build a replication server
         server_factory = ReplicationStreamProtocolFactory(hs)
diff --git a/tests/replication/tcp/streams/test_account_data.py b/tests/replication/tcp/streams/test_account_data.py
--- a/tests/replication/tcp/streams/test_account_data.py
+++ b/tests/replication/tcp/streams/test_account_data.py
@@ -37,11 +37,6 @@ def test_update_function_room_account_data_limit(self) -> None:
         # also one global update
         self.get_success(store.add_account_data_for_user("test_user", "m.global", {}))
 
-        # tell the notifier to catch up to avoid duplicate rows.
-        # workaround for https://github.com/matrix-org/synapse/issues/7360
-        # FIXME remove this when the above is fixed
-        self.replicate()
-
         # check we're testing what we think we are: no rows should yet have been
         # received
         self.assertEqual([], self.test_handler.received_rdata_rows)
diff --git a/tests/replication/tcp/streams/test_to_device.py b/tests/replication/tcp/streams/test_to_device.py
new file mode 100644
--- /dev/null
+++ b/tests/replication/tcp/streams/test_to_device.py
@@ -0,0 +1,89 @@
+# Copyright 2023 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+
+import synapse
+from synapse.replication.tcp.streams._base import _STREAM_UPDATE_TARGET_ROW_COUNT
+from synapse.types import JsonDict
+
+from tests.replication._base import BaseStreamTestCase
+
+logger = logging.getLogger(__name__)
+
+
+class ToDeviceStreamTestCase(BaseStreamTestCase):
+    servlets = [
+        synapse.rest.admin.register_servlets,
+        synapse.rest.client.login.register_servlets,
+    ]
+
+    def test_to_device_stream(self) -> None:
+        store = self.hs.get_datastores().main
+
+        user1 = self.register_user("user1", "pass")
+        self.login("user1", "pass", "device")
+        user2 = self.register_user("user2", "pass")
+        self.login("user2", "pass", "device")
+
+        # connect to pull the updates related to users creation/login
+        self.reconnect()
+        self.replicate()
+        self.test_handler.received_rdata_rows.clear()
+        # disconnect so we can accumulate the updates without pulling them
+        self.disconnect()
+
+        msg: JsonDict = {}
+        msg["sender"] = "@sender:example.org"
+        msg["type"] = "m.new_device"
+
+        # add messages to the device inbox for user1 up until the
+        # limit defined for a stream update batch
+        for i in range(0, _STREAM_UPDATE_TARGET_ROW_COUNT):
+            msg["content"] = {"device": {}}
+            messages = {user1: {"device": msg}}
+
+            self.get_success(
+                store.add_messages_from_remote_to_device_inbox(
+                    "example.org",
+                    f"{i}",
+                    messages,
+                )
+            )
+
+        # add one more message, for user2 this time
+        # this message would be dropped before fixing #15335
+        msg["content"] = {"device": {}}
+        messages = {user2: {"device": msg}}
+
+        self.get_success(
+            store.add_messages_from_remote_to_device_inbox(
+                "example.org",
+                f"{_STREAM_UPDATE_TARGET_ROW_COUNT}",
+                messages,
+            )
+        )
+
+        # replication is disconnected so we shouldn't get any updates yet
+        self.assertEqual([], self.test_handler.received_rdata_rows)
+
+        # now reconnect to pull the updates
+        self.reconnect()
+        self.replicate()
+
+        # we should receive the fact that we have to_device updates
+        # for user1 and user2
+        received_rows = self.test_handler.received_rdata_rows
+        self.assertEqual(len(received_rows), 2)
+        self.assertEqual(received_rows[0][2].entity, user1)
+        self.assertEqual(received_rows[1][2].entity, user2)