Repository: matrix-org/synapse

Issue:
Use new `device_lists_changes_in_room` table for fetching device lists in `/sync`
We added a table that tracks device list changes in a room in https://github.com/matrix-org/synapse/pull/12321. We can use this in `/sync` (and `/keys/changes`) to more efficiently calculate the device list changes without having to calculate the full set of users who share a room with the requester.

c.f. 

https://github.com/matrix-org/synapse/blob/573cd0f92f78de54cbe5dc9d0bf9f904f9d90011/synapse/handlers/sync.py#L1313-L1329

and

https://github.com/matrix-org/synapse/blob/573cd0f92f78de54cbe5dc9d0bf9f904f9d90011/synapse/handlers/device.py#L139-L143

The tricky thing is that since the table has been added recently we need to a) only use it once our minimum schema is `69` and b) only for device list stream IDs that have happened after a. 


Discussion:
Is this ready for testing? I could merge the PR on a fork and try it out. I have an account which can't log in to new devices...
> Is this ready for testing? I could merge the PR on a fork and try it out. I have an account which can't log in to new devices...

Can you try running latest `develop`, https://github.com/matrix-org/synapse/pull/12321 and https://github.com/matrix-org/synapse/pull/12365 should hopefully fix logging in and I'd be interested in how they perform in real world use cases.
Sorry I'm having some trouble with building the docker image so I won't be able to test this soon...
> Sorry I'm having some trouble with building the docker image so I won't be able to test this soon...

If you're running <del>1.56</del> (edit:) 1.57 then there is also a hidden config option that you can set to enable the same functionality: `use_new_device_lists_changes_in_room: true`, which will be enabled in <del>1.57</del> (edit:) 1.58 by default. Note that if you do enable it then you **must not** downgrade your synapse to a prior version  
Is this a hidden config in the release and also in the tag? I'm not using the official releases, I have a fork and I merged... 
> Is this a hidden config in the release and also in the tag? I'm not using the official releases, I have a fork and I merged...

Oh sorry, I'm wrong, its not in 1.56. I'm getting the timeline wrong. It will be in 1.57 and then enabled by default in 1.58.

The commit you need is https://github.com/matrix-org/synapse/commit/5c9e39e6192e952ba8a5bb8e5485bc6067f91699, which is in the tag v1.57.0rc1.
Hey @erikjohnston I managed to build a docker image and test but it seems like I still can't log in... Maybe I need to higher the cache factor? I'm using the defaults...

That's what @squahtx suggested here: https://github.com/matrix-org/synapse/issues/12429#issuecomment-1094903034

Here are the logs:
[synapse.log](https://github.com/matrix-org/synapse/files/8503876/synapse.log)

I found:

```
Apr 18 03:50:59 localhost.localdomain matrix-synapse[19576]: 2022-04-18 01:50:59,075 - synapse.http.server - 690 - WARNING - POST-12397 - Not sending response to request <XForwardedForRequest at 0x7fd634d36820 method='POST' uri='/_matrix/client/r0/login' clientproto='HTTP/1.0' site='8008'>, already disconnected.

```

in line 3425 after about an hour and so since clicking the login.
I think this was fixed by #12365. @erikjohnston: if there is still outstanding work here, please can you clarify what it is.
https://github.com/matrix-org/synapse/pull/12365 only enabled using that table while registering a new device. This issue is about using the new table in a couple of other places where we do similar things.

Resolving patch:
diff --git a/synapse/handlers/device.py b/synapse/handlers/device.py
--- a/synapse/handlers/device.py
+++ b/synapse/handlers/device.py
@@ -123,23 +123,28 @@ async def get_device(self, user_id: str, device_id: str) -> JsonDict:
 
         return device
 
-    @trace
-    @measure_func("device.get_user_ids_changed")
-    async def get_user_ids_changed(
-        self, user_id: str, from_token: StreamToken
-    ) -> JsonDict:
-        """Get list of users that have had the devices updated, or have newly
-        joined a room, that `user_id` may be interested in.
+    async def get_device_changes_in_shared_rooms(
+        self, user_id: str, room_ids: Collection[str], from_token: StreamToken
+    ) -> Collection[str]:
+        """Get the set of users whose devices have changed who share a room with
+        the given user.
         """
+        changed_users = await self.store.get_device_list_changes_in_rooms(
+            room_ids, from_token.device_list_key
+        )
 
-        set_tag("user_id", user_id)
-        set_tag("from_token", from_token)
-        now_room_key = self.store.get_room_max_token()
+        if changed_users is not None:
+            # We also check if the given user has changed their device. If
+            # they're in no rooms then the above query won't include them.
+            changed = await self.store.get_users_whose_devices_changed(
+                from_token.device_list_key, [user_id]
+            )
+            changed_users.update(changed)
+            return changed_users
 
-        room_ids = await self.store.get_rooms_for_user(user_id)
+        # If the DB returned None then the `from_token` is too old, so we fall
+        # back on looking for device updates for all users.
 
-        # First we check if any devices have changed for users that we share
-        # rooms with.
         users_who_share_room = await self.store.get_users_who_share_room_with_user(
             user_id
         )
@@ -153,6 +158,27 @@ async def get_user_ids_changed(
             from_token.device_list_key, tracked_users
         )
 
+        return changed
+
+    @trace
+    @measure_func("device.get_user_ids_changed")
+    async def get_user_ids_changed(
+        self, user_id: str, from_token: StreamToken
+    ) -> JsonDict:
+        """Get list of users that have had the devices updated, or have newly
+        joined a room, that `user_id` may be interested in.
+        """
+
+        set_tag("user_id", user_id)
+        set_tag("from_token", from_token)
+        now_room_key = self.store.get_room_max_token()
+
+        room_ids = await self.store.get_rooms_for_user(user_id)
+
+        changed = await self.get_device_changes_in_shared_rooms(
+            user_id, room_ids, from_token
+        )
+
         # Then work out if any users have since joined
         rooms_changed = self.store.get_rooms_that_changed(room_ids, from_token.room_key)
 
@@ -237,10 +263,19 @@ async def get_user_ids_changed(
                         break
 
         if possibly_changed or possibly_left:
-            # Take the intersection of the users whose devices may have changed
-            # and those that actually still share a room with the user
-            possibly_joined = possibly_changed & users_who_share_room
-            possibly_left = (possibly_changed | possibly_left) - users_who_share_room
+            possibly_joined = possibly_changed
+            possibly_left = possibly_changed | possibly_left
+
+            # Double check if we still share rooms with the given user.
+            users_rooms = await self.store.get_rooms_for_users_with_stream_ordering(
+                possibly_left
+            )
+            for changed_user_id, entries in users_rooms.items():
+                if any(e.room_id in room_ids for e in entries):
+                    possibly_left.discard(changed_user_id)
+                else:
+                    possibly_joined.discard(changed_user_id)
+
         else:
             possibly_joined = set()
             possibly_left = set()
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -240,6 +240,7 @@ def __init__(self, hs: "HomeServer"):
         self.auth_blocking = hs.get_auth_blocking()
         self._storage_controllers = hs.get_storage_controllers()
         self._state_storage_controller = self._storage_controllers.state
+        self._device_handler = hs.get_device_handler()
 
         # TODO: flush cache entries on subsequent sync request.
         #    Once we get the next /sync request (ie, one with the same access token
@@ -1272,21 +1273,11 @@ async def _generate_sync_entry_for_device_list(
                     ):
                         users_that_have_changed.add(changed_user_id)
             else:
-                users_who_share_room = (
-                    await self.store.get_users_who_share_room_with_user(user_id)
-                )
-
-                # Always tell the user about their own devices. We check as the user
-                # ID is almost certainly already included (unless they're not in any
-                # rooms) and taking a copy of the set is relatively expensive.
-                if user_id not in users_who_share_room:
-                    users_who_share_room = set(users_who_share_room)
-                    users_who_share_room.add(user_id)
-
-                tracked_users = users_who_share_room
                 users_that_have_changed = (
-                    await self.store.get_users_whose_devices_changed(
-                        since_token.device_list_key, tracked_users
+                    await self._device_handler.get_device_changes_in_shared_rooms(
+                        user_id,
+                        sync_result_builder.joined_room_ids,
+                        from_token=since_token,
                     )
                 )
 
diff --git a/synapse/storage/databases/main/devices.py b/synapse/storage/databases/main/devices.py
--- a/synapse/storage/databases/main/devices.py
+++ b/synapse/storage/databases/main/devices.py
@@ -1193,6 +1193,65 @@ def get_devices_not_accessed_since_txn(
 
         return devices
 
+    @cached()
+    async def _get_min_device_lists_changes_in_room(self) -> int:
+        """Returns the minimum stream ID that we have entries for
+        `device_lists_changes_in_room`
+        """
+
+        return await self.db_pool.simple_select_one_onecol(
+            table="device_lists_changes_in_room",
+            keyvalues={},
+            retcol="COALESCE(MIN(stream_id), 0)",
+            desc="get_min_device_lists_changes_in_room",
+        )
+
+    async def get_device_list_changes_in_rooms(
+        self, room_ids: Collection[str], from_id: int
+    ) -> Optional[Set[str]]:
+        """Return the set of users whose devices have changed in the given rooms
+        since the given stream ID.
+
+        Returns None if the given stream ID is too old.
+        """
+
+        if not room_ids:
+            return set()
+
+        min_stream_id = await self._get_min_device_lists_changes_in_room()
+
+        if min_stream_id > from_id:
+            return None
+
+        sql = """
+            SELECT DISTINCT user_id FROM device_lists_changes_in_room
+            WHERE {clause} AND stream_id >= ?
+        """
+
+        def _get_device_list_changes_in_rooms_txn(
+            txn: LoggingTransaction,
+            clause,
+            args,
+        ) -> Set[str]:
+            txn.execute(sql.format(clause=clause), args)
+            return {user_id for user_id, in txn}
+
+        changes = set()
+        for chunk in batch_iter(room_ids, 1000):
+            clause, args = make_in_list_sql_clause(
+                self.database_engine, "room_id", chunk
+            )
+            args.append(from_id)
+
+            changes |= await self.db_pool.runInteraction(
+                "get_device_list_changes_in_rooms",
+                _get_device_list_changes_in_rooms_txn,
+                clause,
+                args,
+            )
+
+        return changes
+
 
 class DeviceBackgroundUpdateStore(SQLBaseStore):
     def __init__(


Test patch: