Repository: matrix-org/synapse

Issue:
StreamChangeCache.get_all_changed_entities incorrect usage
The `get_all_changed_entities` returns `None` in the case that the cache cannot provide information on the given stream ordering. Some of the callers treat `None` equivalent to `[]` (which corresponds to *no* changes).

Callers, checked do it correctly. Unchecked are either incorrect or maybe incorrect:

* [x] [`PresenceEventSource.get_new_events`](https://github.com/matrix-org/synapse/blob/618e4ab81b70e37bdb8e9224bd84fcfe4b15bdea/synapse/handlers/presence.py#L1695-L1725)
* [ ] [`PresenceEventSource._filter_all_presence_updates_for_user`](https://github.com/matrix-org/synapse/blob/618e4ab81b70e37bdb8e9224bd84fcfe4b15bdea/synapse/handlers/presence.py#L1769-L1778)
* [x] [`TypingWriterHandler.get_all_typing_updates`](https://github.com/matrix-org/synapse/blob/126a15794c95002560709283640ad412636b29b8/synapse/handlers/typing.py#L423-L435)
* `DeviceWorkerStore.get_cached_device_list_changes`:
	* [x] [`SyncHandler._generate_sync_entry_for_device_list`](https://github.com/matrix-org/synapse/blob/1183c372fa9da01b2667f1b83dab958dad432c68/synapse/handlers/sync.py#L1521-L1552)
* [ ] [`DeviceWorkerStore.get_users_whose_devices_changed`](https://github.com/matrix-org/synapse/blob/13aa29db1ddc925beb35f5f1da8fd1a1bcc91373/synapse/storage/databases/main/devices.py#L831-L846) (I'm fairly certain this is incorrect.)
* [x] [`StreamChangeCache.get_entities_changed`](https://github.com/matrix-org/synapse/blob/13aa29db1ddc925beb35f5f1da8fd1a1bcc91373/synapse/util/caches/stream_change_cache.py#L112-L128)


Discussion:
I think user impact would be missed presence updates and missed device change notifications?
Tangentially related to #14435.
I think in both these situations we need to pull the full list of user IDs from the database.

Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -1764,14 +1764,14 @@ async def _filter_all_presence_updates_for_user(
         Returns:
             A list of presence states for the given user to receive.
         """
+        updated_users = None
         if from_key:
             # Only return updates since the last sync
             updated_users = self.store.presence_stream_cache.get_all_entities_changed(
                 from_key
             )
-            if not updated_users:
-                updated_users = []
 
+        if updated_users is not None:
             # Get the actual presence update for each change
             users_to_state = await self.get_presence_handler().current_state_for_users(
                 updated_users
diff --git a/synapse/storage/databases/main/devices.py b/synapse/storage/databases/main/devices.py
--- a/synapse/storage/databases/main/devices.py
+++ b/synapse/storage/databases/main/devices.py
@@ -842,12 +842,11 @@ async def get_users_whose_devices_changed(
                 user_ids, from_key
             )
 
-        if not user_ids_to_check:
+        # If an empty set was returned, there's nothing to do.
+        if user_ids_to_check is not None and not user_ids_to_check:
             return set()
 
         def _get_users_whose_devices_changed_txn(txn: LoggingTransaction) -> Set[str]:
-            changes: Set[str] = set()
-
             stream_id_where_clause = "stream_id > ?"
             sql_args = [from_key]
 
@@ -858,19 +857,25 @@ def _get_users_whose_devices_changed_txn(txn: LoggingTransaction) -> Set[str]:
             sql = f"""
                 SELECT DISTINCT user_id FROM device_lists_stream
                 WHERE {stream_id_where_clause}
-                AND
             """
 
-            # Query device changes with a batch of users at a time
-            # Assertion for mypy's benefit; see also
-            # https://mypy.readthedocs.io/en/stable/common_issues.html#narrowing-and-inner-functions
-            assert user_ids_to_check is not None
-            for chunk in batch_iter(user_ids_to_check, 100):
-                clause, args = make_in_list_sql_clause(
-                    txn.database_engine, "user_id", chunk
-                )
-                txn.execute(sql + clause, sql_args + args)
-                changes.update(user_id for user_id, in txn)
+            # If the stream change cache gave us no information, fetch *all*
+            # users between the stream IDs.
+            if user_ids_to_check is None:
+                txn.execute(sql, sql_args)
+                return {user_id for user_id, in txn}
+
+            # Otherwise, fetch changes for the given users.
+            else:
+                changes: Set[str] = set()
+
+                # Query device changes with a batch of users at a time
+                for chunk in batch_iter(user_ids_to_check, 100):
+                    clause, args = make_in_list_sql_clause(
+                        txn.database_engine, "user_id", chunk
+                    )
+                    txn.execute(sql + " AND " + clause, sql_args + args)
+                    changes.update(user_id for user_id, in txn)
 
             return changes
 


Test patch: