Repository: matrix-org/synapse
Issue category: Performance Issue

When enabling EDU support for appservices, synapse will start from stream position 0 when fetching presence/receipts
Which is very expensive if you were say, matrix.org. If the appservice hasn't got a stream position in the database, we should go from a reasonable point in time.


Functions to edit:
synapse/handlers/appservice.py:ApplicationServicesHandler._notify_interested_services_ephemeral
synapse/storage/databases/main/receipts.py:ReceiptsWorkerStore.get_linearized_receipts_for_all_rooms

Patch:
diff --git a/changelog.d/8744.bugfix b/changelog.d/8744.bugfix
new file mode 100644
index 000000000000..f8f9630bd6af
--- /dev/null
+++ b/changelog.d/8744.bugfix
@@ -0,0 +1,1 @@
+Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0. 
diff --git a/synapse/handlers/appservice.py b/synapse/handlers/appservice.py
index 9fc844422862..5c6458eb52fc 100644
--- a/synapse/handlers/appservice.py
+++ b/synapse/handlers/appservice.py
@@ -226,7 +226,7 @@ async def _notify_interested_services_ephemeral(
         new_token: Optional[int],
         users: Collection[Union[str, UserID]],
     ):
-        logger.info("Checking interested services for %s" % (stream_key))
+        logger.debug("Checking interested services for %s" % (stream_key))
         with Measure(self.clock, "notify_interested_services_ephemeral"):
             for service in services:
                 # Only handle typing if we have the latest token
diff --git a/synapse/handlers/receipts.py b/synapse/handlers/receipts.py
index c242c409cf26..153cbae7b912 100644
--- a/synapse/handlers/receipts.py
+++ b/synapse/handlers/receipts.py
@@ -158,7 +158,8 @@ async def get_new_events_as(
         if from_key == to_key:
             return [], to_key
 
-        # We first need to fetch all new receipts
+        # Fetch all read receipts for all rooms, up to a limit of 100. This is ordered
+        # by most recent.
         rooms_to_events = await self.store.get_linearized_receipts_for_all_rooms(
             from_key=from_key, to_key=to_key
         )
diff --git a/synapse/storage/databases/main/receipts.py b/synapse/storage/databases/main/receipts.py
index ca7917c9895b..1e7949a3233a 100644
--- a/synapse/storage/databases/main/receipts.py
+++ b/synapse/storage/databases/main/receipts.py
@@ -278,7 +278,8 @@ def f(txn):
     async def get_linearized_receipts_for_all_rooms(
         self, to_key: int, from_key: Optional[int] = None
     ) -> Dict[str, JsonDict]:
-        """Get receipts for all rooms between two stream_ids.
+        """Get receipts for all rooms between two stream_ids, up
+        to a limit of the latest 100 read receipts.
 
         Args:
             to_key: Max stream id to fetch receipts upto.
@@ -294,12 +295,16 @@ def f(txn):
                 sql = """
                     SELECT * FROM receipts_linearized WHERE
                     stream_id > ? AND stream_id <= ?
+                    ORDER BY stream_id DESC
+                    LIMIT 100
                 """
                 txn.execute(sql, [from_key, to_key])
             else:
                 sql = """
                     SELECT * FROM receipts_linearized WHERE
                     stream_id <= ?
+                    ORDER BY stream_id DESC
+                    LIMIT 100
                 """
 
                 txn.execute(sql, [to_key])