Repository: matrix-org/synapse

Issue:
Outgoing presence exploded in 1.35.0rc1
### Description

There seems to be a huge spike in outgoing presence right after 1.35.0rc1 was released. Incoming presence also went up, but not as much.

![image](https://user-images.githubusercontent.com/4224639/121363705-da72a080-c93f-11eb-8736-79a5302ac465.png)

My federation sender is also having trouble with metrics after the 1.36.0rc1 upgrade (which is what I was looking into when I noticed the presence problem), but that might be a separate issue:

![image](https://user-images.githubusercontent.com/4224639/121364408-6e446c80-c940-11eb-9a9a-7abc812600ec.png)

### Version information

- **Version**: 1.36.0rc1
- **Install method**: Docker
- **Workers**: federation sender, 3 generic workers (federation reader, synchrotron, `events` stream writer), appservice sender, media repo


Discussion:
I seem to be seeing the same and incoming presence seems to suggest, I am not alone in that:

![grafik](https://user-images.githubusercontent.com/16312324/121364634-7cd25a00-c927-11eb-8f33-0104aa8d64dc.png)

2 federation senders, 1 event creator and master process (and some other workers).

The two presence related PRs in v1.35.0 are https://github.com/matrix-org/synapse/pull/10014 and https://github.com/matrix-org/synapse/pull/9823. https://github.com/matrix-org/synapse/pull/10014 attempted to prevent presence updates from being accidentally deleted, whereas https://github.com/matrix-org/synapse/pull/9823 deals with optional functionality to send all known user presence to a local or remote user when requested by a Synapse module.

I'm currently looking at the latter as it may explain a large spike in traffic...
I don't see anything unusual in either PR.

#10014 fixed the insertion of rows into the `presence_stream` table, and from what I can tell from my personal homeserver, it seems to be performing as intended (there's no more than one row per user).

#9823 messes about with sending large numbers of user presence out, though I don't see how this could happen unless the `users_to_send_full_presence_to` database table has some rows in it (which is unlikely).

Attempting to reproduce the bug on a homeserver without a federation sender has failed, while another with a federation sender does exhibit similar behaviour (but only *after* upgrading it from v1.35.1 to v1.36.0rc1 + https://github.com/matrix-org/synapse/pull/10149)...
I actually have one duplicated in the `presence_stream` table, would that cause issues?
CPU usage increase (~1.5-2x) since the upgrade 1.34.0 -> 1.35.1

- 4x sync worker (each one increased from ~12% to 17% cpu usage)
- 2x federation_sender worker (each one increased from 7% to 21%)

I also see periodic cpu spikes (exactly every minute) accompanied by a lot of `presence` updates in sync and `/_matrix/federation/v1/send` in federation_sender:

![image](https://user-images.githubusercontent.com/6731714/121498315-984b6c80-c9dc-11eb-9caa-d38616af672c.png)

![image](https://user-images.githubusercontent.com/6731714/121498403-abf6d300-c9dc-11eb-8c44-4758f6f874b7.png)

Open FDs for federation_sender also looks unusual. Before the upgrade open FDs was ~100/worker, with occasional spikes up to 1500/worker. Now they constantly hold ~1500 open FDs.

### Version information

- Version: 1.35.1
- Install method: pip
- Workers: 2 federation_sender, 8 generic workers (4 sync, 1 federation, 1 federation_send, 1 client, 1 send), 1 user_dir
For the sake of completeness, I just heard someone float the idea that federation patterns are likely to change now that the libera.chat bridge is up and running on its own homeserver, rather than being part of matrix.org. But it seems like the observed issues are happening at a variety of times most closely associated with server upgrades.
While I would expect federation patterns to change, I don't think it makes sense, that I am sending 20x as much presence because of an additional server. Inbound traffic barely changed.
I'm also seeing an increase of incoming presence. I have presence disabled, but it is taking a large hit on my system, making it struggle keeping up.



Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -495,9 +495,6 @@ async def notify_from_replication(
             users=users_to_states.keys(),
         )
 
-        # If this is a federation sender, notify about presence updates.
-        await self.maybe_send_presence_to_interested_destinations(states)
-
     async def process_replication_rows(
         self, stream_name: str, instance_name: str, token: int, rows: list
     ):
@@ -519,11 +516,27 @@ async def process_replication_rows(
             for row in rows
         ]
 
-        for state in states:
-            self.user_to_current_state[state.user_id] = state
+        # The list of states to notify sync streams and remote servers about.
+        # This is calculated by comparing the old and new states for each user
+        # using `should_notify(..)`.
+        #
+        # Note that this is necessary as the presence writer will periodically
+        # flush presence state changes that should not be notified about to the
+        # DB, and so will be sent over the replication stream.
+        state_to_notify = []
+
+        for new_state in states:
+            old_state = self.user_to_current_state.get(new_state.user_id)
+            self.user_to_current_state[new_state.user_id] = new_state
+
+            if not old_state or should_notify(old_state, new_state):
+                state_to_notify.append(new_state)
 
         stream_id = token
-        await self.notify_from_replication(states, stream_id)
+        await self.notify_from_replication(state_to_notify, stream_id)
+
+        # If this is a federation sender, notify about presence updates.
+        await self.maybe_send_presence_to_interested_destinations(state_to_notify)
 
     def get_currently_syncing_users_for_replication(self) -> Iterable[str]:
         return [


Test patch: