Repository: matrix-org/synapse

Issue:
Repeated "db txn 'update_presence' from sentinel context" log messages
### Description

After upgrading my homeserver from 0.33 to 0.34.1.1 (and switching to python3 in a new virtual env, if that is relevant) Synapse writes the following log-entries every minute into the logs:
```
2019-01-17 19:40:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 19:40:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
```
The first hour does not contain anything else:
```
2019-01-17 13:44:05,223 - root - 211 - WARNING - None - ***** STARTING SERVER *****
2019-01-17 13:44:05,282 - root - 214 - WARNING - None - Server /home/matrix/synapse/env3/lib/python3.5/site-packages/synapse/app/homeserver.py version 0.34.1.1
2019-01-17 13:44:05,596 - synapse.app.homeserver - 223 - WARNING - None - Not enabling webclient resource, as web_client_location is unset.
2019-01-17 13:44:05,672 - synapse.app._base - 194 - WARNING - None - Failed to listen on 0.0.0.0, continuing because listening on [::]
2019-01-17 13:44:05,701 - synapse.app.homeserver - 223 - WARNING - None - Not enabling webclient resource, as web_client_location is unset.
2019-01-17 13:46:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:46:05,582 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:47:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:47:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:48:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:48:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:49:05,576 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:49:05,580 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:50:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:50:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:51:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:51:05,582 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:52:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:52:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:53:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:53:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:54:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:54:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:55:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:55:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:56:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:56:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:57:05,576 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:57:05,580 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:58:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:58:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 13:59:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 13:59:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 14:00:05,576 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 14:00:05,580 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 14:01:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-17 14:01:05,579 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-17 14:02:05,575 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
```

After a while, the occasional "Not sending response to request ... client already disconnected"-messages start showing as well. 

Before the upgrade I had no such messages in the logs.

Everything seems to work, but I would prefer to get rid of these messages, as they grow the logs very fast.

### Version information

- **Homeserver**: Selfhosted homeserver
- **Version**: 0.34.1.1
- **Install method**:  pip
- **Platform**: raspian, rapsberrypi, bare metal


Discussion:
Thanks for the clear report @zwieberl.

The fact that these are happening reliably at 5 seconds past the minute points to a timer event. So I guess it's this: https://github.com/matrix-org/synapse/blob/v0.34.1.1/synapse/handlers/presence.py#L213
I think you are right. Switched to INFO-level and got this:
```
2019-01-18 09:48:01,593 - synapse.handlers.presence - 350 - INFO -  - Handling presence timeouts
2019-01-18 09:48:01,602 - synapse.handlers.typing - 89 - INFO -  - Checking for typing timeouts
2019-01-18 09:48:06,593 - synapse.handlers.presence - 350 - INFO -  - Handling presence timeouts
2019-01-18 09:48:06,601 - synapse.handlers.presence - 257 - INFO -  - Performing _persist_unpersisted_changes. Persisting 1 unpersisted changes
2019-01-18 09:48:06,603 - synapse.storage._base - 319 - WARNING -  - Starting db txn 'update_presence' from sentinel context
2019-01-18 09:48:06,606 - synapse.storage._base - 355 - WARNING -  - Starting db connection from sentinel context: metrics will be lost
2019-01-18 09:48:06,616 - synapse.handlers.typing - 89 - INFO -  - Checking for typing timeouts
2019-01-18 09:48:06,632 - synapse.handlers.presence - 269 - INFO -  - Finished _persist_unpersisted_changes
```
So it seems to be indeed the the `_persist_unpersisted_changes`-handler. 
Not sure what that means, though

Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -182,17 +182,27 @@ def __init__(self, hs):
         # Start a LoopingCall in 30s that fires every 5s.
         # The initial delay is to allow disconnected clients a chance to
         # reconnect before we treat them as offline.
+        def run_timeout_handler():
+            return run_as_background_process(
+                "handle_presence_timeouts", self._handle_timeouts
+            )
+
         self.clock.call_later(
             30,
             self.clock.looping_call,
-            self._handle_timeouts,
+            run_timeout_handler,
             5000,
         )
 
+        def run_persister():
+            return run_as_background_process(
+                "persist_presence_changes", self._persist_unpersisted_changes
+            )
+
         self.clock.call_later(
             60,
             self.clock.looping_call,
-            self._persist_unpersisted_changes,
+            run_persister,
             60 * 1000,
         )
 
@@ -229,6 +239,7 @@ def _on_shutdown(self):
         )
 
         if self.unpersisted_users_changes:
+
             yield self.store.update_presence([
                 self.user_to_current_state[user_id]
                 for user_id in self.unpersisted_users_changes
@@ -240,30 +251,18 @@ def _persist_unpersisted_changes(self):
         """We periodically persist the unpersisted changes, as otherwise they
         may stack up and slow down shutdown times.
         """
-        logger.info(
-            "Performing _persist_unpersisted_changes. Persisting %d unpersisted changes",
-            len(self.unpersisted_users_changes)
-        )
-
         unpersisted = self.unpersisted_users_changes
         self.unpersisted_users_changes = set()
 
         if unpersisted:
+            logger.info(
+                "Persisting %d upersisted presence updates", len(unpersisted)
+            )
             yield self.store.update_presence([
                 self.user_to_current_state[user_id]
                 for user_id in unpersisted
             ])
 
-        logger.info("Finished _persist_unpersisted_changes")
-
-    @defer.inlineCallbacks
-    def _update_states_and_catch_exception(self, new_states):
-        try:
-            res = yield self._update_states(new_states)
-            defer.returnValue(res)
-        except Exception:
-            logger.exception("Error updating presence")
-
     @defer.inlineCallbacks
     def _update_states(self, new_states):
         """Updates presence of users. Sets the appropriate timeouts. Pokes
@@ -338,45 +337,41 @@ def _handle_timeouts(self):
         logger.info("Handling presence timeouts")
         now = self.clock.time_msec()
 
-        try:
-            with Measure(self.clock, "presence_handle_timeouts"):
-                # Fetch the list of users that *may* have timed out. Things may have
-                # changed since the timeout was set, so we won't necessarily have to
-                # take any action.
-                users_to_check = set(self.wheel_timer.fetch(now))
-
-                # Check whether the lists of syncing processes from an external
-                # process have expired.
-                expired_process_ids = [
-                    process_id for process_id, last_update
-                    in self.external_process_last_updated_ms.items()
-                    if now - last_update > EXTERNAL_PROCESS_EXPIRY
-                ]
-                for process_id in expired_process_ids:
-                    users_to_check.update(
-                        self.external_process_last_updated_ms.pop(process_id, ())
-                    )
-                    self.external_process_last_update.pop(process_id)
+        # Fetch the list of users that *may* have timed out. Things may have
+        # changed since the timeout was set, so we won't necessarily have to
+        # take any action.
+        users_to_check = set(self.wheel_timer.fetch(now))
+
+        # Check whether the lists of syncing processes from an external
+        # process have expired.
+        expired_process_ids = [
+            process_id for process_id, last_update
+            in self.external_process_last_updated_ms.items()
+            if now - last_update > EXTERNAL_PROCESS_EXPIRY
+        ]
+        for process_id in expired_process_ids:
+            users_to_check.update(
+                self.external_process_last_updated_ms.pop(process_id, ())
+            )
+            self.external_process_last_update.pop(process_id)
 
-                states = [
-                    self.user_to_current_state.get(
-                        user_id, UserPresenceState.default(user_id)
-                    )
-                    for user_id in users_to_check
-                ]
+        states = [
+            self.user_to_current_state.get(
+                user_id, UserPresenceState.default(user_id)
+            )
+            for user_id in users_to_check
+        ]
 
-                timers_fired_counter.inc(len(states))
+        timers_fired_counter.inc(len(states))
 
-                changes = handle_timeouts(
-                    states,
-                    is_mine_fn=self.is_mine_id,
-                    syncing_user_ids=self.get_currently_syncing_users(),
-                    now=now,
-                )
+        changes = handle_timeouts(
+            states,
+            is_mine_fn=self.is_mine_id,
+            syncing_user_ids=self.get_currently_syncing_users(),
+            now=now,
+        )
 
-            run_in_background(self._update_states_and_catch_exception, changes)
-        except Exception:
-            logger.exception("Exception in _handle_timeouts loop")
+        return self._update_states(changes)
 
     @defer.inlineCallbacks
     def bump_presence_active_time(self, user):


Test patch: