Repository: matrix-org/synapse

Issue:
"int not iterable" during presence timeout handling
Synapse: 0.33.2 with unrelated patches.

```
homeserver_1 - 2018-08-17 18:27:51,422 - synapse.handlers.presence - 345 - INFO - Handling presence timeouts
homeserver_1 - 2018-08-17 18:27:51,423 - synapse.handlers.presence - 386 - ERROR - Exception in _handle_timeouts loop
Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/presence.py", line 364, in _handle_timeouts
    self.external_process_last_updated_ms.pop(process_id, ())
TypeError: 'int' object is not iterable
```


Discussion:
Updated stacktrace:

```
homeserver_1 - 2018-10-02 16:15:02,799 - synapse.handlers.presence - 350 - INFO -  - Handling presence timeouts
homeserver_1 - 2018-10-02 16:15:02,799 - synapse.handlers.presence - 391 - ERROR -  - Exception in _handle_timeouts loop
Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/presence.py", line 369, in _handle_timeouts
    self.external_process_last_updated_ms.pop(process_id, ())
TypeError: 'int' object is not iterable
```
Traceback with v1.3.1:

```
2019-09-26 00:38:53,827 synapse.metrics.background_process_metrics(202) [ERROR   ]: handle_presence_timeouts-189 Background process 'handle_presence_timeouts' threw an exception
Traceback (most recent call last):
  File "/synapse-venv/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 200, in run
    yield func(*args, **kwargs)
  File "/synapse-venv/lib/python3.7/site-packages/synapse/handlers/presence.py", line 353, in _handle_timeouts
    self.external_process_last_updated_ms.pop(process_id, ())
TypeError: 'int' object is not iterable
```
this code looks completely broken. I'm sure it's never worked. Unfortunately fixing it is going to require grokking what it is meant to do and writing some tests.
We're seeing those exceptions very frequently.

Recently we've also started to experience serious presence update issues (e.g. users on the other side of a federated server don't receive presence updates).

Could this be a cause for that?
Can you help us with this issue? Raiden atm somehow relies on properly working presence states. And somehow since 1 month, this is not working anymore. 

See problem description here https://gist.github.com/ulope/1310338587ee7a794e550aca2a0c5d6d

We would be willing to pay for a quick solution to that problem if needed. 
@Dominik1999 @ulope 
We're working on it as a matter of urgency, will get a fix out as soon as we can, though as above, we don't consider it to be a trivial.

Resolving patch:
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -24,6 +24,7 @@
 
 import logging
 from contextlib import contextmanager
+from typing import Dict, Set
 
 from six import iteritems, itervalues
 
@@ -179,8 +180,9 @@ def __init__(self, hs):
         # we assume that all the sync requests on that process have stopped.
         # Stored as a dict from process_id to set of user_id, and a dict of
         # process_id to millisecond timestamp last updated.
-        self.external_process_to_current_syncs = {}
-        self.external_process_last_updated_ms = {}
+        self.external_process_to_current_syncs = {}  # type: Dict[int, Set[str]]
+        self.external_process_last_updated_ms = {}  # type: Dict[int, int]
+
         self.external_sync_linearizer = Linearizer(name="external_sync_linearizer")
 
         # Start a LoopingCall in 30s that fires every 5s.
@@ -349,10 +351,13 @@ def _handle_timeouts(self):
             if now - last_update > EXTERNAL_PROCESS_EXPIRY
         ]
         for process_id in expired_process_ids:
+            # For each expired process drop tracking info and check the users
+            # that were syncing on that process to see if they need to be timed
+            # out.
             users_to_check.update(
-                self.external_process_last_updated_ms.pop(process_id, ())
+                self.external_process_to_current_syncs.pop(process_id, ())
             )
-            self.external_process_last_update.pop(process_id)
+            self.external_process_last_updated_ms.pop(process_id)
 
         states = [
             self.user_to_current_state.get(user_id, UserPresenceState.default(user_id))


Test patch:
diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py
--- a/tests/handlers/test_presence.py
+++ b/tests/handlers/test_presence.py
@@ -22,6 +22,7 @@
 from synapse.events import room_version_to_event_format
 from synapse.events.builder import EventBuilder
 from synapse.handlers.presence import (
+    EXTERNAL_PROCESS_EXPIRY,
     FEDERATION_PING_INTERVAL,
     FEDERATION_TIMEOUT,
     IDLE_TIMER,
@@ -413,6 +414,44 @@ def test_last_active(self):
         self.assertEquals(state, new_state)
 
 
+class PresenceHandlerTestCase(unittest.HomeserverTestCase):
+    def prepare(self, reactor, clock, hs):
+        self.presence_handler = hs.get_presence_handler()
+        self.clock = hs.get_clock()
+
+    def test_external_process_timeout(self):
+        """Test that if an external process doesn't update the records for a while
+        we time out their syncing users presence.
+        """
+        process_id = 1
+        user_id = "@test:server"
+
+        # Notify handler that a user is now syncing.
+        self.get_success(
+            self.presence_handler.update_external_syncs_row(
+                process_id, user_id, True, self.clock.time_msec()
+            )
+        )
+
+        # Check that if we wait a while without telling the handler the user has
+        # stopped syncing that their presence state doesn't get timed out.
+        self.reactor.advance(EXTERNAL_PROCESS_EXPIRY / 2)
+
+        state = self.get_success(
+            self.presence_handler.get_state(UserID.from_string(user_id))
+        )
+        self.assertEqual(state.state, PresenceState.ONLINE)
+
+        # Check that if the external process timeout fires, then the syncing
+        # user gets timed out
+        self.reactor.advance(EXTERNAL_PROCESS_EXPIRY)
+
+        state = self.get_success(
+            self.presence_handler.get_state(UserID.from_string(user_id))
+        )
+        self.assertEqual(state.state, PresenceState.OFFLINE)
+
+
 class PresenceJoinTestCase(unittest.HomeserverTestCase):
     """Tests remote servers get told about presence of users in the room when
     they join and when new local users join.