Repository: matrix-org/synapse

Issue:
appservice_stream_position can get into such a state that ASes break silently
I wasted 3h on arasphere last night trying to work out why it wouldn't send stuff to ASes. turns out the value in that table was somehow mangled. deleting it didn't help, but manually reinserting max streampos did. Plus it's inconsistently named wrt all the other AS stuff :(


Discussion:
so on my HS atm i have:

```
sqlite> select max(stream_ordering) from events;
811112
sqlite> select * from appservice_stream_position;
X|809618
```

and it somehow got into a state that the value in appservice_stream_position was mangled (missing entirely, iirc)
and i unstuck it by manually inserting max(stream_ordering) from events into it

I've also been hit by this, now for the second time during the last week.

The first time I "solved" this be destroying the DBs and starting from scratch as I was still setting things up anyway.

Now on the second time my DB looked like this:
```
sqlite> select max(stream_ordering) from events;                                                                                                                                                                                                                                                                               
82774
sqlite> select * from appservice_stream_position;
X|6481
```

I was able to fix it by setting the stream_ordering column in appservice_stream_position to max(stream_ordering) from events like @ara4n said even though I was not missing the row.

`
sqlite> update appservice_stream_position set stream_ordering=82774;
`

@kegsay any idea?
I also had this issue, but correcting the values fixed it. 
@ara4n no - I didn't do any of the stream position stuff when I did the original Synapse AS API implementation. You'd need to ask @erikjohnston.
I had the same issue, fixed it via this.

Then I joined a room (#riot:matrix.org) noone on the synapse server was in yet, and it got off-sync again and i had to re-apply the fix. Perhaps this'll help to hunt down the bug
This is still very much an issue. There's absolutely no information in the appservice worker's log, as it seems to successfully send a transaction to an appservice, then it stops trying ever again. It does still receive rdata caches/events, and the following appears between rdata updates, but eventually it also stops receiving rdata:
```
appservice - 2017-10-11 02:55:54,347 - synapse.storage.TIME - 219 - INFO - - Total database time: 0.000% {set_appservice_last_pos(0): 0.000%, get_users_in_room(0): 0.000%, get_user_by_id(0): 0.000%} {}
```

 A restart of the appservice worker did not help too much (it started receiving rdata again, but not sending transactions to appservices). I had to do the update query manually and restart the entire synapse instance.

Are there particular flags/logging I can leave on to try and debug this if it happens again? (this is the 3rd time in ~5 months it has happened).
This happened once again for me (t2bot.io), and I don't see anything useful in the logs. I have exported them for viewing if desired though (they have tokens and such, so I'd rather not post them publicly).

and again :(

```
synapse=# select max(stream_ordering) from events;
   max
----------
 65145242
(1 row)

synapse=# select * from appservice_stream_position;
 lock | stream_ordering
------+-----------------
 X    |        65097572
(1 row)
```

Logs are also available for this event.
This is becoming a trend where every couple months I'm having to meddle with the database. This has once again happened.

Is there a lead on a possible cause for this happening? 
ok so let me just try to understand the symptoms here:

* synapse stops sending events to appservices
* inspection of the db shows that appservice_stream_position is stuck at an old event
* restarting synapse does not fix it
* updating appservice_stream_position to a later value (and restarting synapse) does fix it

Is that correct?

If so, the only plausible conclusion is that synapse's appservice sender gets stuck on particular events; by updating appservice_stream_position, we skip the problematic event and things get going again.

Since synapse handles events in batches of 100, the problem might lie anywhere in the next 100 events after the sticking point (ie, stream_orderings 65097573 to 65097672 in @turt2live 's latest example). I'm not quite sure how we can go about narrowing down the problem further :/.

It's *possible* that some recent improvements to error handling in synapse (in particular, https://github.com/matrix-org/synapse/pull/3138/files#diff-df9c8550f76f9aca4912eaa72d55379eR121) will reveal the problem here.
That's pretty much the problem. The additional issue is that there's no logging whatsoever for an issue occuring: it just... stops

Improved error handling may catch it. I'm also looking at adding much more verbose logging in the area to see if it can catch what's going on.
We got bitten by this again today, because of some events that existed in the `events` table, yet `DataStore._get_events` couldn't find any of them so the stream position couldn't advance. I believe we should skip over any events we couldn't fetch.
Ran into this again today, fast forwarded with this query:
```sql
update appservice_stream_position set stream_ordering = (select max(stream_ordering) from events);
```

No synapse restart required
got crazy setting up mautrix-telegram, thanks @TimePath for the postgresql command

```
[20:37:52] root@matrix:~# sudo -u postgres psql synapse
could not change directory to "/root": Permission denied
psql (11.7 (Debian 11.7-0+deb10u1))
Type "help" for help.

synapse=# update appservice_stream_position set stream_ordering = (select max(stream_ordering) from events);
```

after applying this, I see transactions between synapse and mautrix-telegram happening, phew
please can people who observe this issue share logs from the homeserver.
Checked my logs and I don't see anything relevant. We can coordinate privately about the logs you need
closing this for lack of useful response from the reporters.
it's because the logs are silent - this still happens. What logs are you hoping to see?
ok, well that is useful information in itself. Last I heard we'd made some logging changes and were waiting to hear if that showed anything up.

If you could share INFO-level logs from the time of the last successful transaction to a point where you would definitely have expected there to be another transaction, that would be a good starting point to check we're all on the same page.
my logs have since rolled over (happened last week), but any day now it's due to happen again.
I had this on my homeserver. Updating `appservice_stream_position` solved the problem.

I can share logs from today, but not from around when it stopped working. I don't use Telegram much, so based on my Telegram/Matrix rooms, all I know is that it broke sometime between April 16th and August 30th. I only have about two weeks worth of logs on my HS.
Based on the timestamp off the event that `appservice_stream_position` was stuck on, it broke May 20th

Edit: I hacked together a script to catch this next time it breaks: https://gist.github.com/Twi1ightSparkle/53d33ce50c813f84fcccb3abe8dcdfba
@richvdh This just happened and I have logs this time. Would you like them personally or should I send them to someone else?
Looking at logs and prometheus from @turt2live (thanks!), I'm seeing the following background process in flight:

https://github.com/matrix-org/synapse/blob/a7a913918cf04c6d900223b19e719fafbbe94efa/synapse/handlers/appservice.py#L86-L179

which is the loop that takes streams from the events table and dispatches to the various appservice classes for sending out. However, the outer loop is stuck, as the metric counts at the end of the loop aren't progressing:

https://github.com/matrix-org/synapse/blob/a7a913918cf04c6d900223b19e719fafbbe94efa/synapse/handlers/appservice.py#L164

But, the inner loop *is* progressing, at a rate of one iteration per minute, based on:

https://github.com/matrix-org/synapse/blob/a7a913918cf04c6d900223b19e719fafbbe94efa/synapse/handlers/appservice.py#L137-L139

During that time we see requests in the logs and metrics timing out after one minute:



```
INFO - notify_interested_services-743543- Error sending request to  GET https://<discord_appservice>/users/<user_id>?access_token=<redacted>: RequestTimedOutError 504: Timeout waiting for response from remote server
```

which is probably being triggered from the loop by:

https://github.com/matrix-org/synapse/blob/a7a913918cf04c6d900223b19e719fafbbe94efa/synapse/handlers/appservice.py#L116-L118

Since we won't process the next batch of events until we've finished the current batch, this one minute lag per event processed is enough to cause all appservices to fall behind. (Side note: @turt2live has a patch to drop "old" events from being sent out, so we end up not seeing *any* transactions being sent out)


Other observations:

* Things quite obviously get stuck going slow at a particular instance
* At first we see events going out to the various different appservices, but then for the rest of the time we *only* see membership events for the discord bridge.
* It looks like the user query hit to the discord appservice doesn't at first

I'm not entirely sure what's going on here, but there seems to suddenly be a large number of discord membership events all at once? Does the discord bridge then fail to handle the user querying correctly? Perhaps all user queries to the discord appservice are slow, and its only a noticeable problem due to the sheer number of membership requests all at once.
It still seems to be happening, Synapse 1.31.0 stopped sending events to appservice rooms suddenly.
`appservice_stream_position` mismatched to `select max(stream_ordering) from events`.
Fixed by https://github.com/matrix-org/synapse/issues/1834#issuecomment-295484711.
Thanks so much, just ran into this issue!
We ran into this for the Telegram bridge at SNT, and were able to identify the problem (at least for us): if all events that *would* be returned by `get_new_events_for_appservice` are rejected, the `events` list will be empty.

PR incoming.

Resolving patch:
diff --git a/synapse/handlers/appservice.py b/synapse/handlers/appservice.py
--- a/synapse/handlers/appservice.py
+++ b/synapse/handlers/appservice.py
@@ -87,7 +87,8 @@ async def _notify_interested_services(self, max_token: RoomStreamToken):
             self.is_processing = True
             try:
                 limit = 100
-                while True:
+                upper_bound = -1
+                while upper_bound < self.current_max:
                     (
                         upper_bound,
                         events,
@@ -95,9 +96,6 @@ async def _notify_interested_services(self, max_token: RoomStreamToken):
                         self.current_max, limit
                     )
 
-                    if not events:
-                        break
-
                     events_by_room = {}  # type: Dict[str, List[EventBase]]
                     for event in events:
                         events_by_room.setdefault(event.room_id, []).append(event)
@@ -153,9 +151,6 @@ async def handle_room_events(events):
 
                     await self.store.set_appservice_last_pos(upper_bound)
 
-                    now = self.clock.time_msec()
-                    ts = await self.store.get_received_ts(events[-1].event_id)
-
                     synapse.metrics.event_processing_positions.labels(
                         "appservice_sender"
                     ).set(upper_bound)
@@ -168,12 +163,16 @@ async def handle_room_events(events):
 
                     event_processing_loop_counter.labels("appservice_sender").inc()
 
-                    synapse.metrics.event_processing_lag.labels(
-                        "appservice_sender"
-                    ).set(now - ts)
-                    synapse.metrics.event_processing_last_ts.labels(
-                        "appservice_sender"
-                    ).set(ts)
+                    if events:
+                        now = self.clock.time_msec()
+                        ts = await self.store.get_received_ts(events[-1].event_id)
+
+                        synapse.metrics.event_processing_lag.labels(
+                            "appservice_sender"
+                        ).set(now - ts)
+                        synapse.metrics.event_processing_last_ts.labels(
+                            "appservice_sender"
+                        ).set(ts)
             finally:
                 self.is_processing = False
 


Test patch:
diff --git a/tests/handlers/test_appservice.py b/tests/handlers/test_appservice.py
--- a/tests/handlers/test_appservice.py
+++ b/tests/handlers/test_appservice.py
@@ -57,10 +57,10 @@ def test_notify_interested_services(self):
             sender="@someone:anywhere", type="m.room.message", room_id="!foo:bar"
         )
         self.mock_store.get_new_events_for_appservice.side_effect = [
-            make_awaitable((0, [event])),
             make_awaitable((0, [])),
+            make_awaitable((1, [event])),
         ]
-        self.handler.notify_interested_services(RoomStreamToken(None, 0))
+        self.handler.notify_interested_services(RoomStreamToken(None, 1))
 
         self.mock_scheduler.submit_event_for_as.assert_called_once_with(
             interested_service, event
@@ -77,7 +77,6 @@ def test_query_user_exists_unknown_user(self):
         self.mock_as_api.query_user.return_value = make_awaitable(True)
         self.mock_store.get_new_events_for_appservice.side_effect = [
             make_awaitable((0, [event])),
-            make_awaitable((0, [])),
         ]
 
         self.handler.notify_interested_services(RoomStreamToken(None, 0))
@@ -95,7 +94,6 @@ def test_query_user_exists_known_user(self):
         self.mock_as_api.query_user.return_value = make_awaitable(True)
         self.mock_store.get_new_events_for_appservice.side_effect = [
             make_awaitable((0, [event])),
-            make_awaitable((0, [])),
         ]
 
         self.handler.notify_interested_services(RoomStreamToken(None, 0))