Repository: matrix-org/synapse

Issue:
SerializationFailure on appservice outgoing transaction (updating `last_txn`)
### Description

We are seeing AS' go into recoverer mode after a serialization failure when updating `application_services_state.last_txn` (https://github.com/matrix-org/synapse/blob/v1.48.0/synapse/storage/databases/main/appservice.py#L272). We recently fixed a similar issue in https://github.com/matrix-org/synapse/issues/11195 which targets the same database row, so I suspect the issue now is that this `last_txn` update occaisionally clashes with the stream position update we added the linearizer for.

I see that the `last_txn` implementation was added a few years back (https://github.com/matrix-org/synapse/commit/0a60bbf4fac4262da3fee702ca46d2f019597ef1) and hasn't changed since - as far as I can tell the only use of this column is to check the previous value hasn't incremented more than one. This does appear to get triggered from time to time still - I can see 12 log entires in the last 7 days in our deployment.

### Possible Solutions

- move this column to it's own table, not great but would get around the immediate issue
- fix the ordering issue (or work around it) and remove the column/check entirely

I notice the recoverer just pulls the lowest transaction and sends that - could the entire sending mechanism work just like that instead? Simply have the `_TransactionController` write out transactions to the database in the batches and have process (currently `_Recoverer`) that just pulls from these constantly. This would remove any need to track the last transaction ID for a given AS.

### Version information

- v1.48
- worker based deployment with dedicated appservice pusher worker


Discussion:
Update on this: have been looking at the code here and I think the scheduler/AS handler is likely to drop events during a restart of synapse - notably the events are batched in memory as attributes on the `_ServiceQueuer` instance.

This means if the AS pusher instance is in recoverer mode and during that time is restarted, any events sat in memory will be lost as they don't exist on the database. The AS handler completes the `notify_interested_services` background task without any confirmation the events have been stored as an AS transaction in the database.

Note: this appears to be the case when not in recoverer mode as well, if a request to the AS is ongoing any events currently being held in memory would also be lost upon restart of the instance.
`SerializationFailure`s aren't, in themselves, problems - we will generally retry any transactions that hit a `SerializationFailure`. In any case, we should frame this issue in terms of the user-visible symptom, rather than the cause. You say the AS goes into recoverer (recovery?) mode, but again, I'm not sure what impact that actually has on the system - please could you clarify.

Dropping events sounds like a different problem - I suggest opening a separate issue for that.

In general the AS support code isn't something that changes often, so I for one am not really familiar with how it works (or even how it's meant to work), which means it's hard to answer questions like "could the sending mechanism work differently".

> `SerializationFailure`s aren't, in themselves, problems - we will generally retry any transactions that hit a `SerializationFailure`. In any case, we should frame this issue in terms of the user-visible symptom, rather than the cause. You say the AS goes into recoverer (recovery?) mode, but again, I'm not sure what impact that actually has on the system - please could you clarify.

👍 wasn't clear there - just re-checked and the problem we're seeing looks like:

+ Encounter `SerializationError` in normal transaction handling
+ Enter recovery mode
+ Encounter a second `SerializationError` whilst in recovery
+ Recoverer then triggers the backoff code, which causes transactions to build up

Neither the AS or appservice pusher worker appear to be under any load so it should be possible to process transactions much quicker. The serialization error, although not a direct problem, is causing the backoff which is a problem. I think this better explains the issue we're seeing!

> Dropping events sounds like a different problem - I suggest opening a separate issue for that.

👍 - we did some poking around what transactions generally look like and they're nowhere near the 100 event limit, so the issue here is much reduced, will open another issue to track it.

> Recoverer then triggers the backoff code, which causes transactions to build up

so the end result is slow delivery to the AS?
> > Recoverer then triggers the backoff code, which causes transactions to build up
> 
> so the end result is slow delivery to the AS?

Yep, and this leads to a backlog of AS transactions to process and we end up in a situation where the incoming AS txn rate is greater than processed.

I believe part of the issue is that during high AS traffic the `application_services_state` table is written to by both the handler to update the stream position and also the scheduler to track the latest txn. Thus during these periods of heavy AS traffic the `SerializationError` is much more likely to occur.
Looping back on this, I may have some time later this week to work on a fix so would be good to agree an apporach.

I believe the crux of the problem is that the `application_services_state` table is used both to track ephemeral stream ID position (`read_receipt_stream_id` & `presence_stream_id`) and also the AS transaction state (`state`, `last_txn`). The issue with this approach is that the AS handler and AS sending work independently of each other, leading to serialization errors on the table.

Did a quick scan of the issues - perhaps https://github.com/matrix-org/synapse/issues/11567 is the resolution here to allow parallel updates to the different columns (unsure if that's the case with the changed isolation level)?
> Did a quick scan of the issues - perhaps #11567 is the resolution here to allow parallel updates to the different columns (unsure if that's the case with the changed isolation level)?

Using the `READ COMMITTED` isolation level seems like a good fix:

### With `REPEATABLE READ`
| connection 1                                                             | connection 2                                                                            |
|--------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| BEGIN TRANSACTION;                                                       | BEGIN TRANSACTION;                                                                      |
| SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;                         | SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;                                        |
| UPDATE application_services_state SET last_txn = 2 WHERE as_id = 'test'; |                                                                                         |
|                                                                          | UPDATE application_services_state SET read_receipt_stream_id = 20 WHERE as_id = 'test'; |
|                                                                          | &nbsp;&nbsp;&nbsp;&nbsp;(blocks)                                                        |
| COMMIT;                                                                  | &nbsp;&nbsp;&nbsp;&nbsp;ERROR:  could not serialize access due to concurrent update     |


### With `READ COMMITTED`
| connection 1                                                             | connection 2                                                                            |
|--------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| BEGIN TRANSACTION;                                                       | BEGIN TRANSACTION;                                                                      |
| SET TRANSACTION ISOLATION LEVEL READ COMMITTED;                          | SET TRANSACTION ISOLATION LEVEL READ COMMITTED;                                         |
| UPDATE application_services_state SET last_txn = 2 WHERE as_id = 'test'; |                                                                                         |
|                                                                          | UPDATE application_services_state SET read_receipt_stream_id = 20 WHERE as_id = 'test'; |
|                                                                          | &nbsp;&nbsp;&nbsp;&nbsp;(blocks)                                                        |
| COMMIT;                                                                  | &nbsp;&nbsp;&nbsp;&nbsp;(unblocks)                                                      |
|                                                                          | COMMIT;                                                                                 |

```
SELECT * FROM application_services_state WHERE as_id = 'test';
 as_id | state | last_txn | read_receipt_stream_id | presence_stream_id | to_device_stream_id
-------+-------+----------+------------------------+--------------------+---------------------
 test  |       |        2 |                     20 |                    |
(1 row)
```

Resolving patch:
diff --git a/synapse/storage/databases/main/appservice.py b/synapse/storage/databases/main/appservice.py
--- a/synapse/storage/databases/main/appservice.py
+++ b/synapse/storage/databases/main/appservice.py
@@ -29,6 +29,8 @@
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
 from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
+from synapse.storage.types import Cursor
+from synapse.storage.util.sequence import build_sequence_generator
 from synapse.types import DeviceListUpdates, JsonDict
 from synapse.util import json_encoder
 from synapse.util.caches.descriptors import _CacheContext, cached
@@ -72,6 +74,22 @@ def __init__(
         )
         self.exclusive_user_regex = _make_exclusive_regex(self.services_cache)
 
+        def get_max_as_txn_id(txn: Cursor) -> int:
+            logger.warning("Falling back to slow query, you should port to postgres")
+            txn.execute(
+                "SELECT COALESCE(max(txn_id), 0) FROM application_services_txns"
+            )
+            return txn.fetchone()[0]  # type: ignore
+
+        self._as_txn_seq_gen = build_sequence_generator(
+            db_conn,
+            database.engine,
+            get_max_as_txn_id,
+            "application_services_txn_id_seq",
+            table="application_services_txns",
+            id_column="txn_id",
+        )
+
         super().__init__(database, db_conn, hs)
 
     def get_app_services(self):
@@ -239,21 +257,7 @@ async def create_appservice_txn(
         """
 
         def _create_appservice_txn(txn):
-            # work out new txn id (highest txn id for this service += 1)
-            # The highest id may be the last one sent (in which case it is last_txn)
-            # or it may be the highest in the txns list (which are waiting to be/are
-            # being sent)
-            last_txn_id = self._get_last_txn(txn, service.id)
-
-            txn.execute(
-                "SELECT MAX(txn_id) FROM application_services_txns WHERE as_id=?",
-                (service.id,),
-            )
-            highest_txn_id = txn.fetchone()[0]
-            if highest_txn_id is None:
-                highest_txn_id = 0
-
-            new_txn_id = max(highest_txn_id, last_txn_id) + 1
+            new_txn_id = self._as_txn_seq_gen.get_next_id_txn(txn)
 
             # Insert new txn into txn table
             event_ids = json_encoder.encode([e.event_id for e in events])
@@ -286,25 +290,8 @@ async def complete_appservice_txn(
             txn_id: The transaction ID being completed.
             service: The application service which was sent this transaction.
         """
-        txn_id = int(txn_id)
 
         def _complete_appservice_txn(txn):
-            # Debugging query: Make sure the txn being completed is EXACTLY +1 from
-            # what was there before. If it isn't, we've got problems (e.g. the AS
-            # has probably missed some events), so whine loudly but still continue,
-            # since it shouldn't fail completion of the transaction.
-            last_txn_id = self._get_last_txn(txn, service.id)
-            if (last_txn_id + 1) != txn_id:
-                logger.error(
-                    "appservice: Completing a transaction which has an ID > 1 from "
-                    "the last ID sent to this AS. We've either dropped events or "
-                    "sent it to the AS out of order. FIX ME. last_txn=%s "
-                    "completing_txn=%s service_id=%s",
-                    last_txn_id,
-                    txn_id,
-                    service.id,
-                )
-
             # Set current txn_id for AS to 'txn_id'
             self.db_pool.simple_upsert_txn(
                 txn,
@@ -376,17 +363,6 @@ def _get_oldest_unsent_txn(txn):
             device_list_summary=DeviceListUpdates(),
         )
 
-    def _get_last_txn(self, txn, service_id: Optional[str]) -> int:
-        txn.execute(
-            "SELECT last_txn FROM application_services_state WHERE as_id=?",
-            (service_id,),
-        )
-        last_txn_id = txn.fetchone()
-        if last_txn_id is None or last_txn_id[0] is None:  # no row exists
-            return 0
-        else:
-            return int(last_txn_id[0])  # select 'last_txn' col
-
     async def set_appservice_last_pos(self, pos: int) -> None:
         def set_appservice_last_pos_txn(txn):
             txn.execute(
diff --git a/synapse/storage/schema/__init__.py b/synapse/storage/schema/__init__.py
--- a/synapse/storage/schema/__init__.py
+++ b/synapse/storage/schema/__init__.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-SCHEMA_VERSION = 68  # remember to update the list below when updating
+SCHEMA_VERSION = 69  # remember to update the list below when updating
 """Represents the expectations made by the codebase about the database schema
 
 This should be incremented whenever the codebase changes its requirements on the
@@ -58,6 +58,9 @@
     - event_reference_hashes is no longer read.
     - `events` has `state_key` and `rejection_reason` columns, which are populated for
       new events.
+
+Changes in SCHEMA_VERSION = 69:
+    - Use sequence to generate future `application_services_txns.txn_id`s
 """
 
 
diff --git a/synapse/storage/schema/main/delta/69/01as_txn_seq.py b/synapse/storage/schema/main/delta/69/01as_txn_seq.py
new file mode 100644
--- /dev/null
+++ b/synapse/storage/schema/main/delta/69/01as_txn_seq.py
@@ -0,0 +1,44 @@
+# Copyright 2022 Beeper
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""
+Adds a postgres SEQUENCE for generating application service transaction IDs.
+"""
+
+from synapse.storage.engines import PostgresEngine
+
+
+def run_create(cur, database_engine, *args, **kwargs):
+    if isinstance(database_engine, PostgresEngine):
+        # If we already have some AS TXNs we want to start from the current
+        # maximum value. There are two potential places this is stored - the
+        # actual TXNs themselves *and* the AS state table. At time of migration
+        # it is possible the TXNs table is empty so we must include the AS state
+        # last_txn as a potential option, and pick the maximum.
+
+        cur.execute("SELECT COALESCE(max(txn_id), 0) FROM application_services_txns")
+        row = cur.fetchone()
+        txn_max = row[0]
+
+        cur.execute("SELECT COALESCE(max(last_txn), 0) FROM application_services_state")
+        row = cur.fetchone()
+        last_txn_max = row[0]
+
+        start_val = max(last_txn_max, txn_max) + 1
+
+        cur.execute(
+            "CREATE SEQUENCE application_services_txn_id_seq START WITH %s",
+            (start_val,),
+        )


Test patch:
diff --git a/tests/storage/test_appservice.py b/tests/storage/test_appservice.py
--- a/tests/storage/test_appservice.py
+++ b/tests/storage/test_appservice.py
@@ -169,15 +169,6 @@ def _insert_txn(self, as_id, txn_id, events):
             (as_id, txn_id, json.dumps([e.event_id for e in events])),
         )
 
-    def _set_last_txn(self, as_id, txn_id):
-        return self.db_pool.runOperation(
-            self.engine.convert_param_style(
-                "INSERT INTO application_services_state(as_id, last_txn, state) "
-                "VALUES(?,?,?)"
-            ),
-            (as_id, txn_id, ApplicationServiceState.UP.value),
-        )
-
     def test_get_appservice_state_none(
         self,
     ) -> None:
@@ -277,64 +268,6 @@ def test_create_appservice_txn_first(
         self.assertEqual(txn.events, events)
         self.assertEqual(txn.service, service)
 
-    def test_create_appservice_txn_older_last_txn(
-        self,
-    ) -> None:
-        service = Mock(id=self.as_list[0]["id"])
-        events = cast(List[EventBase], [Mock(event_id="e1"), Mock(event_id="e2")])
-        self.get_success(self._set_last_txn(service.id, 9643))  # AS is falling behind
-        self.get_success(self._insert_txn(service.id, 9644, events))
-        self.get_success(self._insert_txn(service.id, 9645, events))
-        txn = self.get_success(
-            self.store.create_appservice_txn(
-                service, events, [], [], {}, {}, DeviceListUpdates()
-            )
-        )
-        self.assertEqual(txn.id, 9646)
-        self.assertEqual(txn.events, events)
-        self.assertEqual(txn.service, service)
-
-    def test_create_appservice_txn_up_to_date_last_txn(
-        self,
-    ) -> None:
-        service = Mock(id=self.as_list[0]["id"])
-        events = cast(List[EventBase], [Mock(event_id="e1"), Mock(event_id="e2")])
-        self.get_success(self._set_last_txn(service.id, 9643))
-        txn = self.get_success(
-            self.store.create_appservice_txn(
-                service, events, [], [], {}, {}, DeviceListUpdates()
-            )
-        )
-        self.assertEqual(txn.id, 9644)
-        self.assertEqual(txn.events, events)
-        self.assertEqual(txn.service, service)
-
-    def test_create_appservice_txn_up_fuzzing(
-        self,
-    ) -> None:
-        service = Mock(id=self.as_list[0]["id"])
-        events = cast(List[EventBase], [Mock(event_id="e1"), Mock(event_id="e2")])
-        self.get_success(self._set_last_txn(service.id, 9643))
-
-        # dump in rows with higher IDs to make sure the queries aren't wrong.
-        self.get_success(self._set_last_txn(self.as_list[1]["id"], 119643))
-        self.get_success(self._set_last_txn(self.as_list[2]["id"], 9))
-        self.get_success(self._set_last_txn(self.as_list[3]["id"], 9643))
-        self.get_success(self._insert_txn(self.as_list[1]["id"], 119644, events))
-        self.get_success(self._insert_txn(self.as_list[1]["id"], 119645, events))
-        self.get_success(self._insert_txn(self.as_list[1]["id"], 119646, events))
-        self.get_success(self._insert_txn(self.as_list[2]["id"], 10, events))
-        self.get_success(self._insert_txn(self.as_list[3]["id"], 9643, events))
-
-        txn = self.get_success(
-            self.store.create_appservice_txn(
-                service, events, [], [], {}, {}, DeviceListUpdates()
-            )
-        )
-        self.assertEqual(txn.id, 9644)
-        self.assertEqual(txn.events, events)
-        self.assertEqual(txn.service, service)
-
     def test_complete_appservice_txn_first_txn(
         self,
     ) -> None:
@@ -368,13 +301,13 @@ def test_complete_appservice_txn_first_txn(
         )
         self.assertEqual(0, len(res))
 
-    def test_complete_appservice_txn_existing_in_state_table(
+    def test_complete_appservice_txn_updates_last_txn_state(
         self,
     ) -> None:
         service = Mock(id=self.as_list[0]["id"])
         events = [Mock(event_id="e1"), Mock(event_id="e2")]
         txn_id = 5
-        self.get_success(self._set_last_txn(service.id, 4))
+        self._set_state(self.as_list[0]["id"], ApplicationServiceState.UP)
         self.get_success(self._insert_txn(service.id, txn_id, events))
         self.get_success(
             self.store.complete_appservice_txn(txn_id=txn_id, service=service)