Repository: matrix-org/synapse

Issue:
Postgres port script fails to set correct initial values for some sequences
## TL;DR: Uninitialized sequences
* `receipts_sequence` (`receipts_linearized`)
* `account_data_sequence` (`room_account_data`)
* `device_inbox_sequence` (`device_inbox`)

## Steps to reproduce
I have synapse installed through the ubuntu package from the https://packages.matrix.org/debian/ repository, at version `matrix-synapse-py3/unknown,now 1.26.0+focal1 amd64 [installed]`

I ran `synapse_port_db` to migrate my synapse install from sqlite to postgres.

After the script was complete I updated the database section of my homeserver.yaml to point to the new postgres install, and started up synapse.

The problem was that synapse then failed to come up (and, as an aside, `journalctl --since=today -u matrix-synapse` was uninformative as to why it was failing). Looking at the homeserver log I saw an error message beginning with 
```
2021-02-07 16:54:16,547 - twisted - 254 - ERROR - None -  Postgres sequence 'receipts_sequence' is inconsistent with associated
2021-02-07 16:54:16,547 - twisted - 254 - ERROR - None -  table 'receipts_linearized'.
```
It then went on to provide an SQL statement to run to fix that problem.

I checked (`select * from receipts_sequence`) before running that setval statement and the current value of the sequence was in fact 1, indicating it had never been called.

After addressing that issue I got error messages about two more sequences (account_data_sequence and device_inbox_sequence) before I was able to start up my synapse.

The printed error message said "_This can happen if Synapse has been downgraded and then upgraded again, or due to a bad migration._", but I'm quite sure I've never downgraded synapse or manually run a migration against the sqlite db.
Postgres port script breaks the database after synapse 1.26, doesn't allow creating new DMs.
### Description

I upgraded from sqlite to postgres using `synapse_port_db`. The script ran successfully, except for the issues I mentioned in https://github.com/matrix-org/synapse/issues/9344. A few days after that, however, a new user on a different HS tried to initiate a DM with me, and I found myself unable to accept the invite with a "Failed to join room: Internal Server Error". My synapse logs indicate that the problem is `duplicate key value violates unique constraint "event_auth_chains_c_seq_index"`.

Searching the history of #synapse:matrix.org for `event_auth_chains_c_seq_index` turned up a couple more cases where people saw that error and were unable to create new rooms or DMs after migrating to postgres, so I know it's not just me. In the past the advice they had been given was to wipe their server and start fresh, but I think that kind of a data loss bug (or a db state bug for which the only remedy is data loss) is pretty unacceptable for a messaging service.

This is particularly pernicious because the postgres migration script completes successfully, and it's only possibly a few days later, when someone tries to create a new room, that you find the database is borked. This would be less of a problem if there were a clear, documented workaround (like there is for the bugs in #9344), but for this bug the crowd in #synapse:matrix.org does not know of a nondestructive fix for the problem.

### Steps to reproduce

- upgrade from sqlite to postgres on synapse 1.26
- have another user try to start a new DM room with you.
- Observe that this fails, with an error 

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**:

If not matrix.org:

- **Version**: {"server_version":"1.26.0","python_version":"3.8.5"}

- **Install method**: apt: `matrix-synapse-py3/unknown,now 1.26.0+focal1 amd64 [installed]`

- **Platform**: Ubuntu 20.04.2 LTS



Discussion:
I suspect that we did not update the port DB script when we recently added a few sequences. See the code around https://github.com/matrix-org/synapse/blob/7036e24e98fc21855c34876d7024015470721bbe/scripts/synapse_port_db#L622-L632
I faced the same inconsistencies with matrix-synapse 1.26.0-1~bpo10+3 on debian buster.

Synapse gave me the 3 following sql commands in the log - one at a time.

```
SELECT setval('receipts_sequence', (SELECT GREATEST(MAX(stream_id), 0) FROM receipts_linearized));
SELECT setval('account_data_sequence', (SELECT GREATEST(MAX(stream_id), 0) FROM room_account_data));
SELECT setval('device_inbox_sequence', (SELECT GREATEST(MAX(stream_id), 0) FROM device_inbox)); 
```
Then synapse started well.
But when I tried to create a room I got this error:
```
synapse.http.server: [POST-1655] Failed handle request via 'RoomCreateRestServlet': <SynapseRequest at 0x7f11202d6b00 method='POST' uri='/_matrix/client/r0/createRoom' clientproto='HTTP/1.1' site='8008'>
                                      Traceback (most recent call last):
                                        File "/usr/lib/python3/dist-packages/synapse/storage/persist_events.py", line 172, in handle_queue_loop
                                          ret = await per_item_callback(item)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/persist_events.py", line 325, in persisting_queue
                                          item.events_and_contexts, backfilled=item.backfilled
                                        File "/usr/lib/python3/dist-packages/synapse/storage/persist_events.py", line 537, in _persist_events
                                          backfilled=backfilled,
                                        File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events.py", line 177, in _persist_events_and_state_updates
                                          new_forward_extremeties=new_forward_extremeties,
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 664, in runInteraction
                                          **kwargs,
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 740, in runWithConnection
                                          self._db_pool.runWithConnection(inner_func, *args, **kwargs)
                                        File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
                                          result = inContext.theWork()
                                        File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
                                          inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
                                        File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
                                          return self.currentContext().callWithContext(ctx, func, *args, **kw)
                                        File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
                                          return func(*args,**kw)
                                        File "/usr/lib/python3/dist-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
                                          compat.reraise(excValue, excTraceback)
                                        File "/usr/lib/python3/dist-packages/twisted/python/compat.py", line 464, in reraise
                                          raise exception.with_traceback(traceback)
                                        File "/usr/lib/python3/dist-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
                                          result = func(conn, *args, **kw)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 734, in inner_func
                                          return func(db_conn, *args, **kwargs)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 534, in new_transaction
                                          r = func(cursor, *args, **kwargs)
                                        File "/usr/lib/python3/dist-packages/synapse/logging/utils.py", line 71, in wrapped
                                          return f(*args, **kwargs)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events.py", line 379, in _persist_events_txn
                                          self._persist_event_auth_chain_txn(txn, [e for e, _ in events_and_contexts])
                                        File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events.py", line 473, in _persist_event_auth_chain_txn
                                          txn, self.db_pool, event_to_room_id, event_to_types, event_to_auth_chain,
                                        File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events.py", line 677, in _add_chain_cover_index
                                          for event_id, (c_id, seq) in new_chain_tuples.items()
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 891, in simple_insert_many_txn
                                          txn.executemany(sql, vals)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 291, in executemany
                                          self._do_execute(self.txn.executemany, sql, *args)
                                        File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 314, in _do_execute
                                          return func(sql, *args)
                                      psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "event_auth_chains_c_seq_index"
                                      DETAIL:  Key (chain_id, sequence_number)=(1, 1) already exists.

```

I checked all the values of the sequences and I found that user_id_seq , instance_map_instance_id_seq  and event_auth_chain_id have their default initial value (1).
I'm investigating in the database structure and synapse source code to find what table.row to use

@jm-github you should see https://github.com/matrix-org/synapse/issues/9382. You need to run `select setval('event_auth_chain_id', (select max(chain_id) from event_auth_chains));`
Reading through the code introduced in https://github.com/matrix-org/synapse/pull/8868 I'm starting to suspect this is an effect of another improperly-seeded sequence in the postgres migration script; unfortunately the checks that flagged other problem sequences in the migration script didn't catch this one.

If that hunch is right, I think running this query by hand should fix this problem:
```
select setval('event_auth_chain_id', (select max(chain_id) from event_auth_chains));
```
Would anyone who knows the event auth chain schema better than I do like to confirm that fiddling with the database directly there won't mess something else up?
Thanks for digging into this, looks like we've forgotten to `.check_consistency` on the sequence at startup, so none of the checks are happening :facepalm: 

I can confirm that is safe to increase the values of any of our sequences (while the server is offline).
related: #9344 
Ran that `setval` query to update the sequence and creating new rooms works fine now. Thanks for your help!

Resolving patch:
diff --git a/synapse/storage/databases/__init__.py b/synapse/storage/databases/__init__.py
--- a/synapse/storage/databases/__init__.py
+++ b/synapse/storage/databases/__init__.py
@@ -79,7 +79,7 @@ def __init__(self, main_store_class, hs):
                     # If we're on a process that can persist events also
                     # instantiate a `PersistEventsStore`
                     if hs.get_instance_name() in hs.config.worker.writers.events:
-                        persist_events = PersistEventsStore(hs, database, main)
+                        persist_events = PersistEventsStore(hs, database, main, db_conn)
 
                 if "state" in database_config.databases:
                     logger.info(
diff --git a/synapse/storage/databases/main/events.py b/synapse/storage/databases/main/events.py
--- a/synapse/storage/databases/main/events.py
+++ b/synapse/storage/databases/main/events.py
@@ -42,6 +42,7 @@
 from synapse.storage._base import db_to_json, make_in_list_sql_clause
 from synapse.storage.database import DatabasePool, LoggingTransaction
 from synapse.storage.databases.main.search import SearchEntry
+from synapse.storage.types import Connection
 from synapse.storage.util.id_generators import MultiWriterIdGenerator
 from synapse.types import StateMap, get_domain_from_id
 from synapse.util import json_encoder
@@ -90,7 +91,11 @@ class PersistEventsStore:
     """
 
     def __init__(
-        self, hs: "HomeServer", db: DatabasePool, main_data_store: "DataStore"
+        self,
+        hs: "HomeServer",
+        db: DatabasePool,
+        main_data_store: "DataStore",
+        db_conn: Connection,
     ):
         self.hs = hs
         self.db_pool = db
@@ -109,6 +114,12 @@ def __init__(
         )  # type: MultiWriterIdGenerator
         self._stream_id_gen = self.store._stream_id_gen  # type: MultiWriterIdGenerator
 
+        # The consistency of this cannot be checked when the ID generator is
+        # created since the database might not yet be up-to-date.
+        self.db_pool.event_chain_id_gen.check_consistency(
+            db_conn, "event_auth_chains", "chain_id"  # type: ignore
+        )
+
         # This should only exist on instances that are configured to write
         assert (
             hs.get_instance_name() in hs.config.worker.writers.events


Test patch: