Repository: matrix-org/synapse

Issue:
SQLITE3 error "UNIQUE constraint failed: temp_user_filters.user_id" after update to 1.86.0
### Description

Hi,

I am using a Ubuntu server and had installed the matrix-synapse-py3. The update to version 1.86.0 appeared Today and I updated it as normal (via apt).

After the update the server was not getting back.

I've tried to rollback to version 1.71.0+focal1, but it complains the database has a newer version.

### Steps to reproduce

- sudo apt upgrade

After that, server did not started.


### Homeserver

matrix.brunofontes.net

### Synapse Version

1.86.0

### Installation Method

Debian packages from packages.matrix.org

### Database

SQLite

### Workers

Single process

### Platform

Linux mx 5.4.0-152-generic #169-Ubuntu SMP x86_64

### Configuration

_No response_

### Relevant log output

```shell
2023-06-20 19:44:04,875 - root - 348 - WARNING - main - ***** STARTING SERVER *****                                                                                                     
2023-06-20 19:44:04,875 - root - 349 - WARNING - main - Server /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/homeserver.py version 1.86.0                           
2023-06-20 19:44:04,875 - root - 354 - INFO - main - Server hostname: matrix.brunofontes.net                                                                                            
2023-06-20 19:44:04,876 - root - 355 - INFO - main - Instance name: master                                                                                                              
2023-06-20 19:44:04,876 - root - 356 - INFO - main - Twisted reactor: EPollReactor                                                                                                      
2023-06-20 19:44:04,876 - synapse.app.homeserver - 349 - INFO - main - Setting up server                                                                                                
2023-06-20 19:44:04,876 - synapse.server - 337 - INFO - main - Setting up.                                                                                                              
2023-06-20 19:44:04,880 - synapse.storage.databases - 66 - INFO - main - [database config 'master']: Checking database server                                                           
2023-06-20 19:44:04,880 - synapse.storage.databases - 69 - INFO - main - [database config 'master']: Preparing for databases ['main', 'state']                                          
2023-06-20 19:44:04,881 - synapse.storage.prepare_database - 120 - INFO - main - ['main', 'state']: Checking existing schema version                                                    
2023-06-20 19:44:04,882 - synapse.storage.prepare_database - 124 - INFO - main - ['main', 'state']: Existing schema is 77 (+5 deltas)                                                   
2023-06-20 19:44:04,882 - synapse.storage.databases.main - 288 - INFO - main - Checking database for consistency with configuration...                                                  
2023-06-20 19:44:04,882 - synapse.storage.prepare_database - 418 - INFO - main - Applying schema deltas for v77                                                                         
2023-06-20 19:44:04,883 - synapse.storage.prepare_database - 418 - INFO - main - Applying schema deltas for v78                                                                         
2023-06-20 19:44:04,883 - synapse.storage.prepare_database - 513 - INFO - main - Running 78/01_validate_and_update_profiles.py:run_upgrade                                              
2023-06-20 19:44:04,899 - synapse.storage.prepare_database - 513 - INFO - main - Running 78/02_validate_and_update_user_filters.py:run_upgrade                                          
2023-06-20 19:44:04,901 - synapse.app._base - 215 - ERROR - main - Exception during startup                                                                                             
Traceback (most recent call last):                                                                                                                                                      
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/homeserver.py", line 352, in setup hs.setup()
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/server.py", line 339, in setup self.datastores = Databases(self.DATASTORE_CLASS, self)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/databases/__init__.py", line 74, in __init__ prepare_database(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/prepare_database.py", line 141, in prepare_database _upgrade_existing_database(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/prepare_database.py", line 514, in _upgrade_existing_database module.run_upgrade(cur, database_engine, config=config)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py", line 85, in run_upgrade cur.execute(copy_sql, (f"{hostname}",))
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/database.py", line 417, in execute self._do_execute(self.txn.execute, sql, parameters)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/database.py", line 469, in _do_execute return func(sql, *args, **kwargs)
    sqlite3.IntegrityError: UNIQUE constraint failed: temp_user_filters.user_id
```


### Anything else that would be useful to know?

_No response_


Discussion:
I also encountered this issue. It was introduced by PR #15649.

The workaround I used was to delete the `UNIQUE (full_user_id), UNIQUE (user_id)` constraints from the `CREATE TABLE temp_user_filters` statement in `02_validate_and_update_user_filters.py`. The correct composite unique index on `(user_id, filter_id)` is added later. I guess that the extra keys that are causing the problem could be copy–paste errors since this PR also updates the `temp_profiles` table which does the same thing (and for that table those unique keys look like they are correct).
And how do I do that?
having this same issue as well after the 1.86.0 update, I rolled back to v1.85.2 for now and its working. 
That's working for me.
I seem to encounter similar issues. I also rolled back for the moment and started working again.
Same issue here
> I also encountered this issue. It was introduced by PR #15649.
> 
> The workaround I used was to delete the `UNIQUE (full_user_id), UNIQUE (user_id)` constraints from the `CREATE TABLE temp_user_filters` statement in `02_validate_and_update_user_filters.py`. The correct composite unique index on `(user_id, filter_id)` is added later. I guess that the extra keys that are causing the problem could be copy–paste errors since this PR also updates the `temp_profiles` table which does the same thing (and for that table those unique keys look like they are correct).

helps me also - I think it´s a good time to migrate to Postgres... thanks @csnover 
Had the same issue, and I am running the docker image.

Temporarily fixed by rolling back to tag `matrixdotorg/synapse:v1.85.2`

Editiing files internally in the docker image is not really a viable solution, so a fix for this would be nice.
Same issue when building the OpenBSD port. I will wait for a fix before creating a new OpenBSD package.
> And how do I do that?

nano (or vi)
/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py

find:
```
       CREATE TABLE temp_user_filters (
            full_user_id text NOT NULL,
            user_id text NOT NULL,
            filter_id bigint NOT NULL,
            filter_json bytea NOT NULL,
            UNIQUE (full_user_id), 
            UNIQUE (user_id)
```
        )
replace with:
 ```
      CREATE TABLE temp_user_filters (
            full_user_id text NOT NULL,
            user_id text NOT NULL,
            filter_id bigint NOT NULL,
            filter_json bytea NOT NULL
        )
```
(do not forget to remove the comma at the end of "filter_json bytea NOT NULL"..

do a systemctl restart matrix-synapse

that did it for me.

That being said, a proper fix would be nice :)
Got the same issue since last upgrade to the latest version, had to roll-back to matrixdotorg/synapse:v1.85.2 docker image.
Got the same error on 2 nodes using the container image. Might indeed need to look into migrating to psql. Rollback solved the issue tho.

Resolving patch:
diff --git a/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py b/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py
--- a/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py
+++ b/synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py
@@ -61,9 +61,7 @@ def run_upgrade(
             full_user_id text NOT NULL,
             user_id text NOT NULL,
             filter_id bigint NOT NULL,
-            filter_json bytea NOT NULL,
-            UNIQUE (full_user_id),
-            UNIQUE (user_id)
+            filter_json bytea NOT NULL
         )
         """
         cur.execute(create_sql)
diff --git a/synapse/storage/schema/main/delta/78/03_remove_unused_indexes_user_filters.py b/synapse/storage/schema/main/delta/78/03_remove_unused_indexes_user_filters.py
new file mode 100644
--- /dev/null
+++ b/synapse/storage/schema/main/delta/78/03_remove_unused_indexes_user_filters.py
@@ -0,0 +1,65 @@
+# Copyright 2023 The Matrix.org Foundation C.I.C
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+from synapse.config.homeserver import HomeServerConfig
+from synapse.storage.database import LoggingTransaction
+from synapse.storage.engines import BaseDatabaseEngine, Sqlite3Engine
+
+
+def run_update(
+    cur: LoggingTransaction,
+    database_engine: BaseDatabaseEngine,
+    config: HomeServerConfig,
+) -> None:
+    """
+    Fix to drop unused indexes caused by incorrectly adding UNIQUE constraint to
+    columns `user_id` and `full_user_id` of table `user_filters` in previous migration.
+    """
+
+    if isinstance(database_engine, Sqlite3Engine):
+        cur.execute("DROP TABLE IF EXISTS temp_user_filters")
+        create_sql = """
+        CREATE TABLE temp_user_filters (
+            full_user_id text NOT NULL,
+            user_id text NOT NULL,
+            filter_id bigint NOT NULL,
+            filter_json bytea NOT NULL
+        )
+        """
+        cur.execute(create_sql)
+
+        copy_sql = """
+        INSERT INTO temp_user_filters (
+            user_id,
+            filter_id,
+            filter_json,
+            full_user_id)
+            SELECT user_id, filter_id, filter_json, full_user_id FROM user_filters
+        """
+        cur.execute(copy_sql)
+
+        drop_sql = """
+        DROP TABLE user_filters
+        """
+        cur.execute(drop_sql)
+
+        rename_sql = """
+        ALTER TABLE temp_user_filters RENAME to user_filters
+        """
+        cur.execute(rename_sql)
+
+        index_sql = """
+        CREATE UNIQUE INDEX IF NOT EXISTS user_filters_unique ON
+        user_filters (user_id, filter_id)
+        """
+        cur.execute(index_sql)
diff --git a/synapse/storage/schema/main/delta/78/04_add_full_user_id_index_user_filters.py b/synapse/storage/schema/main/delta/78/04_add_full_user_id_index_user_filters.py
new file mode 100644
--- /dev/null
+++ b/synapse/storage/schema/main/delta/78/04_add_full_user_id_index_user_filters.py
@@ -0,0 +1,25 @@
+# Copyright 2023 The Matrix.org Foundation C.I.C
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+from synapse.storage.database import LoggingTransaction
+from synapse.storage.engines import BaseDatabaseEngine, Sqlite3Engine
+
+
+def run_create(cur: LoggingTransaction, database_engine: BaseDatabaseEngine) -> None:
+    if isinstance(database_engine, Sqlite3Engine):
+        idx_sql = """
+        CREATE UNIQUE INDEX IF NOT EXISTS user_filters_full_user_id_unique ON
+        user_filters (full_user_id, filter_id)
+        """
+        cur.execute(idx_sql)


Test patch: