Repository: matrix-org/synapse

Issue:
Schema deltas are no longer transactional on sqlite
https://github.com/matrix-org/synapse/pull/13873 may have introduced a regression where schema deltas are no longer executed inside a transaction on sqlite. This can lead to database inconsistencies and bootloops when Synapse is killed while running database migrations.

Examples:
* https://github.com/matrix-org/synapse/issues/14333
* https://github.com/matrix-org/synapse/issues/14460

Previously: https://github.com/matrix-org/synapse/issues/6467.
Split out from #14401.


Discussion:
> When using sqlite, schema deltas are not fully transactional (see docstring [here](https://github.com/matrix-org/synapse/blob/8c94dd3a277d4e11192f98a9ca32cb6638606b66/synapse/storage/engines/sqlite.py#L127-L144)). The schema delta will end up being half applied and the database will end up in an inconsistent state. Depending on the nature of the schema delta, the next run may fail.

It looks like this may have been introduced by https://github.com/matrix-org/synapse/pull/13873, which replaced the old handrolled code for parsing sql files with something more conventional. That was probably a good idea, but it does break the transactional semantics we were relying on for our migrations...

_Originally posted by @richvdh in https://github.com/matrix-org/synapse/issues/14401#issuecomment-1399276174_
            
> > When using sqlite, schema deltas are not fully transactional (see docstring [here](https://github.com/matrix-org/synapse/blob/8c94dd3a277d4e11192f98a9ca32cb6638606b66/synapse/storage/engines/sqlite.py#L127-L144)). The schema delta will end up being half applied and the database will end up in an inconsistent state. Depending on the nature of the schema delta, the next run may fail.
> 
> It looks like this may have been introduced by #13873, which replaced the old handrolled code for parsing sql files with something more conventional. That was probably a good idea, but it does break the transactional semantics we were relying on for our migrations...

Sorry, that's my mistake. FWIW the old parser still exists, see e.g. https://github.com/matrix-org/synapse/blob/0a38c7ec6d46b6e51bfa53ff44e51637d3c63f5c/synapse/storage/prepare_database.py#L609

The only other option would be to inject an explicit `BEGIN TRANSACTION` and `COMMIT` before and after the script. But that's not a like-for-like replacement for what had before. Maybe it's best to revert back.

(I recall that when I was making the full schema dumps that parser couldn't handle something postgres-specific. Trigger definitions maybe?)

_Originally posted by @DMRobertson in https://github.com/matrix-org/synapse/issues/14401#issuecomment-1399583234_
            
> The only other option would be to inject an explicit `BEGIN TRANSACTION` and `COMMIT` before and after the script. But that's not a like-for-like replacement for what had before

It's not, but I think it would have the right semantics, and would avoid us maintaining our own SQL parser, with all the pitfalls that brings.

_Originally posted by @richvdh in https://github.com/matrix-org/synapse/issues/14401#issuecomment-1402310945_
            

Resolving patch:
diff --git a/synapse/storage/engines/_base.py b/synapse/storage/engines/_base.py
--- a/synapse/storage/engines/_base.py
+++ b/synapse/storage/engines/_base.py
@@ -132,6 +132,9 @@ def executescript(cursor: CursorType, script: str) -> None:
         """Execute a chunk of SQL containing multiple semicolon-delimited statements.
 
         This is not provided by DBAPI2, and so needs engine-specific support.
+
+        Some database engines may automatically COMMIT the ongoing transaction both
+        before and after executing the script.
         """
         ...
 
diff --git a/synapse/storage/engines/sqlite.py b/synapse/storage/engines/sqlite.py
--- a/synapse/storage/engines/sqlite.py
+++ b/synapse/storage/engines/sqlite.py
@@ -135,13 +135,14 @@ def executescript(cursor: sqlite3.Cursor, script: str) -> None:
         > than one statement with it, it will raise a Warning. Use executescript() if
         > you want to execute multiple SQL statements with one call.
 
-        Though the docs for `executescript` warn:
+        The script is wrapped in transaction control statemnets, since the docs for
+        `executescript` warn:
 
         > If there is a pending transaction, an implicit COMMIT statement is executed
         > first. No other implicit transaction control is performed; any transaction
         > control must be added to sql_script.
         """
-        cursor.executescript(script)
+        cursor.executescript(f"BEGIN TRANSACTION;\n{script}\nCOMMIT;")
 
 
 # Following functions taken from: https://github.com/coleifer/peewee


Test patch: