Repository: matrix-org/synapse

Issue:
'sqlite3.OperationalError: near "WHERE"' on sqlite 3.7
Hi, I have problem, on first run:

```
(env) [root@chat synapse]# synctl start
Starting ...
This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
2019-10-24 13:40:10,455 - twisted - 171 - INFO - None - Redirected stdout/stderr to logs
2019-10-24 13:40:10,455 - root - 236 - WARNING - None - ***** STARTING SERVER *****
2019-10-24 13:40:10,455 - root - 237 - WARNING - None - Server /var/www/synapse/env/lib/python3.6/site-packages/synapse/app/homeserver.py version 1.4.1
2019-10-24 13:40:10,456 - root - 238 - INFO - None - Server hostname: matrix.omnichat.tech
2019-10-24 13:40:10,457 - synapse.app.homeserver - 359 - INFO - None - Preparing database: sqlite3...
2019-10-24 13:40:16,312 - synapse.storage.prepare_database - 249 - INFO - None - Upgrading schema to v55
2019-10-24 13:40:16,313 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 55/access_token_expiry.sql
2019-10-24 13:40:16,314 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 55/track_threepid_validations.sql
2019-10-24 13:40:16,316 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 55/users_alter_deactivated.sql
2019-10-24 13:40:16,316 - synapse.storage.prepare_database - 249 - INFO - None - Upgrading schema to v56
2019-10-24 13:40:16,317 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/add_spans_to_device_lists.sql
2019-10-24 13:40:16,318 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/current_state_events_membership.sql
2019-10-24 13:40:16,318 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/current_state_events_membership_mk2.sql
2019-10-24 13:40:16,319 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/destinations_failure_ts.sql
2019-10-24 13:40:16,320 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/devices_last_seen.sql
2019-10-24 13:40:16,322 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/fix_room_keys_index.sql
2019-10-24 13:40:16,323 - synapse.storage.prepare_database - 287 - INFO - None - Applying schema 56/redaction_censor.sql
2019-10-24 13:40:16,324 - twisted - 171 - ERROR -  - Traceback (most recent call last):
2019-10-24 13:40:16,325 - twisted - 171 - ERROR -  -   File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
2019-10-24 13:40:16,326 - twisted - 171 - ERROR -  -     "__main__", mod_spec)
2019-10-24 13:40:16,326 - twisted - 171 - ERROR -  -   File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
2019-10-24 13:40:16,326 - twisted - 171 - ERROR -  -     exec(code, run_globals)
2019-10-24 13:40:16,326 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/app/homeserver.py", line 659, in <module>
2019-10-24 13:40:16,327 - twisted - 171 - ERROR -  -     main()
2019-10-24 13:40:16,327 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/app/homeserver.py", line 654, in main
2019-10-24 13:40:16,328 - twisted - 171 - ERROR -  -     hs = setup(sys.argv[1:])
2019-10-24 13:40:16,328 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/app/homeserver.py", line 363, in setup
2019-10-24 13:40:16,329 - twisted - 171 - ERROR -  -     prepare_database(db_conn, database_engine, config=config)
2019-10-24 13:40:16,329 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/storage/prepare_database.py", line 74, in prepare_database
2019-10-24 13:40:16,329 - twisted - 171 - ERROR -  -     _setup_new_database(cur, database_engine)
2019-10-24 13:40:16,330 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/storage/prepare_database.py", line 171, in _setup_new_database
2019-10-24 13:40:16,330 - twisted - 171 - ERROR -  -     is_empty=True,
2019-10-24 13:40:16,330 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/storage/prepare_database.py", line 288, in _upgrade_existing_database
2019-10-24 13:40:16,331 - twisted - 171 - ERROR -  -     executescript(cur, absolute_path)
2019-10-24 13:40:16,331 - twisted - 171 - ERROR -  -   File "/var/www/synapse/env/lib/python3.6/site-packages/synapse/storage/prepare_database.py", line 427, in executescript
2019-10-24 13:40:16,332 - twisted - 171 - ERROR -  -     txn.execute(statement)
2019-10-24 13:40:16,332 - twisted - 171 - ERROR -  - sqlite3.OperationalError: near "WHERE": syntax error
```



Discussion:
Are you using CentOS 7? I had to work around this by installing sqlite 3.8.

https://kojipkgs.fedoraproject.org//packages/sqlite/3.8.11/1.fc21/x86_64/sqlite-3.8.11-1.fc21.x86_64.rpm
works for me on CentOS with the above rpm package, I also tried to compile sqlite3.30 from the official site but for whatever reason, that doesn't work either. 3.8 works.
Problem with query: 
CREATE INDEX redactions_have_censored ON redactions(event_id) WHERE not have_censored;

I am commnent it, and was abble to install.
Path: synapse/synapse/storage/schema/delta/56/redaction_censor.sql
My SQLite version is 3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668

so it looks like #5934 broke compatibility with sqlite 3.7.

At this point sqlite 3.7 is over 6 years old, and there have been a lot of improvements to sqlite since. 

In any case, the sqlite support in synapse isn't really intended for production use: you should be using postgres for anything other than a test installation.
We even have a shiny new [migration script](https://github.com/matrix-org/synapse/blob/develop/scripts/synapse_port_db) about to land in 1.5.0.

So either way, upgrade sqlite will fix, but I'd _really_ recommend that you migrate to postgres.
I'd suggest making postgres the default then :)
thanks very much for the support 👍
> So either way, upgrade sqlite will fix, but I'd really recommend that you migrate to postgres.

sadly you won't be able to run the migration script without having run the migrations, which means that you'll first need to upgrade sqlite.

Neil: if we're dropping support for sqlite 3.7, we need to update https://github.com/matrix-org/synapse/blob/master/INSTALL.md#centosfedora and possibly add something to https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source.
and a note in UPGRADE.rst
This is terrible.

Resolving patch:
diff --git a/synapse/storage/data_stores/main/events.py b/synapse/storage/data_stores/main/events.py
--- a/synapse/storage/data_stores/main/events.py
+++ b/synapse/storage/data_stores/main/events.py
@@ -1039,20 +1039,25 @@ def _store_redaction(self, txn, event):
             },
         )
 
-    @defer.inlineCallbacks
-    def _censor_redactions(self):
+    async def _censor_redactions(self):
         """Censors all redactions older than the configured period that haven't
         been censored yet.
 
         By censor we mean update the event_json table with the redacted event.
-
-        Returns:
-            Deferred
         """
 
         if self.hs.config.redaction_retention_period is None:
             return
 
+        if not (
+            await self.db.updates.has_completed_background_update(
+                "redactions_have_censored_ts_idx"
+            )
+        ):
+            # We don't want to run this until the appropriate index has been
+            # created.
+            return
+
         before_ts = self._clock.time_msec() - self.hs.config.redaction_retention_period
 
         # We fetch all redactions that:
@@ -1074,15 +1079,15 @@ def _censor_redactions(self):
             LIMIT ?
         """
 
-        rows = yield self.db.execute(
+        rows = await self.db.execute(
             "_censor_redactions_fetch", None, sql, before_ts, 100
         )
 
         updates = []
 
         for redaction_id, event_id in rows:
-            redaction_event = yield self.get_event(redaction_id, allow_none=True)
-            original_event = yield self.get_event(
+            redaction_event = await self.get_event(redaction_id, allow_none=True)
+            original_event = await self.get_event(
                 event_id, allow_rejected=True, allow_none=True
             )
 
@@ -1115,7 +1120,7 @@ def _update_censor_txn(txn):
                     updatevalues={"have_censored": True},
                 )
 
-        yield self.db.runInteraction("_update_censor_txn", _update_censor_txn)
+        await self.db.runInteraction("_update_censor_txn", _update_censor_txn)
 
     def _censor_event_txn(self, txn, event_id, pruned_json):
         """Censor an event by replacing its JSON in the event_json table with the
diff --git a/synapse/storage/data_stores/main/events_bg_updates.py b/synapse/storage/data_stores/main/events_bg_updates.py
--- a/synapse/storage/data_stores/main/events_bg_updates.py
+++ b/synapse/storage/data_stores/main/events_bg_updates.py
@@ -90,6 +90,14 @@ def __init__(self, database: Database, db_conn, hs):
             "event_store_labels", self._event_store_labels
         )
 
+        self.db.updates.register_background_index_update(
+            "redactions_have_censored_ts_idx",
+            index_name="redactions_have_censored_ts",
+            table="redactions",
+            columns=["received_ts"],
+            where_clause="NOT have_censored",
+        )
+
     @defer.inlineCallbacks
     def _background_reindex_fields_sender(self, progress, batch_size):
         target_min_stream_id = progress["target_min_stream_id_inclusive"]


Test patch: