Repository: matrix-org/synapse

Issue:
we should run the 'are_all_users_on_domain' database sanity check before running schema migrations
<!--

**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**:
You will likely get better support more quickly if you ask in ** #synapse:matrix.org ** ;)


This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description
When I run a search on my hosted matrix service from any client I see no results.
I can see the correct count but not the actual results
<!-- Describe here the problem that you are experiencing -->

### Steps to reproduce
- Doing a search
- No results showed

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->
- I should see the results
### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**:

If not matrix.org:

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**:
`{"server_version": "1.9.1", "python_version": "3.7.3"}`
- **Install method**:
<!-- examples: package manager/git clone/pip  -->
Debian apt repo package
- **Platform**:
Debian 10
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



I found the query that's run on sqlite for search and I saw it doesn't work properly:
https://github.com/matrix-org/synapse/blob/v1.9.1/synapse/storage/data_stores/main/search.py#L424
https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg71741.html

I then realised that the bultin sqlite3 function is overidden by https://github.com/matrix-org/synapse/blob/v1.9.1/synapse/storage/engines/sqlite.py#L76

my guess is that somehow the function overide is not applied and so the bultin failing rank function is used


Discussion:
I was finally able to identify the issue.
It's totally unrelated to the rank function but rather to the new `local_current_membership` table introduced in `1.9.1` relase and relative db migration.
I use debian sid repo to manage my synapse installation. I observed that since a few version the server name is not anymore defined in `homeserver.yaml` but rather from `conf.d/server_name.yaml`
The systemd unit `ExecStart` is the following
```/usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/```
Over every update of the synapse package the content of `conf.d/server_name.yaml` is replaced with the name of the server coming from `/etc/hostname` and I have to manually change it back to the production server name and restart the systemd service to make it work otherwise synapse will fail to start due to mismatching between the server name defined in the conf and the one in the db.
Apparently this doesn't stop the migration to be run in background: for the specific case I guess the server name from the configuration was used to populate `local_current_membership` table ending in an empty one since where was no matching local user for the wrong server name in the db.
This would have been prevented if the migration (especially since it is relying on the server name conf) was not run until the invalid server name configuration was fixed.
I'm afraid I don't quite understand your explanation.

If the problem is that `conf.d/server_name.yaml` is being overwritten, please can you raise that with the maintainers of the debian package?
(title updated for clarification: we use 'database migration' to mean moving from sqlite to postgres)
@richvdh I will clarify better
The problem with the `conf.d/server_name.yaml` overwritten has to be raised with the mantainers of the debian package, and I will do.

But there's another problem and you should be able to reproduce with the following steps:
1. Install matrix 1.9.0 with sqlite defining `server_name: whatever.domain.com`
1. Register some users for the server
1. Shut down the server and change to `server_name: another.domain.com`
1. Upgrade to matrix 1.9.1, it should fail to start because of mismatching between the server name in the db and the one in the config
1. Check in the db the existance of the table `local_current_membership`. It should be there but empty
1. Fix the server name and restart matrix. Now it is working.
1. Use search feature in any client: no results

The problem is that at point 4 the database update for `local_current_membership` was applied even if the server name configuration was invalid
Relying on the `config.server_name` the query that populates the table results in inserting no data: https://github.com/matrix-org/synapse/blob/9bae740527c4621f9f8eb8ca936669f2372c42eb/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py#L72
Once you fix the server name and restart properly the server the migration is already run and `local_current_membership` will stay empty breaking the search

My fix was to manually delete the table, remove the migration from the db and restart the server with the proper server name
ah I see. Thanks

Resolving patch:
diff --git a/synapse/storage/data_stores/main/__init__.py b/synapse/storage/data_stores/main/__init__.py
--- a/synapse/storage/data_stores/main/__init__.py
+++ b/synapse/storage/data_stores/main/__init__.py
@@ -20,6 +20,7 @@
 import time
 
 from synapse.api.constants import PresenceState
+from synapse.config.homeserver import HomeServerConfig
 from synapse.storage.database import Database
 from synapse.storage.engines import PostgresEngine
 from synapse.storage.util.id_generators import (
@@ -117,16 +118,6 @@ def __init__(self, database: Database, db_conn, hs):
         self._clock = hs.get_clock()
         self.database_engine = database.engine
 
-        all_users_native = are_all_users_on_domain(
-            db_conn.cursor(), database.engine, hs.hostname
-        )
-        if not all_users_native:
-            raise Exception(
-                "Found users in database not native to %s!\n"
-                "You cannot changed a synapse server_name after it's been configured"
-                % (hs.hostname,)
-            )
-
         self._stream_id_gen = StreamIdGenerator(
             db_conn,
             "events",
@@ -567,13 +558,26 @@ def search_users(self, term):
         )
 
 
-def are_all_users_on_domain(txn, database_engine, domain):
+def check_database_before_upgrade(cur, database_engine, config: HomeServerConfig):
+    """Called before upgrading an existing database to check that it is broadly sane
+    compared with the configuration.
+    """
+    domain = config.server_name
+
     sql = database_engine.convert_param_style(
         "SELECT COUNT(*) FROM users WHERE name NOT LIKE ?"
     )
     pat = "%:" + domain
-    txn.execute(sql, (pat,))
-    num_not_matching = txn.fetchall()[0][0]
+    cur.execute(sql, (pat,))
+    num_not_matching = cur.fetchall()[0][0]
     if num_not_matching == 0:
-        return True
-    return False
+        return
+
+    raise Exception(
+        "Found users in database not native to %s!\n"
+        "You cannot changed a synapse server_name after it's been configured"
+        % (domain,)
+    )
+
+
+__all__ = ["DataStore", "check_database_before_upgrade"]
diff --git a/synapse/storage/prepare_database.py b/synapse/storage/prepare_database.py
--- a/synapse/storage/prepare_database.py
+++ b/synapse/storage/prepare_database.py
@@ -278,13 +278,17 @@ def _upgrade_existing_database(
             the current_version wasn't generated by applying those delta files.
         database_engine (DatabaseEngine)
         config (synapse.config.homeserver.HomeServerConfig|None):
-            application config, or None if we are connecting to an existing
-            database which we expect to be configured already
+            None if we are initialising a blank database, otherwise the application
+            config
         data_stores (list[str]): The names of the data stores to instantiate
             on the given database.
         is_empty (bool): Is this a blank database? I.e. do we need to run the
             upgrade portions of the delta scripts.
     """
+    if is_empty:
+        assert not applied_delta_files
+    else:
+        assert config
 
     if current_version > SCHEMA_VERSION:
         raise ValueError(
@@ -292,6 +296,13 @@ def _upgrade_existing_database(
             + "new for the server to understand"
         )
 
+    # some of the deltas assume that config.server_name is set correctly, so now
+    # is a good time to run the sanity check.
+    if not is_empty and "main" in data_stores:
+        from synapse.storage.data_stores.main import check_database_before_upgrade
+
+        check_database_before_upgrade(cur, database_engine, config)
+
     start_ver = current_version
     if not upgraded:
         start_ver += 1


Test patch: