Repository: matrix-org/synapse

Issue:
Exception from 'rotate_notifs' shown when running the synapse_port_db script
### Description

When migrating an ~5Gb SQLite3 Database to PostgreSQL as described in the [documentation](https://matrix-org.github.io/synapse/latest/postgres.html#using-the-port-script), the script printed an exception repeatedly from the `rotate_notifs` background process.


After about 10 minutes the script  started to output database migration progress and I didn't see this exception again.

### Steps to reproduce

- Set up the Postgres database as described in the [docs](https://matrix-org.github.io/synapse/latest/postgres.html#using-the-port-script)
- `systemctl stop matrix-synapse`
- Added the Postgres configuration section (copied from the docs) in a new file `/etc/matrix-synapse/conf.d/postgres.yaml`
- Concatenated `homeserver.yaml` and all files in the `conf.d` directory into a single configuration for the migration script
- Ran `synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db --postgres-config concatenated_config.yaml`
- After some log messages I've got the exception message printed. As it repeated, I thought that the script was stuck on the exception and killed it and tried again with an updated system. But this also created the same exception. While the migration worked after waiting some time, I don't expect the script to throw any exceptions when migrating (or don't show them, if they are ok).

### Homeserver

Self hosted homeserver

### Synapse Version

1.81.0+bullseye1 and 1.85.2+bookworm1 after server upgrade

### Installation Method

Debian packages from packages.matrix.org

### Database

SQLite3 3.40.1-2

### Workers

I don't know

### Platform

Debian Bullseye and Debian Bookworm on a VM in a Proxmox Cluster

### Configuration

_No response_

### Relevant log output

```shell
synapse.metrics.background_process_metrics - 244 - ERROR - Background proces
s 'rotate_notifs' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/metrics/background_process_metr
ics.py", line 242, in run
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_pu
sh_actions.py", line 1297, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 924,
 in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, 
in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/failure.py", line 518, i
n throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 890,
 in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1019
, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1012, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 752, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1332, in _handle_new_receipts_for_notifs_txn
    min_receipts_stream_id = self.db_pool.simple_select_one_onecol_txn(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/database.py", line 1741, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
```


### Anything else that would be useful to know?

Maybe the issue is caused by the large DB size of ~5Gb and while the DB is loaded the exceptions are thrown due to missing data (just my assumption).


Discussion:
Hmm, this looks like its trying to run the `rotate_notifs` background process, which shouldn't be run during the port DB script.

Resolving patch:
diff --git a/synapse/_scripts/synapse_port_db.py b/synapse/_scripts/synapse_port_db.py
--- a/synapse/_scripts/synapse_port_db.py
+++ b/synapse/_scripts/synapse_port_db.py
@@ -1369,6 +1369,9 @@ def main() -> None:
         sys.stderr.write("Database must use the 'psycopg2' connector.\n")
         sys.exit(3)
 
+    # Don't run the background tasks that get started by the data stores.
+    hs_config["run_background_tasks_on"] = "some_other_process"
+
     config = HomeServerConfig()
     config.parse_config_dict(hs_config, "", "")
 


Test patch: