Repository: matrix-org/synapse

Issue:
Export data admin command is broken.
Running `python -m synapse.app.admin_cmd -c <config_file> export-data <user_id>` results in:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/synapse/app/admin_cmd.py", line 236, in <module>
    start(sys.argv[1:])
  File "/usr/local/lib/python3.8/site-packages/synapse/app/admin_cmd.py", line 190, in start
    and not config.worker_log_file
  File "/usr/local/lib/python3.8/site-packages/synapse/config/_base.py", line 340, in __getattr__
    return self._get_unclassed_config(None, item)
  File "/usr/local/lib/python3.8/site-packages/synapse/config/_base.py", line 363, in _get_unclassed_config
    raise AttributeError(item, "not found in %s" % (list(self._configs.keys()),))
AttributeError: ('worker_log_file', "not found in ['modules', 'server', 'tls', 'federation', 'caches', 'database', 'logging', 'ratelimiting', 'media', 'oembed', 'captcha', 'voip', 'registration', 'account_validity', 'metrics', 'api', 'appservice', 'key', 'saml2', 'oidc', 'cas', 'sso', 'jwt', 'auth', 'email', 'authproviders', 'push', 'spamchecker', 'room', 'groups', 'userdirectory', 'consent', 'stats', 'servernotices', 'roomdirectory', 'thirdpartyrules', 'tracing', 'worker', 'redis', 'experimental']")
```

The effect of this is that its currently not possible to export a users data, e.g due to a GDPR request.

c.f. #10900


Discussion:
When I am ty to fix it by removing this line:
https://github.com/matrix-org/synapse/blob/8aaa4b7b5df5e851a5f3dd74cd3062c9f94f0066/synapse/app/admin_cmd.py#L190

I get this new error:
```
2021-09-28 14:59:19,138 - synapse.handlers.admin - 107 - INFO - command - [@user:server] Handling room !QdAJnGzpNsxiimllTe:server, 1/37
2021-09-28 14:59:20,909 - twisted - 271 - CRITICAL - sentinel - main function encountered error
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/internet/defer.py", line 662, in callback
    self._startRunCallbacks(result)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/admin_cmd.py", line 224, in run
    await args.func(ss, args)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/admin_cmd.py", line 80, in export_data_command
    res = await hs.get_admin_handler().export_user_data(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/admin.py", line 169, in export_user_data
    events = await filter_events_for_client(self.storage, user_id, events)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/visibility.py", line 108, in filter_events_for_client
    ] = await storage.main.get_retention_policy_for_room(room_id)
builtins.AttributeError: 'AdminCmdSlavedStore' object has no attribute 'get_retention_policy_for_room'

2021-09-28 14:59:20,914 - twisted - 271 - INFO - sentinel - Main loop terminated.
```

@erikjohnston Do you have a hint to fix it? Is it a missing import?
It looks like the issue is that we recently changed  `config.worker_log_file` to `config.worker.worker_log_file` in `synapse/synapse/app/admin_cmd.py` however, the `config.worker` object has no attribute `worker_log_file`, nor is there any references to a `worker_log_file` anywhere else in the code. A little digging did uncover that the logging configuration file for a worker will contain a filename/path to a logfile for that worker. 

My question is, since we are already checking that the `config.worker.worker_log_config`  exists, do we really also need to check that the log file that it presumably references exists as well? I.e. can we drop that line, or is it necessary to figure out how to extract the reference to the worker log file from the `worker_log_config` and check for it when this is the only place that reference will be used? I spent some time trying to do just that and couldn't figure it out, which led me to wonder if it isn't necessary in the first place.
> nor is there any references to a worker_log_file anywhere else in the code.

Indeed, but I think this goes back further than the recent changes you mention (#10897?). 

Running

```shell
git log -G worker_log_file
```
the first two hits are bb7fdd821b07016a43bdbb245eda5b35356863c0 and 6a85cb5ef7f9dfe4cd58abc313d66ee270db3549. The former is part of Patrick's PR; the latter is part of (the pleasingly-numberered) #5678 from 2019.

Has this script been broken since 2019? Surely we'd have noticed sooner...

> I.e. can we drop that line

I think so. 6a85cb5ef7f9dfe4cd58abc313d66ee270db3549 seems to be removing `log_file` configurations. And in e484101306787988adacf6d6de4fcd565368dec4 we explicitly said "this option is gone".

So yeah, my vote is to drop that line.



(Though as @dklimpel notes, there might be more fixing required afterwards.)


> Has this script been broken since 2019? Surely we'd have noticed sooner...

seems like it has, and nobody at element uses it. If we're going to have it, we should probably have some CI to check it keeps working (like we do for the port_db script).

Okay excellent. I will write a test case to keep an eye on this functionality going forwards, fix this issue, and hopefully fix anything else that might have broken in the interval. 

Resolving patch:
diff --git a/synapse/app/admin_cmd.py b/synapse/app/admin_cmd.py
--- a/synapse/app/admin_cmd.py
+++ b/synapse/app/admin_cmd.py
@@ -39,6 +39,7 @@
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
+from synapse.storage.databases.main.room import RoomWorkerStore
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
 
@@ -58,6 +59,7 @@ class AdminCmdSlavedStore(
     SlavedEventStore,
     SlavedClientIpStore,
     BaseSlavedStore,
+    RoomWorkerStore,
 ):
     pass
 
@@ -185,11 +187,7 @@ def start(config_options):
     # a full worker config.
     config.worker.worker_app = "synapse.app.admin_cmd"
 
-    if (
-        not config.worker.worker_daemonize
-        and not config.worker.worker_log_file
-        and not config.worker.worker_log_config
-    ):
+    if not config.worker.worker_daemonize and not config.worker.worker_log_config:
         # Since we're meant to be run as a "command" let's not redirect stdio
         # unless we've actually set log config.
         config.logging.no_redirect_stdio = True
@@ -198,9 +196,9 @@ def start(config_options):
     config.server.update_user_directory = False
     config.worker.run_background_tasks = False
     config.worker.start_pushers = False
-    config.pusher_shard_config.instances = []
+    config.worker.pusher_shard_config.instances = []
     config.worker.send_federation = False
-    config.federation_shard_config.instances = []
+    config.worker.federation_shard_config.instances = []
 
     synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
 
@@ -221,7 +219,7 @@ def start(config_options):
 
     async def run():
         with LoggingContext("command"):
-            _base.start(ss)
+            await _base.start(ss)
             await args.func(ss, args)
 
     _base.start_worker_reactor(


Test patch:
diff --git a/.ci/scripts/test_export_data_command.sh b/.ci/scripts/test_export_data_command.sh
new file mode 100755
--- /dev/null
+++ b/.ci/scripts/test_export_data_command.sh
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Test for the export-data admin command against sqlite and postgres
+
+set -xe
+cd `dirname $0`/../..
+
+echo "--- Install dependencies"
+
+# Install dependencies for this test.
+pip install psycopg2
+
+# Install Synapse itself. This won't update any libraries.
+pip install -e .
+
+echo "--- Generate the signing key"
+
+# Generate the server's signing key.
+python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
+
+echo "--- Prepare test database"
+
+# Make sure the SQLite3 database is using the latest schema and has no pending background update.
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+
+# Run the export-data command on the sqlite test database
+python -m synapse.app.admin_cmd -c .ci/sqlite-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
+--output-directory /tmp/export_data
+
+# Test that the output directory exists and contains the rooms directory
+dir="/tmp/export_data/rooms"
+if [ -d "$dir" ]; then
+  echo "Command successful, this test passes"
+else
+  echo "No output directories found, the command fails against a sqlite database."
+  exit 1
+fi
+
+# Create the PostgreSQL database.
+.ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
+
+# Port the SQLite databse to postgres so we can check command works against postgres
+echo "+++ Port SQLite3 databse to postgres"
+scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+
+# Run the export-data command on postgres database
+python -m synapse.app.admin_cmd -c .ci/postgres-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
+--output-directory /tmp/export_data2
+
+# Test that the output directory exists and contains the rooms directory
+dir2="/tmp/export_data2/rooms"
+if [ -d "$dir2" ]; then
+  echo "Command successful, this test passes"
+else
+  echo "No output directories found, the command fails against a postgres database."
+  exit 1
+fi
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -253,6 +253,35 @@ jobs:
             /logs/results.tap
             /logs/**/*.log*
 
+  export-data:
+    if: ${{ !failure() && !cancelled() }} # Allow previous steps to be skipped, but not fail
+    needs: [linting-done, portdb]
+    runs-on: ubuntu-latest
+    env:
+      TOP: ${{ github.workspace }}
+
+    services:
+      postgres:
+        image: postgres
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_PASSWORD: "postgres"
+          POSTGRES_INITDB_ARGS: "--lc-collate C --lc-ctype C --encoding UTF8"
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    steps:
+      - uses: actions/checkout@v2
+      - run: sudo apt-get -qq install xmlsec1
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.9"
+      - run: .ci/scripts/test_export_data_command.sh
+
   portdb:
     if: ${{ !failure() && !cancelled() }} # Allow previous steps to be skipped, but not fail
     needs: linting-done