Repository: matrix-org/synapse

Issue:
update to 1.12 broke workers setup
### Description

I changed my setup for using workers a few days ago. Today, after updating to 1.12 (debian repository from matrix.org), my workers are not starting anymore.

Example of the log for a synchrotron worker :

```
2020-03-24 00:51:40,018 - root - 257 - WARNING - None- ***** STARTING SERVER *****
2020-03-24 00:51:40,018 - root - 258 - WARNING - None- Server /opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/synchrotron.py version 1.12.0
2020-03-24 00:51:40,230 - synapse.app.synchrotron - 245 - WARNING - - Starting daemon.
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - - Traceback (most recent call last):
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 277, in start
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -     hs.start_listening(listeners)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 575, in start_listening
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     self.get_tcp_replication().start_replication(self)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 581, in build_tcp_replication
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     return GenericWorkerReplicationHandler(self)
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 604, in __init__
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     self.send_handler = FederationSenderHandler(hs, self)
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 734, in __init__
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     self.federation_sender = hs.get_federation_sender()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 445, in build_federation_sender
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - -     raise Exception("Workers cannot send federation traffic")
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - - Exception: Workers cannot send federation traffic
2020-03-24 00:51:40,337 - twisted - 192 - CRITICAL - - Unhandled Error
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1282, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1262, in startRunning
    ReactorBase.startRunning(self)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 770, in startRunning
    self.fireSystemEvent('startup')
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 721, in fireSystemEvent
    event.fireEvent()
--- <exception caught here> ---
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 428, in fireEvent
    result = callable(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 297, in start
    sys.exit(1)
builtins.SystemExit: 1
```

/etc/matrix-synapse/worker/synchrotron-1.yaml:
```
worker_app: synapse.app.synchrotron

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8083
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/run/synapse/synchrotron1.pid
worker_log_config: /etc/matrix-synapse/synchrotron1-log.yaml
```

### Steps to reproduce

worked with 1.11 and stopped with the update

### Version information

- **Homeserver**: 

selfhosted

- **Version**:

 {
    "python_version": "3.7.3",
    "server_version": "1.12.0"
}

- **Install method**:

apt, matrix.org repository
<!--

### Description

I changed my setup for using workers a few days ago. Today, after updating to 1.12 (debian repository from matrix.org), my workers are not starting anymore.

Example of the log for a synchrotron worker :

```
2020-03-24 00:51:40,018 - root - 257 - WARNING - None- ***** STARTING SERVER *****
2020-03-24 00:51:40,018 - root - 258 - WARNING - None- Server /opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/synchrotron.py version 1.12.0
2020-03-24 00:51:40,230 - synapse.app.synchrotron - 245 - WARNING - - Starting daemon.
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - - Traceback (most recent call last):
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 277, in start
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -     hs.start_listening(listeners)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 575, in start_listening
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     self.get_tcp_replication().start_replication(self)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 581, in build_tcp_replication
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     return GenericWorkerReplicationHandler(self)
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 604, in __init__
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     self.send_handler = FederationSenderHandler(hs, self)
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 734, in __init__
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     self.federation_sender = hs.get_federation_sender()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 445, in build_federation_sender
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - -     raise Exception("Workers cannot send federation traffic")
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - - Exception: Workers cannot send federation traffic
2020-03-24 00:51:40,337 - twisted - 192 - CRITICAL - - Unhandled Error
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1282, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1262, in startRunning
    ReactorBase.startRunning(self)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 770, in startRunning
    self.fireSystemEvent('startup')
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 721, in fireSystemEvent
    event.fireEvent()
--- <exception caught here> ---
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 428, in fireEvent
    result = callable(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 297, in start
    sys.exit(1)
builtins.SystemExit: 1
```

/etc/matrix-synapse/worker/synchrotron-1.yaml:
```
worker_app: synapse.app.synchrotron

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8083
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/run/synapse/synchrotron1.pid
worker_log_config: /etc/matrix-synapse/synchrotron1-log.yaml
```

### Steps to reproduce

worked with 1.11 and stopped with the update

### Version information

- **Homeserver**: 

selfhosted

- **Version**:

 {
    "python_version": "3.7.3",
    "server_version": "1.12.0"
}

- **Install method**:

apt, matrix.org repository



Discussion:
Ah, looks like we're not correctly ignoring `send_federation` config option (which defaults to True) on non-federation sending workers. As a hacky fix you should be able to add `send_federation: false` to your worker config file

Resolving patch:
diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -860,6 +860,9 @@ def start(config_options):
 
         # Force the appservice to start since they will be disabled in the main config
         config.notify_appservices = True
+    else:
+        # For other worker types we force this to off.
+        config.notify_appservices = False
 
     if config.worker_app == "synapse.app.pusher":
         if config.start_pushers:
@@ -873,6 +876,9 @@ def start(config_options):
 
         # Force the pushers to start since they will be disabled in the main config
         config.start_pushers = True
+    else:
+        # For other worker types we force this to off.
+        config.start_pushers = False
 
     if config.worker_app == "synapse.app.user_dir":
         if config.update_user_directory:
@@ -886,6 +892,9 @@ def start(config_options):
 
         # Force the pushers to start since they will be disabled in the main config
         config.update_user_directory = True
+    else:
+        # For other worker types we force this to off.
+        config.update_user_directory = False
 
     if config.worker_app == "synapse.app.federation_sender":
         if config.send_federation:
@@ -899,6 +908,9 @@ def start(config_options):
 
         # Force the pushers to start since they will be disabled in the main config
         config.send_federation = True
+    else:
+        # For other worker types we force this to off.
+        config.send_federation = False
 
     synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
 


Test patch: