Repository: matrix-org/synapse

Issue:
Synapse exits with zero exit code on config error
If synapse cannot bind to a TCP socket, or there is an error in the listeners config, synapse exits with a zero exit code even if it is configured not to daemonize.

It looks like it doesn't actually exit when `sys.exit(1)` is called but rather does a bunch of cleanup and forgets the exit code in the meantime


Discussion:
Obviously, this is problematic for `systemd` or other process controllers, because they cannot tell that there is an error and will tend to restart synapse in a loop.
(This is still the case)
I've tested blowing up synapse in the config and setup stages while running it as a python module and it's exit code's were correct. The problem is with sycntl, it swallows the error codes.

The simple solution is to have a non-zero error code if it fails to start something. However, syctl can simultaneously try to start up a number of workers of which only a subset might fail. It's unclear whether the command should be considered a complete failure in this case because there will be workers running after it is executed. I presume this was intentional because it allows synapse to limp along.

I think the cleanest solution would be to have two error codes. One if sycntl was a complete failure and the other if only a subset failed. I propose 1 and 4 respectively.

I definitely wasn't thinking of synctl when I wrote this. Perhaps it's been fixed since...
It does not appear to have been fixed since.

I have misconfigured my synapse so that it has:

```yaml
listeners:
  - port: 8
    type: http
    tls: true
    resources:
      - names: [client, federation]
```

I get:

```
(env3)rav@fred:~/work/synapse (develop $%=)$ python -m synapse.app.homeserver -c homeserver.fred.yaml
...
2019-09-06 15:50:12,678 - twisted - 166 - ERROR -  - Traceback (most recent call last):
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1361, in startListening
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  -     skt.bind(addr)
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  - PermissionError: [Errno 13] Permission denied
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  - 
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  - During handling of the above exception, another exception occurred:
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  - 
2019-09-06 15:50:12,679 - twisted - 166 - ERROR -  - Traceback (most recent call last):
2019-09-06 15:50:12,680 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/_base.py", line 269, in start
2019-09-06 15:50:12,680 - twisted - 166 - ERROR -  -     hs.start_listening(listeners)
2019-09-06 15:50:12,680 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/homeserver.py", line 266, in start_listening
2019-09-06 15:50:12,681 - twisted - 166 - ERROR -  -     self._listening_services.extend(self._listener_http(config, listener))
2019-09-06 15:50:12,681 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/homeserver.py", line 137, in _listener_http
2019-09-06 15:50:12,681 - twisted - 166 - ERROR -  -     reactor=self.get_reactor(),
2019-09-06 15:50:12,681 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/_base.py", line 196, in listen_ssl
2019-09-06 15:50:12,682 - twisted - 166 - ERROR -  -     check_bind_error(e, address, bind_addresses)
2019-09-06 15:50:12,682 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/__init__.py", line 49, in check_bind_error
2019-09-06 15:50:12,682 - twisted - 166 - ERROR -  -     raise e
2019-09-06 15:50:12,682 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/synapse/app/_base.py", line 193, in listen_ssl
2019-09-06 15:50:12,682 - twisted - 166 - ERROR -  -     reactor.listenSSL(port, factory, context_factory, backlog, address)
2019-09-06 15:50:12,683 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 522, in listenSSL
2019-09-06 15:50:12,683 - twisted - 166 - ERROR -  -     port = self.listenTCP(port, tlsFactory, backlog, interface)
2019-09-06 15:50:12,683 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 495, in listenTCP
2019-09-06 15:50:12,683 - twisted - 166 - ERROR -  -     p.startListening()
2019-09-06 15:50:12,684 - twisted - 166 - ERROR -  -   File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1363, in startListening
2019-09-06 15:50:12,684 - twisted - 166 - ERROR -  -     raise CannotListenError(self.interface, self.port, le)
2019-09-06 15:50:12,684 - twisted - 166 - ERROR -  - twisted.internet.error.CannotListenError: Couldn't listen on :::8: [Errno 13] Permission denied.
2019-09-06 15:50:12,685 - synapse.handlers.deactivate_account - 159 - INFO - user_parter_loop-0 - User parter finished: stopping
2019-09-06 15:50:12,686 - synapse.app.homeserver - 579 - INFO -  - report_stats can use psutil
2019-09-06 15:50:12,687 - synapse.handlers.presence - 236 - INFO - presence.on_shutdown-0 - Performing _on_shutdown. Persisting 0 unpersisted changes
2019-09-06 15:50:12,687 - synapse.handlers.presence - 247 - INFO - presence.on_shutdown-0 - Finished _on_shutdown
2019-09-06 15:50:12,710 - twisted - 166 - CRITICAL -  - Unhandled error in Deferred:
2019-09-06 15:50:12,710 - twisted - 166 - CRITICAL -  - 
Traceback (most recent call last):
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1361, in startListening
    skt.bind(addr)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rav/work/synapse/synapse/app/_base.py", line 269, in start
    hs.start_listening(listeners)
  File "/home/rav/work/synapse/synapse/app/homeserver.py", line 266, in start_listening
    self._listening_services.extend(self._listener_http(config, listener))
  File "/home/rav/work/synapse/synapse/app/homeserver.py", line 137, in _listener_http
    reactor=self.get_reactor(),
  File "/home/rav/work/synapse/synapse/app/_base.py", line 196, in listen_ssl
    check_bind_error(e, address, bind_addresses)
  File "/home/rav/work/synapse/synapse/app/__init__.py", line 49, in check_bind_error
    raise e
  File "/home/rav/work/synapse/synapse/app/_base.py", line 193, in listen_ssl
    reactor.listenSSL(port, factory, context_factory, backlog, address)
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 522, in listenSSL
    port = self.listenTCP(port, tlsFactory, backlog, interface)
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 495, in listenTCP
    p.startListening()
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1363, in startListening
    raise CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on :::8: [Errno 13] Permission denied.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rav/work/synapse/env3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/rav/work/synapse/synapse/app/homeserver.py", line 435, in start
    _base.start(hs, config.listeners)
  File "/home/rav/work/synapse/synapse/app/_base.py", line 279, in start
    sys.exit(1)
SystemExit: 1
2019-09-06 15:50:12,720 - twisted - 166 - INFO -  - Main loop terminated.
(env3)rav@fred:~/work/synapse (develop $%=)$ echo $?
0

Looks like this is due to us binding ports *after* having started the reactor, at which point we always return with 0. Errors in the config parsing *do* return with a non-zero error code because that happens before we start the reactor.

Resolving patch:
diff --git a/synapse/app/_base.py b/synapse/app/_base.py
--- a/synapse/app/_base.py
+++ b/synapse/app/_base.py
@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2017 New Vector Ltd
+# Copyright 2019-2021 The Matrix.org Foundation C.I.C
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,7 +20,7 @@
 import socket
 import sys
 import traceback
-from typing import Iterable
+from typing import Awaitable, Callable, Iterable
 
 from typing_extensions import NoReturn
 
@@ -143,6 +144,45 @@ def quit_with_error(error_string: str) -> NoReturn:
     sys.exit(1)
 
 
+def register_start(cb: Callable[..., Awaitable], *args, **kwargs) -> None:
+    """Register a callback with the reactor, to be called once it is running
+
+    This can be used to initialise parts of the system which require an asynchronous
+    setup.
+
+    Any exception raised by the callback will be printed and logged, and the process
+    will exit.
+    """
+
+    async def wrapper():
+        try:
+            await cb(*args, **kwargs)
+        except Exception:
+            # previously, we used Failure().printTraceback() here, in the hope that
+            # would give better tracebacks than traceback.print_exc(). However, that
+            # doesn't handle chained exceptions (with a __cause__ or __context__) well,
+            # and I *think* the need for Failure() is reduced now that we mostly use
+            # async/await.
+
+            # Write the exception to both the logs *and* the unredirected stderr,
+            # because people tend to get confused if it only goes to one or the other.
+            #
+            # One problem with this is that if people are using a logging config that
+            # logs to the console (as is common eg under docker), they will get two
+            # copies of the exception. We could maybe try to detect that, but it's
+            # probably a cost we can bear.
+            logger.fatal("Error during startup", exc_info=True)
+            print("Error during startup:", file=sys.__stderr__)
+            traceback.print_exc(file=sys.__stderr__)
+
+            # it's no use calling sys.exit here, since that just raises a SystemExit
+            # exception which is then caught by the reactor, and everything carries
+            # on as normal.
+            os._exit(1)
+
+    reactor.callWhenRunning(lambda: defer.ensureDeferred(wrapper()))
+
+
 def listen_metrics(bind_addresses, port):
     """
     Start Prometheus metrics server.
@@ -227,7 +267,7 @@ def refresh_certificate(hs):
         logger.info("Context factories updated.")
 
 
-def start(hs: "synapse.server.HomeServer", listeners: Iterable[ListenerConfig]):
+async def start(hs: "synapse.server.HomeServer", listeners: Iterable[ListenerConfig]):
     """
     Start a Synapse server or worker.
 
@@ -241,75 +281,67 @@ def start(hs: "synapse.server.HomeServer", listeners: Iterable[ListenerConfig]):
         hs: homeserver instance
         listeners: Listener configuration ('listeners' in homeserver.yaml)
     """
-    try:
-        # Set up the SIGHUP machinery.
-        if hasattr(signal, "SIGHUP"):
+    # Set up the SIGHUP machinery.
+    if hasattr(signal, "SIGHUP"):
+        reactor = hs.get_reactor()
 
-            reactor = hs.get_reactor()
+        @wrap_as_background_process("sighup")
+        def handle_sighup(*args, **kwargs):
+            # Tell systemd our state, if we're using it. This will silently fail if
+            # we're not using systemd.
+            sdnotify(b"RELOADING=1")
 
-            @wrap_as_background_process("sighup")
-            def handle_sighup(*args, **kwargs):
-                # Tell systemd our state, if we're using it. This will silently fail if
-                # we're not using systemd.
-                sdnotify(b"RELOADING=1")
+            for i, args, kwargs in _sighup_callbacks:
+                i(*args, **kwargs)
 
-                for i, args, kwargs in _sighup_callbacks:
-                    i(*args, **kwargs)
+            sdnotify(b"READY=1")
 
-                sdnotify(b"READY=1")
+        # We defer running the sighup handlers until next reactor tick. This
+        # is so that we're in a sane state, e.g. flushing the logs may fail
+        # if the sighup happens in the middle of writing a log entry.
+        def run_sighup(*args, **kwargs):
+            # `callFromThread` should be "signal safe" as well as thread
+            # safe.
+            reactor.callFromThread(handle_sighup, *args, **kwargs)
 
-            # We defer running the sighup handlers until next reactor tick. This
-            # is so that we're in a sane state, e.g. flushing the logs may fail
-            # if the sighup happens in the middle of writing a log entry.
-            def run_sighup(*args, **kwargs):
-                # `callFromThread` should be "signal safe" as well as thread
-                # safe.
-                reactor.callFromThread(handle_sighup, *args, **kwargs)
+        signal.signal(signal.SIGHUP, run_sighup)
 
-            signal.signal(signal.SIGHUP, run_sighup)
+        register_sighup(refresh_certificate, hs)
 
-            register_sighup(refresh_certificate, hs)
+    # Load the certificate from disk.
+    refresh_certificate(hs)
 
-        # Load the certificate from disk.
-        refresh_certificate(hs)
+    # Start the tracer
+    synapse.logging.opentracing.init_tracer(  # type: ignore[attr-defined] # noqa
+        hs
+    )
 
-        # Start the tracer
-        synapse.logging.opentracing.init_tracer(  # type: ignore[attr-defined] # noqa
-            hs
-        )
+    # It is now safe to start your Synapse.
+    hs.start_listening(listeners)
+    hs.get_datastore().db_pool.start_profiling()
+    hs.get_pusherpool().start()
+
+    # Log when we start the shut down process.
+    hs.get_reactor().addSystemEventTrigger(
+        "before", "shutdown", logger.info, "Shutting down..."
+    )
 
-        # It is now safe to start your Synapse.
-        hs.start_listening(listeners)
-        hs.get_datastore().db_pool.start_profiling()
-        hs.get_pusherpool().start()
+    setup_sentry(hs)
+    setup_sdnotify(hs)
 
-        # Log when we start the shut down process.
-        hs.get_reactor().addSystemEventTrigger(
-            "before", "shutdown", logger.info, "Shutting down..."
-        )
+    # If background tasks are running on the main process, start collecting the
+    # phone home stats.
+    if hs.config.run_background_tasks:
+        start_phone_stats_home(hs)
 
-        setup_sentry(hs)
-        setup_sdnotify(hs)
-
-        # If background tasks are running on the main process, start collecting the
-        # phone home stats.
-        if hs.config.run_background_tasks:
-            start_phone_stats_home(hs)
-
-        # We now freeze all allocated objects in the hopes that (almost)
-        # everything currently allocated are things that will be used for the
-        # rest of time. Doing so means less work each GC (hopefully).
-        #
-        # This only works on Python 3.7
-        if sys.version_info >= (3, 7):
-            gc.collect()
-            gc.freeze()
-    except Exception:
-        traceback.print_exc(file=sys.stderr)
-        reactor = hs.get_reactor()
-        if reactor.running:
-            reactor.stop()
-        sys.exit(1)
+    # We now freeze all allocated objects in the hopes that (almost)
+    # everything currently allocated are things that will be used for the
+    # rest of time. Doing so means less work each GC (hopefully).
+    #
+    # This only works on Python 3.7
+    if sys.version_info >= (3, 7):
+        gc.collect()
+        gc.freeze()
 
 
 def setup_sentry(hs):
diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -21,7 +21,7 @@
 
 from typing_extensions import ContextManager
 
-from twisted.internet import address, reactor
+from twisted.internet import address
 
 import synapse
 import synapse.events
@@ -34,6 +34,7 @@
     SERVER_KEY_V2_PREFIX,
 )
 from synapse.app import _base
+from synapse.app._base import register_start
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
@@ -960,9 +961,7 @@ def start(config_options):
     # streams. Will no-op if no streams can be written to by this worker.
     hs.get_replication_streamer()
 
-    reactor.addSystemEventTrigger(
-        "before", "startup", _base.start, hs, config.worker_listeners
-    )
+    register_start(_base.start, hs, config.worker_listeners)
 
     _base.start_worker_reactor("synapse-generic-worker", config)
 
diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -20,8 +20,7 @@
 import sys
 from typing import Iterable, Iterator
 
-from twisted.internet import defer, reactor
-from twisted.python.failure import Failure
+from twisted.internet import reactor
 from twisted.web.resource import EncodingResourceWrapper, IResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File
@@ -38,7 +37,7 @@
     WEB_CLIENT_PREFIX,
 )
 from synapse.app import _base
-from synapse.app._base import listen_ssl, listen_tcp, quit_with_error
+from synapse.app._base import listen_ssl, listen_tcp, quit_with_error, register_start
 from synapse.config._base import ConfigError
 from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.config.homeserver import HomeServerConfig
@@ -414,40 +413,29 @@ async def reprovision_acme():
             _base.refresh_certificate(hs)
 
     async def start():
-        try:
-            # Run the ACME provisioning code, if it's enabled.
-            if hs.config.acme_enabled:
-                acme = hs.get_acme_handler()
-                # Start up the webservices which we will respond to ACME
-                # challenges with, and then provision.
-                await acme.start_listening()
-                await do_acme()
-
-                # Check if it needs to be reprovisioned every day.
-                hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)
-
-            # Load the OIDC provider metadatas, if OIDC is enabled.
-            if hs.config.oidc_enabled:
-                oidc = hs.get_oidc_handler()
-                # Loading the provider metadata also ensures the provider config is valid.
-                await oidc.load_metadata()
-                await oidc.load_jwks()
-
-            _base.start(hs, config.listeners)
-
-            hs.get_datastore().db_pool.updates.start_doing_background_updates()
-        except Exception:
-            # Print the exception and bail out.
-            print("Error during startup:", file=sys.stderr)
-
-            # this gives better tracebacks than traceback.print_exc()
-            Failure().printTraceback(file=sys.stderr)
-
-            if reactor.running:
-                reactor.stop()
-            sys.exit(1)
-
-    reactor.callWhenRunning(lambda: defer.ensureDeferred(start()))
+        # Run the ACME provisioning code, if it's enabled.
+        if hs.config.acme_enabled:
+            acme = hs.get_acme_handler()
+            # Start up the webservices which we will respond to ACME
+            # challenges with, and then provision.
+            await acme.start_listening()
+            await do_acme()
+
+            # Check if it needs to be reprovisioned every day.
+            hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)
+
+        # Load the OIDC provider metadatas, if OIDC is enabled.
+        if hs.config.oidc_enabled:
+            oidc = hs.get_oidc_handler()
+            # Loading the provider metadata also ensures the provider config is valid.
+            await oidc.load_metadata()
+            await oidc.load_jwks()
+
+        await _base.start(hs, config.listeners)
+
+        hs.get_datastore().db_pool.updates.start_doing_background_updates()
+
+    register_start(start)
 
     return hs
 


Test patch: