Repository: matrix-org/synapse

Issue:
Dockerfile-workers not showing logging in early output
### Description

When running a docker image, you can watch logs of output printed(usually to /dev/stdout) by running a `docker logs container_name`. For some reason, when using the Dockerfile-workers build, you can see the output that is logged from start.py but not configure_workers_and_start.py.

### Steps to reproduce

- run docker image built from Dockerfile-workers tagged as synapse, add a `-e SYNAPSE_WORKER_TYPES=omgIneedWORKERZ` as an obvious example of an incorrect worker type
- look at output from `docker logs <container_name>`

### Homeserver

My homeserver, and Complement CI test runs

### Synapse Version

1.69.0rc2

### Installation Method

Docker (matrixdotorg/synapse)

### Platform

Docker Engine on linux with python 3.10.8

### Relevant log output

```shell
[admin@matrix2 ~]$ docker logs synapse
Generating a random secret for SYNAPSE_REGISTRATION_SHARED_SECRET
Generating a random secret for SYNAPSE_MACAROON_SECRET_KEY
Generating synapse config file /data/homeserver.yaml
Generating log config file /data/log.config
Setting ownership on /data to 991:991
2022-10-14 23:59:25,368 INFO Included extra file "/etc/supervisor/conf.d/synapse.conf" during parsing
2022-10-14 23:59:25,368 INFO Set uid to user 0 succeeded
```


### Anything else that would be useful to know?

I expected to see:
```
[admin@matrix2 ~]$ docker logs synapse
Generating base homeserver config
Generating a random secret for SYNAPSE_REGISTRATION_SHARED_SECRET
Generating a random secret for SYNAPSE_MACAROON_SECRET_KEY
Generating synapse config file /data/homeserver.yaml
Generating log config file /data/log.config
Setting ownership on /data to 991:991
omgIneedWORKERZ is an unknown worker type! It will be ignored
Starting supervisord
2022-10-15 00:09:39,165 INFO Included extra file "/etc/supervisor/conf.d/synapse.conf" during parsing
2022-10-15 00:09:39,165 INFO Set uid to user 0 succeeded
```
If this log output isn't being sent, no one will know if the system couldn't find libjemalloc.so.2, misspelled a worker_type or committed some other error while declaring variables to pass to the container during startup.


Discussion:

Resolving patch:
diff --git a/docker/configure_workers_and_start.py b/docker/configure_workers_and_start.py
--- a/docker/configure_workers_and_start.py
+++ b/docker/configure_workers_and_start.py
@@ -230,24 +230,19 @@
 
 # Utility functions
 def log(txt: str) -> None:
-    """Log something to the stdout.
-
-    Args:
-        txt: The text to log.
-    """
     print(txt)
 
 
 def error(txt: str) -> NoReturn:
-    """Log something and exit with an error code.
-
-    Args:
-        txt: The text to log in error.
-    """
-    log(txt)
+    print(txt, file=sys.stderr)
     sys.exit(2)
 
 
+def flush_buffers() -> None:
+    sys.stdout.flush()
+    sys.stderr.flush()
+
+
 def convert(src: str, dst: str, **template_vars: object) -> None:
     """Generate a file from a template
 
@@ -328,7 +323,7 @@ def generate_base_homeserver_config() -> None:
     # start.py already does this for us, so just call that.
     # note that this script is copied in in the official, monolith dockerfile
     os.environ["SYNAPSE_HTTP_PORT"] = str(MAIN_PROCESS_HTTP_LISTENER_PORT)
-    subprocess.check_output(["/usr/local/bin/python", "/start.py", "migrate_config"])
+    subprocess.run(["/usr/local/bin/python", "/start.py", "migrate_config"], check=True)
 
 
 def generate_worker_files(
@@ -642,6 +637,7 @@ def main(args: List[str], environ: MutableMapping[str, str]) -> None:
     # Start supervisord, which will start Synapse, all of the configured worker
     # processes, redis, nginx etc. according to the config we created above.
     log("Starting supervisord")
+    flush_buffers()
     os.execle(
         "/usr/local/bin/supervisord",
         "supervisord",
diff --git a/docker/start.py b/docker/start.py
--- a/docker/start.py
+++ b/docker/start.py
@@ -13,14 +13,19 @@
 
 # Utility functions
 def log(txt: str) -> None:
-    print(txt, file=sys.stderr)
+    print(txt)
 
 
 def error(txt: str) -> NoReturn:
-    log(txt)
+    print(txt, file=sys.stderr)
     sys.exit(2)
 
 
+def flush_buffers() -> None:
+    sys.stdout.flush()
+    sys.stderr.flush()
+
+
 def convert(src: str, dst: str, environ: Mapping[str, object]) -> None:
     """Generate a file from a template
 
@@ -131,10 +136,10 @@ def generate_config_from_template(
 
     if ownership is not None:
         log(f"Setting ownership on /data to {ownership}")
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
+        subprocess.run(["chown", "-R", ownership, "/data"], check=True)
         args = ["gosu", ownership] + args
 
-    subprocess.check_output(args)
+    subprocess.run(args, check=True)
 
 
 def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) -> None:
@@ -158,7 +163,7 @@ def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) ->
     if ownership is not None:
         # make sure that synapse has perms to write to the data dir.
         log(f"Setting ownership on {data_dir} to {ownership}")
-        subprocess.check_output(["chown", ownership, data_dir])
+        subprocess.run(["chown", ownership, data_dir], check=True)
 
     # create a suitable log config from our template
     log_config_file = "%s/%s.log.config" % (config_dir, server_name)
@@ -185,6 +190,7 @@ def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) ->
         "--open-private-ports",
     ]
     # log("running %s" % (args, ))
+    flush_buffers()
     os.execv(sys.executable, args)
 
 
@@ -267,8 +273,10 @@ def main(args: List[str], environ: MutableMapping[str, str]) -> None:
     args = [sys.executable] + args
     if ownership is not None:
         args = ["gosu", ownership] + args
+        flush_buffers()
         os.execve("/usr/sbin/gosu", args, environ)
     else:
+        flush_buffers()
         os.execve(sys.executable, args, environ)
 
 


Test patch: