Repository: matrix-org/synapse

Issue:
`generate` option of docker image lacks a way to specify the data path
When you use the `generate` option of the docker image to generate a config file, there is no way to tell it where you want to put the media store etc, and it uses `/`.

Then, when you try to start synapse using the generated config, you get:

```
PermissionError: [Errno 13] Permission denied: '/media_store'
```

We should probably use `/data` by default (and recommend that a volume is mounted there), with an env var to allow you override the path. We also need to chown it so that synapse can write to it.

A workaround is:

* Add `-w /data` to the `generate` commandline, then
* `docker run <volume_options>  --entrypoint=chown matrixdotorg/synapse:latest 901 /data`


Discussion:
It looks like the docker container wants to use `/data` for all of the persistent stuff (e.g. generated server keys, certs, and `homeserver.yaml`).  It should probably use `/data/media` by default (or something like that).
> It should probably use /data/media by default (or something like that).

by default, the media store is put in a `media_store` subdirectory of the data directory, so if the data directory is `/data`, the media store will be `/data/media_store`. Or is that not what you mean?
That's exactly what I mean, I completely forgot how it works.  Disregard 👍 
Is it not possible to just simply allow the user to map ```/conf``` to an external volume, and edit the existing ```homeserver.yaml``` that is generated inside the container? This is what im doing right now, and it works great (so far anyway). 

I even went as far as copying over some lines from the "normal" ```homeserver.yaml```, into the stripped down "docker version" and Synapse picks them up just fine. So far everything works like id expect it to. I'm not sure if there is a danger in doing it this way though. I did pass the volume as read only so the container cant overwrite my file, just being cautious. 
@relink2013 of course that's fine. The point is just that the defaults in the generated file should be sensible and not *require* you to edit them.
Hey everyone, is there a way to make this work with docker-compose?  Forgive me if I don't understand.... isn't setting the entrypoint to a chown command overriding it?  If I set "entrypoint: chown 901:901 /data", the container just exits with code 0, I think cause it just executes the chown and the container does nothing.

I generated my own homeserver.yaml and corrected the "media_store" path to /data/media_store.  How exactly would I fix the permissions for the container while still maintaining the existing "start.py" entrypoint?

And sorry if this is real obvious, but how does overriding the entrypoint help?  Can you somehow edit that for the container after it runs and does the chown?
I tried to use docker-compose to setup, but didn't work out of the box. The following steps worked for me:

- Generate the config
- Create needed folders in docker volume and change ownership:

```
cd path/to/docker/volume
mkdir {media_store,uploads,logs}
chown 991:991 media_store/ uploads/ logs/
```

- Update `homeserver.yaml`

```
media_store_path: "/data/media_store
uploads_path: "/data/uploads"
```

- Update log config (`<yourdomain>.log.config`):

```
handlers:
    file:
        filename: /data/logs/homeserver.log
```

Notes:
  - 991 is the default uid (see `start.py`)
  - I didn't want to change the owner of the docker volume directly (e.g. synapse/_data) so I moved the logs to an own subfolder.
Thanks for the reply!  You got the right idea.  Trying to modify the entrypoint and/or the docker file is a roundabout way of attempting to fix the permissions.  And for some reason this issue and the linked issue have fixes that think the group:user is 901.

Through trial and error, I figured out the same thing:" fix the homeserver.yaml's path references and the log config, and chown the data folder to 991:991.

Back to the original issue, I believe it would be more friendly for the generate command to allow you to specify the data path.  The bigger issue is when you run it via docker and it doesn't setup the permissions for you.
> I tried to use docker-compose to setup, but didn't work out of the box. The following steps worked for me:
> 
> * Generate the config
> * Create needed folders in docker volume and change ownership:
> 
> ```
> cd path/to/docker/volume
> mkdir {media_store,uploads,logs}
> chown 991:991 media_store/ uploads/ logs/
> ```
> 
> * Update `homeserver.yaml`
> 
> ```
> media_store_path: "/data/media_store
> uploads_path: "/data/uploads"
> ```
> 
> * Update log config (`<yourdomain>.log.config`):
> 
> ```
> handlers:
>     file:
>         filename: /data/logs/homeserver.log
> ```
> 
> Notes:
> 
> * 991 is the default uid (see `start.py`)
> * I didn't want to change the owner of the docker volume directly (e.g. synapse/_data) so I moved the logs to an own subfolder.

I have the same problem. I can start the server without an persistent configuration-file. If I use the generate-command with the config-environment-variable, it generates a config-file but the container doesnt start anymore. I need this configuration file to change, because I want to connect the matrix-server to my Active Directory. I tried to change the values you told us, but it doesnt help at all. 

Resolving patch:
diff --git a/docker/start.py b/docker/start.py
--- a/docker/start.py
+++ b/docker/start.py
@@ -1,102 +1,172 @@
 #!/usr/local/bin/python
 
-import jinja2
+import codecs
+import glob
 import os
-import sys
 import subprocess
-import glob
-import codecs
+import sys
+
+import jinja2
+
 
 # Utility functions
-convert = lambda src, dst, environ: open(dst, "w").write(
-    jinja2.Template(open(src).read()).render(**environ)
-)
+def log(txt):
+    print(txt, file=sys.stderr)
+
+
+def error(txt):
+    log(txt)
+    sys.exit(2)
+
+
+def convert(src, dst, environ):
+    """Generate a file from a template
 
+    Args:
+        src (str): path to input file
+        dst (str): path to file to write
+        environ (dict): environment dictionary, for replacement mappings.
+    """
+    with open(src) as infile:
+        template = infile.read()
+    rendered = jinja2.Template(template).render(**environ)
+    with open(dst, "w") as outfile:
+        outfile.write(rendered)
 
-def check_arguments(environ, args):
-    for argument in args:
-        if argument not in environ:
-            print("Environment variable %s is mandatory, exiting." % argument)
-            sys.exit(2)
 
+def generate_config_from_template(environ, ownership):
+    """Generate a homeserver.yaml from environment variables
+
+    Args:
+        environ (dict): environment dictionary
+        ownership (str): "<user>:<group>" string which will be used to set
+            ownership of the generated configs
+
+    Returns:
+        path to generated config file
+    """
+    for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"):
+        if v not in environ:
+            error(
+                "Environment variable '%s' is mandatory when generating a config "
+                "file on-the-fly." % (v,)
+            )
+
+    # populate some params from data files (if they exist, else create new ones)
+    environ = environ.copy()
+    secrets = {
+        "registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
+        "macaroon": "SYNAPSE_MACAROON_SECRET_KEY",
+    }
 
-def generate_secrets(environ, secrets):
     for name, secret in secrets.items():
         if secret not in environ:
             filename = "/data/%s.%s.key" % (environ["SYNAPSE_SERVER_NAME"], name)
+
+            # if the file already exists, load in the existing value; otherwise,
+            # generate a new secret and write it to a file
+
             if os.path.exists(filename):
                 with open(filename) as handle:
                     value = handle.read()
             else:
-                print("Generating a random secret for {}".format(name))
+                log("Generating a random secret for {}".format(name))
                 value = codecs.encode(os.urandom(32), "hex").decode()
                 with open(filename, "w") as handle:
                     handle.write(value)
             environ[secret] = value
 
+    environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
+    if not os.path.exists("/compiled"):
+        os.mkdir("/compiled")
+
+    config_path = "/compiled/homeserver.yaml"
+
+    # Convert SYNAPSE_NO_TLS to boolean if exists
+    if "SYNAPSE_NO_TLS" in environ:
+        tlsanswerstring = str.lower(environ["SYNAPSE_NO_TLS"])
+        if tlsanswerstring in ("true", "on", "1", "yes"):
+            environ["SYNAPSE_NO_TLS"] = True
+        else:
+            if tlsanswerstring in ("false", "off", "0", "no"):
+                environ["SYNAPSE_NO_TLS"] = False
+            else:
+                error(
+                    'Environment variable "SYNAPSE_NO_TLS" found but value "'
+                    + tlsanswerstring
+                    + '" unrecognized; exiting.'
+                )
+
+    convert("/conf/homeserver.yaml", config_path, environ)
+    convert("/conf/log.config", "/compiled/log.config", environ)
+    subprocess.check_output(["chown", "-R", ownership, "/data"])
+    return config_path
+
+
+def run_generate_config(environ, ownership):
+    """Run synapse with a --generate-config param to generate a template config file
+
+    Args:
+        environ (dict): env var dict
+        ownership (str): "userid:groupid" arg for chmod
+
+    Never returns.
+    """
+    for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH"):
+        if v not in environ:
+            error("Environment variable '%s' is mandatory in `generate` mode." % (v,))
 
-# Prepare the configuration
-mode = sys.argv[1] if len(sys.argv) > 1 else None
-environ = os.environ.copy()
-ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
-args = ["python", "-m", "synapse.app.homeserver"]
+    server_name = environ["SYNAPSE_SERVER_NAME"]
+    config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
+    data_dir = environ.get("SYNAPSE_DATA_DIR", "/data")
 
-# In generate mode, generate a configuration, missing keys, then exit
-if mode == "generate":
-    check_arguments(
-        environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH")
-    )
-    args += [
+    # create a suitable log config from our template
+    log_config_file = "%s/%s.log.config" % (config_dir, server_name)
+    if not os.path.exists(log_config_file):
+        log("Creating log config %s" % (log_config_file,))
+        convert("/conf/log.config", log_config_file, environ)
+
+    # make sure that synapse has perms to write to the data dir.
+    subprocess.check_output(["chown", ownership, data_dir])
+
+    args = [
+        "python",
+        "-m",
+        "synapse.app.homeserver",
         "--server-name",
-        environ["SYNAPSE_SERVER_NAME"],
+        server_name,
         "--report-stats",
         environ["SYNAPSE_REPORT_STATS"],
         "--config-path",
         environ["SYNAPSE_CONFIG_PATH"],
+        "--config-directory",
+        config_dir,
+        "--data-directory",
+        data_dir,
         "--generate-config",
     ]
+    # log("running %s" % (args, ))
     os.execv("/usr/local/bin/python", args)
 
-# In normal mode, generate missing keys if any, then run synapse
-else:
+
+def main(args, environ):
+    mode = args[1] if len(args) > 1 else None
+    ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
+
+    # In generate mode, generate a configuration and missing keys, then exit
+    if mode == "generate":
+        return run_generate_config(environ, ownership)
+
+    # In normal mode, generate missing keys if any, then run synapse
     if "SYNAPSE_CONFIG_PATH" in environ:
         config_path = environ["SYNAPSE_CONFIG_PATH"]
     else:
-        check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"))
-        generate_secrets(
-            environ,
-            {
-                "registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
-                "macaroon": "SYNAPSE_MACAROON_SECRET_KEY",
-            },
-        )
-        environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
-        if not os.path.exists("/compiled"):
-            os.mkdir("/compiled")
-
-        config_path = "/compiled/homeserver.yaml"
-
-        # Convert SYNAPSE_NO_TLS to boolean if exists
-        if "SYNAPSE_NO_TLS" in environ:
-            tlsanswerstring = str.lower(environ["SYNAPSE_NO_TLS"])
-            if tlsanswerstring in ("true", "on", "1", "yes"):
-                environ["SYNAPSE_NO_TLS"] = True
-            else:
-                if tlsanswerstring in ("false", "off", "0", "no"):
-                    environ["SYNAPSE_NO_TLS"] = False
-                else:
-                    print(
-                        'Environment variable "SYNAPSE_NO_TLS" found but value "'
-                        + tlsanswerstring
-                        + '" unrecognized; exiting.'
-                    )
-                    sys.exit(2)
-
-        convert("/conf/homeserver.yaml", config_path, environ)
-        convert("/conf/log.config", "/compiled/log.config", environ)
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
-
-    args += [
+        config_path = generate_config_from_template(environ, ownership)
+
+    args = [
+        "python",
+        "-m",
+        "synapse.app.homeserver",
         "--config-path",
         config_path,
         # tell synapse to put any generated keys in /data rather than /compiled
@@ -107,3 +177,7 @@ def generate_secrets(environ, secrets):
     # Generate missing keys and start synapse
     subprocess.check_output(args + ["--generate-keys"])
     os.execv("/sbin/su-exec", ["su-exec", ownership] + args)
+
+
+if __name__ == "__main__":
+    main(sys.argv, os.environ)


Test patch: