Package: marathon-acme
AutomateACMEcertificates forMarathonapps served bymarathon-lbHow it worksThere is one big requirement for deployingmarathon-acme: there must
be shared persistent storage betweenmarathon-acmeand allmarathon-lbinstances. This will be used to store the certificates.marathon-acmewatches Marathon for changes to app definitions.It collects the values of allMARATHON_ACME_{n}_DOMAINlabels on
apps. This will form the set of domains to fetch certificates for.It generates, verifies and stores certificates for any new domains
using the configured ACME certificate authority.It tellsmarathon-lbto reload using themarathon-lbHTTP
API.It issues new certificates for soon-to-expire certificates once a
day.marathon-acmeis written in Python usingTwisted. The certificate issuing
functionality is possible thanks to thetxacmelibrary.The ACME provider that most people are likely to use isLet’s
Encrypt. Before usingmarathon-acmewith Let’s Encrypt, make sure you are aware of theirrate
limits.The entire certificate-issuing workflow is shown below:Usagemarathon-acmeis available as a pip-installable Python package onPyPI. However, most
users will probably want to use the Docker image available fromDocker
Hub.> $ docker run --rm praekeltfoundation/marathon-acme --help
usage: marathon-acme [-h] [-a ACME] [-e EMAIL] [-m MARATHON[,MARATHON,...]]
                     [-l LB[,LB,...]] [-g GROUP] [--allow-multiple-certs]
                     [--listen LISTEN] [--sse-timeout SSE_TIMEOUT]
                     [--log-level {debug,info,warn,error,critical}]
                     storage-dir

Automatically manage ACME certificates for Marathon apps

positional arguments:
  storage-dir           Path to directory for storing certificates

optional arguments:
  -h, --help            show this help message and exit
  -a ACME, --acme ACME  The address for the ACME Directory Resource (default:
                        https://acme-v01.api.letsencrypt.org/directory)
  -e EMAIL, --email EMAIL
                        An email address to register with the ACME service
                        (optional)
  -m MARATHON[,MARATHON,...], --marathon MARATHON[,MARATHON,...]
                        The addresses for the Marathon HTTP API (default:
                        http://marathon.mesos:8080)
  -l LB[,LB,...], --lb LB[,LB,...]
                        The addresses for the marathon-lb HTTP API (default:
                        http://marathon-lb.marathon.mesos:9090)
  -g GROUP, --group GROUP
                        The marathon-lb group to issue certificates for
                        (default: external)
  --allow-multiple-certs
                        Allow multiple certificates for a single app port.
                        This allows multiple domains for an app, but is not
                        recommended.
  --listen LISTEN       The address for the port to listen on (default: :8000)
  --sse-timeout SSE_TIMEOUT
                        Amount of time in seconds to wait for some event data
                        to be received from Marathon. Set to 0 to disable.
                        (default: 60)
  --log-level {debug,info,warn,error,critical}
                        The minimum severity level to log messages at
                        (default: info)
  --version             show program's version number and exitmarathon-acmeapp definitionmarathon-acmeshould be deployed as a Marathon app.{"id":"/marathon-acme","cpus":0.01,"mem":128.0,"args":["--email","letsencrypt@example.com","--marathon","http://marathon1:8080,http://marathon2:8080,http://marathon3:8080","--lb","http://lb1:9090,http://lb2:9090","/var/lib/marathon-acme"],"labels":{"HAPROXY_GROUP":"external","HAPROXY_0_VHOST":"marathon-acme.example.com","HAPROXY_0_BACKEND_WEIGHT":"1","HAPROXY_0_PATH":"/.well-known/acme-challenge/","HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH":"  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}\n  use_backend {backend} if host_{cleanedUpHostname} path_{backend}\n"},"container":{"type":"DOCKER","docker":{"image":"praekeltfoundation/marathon-acme","network":"BRIDGE","portMappings":[{"containerPort":8000,"hostPort":0}],"parameters":[{"value":"my-volume-driver","key":"volume-driver"},{"value":"marathon-acme-certs:/var/lib/marathon-acme","key":"volume"}],}}}The above should mostly be standard across different deployments. The
volume parameters will depend on your particular networked storage
solution.HAPROXYlabels"labels":{"HAPROXY_GROUP":"external","HAPROXY_0_VHOST":"marathon-acme.example.com","HAPROXY_0_BACKEND_WEIGHT":"1","HAPROXY_0_PATH":"/.well-known/acme-challenge/","HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH":"  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}\n  use_backend {backend} if host_{cleanedUpHostname} path_{backend}\n"}Several specialmarathon-lblabels are needed in order to forward
all HTTP requests whose path begins with/.well-known/acme-challenge/tomarathon-acme, in order to serve
ACMEHTTP
challengeresponses.HAPROXY_GROUPexternalmarathon-lbinstances are assigned a group. Only Marathon apps with
aHAPROXY_GROUPlabel that matches their group are routed with that
instance. “external” is the common name for publicly-facing load
balancers.HAPROXY_0_VHOSTmarathon-acme.example.commarathon-acmeneeds its own domain to respond to ACME challenge
requests on. This domain must resolve to yourmarathon-lbinstance(s).HAPROXY_0_BACKEND_WEIGHT1We want this rule in HAProxy’s config file to come before any others so
that requests are routed tomarathon-acmebefore we do the (usually)
domain-based routing for the other Marathon apps. The default weight is0, so we set to1so that the rule comes first.HAPROXY_0_PATH/.well-known/acme-challenge/This is the beginning of the HTTP path to ACME validation challenges.HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATHacl host_{cleanedUpHostname} hdr(host) -i {hostname}
acl path_{backend} path_beg {path}
redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}
use_backend {backend} if host_{cleanedUpHostname} path_{backend}This is where it gets complicated… It’s possible to edit the templates
used for generating the HAProxy on a per-app basis using labels. This is
necessary because by defaultmarathon-lbwill route based on domain
first, but we don’t want to do that. You can see the standard templatehere.Here, we add an extraredirectrule. This redirects all requests
matching the ACME challenge path tomarathon-acme, except those
requests already headed formarathon-acme. The Let’s Encrypt server
will follow redirects.HAPROXYHTTPS labelsIt is possible to havemarathon-acmeserve ACME challenge requests
over HTTPS, although this is usually not necessary. In this case, a
certificate needs to be issued formarathon-acmeand the HTTP
redirect label needs to be modified:"labels":{...,"MARATHON_ACME_0_DOMAIN":"marathon-acme.example.com","HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH":"  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix https://{hostname} code 302 if path_{backend}\n"}Note that using theHAPROXY_0_REDIRECT_TO_HTTPSlabel formarathon-acmewill break things. This label is difficult for us to
use because of the waymarathon-lb’s templating works.MARATHON_ACME_0_DOMAINmarathon-acme.example.comHere we set upmarathon-acmeto fetch a certificate for itself.HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATHacl host_{cleanedUpHostname} hdr(host) -i {hostname}
acl path_{backend} path_beg {path}
redirect prefix https://{hostname} code 302 if path_{backend}We redirect to the HTTPS address (https://{hostname}) for all
domains (includingmarathon-acme’s) for requests to the ACME
challenge path. Theuse_backenddirective can now be removed since
the backend is never used over HTTP as all requests are redirected.Note that this label can only be set after marathon-acme has
fetched the first certificate for its own domain.In other words, set
theMARATHON_ACME_0_DOMAINfirstand make sure it has taken effect
before setting this one.Docker imagesDocker images are available fromDocker
Hub.
There are two different streams of Docker images available::latest/:<version>: Tracks the latest released version ofmarathon-acmeonPyPI.
The Dockerfile for these is in thepraekeltfoundation/docker-marathon-acmerepo.:develop: Tracks thedevelopbranch of this repo and is built
using theDockerfilein this repo.For more details on the Docker images, see thepraekeltfoundation/docker-marathon-acmerepo.Volumes and portsThemarathon-acmecontainer defaults to the/var/lib/marathon-acmedirectory to store certificates and the ACME
client private key. This is the path inside the container that should be
mounted as a shared volume.The container also defaults to listening on port 8000 on all interfaces.You can override these values by providing arguments to the Docker
container.Certificate filesmarathon-acmecreates the following directory/file structure:/var/lib/marathon-acme/client.key: The ACME client private keydefault.pem: A self-signed wildcard cert for HAProxy to fallback tocerts/www.example.com.pem: An issued ACME certificate for a domainunmanaged-certs/: A directory for certs thatmarathon-acmedoesn’t managemarathon-acmedoes nothing with theunmanaged-certs/directory
after creating it. HAProxy fails if any path in its certificate config
doesn’t exist, so it reduces setup friction to have a standard place to
put unmanaged certificates.marathon-lbconfigurationmarathon-acmerequiresmarathon-lb1.4.0 or later in order to be
able to trigger HAProxy reloads.As mentioned earlier,marathon-lbmust share persistent storage withmarathon-acme. BYONS:bring your own networked storage.The only real configuration needed formarathon-lbis to add the
path tomarathon-acme’s certificate storage directory as a source of
certificates. HAProxy supports loading certificates from a directory.
You should setmarathon-lb’s--ssl-certsCLI option to the
certificate directory path as well as the fallback certificate (if
HAProxy cannot find any certificates in the paths it is given it will
fail to start).--ssl-certs <storage-dir>/certs,<storage-dir>/default.pemApp configurationmarathon-acmeuses a singlemarathon-lb-like label to assign
domains to app ports:MARATHON_ACME_{n}_DOMAIN, where{n}is the
port index. The value of the label is a set of comma- and/or
whitespace-separated domain names, althoughby default only the first
domain name will be considered.Currently,marathon-acmecan only issue certificates with a single
domain. This means multiple certificates need to be issued for apps with
multiple configured domains.A limitation was added that limits apps to a single domain. This limit
can be removed by passing the--allow-multiple-certscommand-line
option, although this is not recommended as it makes it possible for a
large number of certificates to be issued for a single app, potentially
exhausting the Let’s Encrypt rate limit.The app or its port must must be in the sameHAPROXY_GROUPasmarathon-acmewas configured with at start-up.We decided not to reuse theHAPROXY_{n}_VHOSTlabel so as to limit
the number of domains that certificates are issued for.LimitationsThe library used for ACME certificate management,txacme, is
currently quite limited in its functionality. The two biggest
limitations are:There is noSubject Alternative Name(SAN) support
yet (#37). Each
certificate will correspond to exactly one domain name. This limitation
makes it easier to hit Let’s Encrypt’s rate limits.There is no support forremovingcertificates fromtxacme’s
certificate store (#77).
Oncemarathon-acmeissues a certificate for an app it will try to renew
that certificateforeverunless it is manually deleted from the
certificate store.For a more complete list of issues, see the issues page for this repo.TroubleshootingChallenge ping endpointOne common problem is thatmarathon-lbis misconfigured and ACME
challenge requests are unable to reachmarathon-acme. You can test
challenge request routing tomarathon-acmeusing the challenge ping
endpoint.It should be possible to reach the/.well-known/acme-challenge/pingpath from all domains served bymarathon-lb:> $ curl cake-service.example.com/.well-known/acme-challenge/ping
{"message": "pong"}

> $ curl soda-service.example.com/.well-known/acme-challenge/ping
{"message": "pong"}