Traefik 2 High Available Mode
By: Al-Waleed Shihadeh
Tags: Key Value Store, Reverse Proxy, Clustering, Docker, Traefik

My Expectations of the solutions

Based on the documentation for the beta version, I expected to have the same features as in the v1.7 and some more advanced features. Below is a summary of the features that I was looking for.

Support High available mode.

Store Traefik CLI commands in the key-value store and reads them when Traefik instances start.

Store LetsEncrypt SSL certificates in the key-value store instead of a file.

Reality && Limitation

It turns out that Traefik 2.x versions do not include all the above-mentioned features and some of these features were moved to the Traefik Enterprise Edition 😢. To illustrate more, Storing CLI commands into a key-value store was deprecated due to some bugs. While storing SSL certificates in a key-value store included only in Traefik Enterprise Edition.

With the above being said, High available mode only supports storing the following Traefik configurations categories

Middleware configurations.

Routers configurations (HTTP, TCP, and UDP).

Services configurations.

TLS configurations.

The full list of all configuration items that can be stored and managed by a key-value store is listed in the following link.

Hosting Traefik with Redis as a key-value store

Even though I was disappointed 😞 by the fact that not all the features that I need were included in the Traefik v 2.2, I decided to try running Traefik with a key-value store for the sake of evaluating the feature and checking if this feature adds value for hosting Traefik. In the next steps, I will describe my try for running Traefik with the Redis server. I chose Redis because it is easier for deployment, However, the below steps my apply for other key-value stores as well.

The First Step in my attempt to host Traefik with Redis is to deploy both Traefik and Redis and connect them together. Luckily these step is a straight forward task and can be achieved using the below docker stack file.

Connecting Traefik with Redis can be done using the below CLI commands

Once both services are up and Traefik can connect to Redis, we will be able to see the below log messages from Traefik logs.

The root cause of this error is that the Redis server does not store any configuration for Traefik.

The next step is to try to create add Traefik configuration to the Redis store and validate if Traefik loads them or not. Adding keys to the Redis server can be done using the “redis-cli” command as shown in the below image. You can either connect to a shell inside the Redis container or use the “redis-cli” command from a remote server.

Once, The above keys added to the Redis store, the middleware definitions appeared on the Traefik dashboard as shown in the below image.

Manually adding the keys using the Redis command-line tool is not convenient and will be a blocker for automating the deployment of the changes on docker services and their routes. Therefore, I start looking for a way to automate the creation of these keys. Below are the options that I thought of.

Create these key once Traefik is deployed: This idea requires deploying Traefik each time we have change on these configurations and also we need to update Redis docker image to include the Redis command line. Therefore, I dropped this method for the above reasons.

Create a service to create these keys: Building a customized service for managing the Traefik configurations could be a good idea, however it requires a lot of efforts and therefore I dropped this option too.

Let Redis bootstrap the keys: Since Redis Docker image has already the Redis command-line tools and it will be able to communicate with the Redis server using the loopback IP, I thought it will be easier to use a custom docker image for Redis with an entry-point the allows bootstrapping data to the Redis server.

To automate the creation and maintaining of the Redis keys, the customer docker image must support the following features

Add keys to the Redis server for a key-value file. for simplicity reasons, I decided that the format of the entries in the file should be in the following pattern.

key1,value1

key2,value2

Importing or setting keys in a Redis server from a file that follows the above patters can be done easily using a single bash command. The below function can do the job by reading the file content and parse it, then pass it to the Redis command line to create the keys.

Remove all items from Redis that are not in the key-value file. This is needed In case some of the defined routes or middlewares are removed and not need anymore. This task also feasible using bash and rails command-line tools, the below function achieve the task by comparing all the entries in the Redis server with the entries in the bootstrap file and delete each key that is not included in the file.

Updating the configuration keys in Redis requires a Redis deployment now for executing the above functions by the entry point 😢. To improve this, I decided to add a background process in the Redis docker image to keep checking for updates on the bootstrap file and load them in case of any changes. In addition, the bootstrap file will be added as a volume so it can be modified outside the Redis docker image or container. Below is the full implantations of the Redis Docker entry point.

The full implementation of the Traefik with Redis server can be found on Github on the following repository. to deploy the stack locally, clone the repository and execute the below command

DEPLOY_VERSION=ha make deploy

Conclusion

Traefik High Availability features such as storing SSL certs in a key-value store are available in the enterprise editions and not available in the open-source version. However, It is possible to store some of the Traefik configurations in a key-value store such as the middleware, services, and TCP configurations.