Configuration and Secret Management with Consul Template on Kubernetes
By: Onur Yilmaz
Tags: Platform, Consul, Vault, Kubernetes, Consul Template

As the platform team at Trendyol Tech, one of our goals was making application configuration and secret management dynamic, reliable, secure and be abstracted away from applications. We have been using Kubernetes ConfigMaps and Secrets for a long time and they don’t satisfy our requirements completely. For example; when we change a configuration on a ConfigMap, we need to restart our application pods for this change to take effect. Sometimes it takes several minutes for the change to be reflected on all application instances. Another problem is that Kubernetes secrets are not secure enough. We need our secrets to be secure and auditable.

Our main requirements were;

We should be able to change applications configurations dynamically without restarting application processes. When we change a configuration, it should effect all application instances instantly .

without restarting application processes. When we change a configuration, it should effect all application instances . Our application secrets must be secure and auditable. We also want to provide role based access control to secrets.

and We also want to provide to secrets. We need to abstract away configuration management from applications. In other words, applications should not know where and how secrets and configurations are kept and provided for them.

away configuration management from applications. In other words, applications should not know where and how secrets and configurations are kept and provided for them. Our configuration and secret management system must be high available and scalable.

What is Consul Template and why we chose it?

After some research, we came up with 2 applications; envconsul and consul-template.

We already knew that Vault is the industry standard tool for storing secrets. It satisfies all of our requirements and has features like dynamic secrets, which might be useful for us in the future. You can also use Consul as the storage backend for Vault.

Consul is a distributed, highly available and scalable tool for service discovery and configuration. In our case, we will only use the key-value store feature of Consul to keep non-secret configurations.

We could use envconsul or consul-template mainly for the 3rd requirement: abstracting away configuration management from applications.

Envconsul provides a way to launch a subprocess with environment variables populated from Consul and Vault. Consul-template populates values from Consul and Vault into the file system and keep the values sync while working in daemon mode.

A Twelve-Factor App rule is, store configurations in the environment. But the problem with Envconsul (or environment variables) is that you cannot change a process’ environment variables without restarting it. This goes against to our first requirement. We want to be able to change configurations without restarting application processes. Thats why we decided to go with consul-template.

Basic architecture of application process and consul-template daemon running together

In consul-template, you can use a template file for rendering configuration files. This gives us great flexibility. Here is a sample template file;

server:

port: "{{ key "apps/example/serverPort" }}"

toggles:

fooEnabled: {{ key "apps/example/fooEnabled" }}

database:

username: "{{ key "apps/example/dbUsername" }}"

{{ with secret "secret/apps/example" }}

password: "{{ .Data.data.dbPassword }}"

{{ end }}

Consul-template reads this template, replaces placeholders with values from Consul if you use “key” keyword and from Vault if you use “with secret” keyword. You can find out more about templating language here. The rendered output file would look like this;