Implementing gitops the ansible way
By: Osvaldo Toja
Tags: Ansible, Rancher, Gitops

Treating clusters as hosts

Introducing an ansible inventory plugin to manage imported kubernetes clusters in Rancher as hosts.

Context

These days container images seem to be the deployment artifact of choice. Everything runs from an image, hey, even lambda functions!.

Your app might be composed of a single image, so no need to orchestrate containers, right? Things a bit more complicated in real life. Chances are your app will need to register a dns name, get an https certificate, setup an entry point (aka load balancers).

All of the above can be provided by apps, running on … wait of it .. containers. Bottom line is, even when deploying a single pod application, you can still benefit from using a container orchestration tool to manage the lifecycle of your app. Leveraging existing apps to do the job for you, let’s call them supporting apps.

Kubernetes seems to be the winner from the container orchestration platforms war. Your team decides to use a hosted kubernetes offering, after all, you’re in the business of using kubernetes. You care about managing/maintaining your app, not kubernetes itself (the control plane it is).

Let’s say your application will be deployed to multiple clusters, whom might be on different clouds, or maybe a single one, like AWS, but on different accounts. Properly managing access credentials can quickly become a problem. Luckily for us there’re projects like Rancher to help with that.

Using rancher allows you to manage your clusters from a single place. It provides you with access to clusters using rancher provided credentials. Last but not least, it allows you to deploy apps, like the ones we mentioned earlier for supporting your application.

Using rancher catalogs (using upstream helm charts) will allow you to easily deploy apps to your clusters. For a unified environment (all clusters being on the same cloud, sharing the same configuration) that’ll get the job done.

However, once you need to support different configurations between clusters, it might not be simple enough.

Gitops has become the standard way for deploying supporting apps to k8s clusters. It also solves the problem of managing differences between clusters in a flexible way. Several tools are available: Flux, Argo CD, Rancher 2.5 introduces Fleet.

So we’re good, right? Use a gitops tool and called it a day. Well, not so fast my friend.

Problem

Supporting apps take care of tasks like sending logs to a provider, manage load balancers from a cloud providing, DNS records. Those actions have one thing in common: they all require credentials (well, they should).

There’re ways for managing credentials when using gitops tools: SOPS, Sealed secrets and Hashicorp Vault, among many others.

The goal is to deploy a new cluster, import it into rancher server and deploy all required supporting apps.

Using sealed secrets requires extra steps: deploying the controller, managing certificates (creation, distribution, rotation). All of which is ok, but goes beyond the original intention of removing complexity from the workflow, not adding it.

Since we’re using Rancher, fleet seemed the logical choice for implementing gitops. Using Hashicorp Vault on kubernetes can easily be achieved.

Fleet uses tekton pipelines under the hood, injecting secrets from vault could be possible but it will require modifying fleet manifests.

Time being a constraint for finding a solution, we start considering other approaches.

At the same time, something else caught our attention. It’s not only about credentials, configuration data also needs to be taking into consideration. Datadog requires an API token, but for external-dns, you might need an IAM role, which is not a secret per se, but will probably come from your configuration management tool of choice. For our particular use case, we use ansible.

Redefining the problem: we have configuration data and secrets provided by ansible and we want to deploy apps using both to kubernetes clusters, imported in rancher.

Solution

What if instead of deploying a new tool, like gitops, we reuse an existing one, like ansible?

For teams starting down the kubernetes road, with ansible knowledge, this might make sense.

Disclaimer: this probably will not scale for a large number of clusters. But that’s not a common scenario when starting from scratch. By the time you reach that large number, the team will have the expertise to came up with newer, better fitted solutions.

One of the benefits of ansible relies on its easy of use. You define an inventory of hosts, some configuration, write a playbook and/or roles to execute tasks against hosts, and that’s it.

The question was, is it possible for ansible to interact with kubernetes clusters as hosts? The answer is yes.

Ansible has support for dynamic inventories.

The solution was to write an rancher inventory plugin.

Ansible Inventory Plugin

The plugin provides a list of clusters from a rancher server as an ansible inventory.

Rancher clusters become ansible hosts. Ansible groups are dynamically created based on a set of labels provided by the inventory plugin configuration file.

Ansible Inventory Group/Host variables

$ tree .

.

├── group_vars

│ ├── all.yml

│ ├── aws.yml

│ ├── azure.yml

│ ├── production.yml

│ └── staging.yml

├── host_vars

│ └── demo1.yml

└── rancher.yml

rancher.yml: The inventory plugin configuration file. Its presence enable the plugin.

group_vars: Values are automatically assigned based on group membership for clusters.

host_vars: To override and provide custom variables for specific clusters.

Cluster Labels

The inventory plugin relies on labels assigned to the imported cluster.

Example labels:

environment_stage=production

k8s_distro=eks

cloud_provider=aws

ansible_managed=true

Only clusters with ansible_managed=true label will be added to the inventory, otherwise they will ignored. This is nice safety measure when working with an existing production rancher server.

The configuration file for the inventory plugin provides the list of labels to be used for creating host groups in the inventory.

The values of the labels will be used to create host groups, but the one defined as managed_label, from the inventory plugin file rancher.yml :

labels_groups: [‘environment_stage’, ‘cloud_provider’, ‘ansible_managed’]

managed_label: ‘ansible_managed’

Demo

A fully working demo is available on the repository: https://github.com/osvaldotoja/ansible-rancher-inventory/

It contains an installation script infra.sh and a configuration file clusters.json . The Makefile simplifies execution of the workflow as detailed on the README file.

The script uses docker to deploy an instance of a rancher server, alongside a number of k3s clusters, using k3d as defined in clusters.json .

Conclusion

For small to medium teams, already using ansible, starting up with kubernetes, with more work to do than time to learn new technologies, reusing what’s already available do provides benefits.

Deploying a change to all clusters in azure might not be simple to explain to someone with no previous knowledge of kustomize (or similar gitops tool in use) but the following is intuitive enough for anyone familiar with ansible:

ansible-playbook deploy.yml — limit azure

Mastering kustomize is not a single day task, but if running ansible playbooks is something the team knows about, that’s good enough for starters.