Why your Kubernetes configuration strategy is broken…
By: Alessandro De Maria
Tags: Kustomize, Helm, Microservices, Kubernetes, DevOps

...and here’s how to fix it

At kapitan.dev we believe the current way to manage Kubernetes configurations is broken. Actually, it probably goes even deeper than that, and you will see soon why.

I have a strange way to look at Kubernetes: for me Kubernetes is something that allows me to define, package and distribute complex applications with a set of configuration files. Meaning, I can define every aspect of the deployment of a complex multi-components application with Kubernetes resources: the services that make the application, configuration files, network policies, services, load balancers, RBAC, monitoring, alerting, auth.

Everything can be captured, packaged and distributed using declarative Kubernetes resource definitions.

And yet, when we use tools like helm and kustomize, we tend to focus on one specific component at the time, effectively losing the big picture of what Kubernetes really is all about.

To draw a parallel with “old” tech, we are ~6 years into the age of Kubernetes, which completely disrupted the way we deploy our services, and yet we are still at the “rpm” or “deb” stage. To be fair, at least helm gets closer to a “yum” or “apt-get”, but doesn’t go much further than that.

Let me give you some examples of where these traditional approaches fall short.

Example 1: Adding a new component to your infrastructure

Imagine that you just created a new component/service and you want to deploy to your infrastructure. Fine, get your helm/kustomize configuration and deploy it, right? Pronto! Presto!

But here is the “expectation vs reality” moment: Life has a way to get to you, and for the new component to work, you also need to:

Add an env variable to another service

Add a new route to your ingress

Add a new annotation to all other services you have

Create a new DB username/password for the new service to use

Create a new network policy associated with the service

Add a CD step to deploy the new component

Please take a second to let it sink in, and answer these questions:

How many steps, tools and pull requests you will have deal with in order to fulfil a “business as usual” operation.

Could anyone in your company/team fulfil this request?

If the answer is not “1, 1, 1” and “yes” please keep reading.

Example 2: Enabling a feature flag

This other example is nothing different from the previous one, just something that is expected to happen more often.

Let’s imagine you have worked weeks cross-teams to define a new behaviour for you application, and it’s behind a feature flag: feature flags to be precise.

Because you have fully embraced microservices like a boss, you need to do the following to enable the new behaviour, which we should call “holiday sales reporting”

set the FLAG_HOLIDAY_SALES_REPORTING=true on fronted component

on component add the --enable-json-output on backend component

on component point the /report route to a new service

Same questions as before really, how easy is it for you to achieve this with your current setup? How much coordination is needed?

Now a bonus question: what if, after the holidays, you need to turn off this flag? How many steps will you have to go through? And what if, in the meanwhile, --enable-json-output has actually been required also by another feature?

And what if you have only added this feature to a couple of environments? How do you document that the feature is enabled?

Solving it with Kapitan

When you use kapitan , you don’t just capture the configuration of one single component, but rather the full configuration of everything else it is needed to run the whole application: Kubernetes resources, Terraform files, documentation, scripts, secrets.

The typical setup uses a “target file” to capture “at least” everything that you would normally put in one namespace, but you can easily track resources that need to be created in other namespaces (i.e. istio-system)

A typical target file (i.e. targets/production.yml ) would look like this:

classes:

- common

- profile.production

- location.europe

- release.production - component.frontend

- component.backend

- component.mysql parameters:

description: "Production environment"

Head over to our repository https://github.com/kapicorp/kapitan-reference for a more complete example i.e. Weaveworks “sock shop”

A class (i.e. component.frontend ) points to a file on the local disk, so you would expect to find a file inventory/classes/component/frontend.yml to capture the configuration needed for the frontend component, which would look like this:

parameters:

components:

frontend:

image: company/frontend:${release}

port:

http:

service_port: 80



ingresses:

global:

paths:

- backend:

serviceName: frontend

servicePort: 80

path: /web/*

Head over to our repository https://github.com/kapicorp/kapitan-reference for a more complete example.

Solving Example 1: Adding a new component to your infrastructure

With kapitan adding a new component consists on creating a new class to define that component: inventory/classes/components/report.yml

parameters:

component:

users:

# define new users

report:

username: report

password: ?{gkms:${target_name}/report||randomstr|base64} # definition of the component itself

report:

image: company/report:${release}

env:

MYSQL_USERNAME: ${users:report:username}

MYSQL_PASSWORD:

secretKeyRef:

key: mysql_password # Create a new Secret resource

secrets:

secret:

data:

mysql_password:

value: ${users:report:password} # Create new network policy

network_policies:

default:

ingress:

- from:

- podSelector:

matchLabels:

role: frontend

ports:

- protocol: TCP

port: 80 # <add more report component configurations here>

frontend:

env:

REPORT_SERVICE: # Add a new env variable in the fronted componentfrontend:env:REPORT_SERVICE: http://report:80 # Add a new ingress

ingresses:

global:

paths:

- backend:

serviceName: report

servicePort: 80

path: /report/* # Add a new annotation to all components in this target

generator:

manifest:

default_config:

annotations:

company.com/report: active

Notice how this one file captures everything you need to do to configure the new report service in your setup.

When you add the component.report class to a target file (or to another class file, i.e. application.website ), Kapitan will take care of configuring everything that is needed for the component to work in one go.

class to a target file (or to another class file, i.e. ), Kapitan will take care of configuring everything that is needed for the component to work in one go. As you might have guesses, when you remove it, all that extra configuration goes away.

Secrets are automatically generated for you (and in the example, encrypted using Google KMS

If you have a Kapitan integration with your CD software (i.e. the yet-to-be-released spinnaker integration), your CD pipelines will also be modified to include the new component

Solving Example 2: Enabling a feature flag

The solution here is pretty similar in principle to the Example 1: you create a class and you add it to your target.

The name that class: features/holiday_report.yml

parameters: frontend:

env:

FLAG_HOLIDAY_SALES_REPORTING: 'true' backend:

args:

- --enable-json-output # Add a new path to the ingress

ingresses:

global:

paths:

- backend:

serviceName: report

servicePort: 80

path: /holiday_report/*

Now when you want to enable/disable this feature, all you need to do it to add/remove the class from the target file:

classes:

- common

- profile.production

- location.europe

- release.production - component.frontend

- component.backend

- component.report

- component.mysql - features. holiday_report parameters:

description: "Production environment"

Notice how it becomes much cleaner and easy to understand which features are enabled where.

If you have docs that are automatically generated by Kapitan, you could add information related to the holiday_report features only to the affected targets.

Final words

I hope you have enjoyed this article, and you have been able to understand what Kapitan can offer. If you want to learn more, please check out our blog and our website https://kapitan.dev