Kubernetes: Change base YAML config for different environments prod/test using Kustomize
By: Kim Wuestkamp
Tags: Containers, Kustomize, Continuous Integration, Continuous Delivery, Kubernetes

Content

Part1: Merge multiple YAML files into one

Part 2 (this article): Change base YAML config for different environments

Install Kustomize

Make sure to have the latest kubectl installed because Kustomize is already included ( kubectl kustomize ). Otherwise install it manually.

What does Kustomize need server side?

Nothing. All Kustomize does is merge various YAML files into one. You can then kubectl apply -f merged.yaml this file. It doesn’t matter if you created the merged.yaml manually or using Kustomize.

Today’s Scenario

We will have three environments:

base : contains a set of YAML configuration on which other environments can be based on

: contains a set of YAML configuration on which other environments can be based on test : extends base, will change the image of the base deployment

: extends base, will change the image of the base deployment prod: extends base, will change image, service type, and adds a completely new resource

Github project

The whole example is finished in this repo: https://github.com/wuestkamp/kubernetes-kustomize-example

Base Env

Create a folder base containing three files:

deployment.yaml :

apiVersion: apps/v1

kind: Deployment

metadata:

name: app

spec:

replicas: 1

selector:

matchLabels:

id: app

template:

metadata:

labels:

id: app

spec:

containers:

- image: XXX # needs to be overridden by environments

name: nginx

service.yaml :

apiVersion: v1

kind: Service

metadata:

labels:

app: service

name: service

spec:

ports:

- name: 80-80

port: 80

protocol: TCP

targetPort: 80

nodePort: 30080

selector:

id: app

type: NodePort

kustomize.yaml :

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization



resources:

- deployment.yaml

- service.yaml

Now when we run kubectl kustomize build base , it will output a Multi Resource File containing both the Service and Deployment. Deploying this would horribly fail though, because there is a wrong image name set:

compiled base env yaml

Test Env

Create a new folder overlays . In it create a new subfolder test and two files:

It seems to be common to use a folder named overlays , but you can name and change the directory structure as you like.

deployment.yaml :

apiVersion: apps/v1

kind: Deployment

metadata:

name: app # necessary for Kustomize identification

spec:

template:

spec:

containers:

- image: nginx:1.17.3

name: nginx # necessary for Kustomize identification

kustomization.yaml :

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization



bases:

- ../../base # relative path to the base folder



patchesStrategicMerge:

- deployment.yaml

Running kubectl kustomize build overlays/test will now use all the base resources and replaces just the image name for the Deployment to nginx:1.17.3, which should result in this:

compiled test env yaml

Because we defined a patchesStrategicMerge entry for deployment, Kustomize will look for the file deployment.yaml and overrides specified values from the base deployment.yaml . The name attributes are important to specify for the to-be-overridden resource identification.

Prod Env

In production we would like to do even more. We would like to replace the image name, the replicas count, the service type and add a new resource. For this we create four files:

deployment.yaml :

apiVersion: apps/v1

kind: Deployment

metadata:

name: app

spec:

replicas: 3

template:

spec:

containers:

- image: nginx:stable

name: nginx

service.yaml :

apiVersion: v1

kind: Service

metadata:

name: service

spec:

type: LoadBalancer

storage.yaml:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: pvc

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Mi

kustomization.yaml :

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization



bases:

- ../../base



resources:

- storage.yaml # a new resource



patchesStrategicMerge:

- deployment.yaml

- service.yaml

Now running kubectl kustomize build overlays/prod will compile the production yaml, which should look like this:

compiled prod env yaml

Deployment

For deploying an environment configuration you can run:

kubectl kustomize build overlays/prod | kubectl apply -f -

Or

kubectl kustomize build overlays/prod > tmp_prod.yaml

kubectl apply -f tmp_prod.yaml

There is always more

We only used one patching strategy (patchesStrategicMerge), but there is also a JSON Patch one.

(Helm) Templating vs Kustomize

When using Helm for its templating engine, YAML files can become very ugly very soon, check out this example.

Using Kustomize it can be much easier to keep configuration clean, sure always depending on the use case.

The end.

Are you using Kustomize in production?

Become Kubernetes Certified