Correct Place to run your Database Migrations in Kubernetes!!
By: Abhishek Chandel
Tags: Kubernetes Job, Database Migration, Kubernetes, Migration

What are Database Migrations??

In a code-first approach of development, we keep all our database changes in the form of code, these scripts are called database migrations. They help us rollback changes rather “easily” as the script that has run, will always have its footprint logged in the code. I just came up with this definition, I don’t know if there’s a official one.

The Problem

Migrations are required to run just once, before the code deployment, so that code changes work properly.

Where to run them??

Database migrations can vary in complexity, they can be small scripts such as just adding a column to updating millions of rows using a custom logic. And often, these migrations are needed for the code shipped in the same build to work. So our migrations must run before the code deployment.

initContainers

What is it?

Specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.

Let’s discuss some pros and cons of this approach.

Pros:

Easy to setup, just write few lines of configurations like below

apiVersion: v1

kind: Pod

metadata:

name: myapp-pod

labels:

app: myapp

spec:

containers:

- name: myapp-container

image: busybox:1.28

command: ['sh', '-c', 'echo The app is running! && sleep 3600']

initContainers:

- name: init-mydb

image: busybox:1.28

command: <command to run your migration>

2. Due to the way it works it does not need an extra stage in the CI/CD pipeline.

Cons:

It is not meant for long running script as Kubernetes poses certain time limitation within which it has to complete, if it fails to do so, kubelet will restart the script again and again, and your actual container will never start. Basically you are stuck.

Each init container must exit successfully before the next container starts. If a container fails to start due to the runtime or exits with failure, it is retried according to the Pod restartPolicy . However, if the Pod restartPolicy is set to Always, the init containers use restartPolicy OnFailure.

2. Pod scaling will take longer, as it has to connect to database and run script.

3. It will run for each pod restart, which is waste of resources.

Kubernetes Jobs

What is it??

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created.

Kubernetes jobs addresses all of the cons that initContainers has. Pod-scaling is faster, You can run long running scripts. Although it will take an extra stage in CI/CD and a bit involved setup than initContainers. However, this decouples your database migrations from code deployment. Now other builds can be deployed if long running migration is executing, which was not the case previously.

Example:

apiVersion: batch/v1

kind: Job

metadata:

name: pi

spec:

template:

spec:

containers:

- name: pi

image: perl

command: <database migration command>

restartPolicy: Never

backoffLimit: 4

and above can be applied using:

kubectl apply -f https://kubernetes.io/examples/controllers/job.yaml

If you are using helm and want to use all those db-secrets and pull secrets that you have for your deployment, then you can use helm to deploy your kubernetes jobs as well using:

helm upgrade --install --wait --namespace ${K8_NAMESPACE} ${HELM_OPTS} ${HELM_RELEASE_NAME} ${HELM_PATH}

Conclusion

Running migrations using Kubernetes jobs has worked really well for us. This provides flexibility to execute long running database scripts, without blocking your CI/CD pipeline.

References