How To: Kubernetes Pods as Jenkins Build Agents
By: Luke Loresch
Tags: Jenkins, Kubernetes

Once installed, head over to Manage Jenkins -> Configure System and scroll down until you see the ‘Cloud’ heading and update the values to meet your configuration.

Required values

Kubernetes URL : URL of your Kubernetes API Server

Kubernetes Namespace : Namespace where your build pods will get created

: Namespace where your build pods will get created Jenkins URL : URL for your Jenkins master

: URL for your Jenkins master Container Cap : Max number of agent containers that Kubernetes is allowed to run from Jenkins

: Max number of agent containers that Kubernetes is allowed to run from Jenkins Pod Retention : Controls how pods are retained

: Controls how pods are retained Max connections to Kubernetes API : Number of simultaneous connections Jenkins is allowed to send to Kubernetes

Jenkins settings for Kuberntes Plugin

If you want to run this from a Jenkins instance running on your host, you can change Jenkins URL to http://<your_ip_address>:8080 with your machine’s IP address.

Using the Plugin

Create a new Pipeline Job by clicking on ‘New Item’ on the Jenkins home page and selecting the Pipeline type.

On the Configure screen, scroll down underneath the Pipeline settings, and for ‘Definition’ choose ‘Pipeline script from SCM’, which will clone your repo defined into the workspace when your build starts. This is the only setup your job will need as everything else is in your Jenkinsfile.

This plugin integrates very nicely with the Declarative Pipeline syntax, allowing you to define your agents pod spec from your project’s source control, as shown in the gist below.

How it works

Here’s the breakdown of what will happen when you click ‘Build Now’ on our pipeline, shown in the Jenkinsfile gist below.

Jenkins clones your project into the workspace, reads the Jenkinsfile to determine the agent type

A Kubernetes Pod with the spec defined in build-pod.yaml is created in the namespace defined in the general settings

is created in the namespace defined in the general settings Project is build using mvn clean install on our project which will create a jar file inside the target directory.

on our project which will create a jar file inside the directory. A Docker Image is produced by copying the jar file from the target directory

The Result

The first part of the build logs show that first the Jenkinsfile is loaded then any configured Jenkins Shared Libraries. Finally, your Kubernetes Pod spec gets sent to the Kubernetes API Server to get created.

Console Output from start of build with internal names removed

Jenkins Pipeline creating a Kubernetes Pod

After all the source code is loaded and the agent is created, your pipeline will begin executing any stages you defined, building our Java app with Maven and packaging it with Docker.

And there you have it, with only a little bit of setup we were able to run our Pipeline in a Kubernetes Pod.

Tips and Tricks

We’ve been using this plugin for a few months at Vivid Seats, and we’ve learned a few key things along the way. Take these tips with you for your setup:

Set resource requests and limits on each container in your Pod

Try to use provided images from dockerhub when appropriate, the less you have to maintain, the easier

Certain optimized images will not have tools like git or curl , so for troubleshooting you can use the jnlp container, which is a container created by default inside your Pod.

Immediate Impact

After adopting this new workflow, we’ve been able to reduce our fleet of traditional Jenkins agents from six large VMs to only two (5 executors on each with 32GB RAM). Maintenance has also become a pain-free experience. No longer do we need to patch and upgrade agents; we can simply change the docker image in the agent’s config. Another benefit we’ve seen is that developers are quickly able to test and create new products without having to spend hours carefully adding new software or updating dependencies on the agent machines.

Was this information helpful?

We hope sharing the details of how we have been using the Kubernetes plugin for Jenkins at Vivid Seats can help others along the way.

Additionally, we are actively hiring for several roles on our engineering team. If you’re interested to learn more about open positions, check out our careers page at vividseats.com/careers for more information.