5 Tips for Hosting Your Own Jenkins Instance
By: Al-Waleed Shihadeh
Tags: Testing, Tdd, Jenkins, Jenkins Pipeline, Programming

1. Distributed Builds

Jenkins is designed to follow the Master-Agent architecture. The Master is responsible for coordinating the executions of the jobs and hosting the Jenkins UI and API server. On the other hand, the Agent (also called node or worker) is responsible for performing the actual work.

Despite the fact that Jenkins is a Master-Agent application, it can be hosted in a single server or a Linux machine (both agent and master run on the machine), and it’s commonly used that way, especially for small-scale projects or when starting using Jenkins for reasons of simplicity.

However, running Jenkins in a single server has several disadvantages and drawbacks:

Scalability : A single machine or server has a specific amount of resources. Adding more resources to the machine may improve the performance of the Jenkins cluster but it’s definitely not a solution to scaling the Jenkins cluster to handle hundreds of jobs, due to limits on the resources that can be attached to the server. On the other hand, scaling the Jenkins cluster by adding more workers/agents is much easier.

: A single machine or server has a specific amount of resources. Adding more resources to the machine may improve the performance of the Jenkins cluster but it’s definitely not a solution to scaling the Jenkins cluster to handle hundreds of jobs, due to limits on the resources that can be attached to the server. On the other hand, scaling the Jenkins cluster by adding more workers/agents is much easier. Maintainability : Usually, most of the jobs executed on Jenkins require special or specific third-party software. Installing all the needed packages on one node can complicate the installation setup. For instance, some libraries, packages, and software may have conflicts and can’t be installed together on the same machine. On the other hand, with the help of multiple agents setup, we can have different agents for different purposes. For example, we can dedicate one agent for compiling and building the source code for the Java programming language, another agent for building Docker images, and another one to build software for the macOS operating system.

: Usually, most of the jobs executed on Jenkins require special or specific third-party software. Installing all the needed packages on one node can complicate the installation setup. For instance, some libraries, packages, and software may have conflicts and can’t be installed together on the same machine. On the other hand, with the help of multiple agents setup, we can have different agents for different purposes. For example, we can dedicate one agent for compiling and building the source code for the Java programming language, another agent for building Docker images, and another one to build software for the macOS operating system. Security: A single node Jenkins cluster has some security risks, such as the risk of accessing production environments and testing environments from the same node. On the other hand, a Jenkins multi-node cluster improves the security of the Jenkins installation. Firstly, by separating the Master from the agents which perform the actual work (no direct access from master to agents). Secondly, by separating the tasks or jobs based on the environments. For instance, production jobs are executed on a dedicated agent where only production jobs can be executed.

As mentioned above, running Jenkins with multiple agents gives more advantages than running it in single-node mode. This Jenkins wiki page explains exactly how to configure Jenkins to support distributed builds and how to add agents to the Jenkins master in a traditional Jenkins installation (Installing Jenkins on multiple virtual machines).

Following the wiki page from Jenkins and adding more agents to the installation will improve it in several areas. However, you still need to maintain the agents, install the required packages and software on these agents, add labels to them, and monitor their health status. That seems like a lot of work!

That’s why I recommend configuring Jenkins to use dynamic agents instead of using static virtual machines. This idea can be implemented by using some of the plugins and extensions that support running Jenkins in cloud environments, For instance, Kubernetes-plugin is a Jenkins plugin that can be used to run dynamic agents in a Kubernetes cluster based on container images. Moreover, the Jenkins helm chart also supports Jenkins's dynamic agents.

In my opinion, the main advantage of dynamic agents is that they help to reduce the maintenance efforts that need to be performed on the agent nodes. To illustrate more, instead of updating a software package on all agents, the Docker template for the agent can be updated. In addition, supporting multiple platforms and OS is much easier with this approach.