The Evolution of CI/CD
By: Chris Cooney
Tags: Continuous Integration, Continuous Delivery, Agile, Software Development, Programming

The Evolution of CI/CD

Photo by Eugene Zhyvchik on Unsplash

Continuous Integration and Continuous Delivery have become a staple of any competent software house. They have become the lifeblood of the team and the engine that drives changes out into production.

Modern CI/CD solutions are all singing, all dancing masters of delivery. They come packed with useful, funky features. Yet, it’s easy to forget that the origins of CI/CD were very humble. This article is a high-level look at how CI/CD has changed over the years, to give a perspective of the direction of travel.

In the beginning…

In the early stages of continuous integration, build “pipelines” were nothing more than simple scripts.

#!/bin/bash unzip file.zip

cd file/

mvn test

mvn package

ftp # And so on...

It would be easy for me to bash these scripts (pun definitely intended). To point at them and laugh at how horrible it must have been to work with them. To wonder, how the engineers couldn’t possibly see the gaping issues with this approach.

I hate to disappoint, but I won’t be doing that. These scripts worked. They were written in well understood, simple languages. They ran on the tin that was always there so when they did run, they ran predictably. Not to mention that they were blazing fast. They had absolutely no overheads. No docker containers to spin up, no VMs to wait for. In short, they did the job.

So what happened?

The job changed. As more and more organizations started to realize the power of these scripts, a set of needs emerged. Needs that would have been difficult to fulfill with only the bash shell.

Visibility

As more and more of these scripts began to break halfway through, we needed some sense of where the breaks were. It wasn’t enough to dig through the bash stderr stream and try to work out where the fire was. The build needed feedback quickly.

Centralization

As these scripts were being made, a common problem occurred. Drift. Drift is what happens when ten engineers solve the same problem in ten different ways. The absence of any strategic solution creates a problem. Soon, you’ve got scripts in every corner of your office. Every single developer machine could potentially dismantle the production environment with a single command. A knife in every shadow. So there arose the need to bring all of these scripts into one place.

And the CI server was born…

CI servers have taken many different forms, with many different responsibilities. Tools like TeamCity and Hudson began to take over the horizon. The functionality that tools like these (and many more!) brought to the table was fantastic.

At the time, this felt like strapping rockets to your engineering practice.

CI servers have morphed and shifted with the times, introducing ideas like “Stages” into “Pipelines” that create a finer level of granularity in the software. For those of you who recognize the Hudson UI above and don’t know why — Hudson is now known by its more modern moniker, Jenkins.

Keeping these damn servers going

In the early days, the problem wasn’t with the features and functionality. It was in the build quality of some of these applications. The nature of their jobs meant holding onto resources for minutes at a time. As we know, computers sometimes like to cling onto resources… until there are no more left. This is known as a memory leak.

And by God, there were memory leaks all over these tools. It became entirely commonplace to restart Jenkins regularly. I’ve worked in offices where a nightly cron job would restart the Jenkins server every night because their Jenkins instance had a half-life of about 30 hours.

But this wasn’t the fault of the vendors

Jenkins (or Hudson for that matter) were not bad pieces of software. Again, they solved new problems that were being posed by the industry. Now, all of your builds ran from the same place.

Jenkins had (and still has) a plugin-based system, where open source tools can be installed onto your Jenkins server via the UI. This creates a highly extensible platform that can perform all manner of interesting tasks, but it also means there is a risk of clashing.

Anyone who has worked with Jenkins for more than a few months will have experienced this. Install Plugin A, then plugin B and suddenly, Plugin A is broken. So is plugin Z, which came with the Jenkins core.

So then we found some new problems to solve…

Spinning up Jenkins takes knowledge. Keeping it running takes a little care and attention. You need to have a rough idea of what you’re doing. It isn’t rocket science, but it isn’t a walk in the park either. It’s somewhere in between. Picture yourself riding a rocket through a park. It’s like that.

As the industry moved to small, lean companies that were looking to get products quickly to market, the tooling needed to evolve. The constraints in CI/CD became obvious:

The time it took to get a CI/CD server running.

The trial, error and agony involved in building those first CI pipelines.

Jenkins responded, with vigor.

The introduction of Jenkins pipelines in Jenkins 2 was a big step forward. Like many other CI solutions, the focus became defining the pipeline in code. This involved far less work in Jenkins’ clumsy UI and far more coding. Something engineers are more comfortable with.

So the first problem still existed, but the second problem was limited. Engineers would quickly iterate on code, rather than clicking around a complex, procedurally generated UI that could be very confusing for even the most simplistic pipelines.

Interestingly, Jenkins’ solution was for you to write your pipeline in a groovy script and have the Jenkins server execute it for you. This is very similar to the old method we discussed at the start of this article, isn’t it?

So what does the future look like?

In recent years, new offerings have come out. Tools that lift a lot of the burden off the engineer. They come in the form of a SaaS offering. CodePipeline, CircleCI or Buddy, to name a few. You follow some conventions, use a couple of yaml files and boom — you’ve got a CI pipeline.

This low barrier to entry is a huge selling point for smaller organizations that don’t want to take on the burden of running a CI server. This is why many organizations are moving closer to a Serverless and Managed service approach. Offloading the maintenance burden onto a third party for a fee is definitely an empowering choice for any company.

These offerings try to tackle the two more modern problems listed above. You don’t need to run your CI server any more, so that issue evaporates into thin air, but what about the ease with which you can build pipelines?

Configuration rather than code, it seems

The common approach to modern CI/CD solutions is to provide a configuration file inside your application repository. yaml appears to be the syntax of choice for these tools. This approach offers the lowest possible barrier to entry.

CI/CD is moving in a declarative direction and, with modern tools like Buddy, we can simply declare what we would like to happen and see it done.