An Introduction to CI, with Comparison of 17 Major CI Tools in 2020, and How to Choose the best CI
By: Tiexin Guo
Tags: Continuous Integration, DevOps, Software Engineering, Software Development

An Introduction to CI, with Comparison of 17 Major CI Tools in 2020, and How to Choose the best CI Tiexin Guo Follow Jun 5, 2020 · 11 min read

First of all, I want to start with an apology — for such a long title. I couldn’t do better (in fact I could, but it would leave important information out).

In this article, I would start with CI, with Jenkins, with what isn’t ideal with Jenkins in 2020, and then try to find out the best choice (for me and for you) by comparing 17 major CI tool-chains.

Let’s start.

Definition of CI

Continuous integration (CI) is the practice of merging all developers’ working copies to a shared mainline several times a day.

We will cover this again later, but it seems CI doesn’t really do much, it integrates and automates a series of tasks so that all devs work can be merged on a daily, if not hourly, basis.

What’s Wrong with Jenkins

This is the elephant in the room: you kind of have to talk about Jenkins when you are talking about CI.

A quick recap on the beginning part of the history of Jenkins for new devops guys:

Jenkins was originally developed as the Hudson project. Hudson’s creation started in summer of 2004 at Sun Microsystems. It was first released in java.net in Feb. 2005. Around 2007 Hudson became known as a better alternative to Cruise Control and other open-source build-servers. At the JavaOne conference in May 2008 the software won the Duke’s Choice Award in the Developer Solutions category. During November 2010, after the acquisition of Sun Microsystems by Oracle, an issue arose in the Hudson community with respect to the infrastructure used, which grew to encompass questions over the stewardship and control by Oracle. Negotiations between the principal project contributors and Oracle took place, and although there were many areas of agreement a key sticking point was the trademarked name “Hudson,” after Oracle claimed the right to the name and applied for a trademark in December 2010. As a result, on January 11, 2011, a call for votes was made to change the project name from “Hudson” to “Jenkins.” The proposal was overwhelmingly approved by community vote on January 29, 2011, creating the Jenkins project.

Why do we even bother to find which CI is best? Can’t we just use Jenkins, which has a long history, a big user base, and more importantly, maybe many of us have been already using it for years?

What is actually wrong with Jenkins so that you have to choose another (or at least try to find which could be better)?

Very good and very tough questions. But short answer is: there is nothing wrong with Jenkins.

In fact, there is nothing wrong with literally ANY CI.

What Does CI Do

CI is just a tool for executing some automated tasks, which can be integrated with source code management tools, triggering jobs automatically based on certain criteria, and execute some configurable commands.

Jenkins does and has been doing exactly that, perfectly, for over 15 years.

Yet, this might precisely be the problem:

At the age of Hudson, and the beginning years of Jenkins, many things had not been invented, or at least, widely used. For example, VMs, public cloud, containers, kubernetes, etc.

So, a tool which was designed many years ago, nowadays, in order to deal with all these new things we use, must rely on some add-ons and plug-ins.

As the administrator and user for Jenkins, you must have more knowledge about them, maybe writing a little bit more CI code than needed in order to make things work, maybe knowing exactly which plugin does what, how to exactly configure it so that it does what you need.

This, in my opinion, is what is “wrong” with Jenkins. Or at least, not ideal.

The Goal of CI

Photo by Andy Hu on Unsplash

Is to frequently integrate in an automated fashion.

If you spend a lot of time configuring the tool you use to achieve that, it doesn’t make much sense anymore. And this is where Jenkins starts (has already started long time ago) to show its disadvantages.

For example, in order to make Jenkins slave runs in kubernetes, you need the Kubernetes plugin for Jenkins, and the configuration of the job might look like this:

In short, in order to use both maven and golang, you must first define the containers in your pipeline code.

In this example, the definition of where the command would run takes almost the same number of lines of code as the pipeline itself. In most cases, if you are running a pod as a slave of Jenkins, you probably need a third container for JNLP.

Imagine you want to build a more complicated pipeline which has more steps and utilizes more tools hence more containers. Things could only get worse.

On Choosing the Next CI

Photo by Victoriano Izquierdo on Unsplash

We have the following things in mind when making decisions:

easy to install (if you are not gonna use the SAAS/PAAS solution)

easy to use

SAAS/PAAS solutions: you don’t even need to maintain it yourself if you choose to not to

install in cloud (in case you want to own your infrastructure but in the cloud)

install in k8s (in case you want to run your infra tool-chain in k8s)

install on-premise (since many big old companies must still have it on-premise)

open source (more community support, can fix bugs that block you by yourself)

I have put a few reasons already in the brackets behind each criterion, and there are more reasons behind this list:

As a tool, the ease of use is of top priority. Otherwise it wouldn’t increase your productivity

Also it must be easy to install. If installing the tool is very complicated, can’t be automated or hard to be done so, it is not very scalable into multiple projects and environments

For many small teams, they don’t want to maintain their CI at all, they want to use some existing cloud service for that

For larger companies, they might want to or must have it under their own control, be it in the cloud or on-premise

K8s is so hot that you also might want your CI tool-chain to run as container in k8s

You get more community support or even do it your own if it’s open source.

Now we are going to consider some options, and we will mainly focus on the “what’s not so good” part of each.

Candidates

1 Jenkins

We have talked about Jenkins already, mostly disadvantages, but still, it is still a very strong contender.

It’s free, open source, many people like it (15k stars on GitHub), many people use it (if you search Jenkins in stack overflow, you find more than 400k questions).

2 CircleCI

Relatively cheap, with free plans, easy to use (config based on YAML, although not everyone likes YAML) and read.

Reasons to NOT use it:

limited enterprise deployment options: only supports in AWS with terraform

not easy to deploy locally if you want to have a try

no option to deploy in k8s

core components written in clojure, which is cool, but relatively not widely used, so if you don’t know it, you can’t contribute; also clojure is slower than Java

3 TeamCity

Free; written in Java.

If you want to run it as a container in k8s, the image is big. In general, java applications don’t suit containerization because of the JRE/JVM.

4 Bamboo

Similar to TeamCity, coming from a big well-known company but written in Java.

5 GitLab CI

Support creating runners in k8s which is nice.

But if you are using other source code management tools it doesn’t make any sense to switch only for the CI.

Personal suggest is, of course, to use it only if you are already using GitLab CI.

6 Buddy

For on-prem install, you pay for number of users, which can be very expensive.

Given there are many opensource free options, there isn’t a very strong reason for this.

7 TravisCI

Written in Ruby, not as performant as Java.

Limited on-premise/cloud deploy options.

8 Codeship

Not open source, can’t be deployed in your cloud or on-premise.

9 GoCD

Even though the name suggests it might be written in Golang, it’s not. It’s actually in Java and Ruby.

It’s open source and free, can be deployed in k8s.

Only reason to not consider is, similar as mentioned above for TeamCity, Java apps doesn’t make much sense as others if you want to deploy it in k8s.

10 Wercker

Oracle buys it. Probably not much reason to consider.

Good thing is, it’s written in Golang; bad thing is, compared to other Golang opensource options, it has much less stars so I guess much less user base as well.

11 Drone

Cloud, on-prem, in k8s deploy all supported.

Small images, written in go, suitable to run in k8s.

More than 20k stars, open source.

Good integration with k8s, for example, secrets used in the CI jobs can be stored as a secret in k8s.

No obvious downside. If anything, examples/official doc isn’t perfect. This would be my personal choice.

12 Semaphore

Not open source, till this day, no on-prem deployment option.

13 Nevercode

No on-prem, not opensource.

14 Spinnaker

This one seems hard to fault, created by Netflix, open source, multi-cloud.

However, to do a trail-run deploy locally in your k8s, it needs to pull many images, very slow to setup, and it’s in Java.

It seems also they are not quite confident about their code. Because you would guess the https://github.com/spinnaker/spinnaker should be the main repo but it’s not. You need to dig a lot in order to find the core code.

15 Buildbot

Written in python, easy to read, less performant than Java/Golang, less stars than other options, also the UI https://buildbot.net/ isn’t so nice on their homepage.

I personally wouldn’t trust people who don’t pay enough attention to their face.

16 Bitrise

Seems to be designed specifically for mobile apps; might not suit for general usage. Not open source.

17 CodePipeline/CodeBuild from AWS

I personally won’t use, just in case the project will migrate to other cloud providers. In a fast-changing world, stay non-vendor-lock-in isn’t the worst idea.

18 Others

It is for sure that there are way more than 17 CI tools out there in the market as of year 2020, and I’m sure I might have missed a few famous ones, like Google CloudBuild, and it’s not quite possible to cover every single one of them, but you get the idea: based on the criteria we listed in the beginning, it’s easy to reach to the conclusion all by yourself with minimum amount of testing for any CI tool-chain.

Conclusion

Here is the tricky part, because people have ideas, and everyone has a different opinion. There definitely isn’t a one-size-fits-all solution; not to mention, people have different taste.

Nonetheless, I will still try to sum up a conclusion here:

For many people, CircleCI or TravisCI is good enough, and the drawbacks aren’t that huge anyway

For some who are in pursuit of agile and quick, drone.io might win the competition, because of its easiness to install, config, use, its flexibility to deploy on-prem or use in the cloud to fit a wider audience, its open-source with many likes on github and good community support, and its cloud-native/12-factor-app properties: small image, fast to setup, written in Golang, more performant than some options like Ruby/Python.

Still, for a lot of even more people, Jenkins is not a bad choice, because it’s very flexible which kind of compensates the relatively complication, on the matter of configuration. More importantly, it seems everyone is trying to do a CI tool to make some moeny, but although Jenkins is in Java, which is not ideal to run in k8s, it’s still not worse compared to other options.

If you have a solid pain point with Jenkins, maybe give travis/circle/drone a try.

I know I have talked maybe too little on some of the options, however, in my defense, for some options, just one reason, can be as simple as just two words, are more than enough to deny it, because after all, we don’t have the energy to give every single product a detailed trail run.

I also know I have talked a lot about drone/travis/circle, maybe a bit too much than necessary so that you might think these options sponsored this article, but trust me, this isn’t a sponsored article; I reached my conclusion above all by my personal, unbiased, professional research and review. On the other hand, drone/travis/circle, if you are watching this and are happy to pay me some sponsorship, I’d be open for some extra breakfast money to talk more about the products that I like.

Give the article a like, comment, subscribe, and share, if you like it and want to read more quality articles on devops / cloud topics in the future; thanks a lot!