DevOps: CI/CD Trade Study For The Top 15 Platforms
By: Andrew Kirima
Tags: DevOps, Continous Integration, Continous Delivery, Technology, Software Development

Trade Study Objective

My objective, on behalf of my work as Head of Product at Sencha Credit, is to implement an automated CI/CD Pipeline to our DevOps tool stack, so that we can perform product deployments in the most optimal time possible. As a startup, we are looking for a free or cheap solution for mobile app development. The app being developed is for fintech and is built with React-Native. Also, the pipeline chosen needs to be compatible with iOS and Android deployment simultaneously.

Trade Factors

Pricing: What’s the cost model? Depending on the number of builds you will be doing certain tools will charge you differently. On larger projects, you may end up running builds many times a day, which is important to be aware of as you select your CI/CD tool because some vendors will charge based on this. Do you need to be building concurrently? If you have more than 1 team in the development, then they may all want access to the build toolchain at the same time. This means you may need more infrastructure or more resources from a SaaS vendor. How many users do you need? Many vendors will charge you more the more users you have. Is there a free tier if you don’t have a budget for this? Reliability: Is your tool Open-Source or a Proprietary tool? Open source is free to use but if it’s not a popular one then it could lose support. However, a proprietary tool has constant support, but usually it will cost more. How well does it work? What does everyone else say about it? Usability & Learning Curve: How easy is it to use and if you have questions is there a big enough community so you can get support? How helpful are they? Flexibility: Does it integrate well with our existing setup? How easy is it to integrate? What other features does it provide? What can it deploy to? Cloud, On-Premises, or Saas? Security: How secure is it? How often does its security fail? If you’re using a tool that runs on a virtual infrastructure, then you need to consider the data and access keys. It may need access to the SSH keys in the coding repository and credentials from the Github repo.

Photo by Yancy Min on Unsplash

CI/CD Platforms

Here is a quick summary of the key things I picked up from my study about each CI/CD platform.

Jenkins

Pricing: Free since it is open source.

Free since it is open source. Reliability: The leading cross-platform open-source CI/CD tool that is java based. Preferable for startups as well as large-scale organizations. Available for Windows, Linux, and macOS platforms.

The leading cross-platform open-source CI/CD tool that is java based. Preferable for startups as well as large-scale organizations. Available for Windows, Linux, and macOS platforms. Usability & Learning Curve: Easy to install and upgrade. Simple UI. You need a server to host it. Maintaining and updating is your responsibility. It has a good ecosystem (1500+ plugins) and a good community surrounding it.

Easy to install and upgrade. Simple UI. You need a server to host it. Maintaining and updating is your responsibility. It has a good ecosystem (1500+ plugins) and a good community surrounding it. Flexibility: Integrates well with AWS, Google Cloud, Azure, Digital Ocean, etc. Can be leveraged for performing work in parallel and realizing complex CD requirements.

Integrates well with AWS, Google Cloud, Azure, Digital Ocean, etc. Can be leveraged for performing work in parallel and realizing complex CD requirements. Security: Even if you run Jenkins on a private network and trust everyone in your team, security issues in Jenkins can still impact you. You have to manage the security manually, so it’s important to keep in mind where you will be hosting the codebase.

IBM UrbanCode

Pricing: Must contact for pricing.

Must contact for pricing. Reliability: One of the best-suited tools for large enterprises for complex builds.

One of the best-suited tools for large enterprises for complex builds. Usability & Learning Curve: Provides project templates that make it easy to manage different configurations. Also, support from a large enterprise community.

Provides project templates that make it easy to manage different configurations. Also, support from a large enterprise community. Flexibility: Integration shouldn’t be too difficult. Had drag and drop automation. Can increase the frequency of software delivery by automated, repeatable deployment processes.

Integration shouldn’t be too difficult. Had drag and drop automation. Can increase the frequency of software delivery by automated, repeatable deployment processes. Security: Enterprise-level of security and scalability.

AWS CodeBuild

Pricing: Has a wide variety of prices based upon build per minute, with a free tier for 100 build minutes of build.general1.small per month. 1 free year provided by the AWS Free Tier term.

Has a wide variety of prices based upon build per minute, with a free tier for 100 build minutes of build.general1.small per month. 1 free year provided by the AWS Free Tier term. Reliability: Holds support for Java, Python, Node.js, Ruby, Go, Android, etc.

Holds support for Java, Python, Node.js, Ruby, Go, Android, etc. Usability & Learning Curve: Supported by a large enterprise so you have access to nearly unlimited documentation and help to answer questions.

Supported by a large enterprise so you have access to nearly unlimited documentation and help to answer questions. Flexibility: Built into AWS projects. Can be pre-configured and customizable.

Built into AWS projects. Can be pre-configured and customizable. Security: Has artifact encryption through integration with the AWS Key Management Service. Built-in the ecosystem of the top 3 largest cloud providers.

Azure DevOps

Pricing: Free for open source and small projects (up to five users). For larger teams, the cost ranges from $30 per month (10 users) to $6,150 per month (1,000 users).

Free for open source and small projects (up to five users). For larger teams, the cost ranges from $30 per month (10 users) to $6,150 per month (1,000 users). Reliability: Has support for a wide variety of version control systems, programming languages, and deployment targets.

Has support for a wide variety of version control systems, programming languages, and deployment targets. Usability & Learning Curve: Tailored to use with Visual Studio. Easy to use, with a wide enterprise for support and near-limitless documentation.

Tailored to use with Visual Studio. Easy to use, with a wide enterprise for support and near-limitless documentation. Flexibility: Can host build agents using Docker. Very flexible. Capable of working with any development stack beyond Microsoft technologies.

Can host build agents using Docker. Very flexible. Capable of working with any development stack beyond Microsoft technologies. Security: With a large enterprise of cloud space, it is known to be one of the safest. Chosen by Smithfield and Merrill Corporation due to its platform security.

TeamCity

Pricing: Free to set up for small teams and open source projects. 100 builds for free. $299 for Build Agent License.

Free to set up for small teams and open source projects. 100 builds for free. $299 for Build Agent License. Reliability: Server-based pipeline tool in Java that was developed by JetBrains, one of the most accomplished companies in developing IDEs, so it has a reliable network and interface.

Server-based pipeline tool in Java that was developed by JetBrains, one of the most accomplished companies in developing IDEs, so it has a reliable network and interface. Usability & Learning Curve: Trivial to setup. Integrates well with Jira Software Cloud and Azure DevOps. Has a large community for support as well as provides good documentation for any issues you may have.

Trivial to setup. Integrates well with Jira Software Cloud and Azure DevOps. Has a large community for support as well as provides good documentation for any issues you may have. Flexibility: Can integrate well with GitHub, GitHub Enterprise, and BitBucket. Can be hosted on a cloud. Fast testing that is automated in parallel, and customizable.

Can integrate well with GitHub, GitHub Enterprise, and BitBucket. Can be hosted on a cloud. Fast testing that is automated in parallel, and customizable. Security: Can run on a private infrastructure firewall.

CircleCI

Pricing: Has a freemium model for Linux and Windows where you can run one job at a time with no charge. Open source projects have a limited number of containers. 2,500 free credits/week. $30 per month and has an option for large organizations.

Has a freemium model for Linux and Windows where you can run one job at a time with no charge. Open source projects have a limited number of containers. 2,500 free credits/week. $30 per month and has an option for large organizations. Reliability: One of the best tools for open-source and large-scale projects. It supports languages that can be built on Windows, Linux, and macOS platforms. Recognized by Forrester Wave as a leader in Cloud Native continuous integration (CI).

One of the best tools for open-source and large-scale projects. It supports languages that can be built on Windows, Linux, and macOS platforms. Recognized by Forrester Wave as a leader in Cloud Native continuous integration (CI). Usability & Learning Curve: Easy to set up with its proprietary YAML syntax for its pipelines. Has a large community for support.

Easy to set up with its proprietary YAML syntax for its pipelines. Has a large community for support. Flexibility: Offers a modern software development architecture that lets teams ramp up their development operations. Good for scaling. Offers a cloud-based and on-premises solution.

Offers a modern software development architecture that lets teams ramp up their development operations. Good for scaling. Offers a cloud-based and on-premises solution. Security: All data is encrypted with TLS and SSH such as environment variables like keys, tokens, and other credentials.

Travis CI

Pricing: Free for open-source projects and if you have a CircleCI Cloud account, with up to 1000 build minutes per month, 1 container, and 1 concurrent job.

Free for open-source projects and if you have a CircleCI Cloud account, with up to 1000 build minutes per month, 1 container, and 1 concurrent job. Reliability: Written in the Ruby programming language and known to be one of the best for open-source projects & enterprise-level projects. Can host on their cloud variant for open-source projects and enterprises with small team sizes.

Written in the Ruby programming language and known to be one of the best for open-source projects & enterprise-level projects. Can host on their cloud variant for open-source projects and enterprises with small team sizes. Usability & Learning Curve: Large community with plenty of resources. Known for ease of use.

Large community with plenty of resources. Known for ease of use. Flexibility: Supports a whole range of programming languages. Can integrate with Lambda Test to perform testing on browsers, platforms, and devices(emulators). Incorporates a build matrix for seamless use.

Supports a whole range of programming languages. Can integrate with Lambda Test to perform testing on browsers, platforms, and devices(emulators). Incorporates a build matrix for seamless use. Security: For more security, you can opt for their enterprise package.

GoCD

Pricing: Free since it’s open source.

Free since it’s open source. Reliability: One of the best CI/CD tools, it can work with Windows, macOS, Docker, etc.

One of the best CI/CD tools, it can work with Windows, macOS, Docker, etc. Usability & Learning Curve: Has an active community in their Google Group with adequate documentation.

Has an active community in their Google Group with adequate documentation. Flexibility: GoCD streamlines your CD workflow on popular cloud environments such as Kubernetes, Docker, AWS, and more. It has a virtual stream map (VSM) to view pipelines graphically.

GoCD streamlines your CD workflow on popular cloud environments such as Kubernetes, Docker, AWS, and more. It has a virtual stream map (VSM) to view pipelines graphically. Security: Has 2 methods of authentication built into it, to mitigate security risks.

CodeShip

Pricing: Freemium pricing model where 100 builds per month is free for unlimited projects. Unlimited builds for $49 a month.

Freemium pricing model where 100 builds per month is free for unlimited projects. Unlimited builds for $49 a month. Reliability: A hosted continuous delivery service that focuses on speed, reliability, and simplicit y. You can set up teams and assign permissions with the teams to specific members.

A hosted continuous delivery service that focuses on speed, reliability, and simplicit You can set up teams and assign permissions with the teams to specific members. Usability & Learning Curve: Known to be super easy to use. It will push out builds & deployments within a few minutes.

Known to be super easy to use. It will push out builds & deployments within a few minutes. Flexibility: Goes well with GitHub version control systems. It can directly test, build, and deploy from GitHub projects. Has a ton of features especially with multi-stage builds. Also, will help reduce the image size of Docker images.

Goes well with GitHub version control systems. It can directly test, build, and deploy from GitHub projects. Has a ton of features especially with multi-stage builds. Also, will help reduce the image size of Docker images. Security: It is not installed on your VPC instead it runs on their secure build machines.

GitLab CI

Pricing: Can have unlimited Private & Public repos for free. Offers a free tier package that includes all stages of the DevOps lifecycle and 400 CI/CD minutes. Then the rest of the packages range from $4 to $99.

Can have unlimited Private & Public repos for free. Offers a free tier package that includes all stages of the DevOps lifecycle and 400 CI/CD minutes. Then the rest of the packages range from $4 to $99. Reliability: Widely known as one of the best CI/CD tools. Available for Windows, Linux, and macOS.

Widely known as one of the best CI/CD tools. Available for Windows, Linux, and macOS. Usability & Learning Curve: Easy to use. To use it you need to host the codebase in a Git repo and specify the scripts for which to build, test, and deploy.

Easy to use. To use it you need to host the codebase in a Git repo and specify the scripts for which to build, test, and deploy. Flexibility: You can easily move from tools like Jenkins and/or CircleCI to Gitlab. Can also integrate with Jenkins CI. Has a variety of features such as code reviews, deployment, developer APIs, etc. It has the whole package and some.

You can easily move from tools like Jenkins and/or CircleCI to Gitlab. Can also integrate with Jenkins CI. Has a variety of features such as code reviews, deployment, developer APIs, etc. It has the whole package and some. Security: Known to be safe and secure, but will have to upgrade to the Gold package to get advanced application security.

Appcircle

Pricing: The starter package is free with 1 concurrent build, 25 minutes max build time per build, 5 minutes per month device previews, etc. for individual use. Need to upgrade to the Professional package, $59 a month to make sure it’s suitable for team use.

The starter package is free with 1 concurrent build, 25 minutes max build time per build, 5 minutes per month device previews, etc. for individual use. Need to upgrade to the Professional package, $59 a month to make sure it’s suitable for team use. Reliability: Known to be one of the best tools for mobile app development.

Known to be one of the best tools for mobile app development. Usability & Learning Curve: Large community with documentation to get help from. Simple and easy to use build configuration interface. Builds with react-native. Has an in-browser simulator/emulator for iOS and Android.

Large community with documentation to get help from. Simple and easy to use build configuration interface. Builds with react-native. Has an in-browser simulator/emulator for iOS and Android. Flexibility: Has a very large landscape of flexibility, more so than App Center.

Has a very large landscape of flexibility, more so than App Center. Security: Include private and on-premises cloud support. Reliable security authentication.

Fastlane

Pricing: Free and Open Source under the MIT License.

Free and Open Source under the MIT License. Reliability: Known to be one of the best to ship out updates to the app store. Works best in tandem with another pipeline.

Known to be one of the best to ship out updates to the app store. Works best in tandem with another pipeline. Usability & Learning Curve: Integrates well with major CI platforms such as Bitrise, CircleCI, Jenkins, and Travis CI. The community isn’t big enough so it may be difficult to learn, but documentation is provided.

Integrates well with major CI platforms such as Bitrise, CircleCI, Jenkins, and Travis CI. The community isn’t big enough so it may be difficult to learn, but documentation is provided. Flexibility: Can automate apps builds and releases for iOS and Android.

Can automate apps builds and releases for iOS and Android. Security: Need to configure with another pipeline to ensure security.

NeverCode (Transitioning into CodeMagic starting January 2021)

Pricing: Pricing concludes to CodeMagic. Which is pretty cheap if you don’t have too many builds.

Pricing concludes to CodeMagic. Which is pretty cheap if you don’t have too many builds. Reliability: One of the best tools for app developers. UI tests are performed on emulators, device simulators, or actual devices without any manual intervention.

One of the best tools for app developers. UI tests are performed on emulators, device simulators, or actual devices without any manual intervention. Usability & Learning Curve: Known to have excellent support and documentation.

Known to have excellent support and documentation. Flexibility: No need to maintain local infrastructure since it only works on the cloud. Customizable, and can accelerate the build process with its smarter build logic.

No need to maintain local infrastructure since it only works on the cloud. Customizable, and can accelerate the build process with its smarter build logic. Security: Known to be pretty secure.

AutoRABIT

Pricing: Have to contact them to get a plan figured out.

Have to contact them to get a plan figured out. Reliability : Taylor made for a salesforce platform, by delivering the fastest CI/CD solution. Can take some time to fetch changes from branches and also to merge/commit changes.

: Taylor made for a salesforce platform, by delivering the fastest CI/CD solution. Can take some time to fetch changes from branches and also to merge/commit changes. Usability & Learning Curve: Not as easy to use as the other options but does have a support team and documentation.

Not as easy to use as the other options but does have a support team and documentation. Flexibility: Supports more than 120+ metadata types. Great to pair with salesforce.

Supports more than 120+ metadata types. Great to pair with salesforce. Security: One negative is security. With the speed of how this type of transaction works, I worry about security vulnerabilities.

Bitrise

Pricing: No free tier. Has a plan for solo developers, organization standard, and organization elite ranging from $36 — $270.

No free tier. Has a plan for solo developers, organization standard, and organization elite ranging from $36 — $270. Reliability: Great for mobile app development with coverage over most programming languages.

Great for mobile app development with coverage over most programming languages. Usability & Learning Curve: Provides a wide community to answer questions. Easy to use and implement.

Provides a wide community to answer questions. Easy to use and implement. Flexibility: Includes full mobile stack coverage. You can add emails for tester groups. Can deploy multiple apps in the same build.

Includes full mobile stack coverage. You can add emails for tester groups. Can deploy multiple apps in the same build. Security: Built-in secure and scalable public cloud instances.

Conclusion

There is no such thing as one size fits all with DevOps tools, so you may have to choose multiple CI/CD tools to meet varied needs rather than a single solution. If you’re a solo developer or small business you should look into options that offer free or cheap pricing packages, be frugal. Also, It is heavily recommended to integrate security into your DevOps model, if the tool you choose doesn’t already have it, with a comprehensive security solution like Threat Stack’s Cloud Security Platform.

From the result of my study, I was able to decide upon some top choices that will work best for my team and our current setup, which we’re Jenkins, CodeShip, GitLab, and CircleCI. Whichever one I choose will be added to FastLane as an added addition to automate the build process for both iOS and Android.

As for someone who knew little about CI/CD pipelines before taking on this study, I can now say I am comfortable moving forward on the decisions I should make when it comes to my DevOps tool stack. This study was effective in helping me make the right decision while upping my knowledge of DevOps, so I hope it can do the same for you, good luck!

Resources: