What CI/CD is not.
By: Soumadri Roy Chowdhury
Tags: Continuous Deployment, Continuous Delivery, Ci Cd Pipeline, Agile Development, Continuous Integration

There are plenty of articles on the internet to explain CI/CD with great details. While I was pondering to write about this topic, it didn’t make sense to add one more article in this already crowded space. So I thought of focusing on the key points which usually people misunderstand.

Just to give a little background on CI/CD, it is a software engineering practice through which we continuously integrate, test, deliver & deploy working application code to production in small increments. This is directly in line with some of the foundational principles of agile.

Space shuttle launching (© Pixabay)

So here are the 6 points about CI/CD, which are often misunderstood,

1. There are actually 3 terms, not 2

Although the buzzword has 2 words in it, namely “CI” and “CD”, CD actually has 2 variants, “Continuous Delivery” and “Continuous Deployment”. And these 2 are slightly different in terms of what it includes as a process. For more details on these refer to this guide.

2. CI/CD ≠ Automated deployment

There is a common misconception that if we have automated deployment procedures/scripts, we can say that we are having CI/CD practice. This is actually not true. While CI/CD asks for automated deployments. That alone is not sufficient. Think of it in this way, CI/CD stems from the following agile principles as described in the Agile Manifesto,

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Working software is the primary measure of progress.

Continuous attention to technical excellence

and good design enhances agility.

If you notice the common themes in all of the above principles, it points towards frequent short interval delivery of working software to customers (Speed) & focus on good design, technical excellence (Quality). If you think with a conventional mindset, this would sound a bit absurd. How can we have speed and frequent delivery without compromising quality? These two generally are inversely proportional. That’s where automated testing comes to the rescue. The idea here is that for every story or bug, the definition of done must include various levels of automation testing (unit testing, acceptance etc.). Your stories/bugs will not be marked as done unless it has corresponding automated testing scripts. Why automated and not manual, you may ask? That’s because it is nearly impossible to cope up with the speed of development and also ensuring all kinds of testing is done manually. Even if you try to put a big testing team to cope up, the quality of testing cannot be guaranteed.

CI/CD process at a high level looks like below,

Conceptual depiction of a typical CI/CD pipeline

As you can see above that all the steps are executed serially, and has feedback loop from every step. Meaning if any of these steps do not meet the defined criteria, the process terminates after sending the feedback to the appropriate system. Also, note that Unit and Acceptance testing are an essential element of this process, and without them, it’s simply not CI/CD. What you may have is just automated deployment.

Common adoption issue One common issue many team faces while adopting CI/CD practices is to get their head around with the idea that without automated unit and acceptance test scripts, their deliverable is simply not complete. I have seen that many teams tend to consider the code development and writing tests as two different activities with different planning for each. That simply doesn’t work. Unless we consider that both are integral to the completion of items, it is not possible to take full advantage of CI/CD. This may require educating various stakeholders of the project and get their explicit buy-in.

3. It is not effective much with a long release cycle

As I have mentioned before a shorter release cycle (typically a few weeks) is one of the key tenets of agile and CI/CD practices implement the same. Now the thing to note here is, the longer your release cycle, the lesser the benefits you will get out of this. The reason for this lies mainly with the length of the release cycle itself, not so much with CI/CD process per se. A lengthy release cycle means that the stakeholder/customer feedback is getting delayed. So the goal here should be to develop as little as possible and deliver it fully to the end-users and get their feedback. Do not bunch up a huge amount of items and then try to make a release. By the time you will get the feedback, it will be too late to incorporate those (due to business or technical reasons).

A sidenote There are certain kinds of projects which may not be suitable for a shorter release cycle. An example can be mission-critical projects like launching a space shuttle etc. But those are anyways outside of the scope of Agile, so no question of CI/CD implementation there. Although there are new generation space tech companies such as SpaceX, who have adopted agile approaches.

4. CD without enough self-check, monitoring & feature toggles may lead to a disaster

This one is specifically about CD (Continuous Delivery & Deployment). One pitfall of releasing in a short cycle is that it is difficult to prepare all kinds of comprehensive tests and run them before the change is deployed to production. The sheer volume of tests & the time it would take to execute would make it difficult to meet the short cycle. That's why a test pyramid approach should be taken.

Now the question here is since we are deploying so frequently, how to keep a tab on which particular deployment has gone wrong, and how can we quickly take action on it. Consider this, in a conventional approach a team may decide to make 1 production deployment every 6 months, effectively 2 production deployments a year. But let’s say they are deploying to production once every 10 days, there will be ~26 releases a year. Some companies like Amazon are deploying some code to live every 11.7 seconds. Now you can imagine the headache of tracking these down when things go wrong.

So depending on the scale of CI/CD adoption, you may require the following approaches to be adopted, otherwise, it may lead to complete chaos,

Automatic health check monitoring at a granular level

Self-check for all services in an interval, to ensure the critical components are operational (e.g. synthetic transaction monitoring)

Implement Feature toggle/Feature flags, to have a mechanism where you can keep deploying your code without worrying whether it needs to be exposed to the end-user or not (as it may depend on a variety of business factors)

5. Not all organizations are fit for CD

CD (both Continuous Delivery & Continuous Deployment) is not for every organization. There are many organizations where each deployment goes through various approvals from different departments. Not only that, they also do not have any DevOps team who has enough rights to deploy stuffs to production. Instead, it is done via a completely different team, who are completely detached (or siloed) from the project teams. In this setting, there isn’t enough trust in the project teams and/or DevOps team to allow them to deploy things on production. In such cases, one can have CI at best but not CD. CD needs the organization-level processes optimized for it. The first challenge with CD is not technical but organization culture/policies.

Read more about this thought process and values here.

6. Don’t try to implement in one go

Last but not the least, like everything else in software development, don’t try to implement the entire CI/CD pipeline and associated culture in one go. Consider the CI/CD-related work also as part of the backlog. Get an explicit buy-in from your product owner on this, and in a balanced fashion implement these. Also, for every release that you make, it may need some adjustments/changes to be done in the CI/CD pipeline. Consider that also in-scope for development. Consider the following items as a checklist for feature/bug development completion,