Deployment maturity levels
By: Raphael Yoshiga
Tags: Software Development, DevOps, Programming, Continuous Delivery, Software Engineering

As software engineers, we love to develop features for making our customer lives easier, but until those features are available to end-users, it adds no value to the business. Therefore the deployment process is an essential part of succeeding as a development team.

With six years of experience in Continuous Delivery, this is how I would classify the maturity level for deployments:

Level 0 – YOLO (You only live once)

Diagram of deployments direct from the developer machine

The deployment process is quite simple; the developer builds the code on his machines, copies and pastes into the production server. That can be quite fast, it’s a great way to prove a concept, but in the long run, it’s not repetitive and unauditable.

It is also reasonably risky; no tests are running after the deployment, so the developer has to test manually. And if it the application broke, pray that he can restore the production environment.

Another aspect of this level with production systems, is talking about spaced deployments of once to twice a month. Deployments require coordination across multiple teams, increasing the time a feature is requested until it is available to users. That is what often ends up happening to monoliths because multiple teams maintain one codebase.

Let’s speak about deploying once a month.

What is safer, to deploy once a month or to deploy every week?

Let’s say every week the development team can develop a single feature.

Once we deploy those features at once, many changes happen simultaneously, increasing risk exponentially, and once things go wrong, which feature caused the problem? Sometimes isn’t even a single feature issue, but a combination of multiple changes that were not tested together.

Big-Bang deploys often are hard to rollback because they touch multiple applications and databases. Imagine breaking production in your Saturday, without an easy rollback; you will have a hard time.

It is counter-intuitive, but often deploying small changes divides the risk, taking little by little in a much more controllable manner.

Another problem with compiling the releases is for the development team to have to context switch. Once they finish the feature, it still isn’t released, and sometimes during the deployment or testing problems are found, impacting the current work they are doing. And as developers, our memory gets tested once we are trying to remember the purpose of the feature developed a month ago.

Level 1 —Continuous delivery

Continuous delivery diagram with a commit, build and deploy the artefacts step.

Automated build and deployments, either through Powershell/Bash scripts or with a CI (Continuous Integration)/CD (Continuous delivery) tools as Octopus deploy, Circle CI or Azure DevOps.

We can go to any version of the application through the click of a button. (That’s the definition of what we call continuous delivery.

The transition from YOLO to continuous delivery:

Once transitioning from manual deploys, I guarantee you there are steps in the deployment process, only the person deploying knows.

A real-world example

This is fine meme

In 2015, we were automating a web API deployment, with a vast customer base, so we waited until dawn to start the first automated deploy.

Before doing the automated deployment we tested in the test environment, everything looked promising and great. We did the first deploy at 2 AM, I checked the individual machines through the browser, and the API worked. I was happy and confident.

At 4 am, we discovered we were not receiving any traffic. Confusion and desperation, we contacted our ops team, which were speaking to the hosting company. After long agonising 2 hours, someone found the problem; the load balancer couldn’t see our web API machines.

The load balancer would ping a specific URL as http://api.com/index.html and expect a 200 OK, if not it would remove the machine from the load balancer pool.

Load balancer not forwarding traffic because the API’s didn’t expose a specific URL

The reason was, the deployment person always remembered to copy and paste that index.html file every deployment, so again:

Manual deploys have steps only the person that deploys know.

Of course, the next day, I added the file to source control and that future deployments didn’t have the problem.

Level 2 – Zero Downtime

The previous models even though automated, had some disruption to the service. Which leads to our users to see something like this:

Once our deployments were scarce, this worked fine, maybe done once or twice a month during the weekends. But in the digital transformation world, our market is way more competitive, releasing new features fast and often became a competitive advantage.

Can you imagine working at Amazon.com, and telling senior management we will deploy three times a day with 5 minutes maintenance windows? That would cost the business millions of dollars quickly. And remember, you are part of one team, Amazon has hundreds of teams, imagine the disruption if each one likes to deploy daily.

That’s why we have this concept of Zero downtime, so we can roll out changes without causing disruption to the final users.

We can achieve this in multiple ways; two common ones are:

Blue/Green deployment

Blue/Green deployment diagram

In Blue/Green deployment, you will have your production environment having two instances of the application. One is live with users accessing. The other “staging” is a passive copy used for deployments and tests.

After we deploy the new version to staging, we can warm up the application to avoid downtime and run smoke tests to validate the application is healthy. That’s an excellent safety net, in this way you won’t be able to break the entire application because of a wrong deployment.

Once we are happy with the blue slot, we can swap it with the green one and start sending traffic. We can apply the swap at load balancer level, for example.

Canary/Rolling deployment

Canary deployments can be useful when you have multiple machines. You deploy to them incrementally, in a rolling manner. So you don’t take the application offline and minimise risks by deploying to only a subset of the traffic at a time.

Before going to the next batch of machines to deploys, we can also run smoke tests before the next stage.

Canary deployment term comes from mining, a dangerous profession; in some caves, there are poisonous gases that could kill the miners. So they would take a canary (little bird) with them, leave the bird in the cave for a while, if the bird is still all right, it would be safe for them. We are using a similar approach by deploying to a subset first.

Level 3 – continuous deployment

Photo by Oscar Sutton on Unsplash

At this point, I think it is useful to make a clear distinction:

Continuous Delivery — It is the ability to go to any application version at any time, as stated in level 1.

Continuous Deployment — All code changes get automatically deployed.

All right, you got a mature project. You have a high code-coverage, you trust your test suites will pick issues during the build/release pipeline. Every code change goes all the way into production without anyone’s approval.

A small change takes less than 30 minutes, maybe even 10 minutes to go to production. Well done! Just ensure you keep up with the quality, and I am sure you will have delighted customers.

Authors Note: Continuous Deployment is not a must and maybe I wouldn’t even recommend to every client.

Bonus

Although those aren’t continuous delivery specific subjects, if you want to do a microservices architecture with continuous delivery, you will need to up your game in those areas:

Logging and Metrics

You aren’t Daredevil, if you can’t see it, you can’t fix it.

At any of those levels, you got to have basic logging. Preferably not logging to a virtual machine file that you need to connect to check the logs remotely.

Preferably you will have a tool to centralise logs across applications, and even better if you have a distributed trace system, where we can see all the steps of a user request in the downstream system.

App Insights example of an end-to-end request tracing

Alerting

In a monolith architecture, you have one service to make sure it is working. In a microservice architecture, you have dozens if not hundreds. You will need a way to view and be alerted about the system health.

If the application goes down, I am sure you prefer to receive an SMS from your alerting system than receive a call from the CTO asking what is going on.

Automated tests

Not only for the smoke tests, for us to start deploying often, we also need short testing times. It wouldn’t be practical to have a manual testing process that takes weeks. Testing needs to be automated, run quickly and provide the safety net for changes.

I speak about my current testing strategy here.

Conclusion

Making the deployments smaller, more often and faster, reduces the risk on the deployments while at the same time enabling development teams to deliver. Automation also makes it more secure, enabling you to audit your deployments, so you know who pushed what change and when.

Tracking teams progress

I like to use an excel to track my team’s maturity levels, so we can prioritise which project needs improvements depending on how often we apply changes and how important the project is for the customer.

Excel matrix for the continuous delivery maturity levels

References: