Canary in a Coal Mine
By: Saskia Leisewitz
Tags: Development Environment, Hellofresh, Continuous Delivery, Deployment, Software Development

by Peter Caron

You have certainly seen on a box or bottle somewhere a label which read “inflammable material”.

Does the label mean that the contents can explode in flames? Or does it mean the opposite, that they cannot? If you aren’t entirely sure, don’t worry, many native English-speakers (mistakenly) assume the word inflammable means that contents will not burn. In fact, the exact opposite is true. Light a match next to one and see for yourself. The result of an inaccurate definition is not usually so dramatic as the confusion around flammable/inflammable — which are in fact synonyms, but problems resulting from two people meaning something different when they use a word can have serious consequences. It seems reasonable even in software development that we should agree on definitions at the start of a project rather than be surprised when something blows up — even if the resulting “explosion” is only figurative.

As software developers, it is our job to help companies manage, plan, and orchestrate software development and deployments. One of the key elements on which we base our actions is the development methodology or ways of working that we choose to employ. We also assume that to be successful, our ways of working need to be built on common understandings and implementations of our chosen methodology, for example Continuous Integration (CI). An effective CI methodology, when properly implemented, uses automated tests, practices small commits and encourages continuously refactoring code as a time- and labour-saving practice. However, the devil, as they say, is in the details. An example of an important detail is the disparity of meaning that we often assign to common CI terminology: the words we employ to describe our development practices. Different people mean different things when they seem to be saying similar things. We can call this inconsistency a terminology failure and it constitutes a significant problem in many software development teams. For example, Continuous Delivery and Continuous Deployment are two concepts in software development that are regularly conflated but whose meanings vary from person to person, depending on who is listening and who is speaking. These are key principles, however, and should have a common definition across the team.

A fuzzy definition of deployment (not only continuous) could easily refer to a change to any environment, test or production. So, when a developer says deployment but means deploy something to staging, whereas a PO hears deployment and thinks to production, i.e. where users can actually see it, the elements are in place for a perfect storm. Obviously then, it is important to agree on a common definition. Because a deployment can mean one or many set or sets of services or one or a set of features deployed to one or many different environments. In a case like that, it is advisable to get that little bagatelle agreed up front. Just as when handling inflammable things, we must be clear about what we mean when we use particular words. Technically, this ambiguity of meaning is referred to as semantic diffusion.

Continuous Delivery, Deployment, and Release

Regularly, I hear these terms used interchangeably which can lead to confusion and ultimately poor quality software. Starting from Humble’s definition, let’s try to pick apart the meanings a bit. Concomitantly with the use of the phrase Continuous Integration usually comes the notion (but usually not the practice) of Continuous Delivery. The two are conflated so often, that they seem to have lost their original sense. The core concept of the latter can be best explained with a question, “why wait for everyone else to finish?” If a feature does not have any dependencies and all the other teams are so slow why not deploy it immediately. That is the simplest explanation of Continuous Delivery. If it is done, don’t sit on it. Deliver it. The question is then, deliver it where? Hopefully, to a production environment. Once continuous deliveries are achieved, Continuous Deployment is only one business decision away. Of course, things are rarely so simple. Even though I may deploy a new microservice to production, it may not be not technically released until other services start using it. For simplicity’s sake, however, let’s settle for the following definition: continuous integration is regular merging of code into a single, common code base; continuous delivery means keeping code in a deployable state and is the natural consequence of CI — if the build and all tests pass, the compiled code is available for deployment to an environment; continuous deployment is automatically deploying that compiled code (continuously delivered from your continuous integration process) to an environment which may or may not be production; and a release is essentially a business decision to enable a feature-set in a production system used by customers usually via a deployment or by enabling feature toggles.

It is an unfortunate truth that most so-called CI systems are far from continuous; pull requests, manual testing and excessively complex, dependent on branch development. All these things slow down the process because they are ultimately detrimental in terms of resources required and time spent. Testing at the end, for example, is far more expensive than tests that run early in the development lifecycle (SDLC); it is both wasteful and ineffective in time and money. As a side note, we sometimes forget that tests exist not to find bugs, but to prevent them from going into production. So too, long-lived branches which require complicated and error-prone merges which increases both the time required to deliver software and raises the chances of introducing bugs. In other words, we spend a lot of time waiting for others to finish.

Continuous Delivery logically should be the end result of a smoothly running, efficient Continuous Integration development process in which many small changes are made, tested, and ready for use. Whether one decides to use them is a deployment decision. Or even a deployment and release decision. Or a business decision. Or all of the above. That decision may be to deploy continuously or at some other cadence.

There are legitimate reasons one might choose not to take that last step from continuous delivery to use continuous deployments irrespective of the target being a web environment or not including some basic requirements for continuous deployment integration.

Making only small changes to code (on short-lived branches) Having sufficient enough automated unit and integration tests to give everyone a warm and fuzzy feeling that the deployment will work Services are independent microservices and not microliths or distributed monoliths Comprehensive and continuous monitoring is in place — and works consistently in all stages of the SDLC Integration and performance testing is automated and complete In-production testing exists — and works

If we agree to these requirements, we have in effect our common definition of continuous deployment. And, the simplest definition of a software release is simply a business-friendly description of a deployment to production. None of the above should be a revelation to anyone familiar with CI/CD practices and is not especially controversial.

A deployment (not continuous) can be a change to any environment, test or production: complete or partial; one feature or many; to different environments or to a single production environment. But at this point common sense starts to, if not unravel completely, then to go off the rails in terms of inconsistent nomenclature. We regularly use terms to describe a couple of different ways to make a software deployment. As with their continuous cousins, we are not always consistent in defining there either.

Releases

A release in our definition means a business decision deployment of one or more features to a production environment. In addition to continuous as discussed above, there are at least four other significant methods used to release software to production environments: big-bang, blue/green, canary, and progressive. Each of these others is common across the industry yet lack agreed upon definitions. Let’s address each in turn.

Big Bang

Like the original one which launched our own Universe, a big bang software release is a launch in which a complete change of the current feature, component or even entire system to something else is attempted. Big bang software releases are often used for developing and deploying operating systems, embedded devices and packaged, end user software — with printed manuals. Suppose you wanted to change a poorly architected system to use the latest greatest REST APIs. You collect all the requirements, plan a brilliant architecture, spend 6–9 months developing and testing and then big-bang deploy to your customers. Imagine that in the morning your applications are all calling APIv1 linked to DB1.0. At the stroke of noon, you throw the virtual switch and bang! all the apps not call APIv2 linked to DB2.0. What could possibly go wrong?

This methodology was (and remains) quite common in many companies and is usually associated with waterfall development. Contrary to some opinions, development which leads to so-called big-bang release does have value and, under certain circumstances has valuable uses, especially in the physical world: software for rocket testing or nuclear power plants where transitional phases are not a viable or cost-efficient option.

Blue/Green

Another common deploy-to-production release strategy uses a blue/green paradigm: a parallel replica of a production environment that can be an ingress gateway or even a simple change to DNS to switch traffic from blue to green. Simple, easy, slow, and expensive. Also, if you have bugs in green, they are released into the wild immediately. This need not be coupled to a waterfall methodology or big-bang release because there is theoretically no lower limit to the time delta between releases: I can as easily blue-green deploy once per day as once per month, quarter or year. Ok, not actually exactly easily, but the point is I don’t have to wait to blue-green deploy. As soon as a feature or service is done, I could deploy it in this way. I am limited only by the cost of continually running dual environments and the willingness of my operations team not to throw up their hands in disgust and quit. References to Blue/Green releases are sometimes used synonymously with A/B testing. Though they share many similarities in practice, they are quite distinct in purpose.

Canary

The metaphors we choose to describe things necessarily shape the way we think about those things, so it’s important that we choose an appropriate metaphor. Industry descriptions of this particular method, so-called canary metaphor vary widely. A canary release is actually a step-child of a blue/green release. Canary releases made to a production environment are a risk-averse way to do a blue-green release. You may have read or heard previous rants by me against canary releases. Canaries, avian and methodological variants alike, are elegant and effective monitoring systems as opposed to testing systems. When birds were used as monitors for noxious gas, nobody sent canaries into mines hoping to find gas. They were controls. Miners did not expect the canary to die each time they went into the mines. The opposite, in fact. Usually, the canary survived. One can only assume that they did not actually want the canaries (or the humans, for that matter) to die. The canary (bird or release) therefore should be to monitor for something — gas or usage patterns: not die or find a bug. Some people might disagree with this conceptualisation and they are free to write their own blog which I will happily read and consider. In the meantime, canary releases are to monitor usage and the uptake of a feature.

I believe that the concept of canary releases has been incorrectly usurped and ultimately corrupted over the years by well-meaning commentators and villainous pundits alike. This may not be entirely fair (but that is sort of expected in a rant, I guess). In any case, I think that in the IT world, the misconceptions surrounding canary release developed in a time when we had to deploy physical servers one at a time. At the time, ramping up physical machines was expensive and blue/green was not a cost-effective option. Imagine trying to sell a purchasing department on the idea of duplicating, say, an entire 100+ server farm so that you could rollback deployments if there were too many bugs released to production. They would think you are crazy and ask “why not release it to production after you find all the bugs?” Well, why not? The answer was to deploy a single server (with the expectation that it would work). But, just in case it failed, we only had one deployed. A canary.

In the mid-2000s, when we (by “we” I mean older folks like myself) began to use virtual servers and then EC2 instances, the costs were still relatively high in terms of hardware requirements, and we could not seriously consider full blue/green releases. Yet, the persistence or pernicious bugs made operations nervous. Risk-averse operations people, because they did not trust the developers to deliver bug-free code, decided to only release one instance into the monolith and find the bugs that they knew were going to be delivered by lazy and shameless developers.

According to Danilo Sato

“Canary releases can be used as a way to implement A/B testing due to similarities in the technical implementation. However, it is preferable to avoid conflating these two concerns: while canary releases are a good way to detect problems and regressions, A/B testing is a way to test a hypothesis using variant implementations. If you monitor business metrics to detect regressions with a canary,1 also using it for A/B testing could interfere with the results. On a more practical note, it can take days to gather enough data to demonstrate statistical significance from an A/B test, while you would want a canary rollout to complete in minutes or hours.”

The key statement is, “canary releases are a good way to detect problems and regressions.” Risk aversion is the primary motivation in this context. As we will see, this is not necessarily synonymous with fast release cycles which can be better served with other kinds of release strategies.

Canary releases in microservice architectures should be limited to business experiments. They are used either as a control (heavily monitored and logged pods) or to try out a new feature in a limited user environment, for example, in a single geographic region with minimal exposure (i.e. every x user gets access to a promotion). It is also possible to use a canary release when the operations team still doesn’t trust the developers’ fancy new algorithm to automatically shut down the nuclear power plant, or to just roll-out a new feature slowly — perhaps a shadow release to allow in-production testing. This is similar, I suppose to what we used to call beta testing. In a microservices world, a canary release that exposes a critical problem in production would be the equivalent of “stop the assembly line,” I have to sneeze. Or the bird died.

Remember, we don’t want the canary to die. We want to ensure the health of the environment before features are deployed to production and monitor them thoroughly in prod. Canary releases are similar to using a staging environment. But different. The difference is that staging environments are usually dedicated to the task; a staging web server never becomes a production server. Ideally, if a service works, activate it in production right away.

I can already hear the naysayers saying “hold on a sec!” They are thinking, “then why do so many writers define canary differently?” Fair question. I think it depends on the context. For example, Susan Fowler says “to counteract any instability introduced by deployment, we can ensure our microservices are deployed carefully with proper staging, canary (a small pool of 2%–5% of production hosts), and production rollouts” Production Ready Microservices, chapter 2. Read her sentence again and you’ll see why I think it is wrong and cannot work well with container-based microservices. Hint: focus on hosts and imagine it’s 1999. Don’t believe everything you read. That being said, Fowler has much sound advice for SRE and development teams and her proposals, even for canary releases, have merit in many contexts and system environments. Nevertheless, what we want is something a bit different. This brings us to the final release strategy I want to discuss, which is a variation on the canary idea but without the negative expectations — in other words progressive releases.

Progressive Delivery

The concept of a progressive pattern or progressive delivery is the alternative (hidden in plain sight) to the challenges implicit in flawed canary releases or clunky blue / green ones with and have the added benefit that you can better guarantee performance at scale. Carlos Sanchez, a Sr. Cloud Software Engineer at Adobe defined progressive delivery this way.

“Progressive delivery is the next step after continuous delivery, where new versions are deployed to a subset of users and are evaluated. in terms of correctness and performance before rolling them to the totality of the users and rolled back if not matching some key metrics.” (Carlos Sanchez, Blog)

There may be only a semantic (or even pedantic) difference of opinion between canary and progressive. But, if that is the case, I prefer to describe experimentation as a canary and the rest as a progressive release. A progressive delivery (to any environment but especially to production) is exactly what it implies; a rollout of services or features progressively either a percentage of total traffic, to selected users, in specific geographic regions, or any combination of the above. Slowly or rapidly, depending on your appetite for risk, rollout to full production in steps — say 1%, 2%, 5%, 20%, 50%, 75%, 100%. The idea is not to find problems but to understand how a service copes with the platform’s requirements in a controlled manner.

The harsh reality of most platforms though is that we need to remember that we still have many services tightly coupled through a monolithic codebase. For that reason, we are unable to do several things: independent deployment of microservices, data decoupling, and continuous integration for example. Our tests are neither robust nor comprehensive enough either to give us a secure feeling that we can deploy to production without going through that extra stage, creating a little Frankenstein-like limited delivery or (gasp) partial deployment. You either deliver/deploy or you don’t. There is no middle ground. Any deployment should be self-contained and therefore “full”. When it is not, and we resort to talking about partial deployments (like what many means by canary), what are we actually doing? Indeed, what are we trying to even say? Are we inherently assuming that these deployments don’t count somehow? Or that they are incomplete? An incomplete deployment. That is, by my reckoning, a failed deployment.

So, is there a role for canary-like releases in our environments? Our staging environment perhaps? Is there even such a thing as a canary release that we can use? Perhaps, but only as long as we use the word in a way that, well, the bird doesn’t die. Or at least not unnecessarily.

Staging or Development Environments

Contrary to some comments I’ve made in public (and in a number of extended rants online) concerning the detrimental impacts of staging environments, my misgivings with this kind of separation of environments is primarily operational: i.e. dealing with things like configuration drift and scalability. Not surprisingly to me, staging as a concept can be compared to the concept of canary releases. I.e., “let’s see if the bird dies when we release it” kind of thing. Unfortunately (for the bird), we expect our little avian helper to die because we believe bugs are inevitable and we willingly send deployments out to die. By this logic, we should keep them as far away from customers for as long as possible and furthermore by pushing software we know or suspect is not ready onto a customer is, by its very concept, flawed. Therefore, we should strive to keep bugs out of production by proper ways of working, design patterns and thorough testing long before they ever reach staging or production.

Does a staging environment have a place in the SDLC? It depends. If it is used as a development environment to which you can deploy your software to see if it works. There are, of course, other ways to employ a development environment in a useful way: Infrastructure as Code (IaC) or h GitOps are two examples. Both approaches offer application deployments and promote services through environments (Development -> Production) sometimes can be a productive step in container-based systems. The environment must be an identical configuration to production though the scale may be less.

Both IaC and GitOps mean that you store the configuration of your declarative infrastructure environment in a version control system. The configuration describes what should be deployed into a given environment. You need to specify the expected state of the environment you want to maintain, leaving it up to your orchestration system and other tooling to achieve that state in a reconciliation loop. In this way, if you have a collection of unstable or coupled macro- or microservices you can run a full, production-like test in staging prior to rollout via a production canary or other method. The key to both processes is that the infrastructure is immutable and you avoid configuration drift.

Continuous Deployment

The next logical step after Continuous Delivery would be to deploy continuously. How do we know when to try continuous deployment? First, we should decide if it is advisable or even possible in our environment. Continuous Deployment is best suited to web environments and probably less optimal for medical equipment software or nuclear reactor cooling systems. Second, we need to make sure we are ready not just technically but also commercially.

If we are still asking questions like:

How can I do continuous deployment as a canary release?

How should we version continuous deployments?

How can I do canary with conflicting versions of deployments?

Can I continuously deploy my mobile apps?

But we cannot answer them satisfactorily, we most likely still have a continuous integration problem, not a continuous deployment opportunity. Continuous Deployment is probably not a viable option for many companies or services and should be only considered after all of the above are well-understood and solved. Sometimes the business will decide that even if the technical teams can deploy continuously, regulatory or marketing requirements (or fissile isotopes like uranium-233) do not make it the right choice.

Final Thoughts

In software development just as in daily life, how we employ words has consequences. Definitions matter. When we don’t establish agreed upon meanings in software development early in the process or we use vague and variable definitions, there will be consequences. There will be impacts — and they will probably not be good ones — on the quality of our product (Jean Paul Sartre wrote: “Chaque parole a des retentissements” — which, loosely translated means “each word has repercussions. I think it is applicable here. Choose your words wisely).

Here are some definitions worth considering when discussing software integration, delivery, deployment or releases:

Problems with canary releases will usually mean that your CI is not working, that services are too tightly coupled or you still have the remnants of a monolithic codebase.

Canary deployments to production are NOT meant to catch bugs! Remember, canary deployments are a risk-averse version of Blue/Green deployments and are most definitely NOT “Continuous Deployment (CD)”.

Continuous Delivery is not Continuous Deployment

Continuous Deployment will not work with monoliths

Canary deployment when used for physical production systems, guided missiles and nuclear power-plants is not a good idea!

Continuous Deployment is primarily intended for web sites, streaming, consumer-facing retail, and rapidly changing systems.

We’ve all heard some version of a well-meaning manager saying, “great, finally we’re doing Agile/CI/DevOps” or some variation followed ominously by “now, how can we publish our roadmap to customers and to commit to the next three years?” But, what a product owner, manager or technical lead might mean by that will not always agree with the definition used by a developer or software architect. Likewise, the phrase Continuous Delivery is similar in at least another respect: Many people push to achieve CD in their company but don’t always understand why they are doing so nor do they perceive the implications for the company or its customers. Teams will certainly suffer the consequences of such misguided enthusiasm so they must be the ones to set realistic expectations for stakeholders: what CI/CD can and cannot do. The definition of these terms like canary, continuous, deployment, release, staging, and a host of others is important. If you cannot agree within your organisation about their meaning, the implementation will probably fail.

I once read that Johann Wolfgang von Goethe reportedly possessed a vocabulary of more than 90.000 words.

So, when that great writer proffered, “if possible, speak a few sensible words”, I think he would have agreed that it makes sense that those sensible words make sense to others. Let’s let the canaries live and agree on a consistent meaning for deliveries and deployment. Otherwise, let’s banish them from software-speak and simply get back to creating great software, delivered at speed.