Canary Release Patterns for APIs, Applications, and Services with Ambassador
By: Daniel Bryant
Tags: Cloud Native, Continuous Delivery, Api Gateway, Canary Release, Kubernetes

With an increasing number of organisations adopting the Ambassador API Gateway as their main ingress solution on Kubernetes, we are now starting to see a series of patterns emerge in relation to continuous delivery and the testing and releasing of new functionality. The canary releasing pattern is one of the most popular, and so in this article I will outline a series of related sub-patterns, implementation details, and things to watch out for with this approach.

Canary Prerequisites

The canary releasing pattern has been made popular by the cloud native vanguard, such as Netflix, Amazon, and Google, but obviously not every organisation is running with this much experience or as many customers. From my experience, canary releasing is a genuinely useful pattern, but there are several prerequisites:

Basic automated (continuous) delivery pipelines: canary releasing relies on deploying and running of multiple versions of a service, which is challenging to do (and not much fun) without some form of automation. Implementing a delivery pipeline that does basic continuous integration and quality assurance on a service will also reduce the amount of issues you see within your canary experiments

Basic observability: in order to decide if a canary release is causing problems you will need to implement basic monitoring (observability) that provides you with data in relation to the golden signals (latency, traffic, errors and saturation). For example, you don’t want to continue migrating traffic to a canary service that is causing the CPU or I/O to spike above tolerable levels.

A codified definition of business success: in a similar fashion to basic observability, you will need to identify a series of metrics that indicate success in relation to a business hypothesis or goal. These are often referred to as key performance indicators (KPIs). For example, when rolling out new functionality you will typically have some kind of goal in mind, such as an increase in customer spending or a lowering of user complaints. Even if you aren’t experimenting with improving a specific business goal (for example, you are simply releasing code with architectural improvements) you will still want to be able to monitor for negative impact on your metrics of success, and roll back the release if this occurs.

A sufficiently large amount of representative user requests: although the likes of Netflix receive millions of user requests each day, many of us are not building systems for this scale, and this can mean that it is difficult to push enough traffic into a canary release in order to fully validate it. In addition to a sufficiently large traffic volume, you will also have to ensure that you test your canary with diverse and/or representative types of user requests in order to test all of the execution paths. For example, if you are canary testing a feature that each customer only uses at most once a year, then you will need a very large amount of users in order to generate a statistically significant success rate.

I’m sure there might be a bit of nuance or disagreement with these prerequisites, and so please do reach out to me if you have any comments.

Canary Releasing Patterns

I’ve written about the fundamentals of canary releasing in several other blog posts, but I often get asked specific questions via the Datawire OSS Slack about how to implement canary releases with Ambassador. The general approach I have seen taken by organisations follows the path from manual canaries, to semi-automated, to the fully-automated canaries (often by way of shadow canarying).

So, assuming you have the above mentioned prerequisites covered, let’s examine the typical evolution of canary releasing with Ambassador in more detail.

Step 1: Manual Canary

This approach to canary releasing is typically the easiest approach, and often the best place to begin your journey. The key mindset change when adopting canaries is to separate the deployment and release of new functionality within a service. You may also want to separate your deployment (build or delivery time) and release (runtime) configuration files and scripts into separate version control repositories, as this leads to a clearer separation of concerns and potentially a cleaner VCS history.

Assuming that you use Kubernetes as your deployment platform of choice, the implementation is as simple as deploying a new “canary” service alongside your existing service, and initially routing no traffic to this. At this point in the canary process, you have not “released” the canary, and you will have only modified your deployment configuration. It’s also worth mentioning that you don’t necessarily need to perform a rolling update of the service via Kubernetes Deployments at this stage, as this is a completely new service being deployed. You will, however, need to name and label this service as appropriate, for example, appending a version number or UUID to the service name to make this unique.

At this point in the process you add a new Mapping for the canary service into your release configuration Kubernetes Ambassador YAML files, and also add an appropriate traffic shifting “weight” property that will route the related percentage of traffic away from the original service to the canary (note that the prefix target URL path is identical for the canary and existing service):

---

apiVersion: ambassador/v1

kind: Mapping

name: shopping-basket-canary

prefix: /basket/

service: shopping-basket-v0023

weight: 10

---

apiVersion: ambassador/v1

kind: Mapping

name: shopping-basket

prefix: /basket/

service: shopping-basket-v0010

You then apply the updated YAML to your cluster via kubectl , or ideally trigger this via a delivery pipeline for your release configuration. The emerging best practice for this style of automating all aspects of configuring a cluster and using a version control repository as the single source of truth is being referred to as “GitOps”.

As the specified percentage of user traffic is routed through the canary, you then begin watching your service dashboards and observing top line metrics (e.g. amount of request being routed to the canary, request handling latency, 5XX errors, resource usage etc) and also related business KPIs (e.g. is the new shopping basket improving checkout conversion). If the metrics look favourable (including the amount and type of requests being sent to the canary), then you will increase the traffic shifting weight in the canary service Mapping, and iterate again on this process. If the metrics show a negative impact, then you can change the traffic shifting weight to 0 and investigate the issue. You can then either begin the experiment again, or potentially delete the canary service with the goal of deploying an improved version later.

It’s worth mentioning that you will need to account for the time lag between incrementing a shift in traffic and being able to observe the effects in the metrics — for example, if you only scrape metrics from your services every 30 seconds, then there could potentially be a 30 second window where your new release is causing an issue and you don’t know about this unless you have more fine-grained alerts. You will also typically want to keep an eye on other (supposedly unrelated) alerts that you have configured, as it’s not unknown for a subtle change in a canary service to cause an unintended issue in an upstream or downstream service.

Once you are routing 100% of the traffic to the canary service and are happy with the result, you can delete the original service in Kubernetes. You can then change the name of the Mapping, removing the word “canary” and switching this to the appropriate “production” name. (The alternative is to use a “blue/green” style naming convention, which doesn’t require the Mapping name to be changed upon a successful release):

---

apiVersion: ambassador/v1

kind: Mapping

name: shopping-basket

prefix: /basket/

service: shopping-basket-v0023

You will also need to make sure that you synchronise all of these config update tidying actions with your VCS and the state of the cluster before you begin more experiments or releases.

The canary release cycle then begins again with the testing of the next version of the service.

Step 2: Semi-automated Canary

The next evolutionary step with canary releasing is to automate the shifting of traffic to a new canary candidate from 0 to 100 percentage of traffic. This can be implemented at a crude level simply with bash scripts, or ideally triggered via updated config files which are then actions via a continuous delivery pipeline (for example, using Jenkin’s pipeline-as-code scripting), and can be as simple as incrementing the percentage of traffic shifted within the Ambassador Mapping at a specified time period e.g. 60 seconds. You will still have to manually monitor associated top line and business metrics, and abort the release if you identify anything negative.

Some organisations also incorporate an automated simple canary release rollback into the script. You can implement similar, providing that you can programmatically access and test the new functionality or associated metrics.

I’ve included a very pseudo-script example below, which provides a (very raw) template as to how this can be implemented

for (i = 0; i < 100; i=i+ 10) { cat <<EOF | kubectl apply -f -

...

---

apiVersion: ambassador/v1

kind: Mapping

name: shopping-basket-canary

prefix: /basket/

service: shopping-basket-v0023

weight: {i}

...

EOF //trigger deployment of new config

git commit -m “Increment canary of service X to {i}”

git push sleep 60

// potential curl check to canary endpoint

}

When automating the rollout of a canary you will need to ensure that enough user requests have been shifted to the canary release. This can be a particular issue for time-based traffic shifting increments, where an unexpected reduction in traffic can result in a false negative result. For example, if your user traffic fluctuates greatly at off peak hours, then releasing a canary here could mean that it does not see a representative load even with a high percentage of traffic being shifted.

Step 3: The Fully Automated “Robo” Canary

The ultimate goal that many organisations are aiming for is the fully-automated canary release, where developers simply flag an updated service as ready for deployment, define associated operational and KPI metric thresholds, and then rely on the deployment pipeline to autonomously manage the entire canary release/rollback lifecycle. This is what the Netflix team have talked about with Spinnaker and Kayenta, and also what the Weaveworks team are working on with Flagger.

While the fundamentals of this approach are not much different than the semi-automated canary mentioned above, there is a significant amount of extra sophistication required. This is typically focused around the automated metrics collection and analysis, and “outlier detection”. Humans are typically very good at eyeballing a series of dashboards and deciding whether a new canary is performing positively or negatively. However, outside of the obvious cases, such as a service crashing or a massive increase in latency, it can be challenging for a computer to determine whether a canary rollout should be continued or halted.

Another often underestimated problem is identifying the traffic and data thresholds that are sufficient to make a decision. During a canary release, every tick of the clock provides incrementally more information about the canary. At some point, though, enough data has been gathered and a decision should be made. How exactly do you identify that point?

In the real-world, we’ve encountered very few organizations who have successfully implemented fully automated canary testing, and I see several barriers to widespread adoption. The first is the lack of general adoption of sophisticated traffic shaping systems such as Ambassador, although the prevalence of this issue is rapidly decreasing as organizations adopt more cloud-native infrastructure. The second is the actual metrics analysis technology, which tends to be domain-specific, and is typically a challenging engineering problem to implement successfully. The final barrier we see is cultural — getting organizations to accept the notion of testing in production, and trusting automated systems to get it right.

Alternative Patterns

Not every organisation follows the evolution described above. For example, some opt to begin their journey with the semi-automated canary. Others encounter different challenges, and may instead adopt some of these alternative patterns.

Not Comfortable Testing In Production? Use a Shadow Canary

Several of the organisations we have been working with are not yet fully comfortable with testing with production traffic, or can’t do this for regulatory or compliance reasons. An alternative pattern that still provides some of the benefits is the “shadow canary”. With this approach you can either deploy your canary service into production or within a staging environment, but you shadow or “mirror” a percentage of production traffic to the canary. Although you observe the canary, you do not return any of the results to the end user, and all of the related side effects are enacted on non-production systems or are mocked or implemented as no-ops.

With this pattern 100% of the production user requests are served by the original service running in production at all times. This often satisfies concerns around testing in production, but in some cases you may also have to anonymize data within user requests before it can be sent to a non-production environment. Ambassador Pro supports the addition of custom “filters” to the request handling process, which can be used to implement data cleansing and transformation.

Upon a successful shadow canary release test, you can then deploy and release the newly canaried service using your existing processes.

Advanced Shadowing: Tap Compare

You can implement a shadow canary in a manual, semi-automated, or fully-automated manner. The fully-automated approach enables the use of additional techniques such as “tap compare” testing, where you can verify that the response of a new canary candidate does not deviate inappropriately from the existing version (often with routing to an additional secondary instance of the existing service in order to identify the “non-deterministic noise” in the response, such as UUIDs and dates).

Diagram showing how Diffy operates (taken from Twitter Engineering Blog)

This is an approach that was popularised via Twitter and their Diffy tool, and there are also additional implementations such as Diferencia.

Limited by Traffic Volume? Use Synthetic Canarying

It is not uncommon for organisations to have important services that either receive a small number of user requests or a small amount of a specific request, and therefore canary releasing isn’t an effective testing strategy. An alternative pattern that helps in this situation is the “synthetic canary”, which relies on automatically generating synthetic user requests, or “synthetic transactions”, that are sent to the canary.

This pattern is often combined with the “shadow canary” pattern discussed above. You can deploy a canary release into production or a staging environment, and only allow access to this from requests with a specific header set or with a specific authentication/authorization — i.e. no production traffic will be sent to the canary. You can then run a series of tests, synthetic transactions or a load test that targets the canary release, and observe the results. This can be a manual process, or you can semi- or fully-automated this.

Conclusion

This article has presented a whistle-stop tour of how I see organisations implementing the canary releasing of functionality with the Ambassador API gateway. There is no single “right way” for beginning the journey to testing functionality with real user traffic, but there are definitely several gotchas to watch out for, such as the need for the prerequisites mentioned. The Datawire team are working on related solutions with a number of Ambassador users, and so please get in touch if you want to share any experience of comments with implementing automated canary releasing.

As usual, you can also ask any questions you may have via Twitter (@getambassadorio), Slack or raise issues via GitHub.