Continuous Delivery and Automated Canary Analysis using Spinnaker and Prometheus
By: Eric Irwin
Tags: Canary Deployments, Continuous Delivery, Engineering, Prometheus, Spinnaker

At Quantum Metric, we have started rolling out Spinnaker as our primary tool for Continuous Delivery.

With the rapid growth of our customer base and the teams contributing to our platform, it has become more important than ever to ensure that we are continuing to deliver value quickly and reliably.

Per the Official Spinnaker Website

Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence.

Though Spinnaker was originally created by Netflix, it has accumulated a strong community of supporters including Google, Amazon and Microsoft when it was made open-source.

While many factors were considered when choosing Spinnaker, one specific feature we were especially intrigued by was the native support for running Automated Canary Analysis (ACA) within our deployment pipelines during a canary deployment.

Simply put, a canary deployment allows us to partially roll out a change against a subset of traffic so that it can be evaluated against the current deployment to ensure it is operating as expected.

Spinnaker uses Kayenta to query one or more metric stores (e.g. Prometheus, NewRelic, SignalFX) against a current instance of the service (the baseline) and a new version (the canary) and then runs statistical analysis to automate the comparison.

At the time of this writing, Quantum Metric uses both Prometheus and SignalFX as metric stores for ACA, however this article will only focus on Prometheus.

Overview

The purpose of this article is to share how we are leveraging ACAs with Kayenta and Prometheus to increase confidence of our production deployments.

However, we will not be covering specific details for configuring Spinnaker, Prometheus, or digging into best practices for running ACAs as much of this can be very contextual.

The links below provide more details on the above:

The Service

With some minor details omitted, the service we will be referencing is our Resource Manager service which is responsible for enforcing resource integrity for specific data within our platform.

It has simple, but well-defined Service Level Objectives (SLO) around Availability and Latency that serve as excellent indicators we can use for canary deployments.

In other words, during a canary deployment, we will automate the analysis of the following Prometheus metrics supporting these SLOs to understand the risk of a deployment before it is fully released by comparing behavior between the baseline and canary deployments.

# Availability

resource_manager_request={status="success"} # Latency

resource_manager_request_duration{duration="bkt_0_to_50ms"}

We will come back to these metrics when we discuss configuring the canary analysis.

Canary Deployment

Before we discuss configuring canary analysis, we need a pipeline that will automate the deployment of a baseline and a canary version of the service.

To recap, the baseline should use the same version (and configuration) that is currently running in production, whereas the canary is the new version.

To read more about why this is preferred rather than running comparisons against production, read Canary Best Practices.

Additionally, we want to ensure that our pipeline is configured to tear down these deployments following the automated analysis and judgement has been made on whether to continue the release or not.

The following image reflects the different stages of our canary deployment pipeline. For simplicity, early stages responsible for baking our Helm charts and deploying to development environments are omitted.

Canary Deployment Pipeline

First, a brief explanation of each of the stages in this pipeline:

Configuration — Manages parameters and other configuration used by the downstream stages such as the Docker Image tag that triggered the deployment pipeline.

Manages parameters and other configuration used by the downstream stages such as the Docker Image tag that triggered the deployment pipeline. Find Baseline — Retrieves the current version of the service running in production for use in downstream Deploy Baseline stage.

Retrieves the current version of the service running in production for use in downstream stage. Deploy/Baseline Canary — Deploy the two versions with identical replica counts for comparison during Canary Analysis.

Deploy the two versions with identical replica counts for comparison during Canary Analysis — Run ACA against the previously deployed baseline and canary versions.

Run ACA against the previously deployed and versions. Canary Judgement — Perform manual judgement on the results of the ACA to determine whether to continue or not. Note that this stage is not necessary and execution can be determined directly based on the results of the Canary Analysis stage.

Perform manual judgement on the results of the ACA to determine whether to continue or not. Note that this stage is not necessary and execution can be determined directly based on the results of the stage. Delete Canary/Baseline — Tear down the previously deployed baseline and canary deployments.

While there are more advanced traffic routing patterns and policies that can be used when running a canary deployment (see Istio), we are just using Kubernetes load balancing to route traffic between the current, baseline and canary deployments.

This gives us the ability to determine the amount of traffic we want to send to the canary deployment using simple math.

For example, if we decide that we want to route 20% of traffic to canary deployments, and there are 3 replicas currently running in production, we would calculate the target replica counts as follows:

3 (current) + 1 (baseline) + 1 (canary) = 5 total replicas 1 (canary) / 5 (total) = 20% of traffic when load balanced

Canary Configuration

Before a canary stage can be added to a pipeline, a Canary Configuration must be defined where we configure the following attributes as input for our Canary Analysis stage.

Specific metrics and groupings for evaluation during analysis.

Scoring thresholds and weights for metric groupings.

Reusable metric filter templates to provide reusability.

The configuration we are using is relatively straight forward, and uses a single Request Check group, though you can leverage multiple metric groups to configure specific weights that are applied when scoring. To learn about more advanced configurations, read through Configuring Metric Groups and Weighting.

Here is the canary configuration we will be using:

Canary Configuration for Resource Manager

The image above reflects that we have a canary configuration with a single Request Check group consisting of two metrics, Request Success and Request Latency, and a default weight of 100.

Let’s dig further into each of these metrics…

First, we configured the Request Success metric as shown below:

Metric Configuration for Request Success

Some specific details for how this metric is configured:

Fail on — Since this is directly tied to our Availability SLO, we are configuring this to only fail when it is determined that there is a decrease in success rate when comparing the canary and baseline deployments.

Query Type — We are using PromQL so that we can specify the templated Prometheus query we want to run directly.

Template — We provide the PromQL query template here that will be used to retrieve the metric that is being analyzed during Canary Analysis. The template includes ${scope} and ${location} which are expressions that will be interpolated at runtime in order to differentiate between the baseline and canary deployments. These will be covered in more detail in the later sections.

Similarly, the configuration for the Request Latency metric looks the same, but runs a query specific to the resource_manager_request_duration metric.

Metric Configuration for Request Latency

For more, visit Canary Configuration

Configuring Canary Analysis

Now that we have our resource-manager canary configuration, we can continue to the Canary Analysis stage for our deployment pipeline, which is configured to perform canary analysis for our deployment using the Request Check configuration.

Canary Analysis Configuration

While our resource-manager canary configuration specifies the metrics, groupings and default scoring thresholds, the Canary Analysis Configuration provides the ability to actually configure how to run canary analysis within our deployment pipeline.

Here are some details for how we are configuring our Canary Analysis stage.

Analysis Type — Configure the analysis to run in real time or against a specified time interval retrospectively. Real Time is perfect for what we need.

Config Name — We are referencing the resource-manager canary configuration.

Lifetime — The total duration in which we will run this Canary Analysis stage. Generally speaking, the longer you can run this, the more data points that can be considered, though it is certainly contextual. In our case, 30 minutes is more than sufficient due to the high volume of the service.

Interval — The frequency at which a new canary score is generated.

Baseline/Canary Location — Used to identify the location of the control (the baseline) and the experiment (the canary) and directly maps to the ${location} expression previously mentioned in our PromQL template. In our case, both of our deployments will be located in the resource-manager Kubernetes namespace which will map to the namespace label on our Prometheus metric.

Baseline/Canary — Similarly, these are used to identify the scope of the control and experiment and directly map to the ${scope} expression used in our PromQL template.

This will result in the following Prometheus queries being run as part of the canary analysis:

# Request Success

sum(delta(resource_manager_request{result="success", pod=~"canary",namespace="resource-manager"} [30s])) sum(delta(resource_manager_request{result="success", pod=~"baseline",namespace="resource-manager"} [30s])) # Request Latency

sum(delta(resource_manager_request_duration{duration="bkt_0_to_50ms",pod=~"canary",namespace="resource-manager"} [30s])) sum(delta(resource_manager_request_duration{duration="bkt_0_to_50ms",pod=~"baseline",namespace="resource-manager"} [30s]))

Configuring Analysis

We now have everything we need to run a full production deployment that will run an ACA across these metrics to help us better decide whether or not to proceed releasing it to 100% of traffic.

Once the deployment pipeline is triggered, and the baseline and canary deployments are successfully running, our Canary Analysis stage will run analysis and calculate a score for the within the provided interval.

We have configured analysis to run using five minute intervals as shown below.

In Progress Canary Analysis

One important capability to note is that you can configure the canary to fail if any of the individual runs fail. This will ensure that if there is an immediate impact, that it does not need to run the full Canary Analysis stage before we can tear down the problematic deployment.

Successful Canary Analysis

If needed, we can drill into any of these individual canary runs to understand how the baseline and canary compared and were scored across both our Request Success and Request Latency metrics .

Canary Analysis Report

Once the Canary Analysis stage is completed, the pipeline is configured with a Canary Judgement stage that allows for any additional analysis to be done and provide manual intervention if needed. Even though this is not necessary, in many cases it can still be useful.

Canary Judgement

Since our Canary Analysis succeeded with a score of 100, and there is only 0.2–0.3% deviation between the baseline and canary, we can feel confident in proceeding with rolling this out to 100% of traffic, and tearing down those deployments.

In the event one or more of the metrics had failed, we would have seen something similar to the following reflecting a significant deviation of one of the metrics.

Failed Canary Analysis — Low Request Success

The image above is specifically reflecting that there was a deviation to Request Success in the canary which is indicative of a potential issue in the new version.

Summary

So there it is!

From a high level, we have shared how we are using Spinnaker, Kayenta and Prometheus to run automated canary analysis to increase confidence during deployments when delivering value to our customers.

We are really only in the early stages of fully utilizing the capabilities of what Spinnaker has to offer, but are excited to see increased adoption across Quantum Metric and know it will further enable us on the path towards Continuous Delivery.

Working with Spinnaker has been both challenging and fulfilling for our team, We have leaned a lot on the community, as well as some great tooling provided by companies like Armory (check them out, they are doing some great work!) to get to this point.

Our Delivery Engineering Team is growing quickly!

If you are passionate about Continuous Delivery, Observability and enabling those around you, definitely check out the opportunities we have available, as well as other open positions at Quantum Metric!

References