Measuring Reliability in GCP: Step By Step SLO creation guide using Cloud Operation Sandbox
By: Ayelet Sachto
Tags: Gcp, Slo, Sre, Reliability, Cloud Operations

SLO Creation Process

To ensure user happiness, you have to define it first. Our users are using our service to achieve a set of goals, the most important of which are the Critical User Journeys (CUJs).

After you have CUJs defined, you want to identify the metrics which most closely describe the user experience, these metrics are our SLIs. You will then use those SLIs to define our SLO targets.

1. SLO Process -Identify our CUJ

A few of the critical actions that users can do as part of shopping on Hipster Shop are:

Browse products, Check out, and Add to cart.

Next, you should ask yourself, which are the most important to the business? And use this as the prioritization criteria for CUJs.

Checkout is not the only critical user journey, but it is tied directly to the businesses revenue, so we have prioritized it first.

If you find it challenging to prioritize CUJs, try to put yourself in the user’s shoes. In our example, as a user, how many times have you browsed or added items to a shopping cart online without purchasing the items?

For the rest of this example, ‘Checkout’ will be the basis of the SLOs, but you could apply the same techniques to other critical user journeys.

The Checkout CUJ uses the following components of our architecture

2. SLO Process -SLI creation

After you identify the CUJ and the interaction you will focus on, you’ll want to choose which metrics will represent the user experience most accurately.

Given that our application, Hipster Shop, serves end-user e-commerce traffic, the users’ experience when performing different actions on the frontend during checkout should remain consistent. For that reason, we should define SLIs for request availability (how many requests are successful), latency (how long a request takes), quality, and other indicators.

When deciding what SLI types to use, I recommend reviewing the SLI Menu from The Art of SLOs handbook, page 7. The handbook has a handy guide for drafting SLIs depending on the type of interaction: request/response, data processing, or storage.

The next step is to define the SLI Specification. The SLI Specification is an assessment of a service outcome that you think matters to your users. You will want to represent those as a proportion of: the number of `good` events divided by the total number of valid events.

And lastly, you will refine that specification into a detailed SLI implementation by adding a measurement method, for example, Application-level Metrics, Logs Processing, Front-end Infra Metrics, Synthetic Clients/Data, Client-side Instrumentation (To read more, read: The Art of SLOs handbook, page 16).

In the case of the checkout CUJ, the goal is for the checkout functionality to maintain acceptable availability. To achieve that, you need to measure how many users tried to check out, and how many of those requests succeeded, so the number of successful requests is the ”good” metric.

It’s important to detail what specifically is going to be measured and where you are planning on measuring it.

Why are 3XX and 4XX status codes excluded? Since 3xx and 4xx indicate redirects and client-side errors, in this specific case, you can treat these as “the system working as intended”. Therefore you don’t want to consider these types of requests as errors against our availability SLO. Please note, that this shouldn’t be a generalization, you should be mindful of making this decision, and it is recommended to review these errors in your system to check if they represent a system failure.

3. SLO Process — SLO

After defining the SLIs, it’s time to set the SLO (Service Level Objective), the target for the SLI during a specified time window, for example, a month or a quarter.

The SLO should include a target that is represented as a percentage, a ratio of good/total, for example, 99.99%. In a specific measurement window, that window should be a duration that will allow us to make strategic decisions and prioritize reliability when needed (To learn more, read: Choosing an appropriate time window).

If you want to make changes to your system based on tracking these SLOs, it is important to set an achievable target. If your target is overly ambitious, the SLOs will quickly become an ignored nuisance. If your customers are happy with your service today, you’re probably doing OK, and you might want to base your targets on historical data that align with your users’ expectations. Those targets are not set in stone, and you should revisit those targets and iterate on them to align with the business requirements.

In our case, you can assume our SLO will be 99% availability, according to the historical data trends. If you do not have historical data, try to check competitors and benchmark for the type of journey in question to examine the user’s expectations. You might consider breaking up journey phases, for example, if a manual action is needed between interactive services versus report generation (For more: refer to the Buy-In currency CUJ in the The Art of SLOs handbook, page 20).

Step 4:

After you have defined your SLIs and SLOs, It’s time to turn those definitions into tangible dashboards and alerts. In GCP, our tool to implement SLIs and SLOs is Cloud Operations Suite.