Perfect pacing: the secret to great product OKRs
By: Ed Biden
Tags: Planning, Management, Impact, Delivery, Product Management

Lots of product teams use OKRs. Few enjoy the experience. In large part, I blame the myth that product teams should be set a success metric and then set free to improve it. Sometimes, that’s the right way to do things, but in my experience, often not…

Principles

Before we dive into what makes great product OKRs, let’s go back to basics. What are we hoping to get out of planning and OKRs in the first place?

Clarity around what we want the team to achieve (the Objective) Visibility on how the team is pacing towards the goal (the Key Results) Flexibility around how the team achieves it (the initiatives)

Once OKRs are set, these three ingredients facilitate productive and enjoyable conversations between product teams and leadership about how to reach the goal faster. They focus the conversation on how well we understand our users, what ideas we’ve got, and how we can shift resources to exploit opportunities we see. Good quality conversations like this accelerate the team, and increase the chances of them hitting the goal.

Without them, progress check ins are awful, because we’re always arguing about what we’re trying to achieve, how close we are to getting there, or mindlessly executing the ideas of the leadership, regardless of how good they are. Bad quality conversations like this waste time and energy. They slow the team down and decrease its chances of reaching the goal.

Getting clarity (1.) and flexibility (3.) right is fairly simple: you define your Objective in your OKR as specifically and measurable as possible, and let the team work out what it needs to do to reach this. Finding the right indicator to measure pacing (2.) is more difficult.

Ideally you would have a metric that told you how close the team was to the goal, and if the team was on track this would increase steadily through the quarter to 100%. “Perfect pacing” would be where pacing towards the goal equals progress through the quarter. So 20% of the way through the quarter, the team will be 20% towards its goal. If we’re only 15% of the way towards our goal, we need to speed up to reach it by the end of the quarter.

This ideal state would be would be great — we’d always know exactly whether the team was on track or whether we needed to course correct. However, finding an indicator that is even close to perfect pacing is difficult, and it’s often not the same metric you hope to move by reaching your objective. Don’t confuse the two.

Timing of impact from product work

So, what stops the metric you use to measure the impact of the team also acting as a measure of pacing? Well, it all depends on two key variables:

How often you ship changes How fast you see impact after those releases

If you can ship multiple times a day and see results instantly, fantastic — set the team a success metric and leave them to it. But often, despite the best will in the world, this isn’t the case. For example:

The features you are building require 1–2 weeks work

You need to wait for the next release train of your mobile app

You need to wait for significance in your AB test

In these sorts of cases, impact lags delivery, and in a quarter (let’s assume this is your planning cadence) you might only have 5–6 releases.

If you charted out the releases for a typical product team, and when you saw the impact from the releases they made, you might see something like this:

Releases (blue circles) correspond to impact (red circles). For simplicity, each feature takes 2 weeks to build and release. Let’s assume the team is AB testing each feature for a few days or weeks after release, which explains the lag between delivery and impact.

Release numbers:

Released week 3, impact in week 6: success! Released week 5, impact in week 11 (long AB test): ok Released week 7, no impact: fail :-( Released week 9, impact week 13: success! Released week 11, impact week 16: ok

This is a successful team — by week 12 (the end of the quarter), it’s released 5 features, and these go on to have more impact than we hoped for. This is only obvious some time later though. Let’s have a look as the quarter ends:

At this point, measuring the team on impact only gives the impression that the team has missed its goal significantly. It’s only released a couple of features that have had any impact throughout the whole quarter. The team appears to have significantly underdelivered.

Let’s jump back to how things looked in week 5 — well into the quarter:

At this point, if the team is only being measured against impact, things aren’t looking good. There have been a couple of releases, but with nearly half the quarter gone, there’s no impact to celebrate. Of course the team is convinced that the AB on feature 1 is looking good, but leadership have heard the same story for the past 3 weeks and are losing patience.

The central tension

Ultimately, of course, we want impact. Delivery means nothing if it doesn’t result in value for our users and the business. But we need a leading metric we can use day-to-day to pace the team, and we can’t be waiting weeks at a time without any hint as to how they are doing. Impact is all too often a seen in lagging metrics that shift much later, and does not fulfil this role.

Types of product work

So what’s the solution? Recognise that not all product work is similar, don’t confuse the Objective for the Key Results, and pick appropriate Key Results to pace yourself given the type of work you are doing.

Most teams I’ve worked with are doing one of three types of work: feature work, project work or experimental work.

1. Feature work

A team doing this sort of work understands its problem space well, and wants to build 3–8 well planned features in a quarter. The chances of each being successful is high, but not guaranteed. If we chart out delivery and impact, and compare these to a line for perfect pacing we see the following:

When selecting KRs for this team we could pick something purely delivery focused. But delivery, whilst it can be a good leading metric, doesn’t matter unless it generates impact eventually. So in this case we might select a mix of delivery and impact KRs. e.g.

Objective: Substantially improve the discovery experience for users Key Results: * Launch 5 major improvements to the discovery experience (50% of KRs; the team probably already knows what these are, but we’re not preventing them changing their mind if they uncover new insights in the middle of the quarter) * Increase conversion rate from 8% to 8.5% (50% of KRs)

2. Project work

Teams doing this sort of work are usually building out new products, or making substantial changes to infrastructure. It may be more than a quarter before the first release to users — though usually there will still be chunks of work that you can track delivery of. Impact metrics give you no indication of pacing here, as they will not move until an MVP is shipped.

In this case I’d suggest going with purely delivery metrics.

Objective: Launch a new mobile app on the App Store by the end of Q3 Key Results: * Complete interactive prototype * Complete account management and registration * Complete search experience * Complete product page * Complete check out * Launch Completed app

3. Experimentation work

Teams doing this sort of work are that rare exception where they are actually shipping experiments multiple times a week, and seeing the results of those releases within a few days. A team of experienced engineers working on a very early stage start up might be a good example here.

In this case we can actually use the metric we’re trying to influence as the pacing metric.

Objective: Grow conversion to purchase from 2% to 5% Key results: Grow conversion to purchase from 2% to 5%

Summary

To wrap up, setting OKRs is more than defining where you want to get to. It’s also about working out what’s going to lead to productive conversations along the way. Think about what will give you fast feedback about whether you are on the right path, whether that is impact, delivery or something else, and set your key results accordingly.