What’s the Definition of Success for a Success Metric?
By: Sari Harrison
Tags: Metrics, Product Management, Innovation, Technology

What is often the case for success metrics is that they skip the value for the user and jump right to the value for the business. AARRR. Engagement. Usage. Revenue. Clicks…

It’s a natural tendency in a world where everyone assumes that business success means you are providing the value you intend. Fine. But let’s test whether these metrics meet their own goals as stated above.

Do business metrics meet the goal of hypothesis validation?

There are two dramatically different worlds here. The world where features are A/B tested and the world where they aren’t (or can’t be).

Where you can and do A/B test, it’s natural to assume the answer to this question is yes. You test the new checkout flow of your eCommerce site and you get a definitive answer to whether it increases revenue. Love it.

But you don’t know if the flow you decided not to implement would have performed even better. You can only test what you have decided to implement. See answer to prioritization below.

Where you can’t A/B test, even if you were right about the impact, you wouldn’t know it. In this world (where I lived most of my career), it is not possible to attribute any improvement in these metrics to any one thing.

It could have been the marketing push that was done and has nothing to do with the new features. It could have been some features from the last release that are finally taking hold in the market. Competition could increase or decrease. And so on.

You don’t release one feature at a time and then wait patiently for a few months to see what happens to your metrics.

You might argue that if the sale goes through as planned, then you have validation. But I was in B2B for 7 years and I know that there are many other reasons the sale could have gone through.

Instead, you need to rely on more feature specific metrics fro validation in a world of no A/B. Typically, that’s usage. There could be specific revenue associated with the feature. But again, you don’t know how much more usage or revenue would have been generated by the features you didn’t do.

Answer: Maybe

Do business metrics meet the goal of inspiring ideas?

When I first joined Apple Maps in November 2014, I was handed the feature list for the year. Assuming we all agreed that each feature had a clear reason for existing, I went about finding out what those reasons were. “Why are we doing feature [feature name]?” I would ask. This was an invalid assumption on my part and therefore not a popular question, but that’s another story.

Apple Maps Nearby Feature

I asked the DRI (Directly Responsible Individual) of the proposed “nearby” feature this question and he said “we want people to use the Maps app every day”.

I said, “let’s give them a cookie every time they launch it then”.

Was I being snarky? Certainly. But you get the point. Never mind that Apple would almost certainly not A/B test prominent client UI like that. So we wouldn’t be able to validate that definition of success.

This is my biggest fear about using business metrics as success metrics. You stop talking about value and you start talking about cookies. Or whatever your equivalent of cookies is.

The only reason a truly good idea (user and business value is delivered) is inspired by business metrics is because the person is actually thinking about user value. It’s coincidental.

A slightly deeper reason for the nearby feature was that our data showed that people didn’t often type category searches. The hypothesis was that this was because they didn’t know they could and the buttons would teach them. That hypothesis turned out to be incorrect.

One of our user value goals in Apple maps is to help people find out what’s around them in order to decide where to go eat, shop, get gas, whatever it is that they need. From that perspective, the feature made some sense. But it wasn’t the inspiration for Nearby. And it probably would not have been the one that we picked had we started from value instead of usage.

In hindsight, the number of copycats that this feature inspired (including Google Maps) is a potentially interesting success metric. But we would have had to wait a long time to know if it had been met.

Answer: No

Do business metrics meet the prioritization goal?

When you are focused on the “business” or “vanity” metrics, you can end up in a game that may sound familiar to some of you. People making s**t up about how many users their feature is going to acquire or how much revenue will be generated.

When I was in Bing Ads this happened to me. Every feature being proposed in our yearly planning exercise required a spreadsheet that estimated how much money was going to be made. Some of the entries were laughable. And all of them were wrong. This just isn’t knowable up front. Can we all admit that?

We don’t know what percent of users is going to use the new service. We don’t know how this feature is going to increase subscriptions. We don’t know how it will improve referral rates. That’s all complete BS. We are guessing.

Educated guesses? Maybe. Maybe not. When you get attached to your idea being green-lit as validation that you are doing a good job… Probably not.

In the land of B2B, “y” customers may have said they were going to sign if you have feature “x”. But it could be that none of them do.

Sales really wants the feature and you can’t control the reality of what they promised it will do. No matter how hard you try.

You have said no to a supposedly promising feature because you knew somehow it wasn’t the right thing to do. Haven’t you? I certainly have.

Answer: No

OK, what about unmeasurable success criteria

So how well do my proposed “unmeasurable” success criteria fare?

Prioritization: better than business metrics

My favorite framework for prioritization involves rating features against their ability to impact weighted goals that relate to product strategy.

Similar to business metrics, it’s very hard to know what the impact will be. But because there are usually more of them and they are closer to what you are actually doing, it is less guesswork.

For example, you cannot argue that the Nearby feature has any impact on the freshness or accuracy of data or the user’s ability to navigate while driving. It does have some impact on planning and it has a lot of impact on finding a place to go. Depending on what goals we have decided need the most focus right now, Nearby makes the cut or it doesn’t based on this.

And this discussion has the added bonus of creating alignment and reenforcing the vision and strategy. The conversation around lift of 2% versus 3% is not going to do that as well. Because it’s one step removed from the why.

Inspiring ideas: way better than business metrics

Do I even need to elaborate on this?

Hypothesis validation: draw

You need both.

Of course, you still A/B when you can to make sure the impact to the business metrics isn’t negative.

The best way to use unmeasurable success criteria for validation would be to survey users. Simply ask them if the product is delivering the value you wanted to deliver. But that’s a delay that some people aren’t comfortable with.

Watching social media can also work — a bit less accurate perhaps, but faster.

The ultimate validation though, is the business metrics. But they are trailing indicators and as already said, have way more variables associated with them than just the features you ship.

Winner: Unmeasurable Success Criteria!

Just kidding, this isn’t really about winning or losing 😉. But I hope I’ve made my case that unmeasurable success metrics are valuable and should be an active, daily conversation.

Use them to complement your existing focus on the business metrics and you will be on your way to building your business and making the world a better place.