What you measure is what gets done
By: Donatas Stirbys
Tags: Product, Scrum, Agile, Product Development, Product Management

What you measure is what gets done

Photo by Charles Deluvio on Unsplash

It is in our nature to improve what we measure, so companies need to choose criteria that reflect the value it strives to create for the customers and for the business. Therefore, organizations still tend to focus on arbitrary metrics (like Scrum outputs) or just keeps churning features without realizing the impact of specific product increments by measuring company-level metrics only (etc. MRR).

Do not make scrum outputs your measurement of success

This is what I call a beginners mistake and to some extent, most companies make these blunders when trying to adopt agile into their daily life in the beginning. I’ve seen people who were too focused on monitoring the direct use of different processes and practices, for example, tracking velocity or the number of story points delivered. Improving the rate of delivered story points and delivering business value are two worlds apart. If you make your teams worry about story points on a daily basis that’s what is going to consume most of their attention and that is what your teams will optimize for.

If you are measuring your teams for the zero-bugs backlog, this is what the teams will optimize for too. More often than not fixing one bug introduces three bugs more. The company just not realizes this at the time, but eventually will. This easily can become a neverending cycle. Fix critical bugs only and focus on improving long term quality instead.

If you are keeping your product people accountable for outputs like super detailed stories or backlog being prepared much in advance — this is what is going to get done. If you still do not see a fundamental flaw in here I can point this out — by doing this the organization shifts internal values for the product specialist from maximizing the value for the customers to being the best backlog operator she can be.

Humans adjust behavior based on the metrics that they are held against. What you measure is what gets done. On the other hand, the main idea of Scrum is to maximize value as fast as possible. That’s why your company should strive to deliver small increments focused on improving a specific part of the product, then measure the results and evaluate how the business value increases with time.

Constantly measure and evaluate how business value evolves

Many companies still value the need for speed over the need for understanding if recently released product increment was actually viable. When a lot of features are shipped without measuring the impact, something, that the customers will end up using is bound to stick anyway. Though, all the features that are not going to stick are going to end up in the product too. Those unused features when creates extra complexity for development teams which when directly impacts the very one thing that the company is valuing — the speed of delivery!

When an organization runs like this for a long time it ends up with a low-quality product. For such a product, customer acquisition costs are high, and due to this, the sales department usually needs to put more effort in order to close the deals by pressurizing the price of the product. This also directly impacts the customer support department which is forced to double the efforts in order to care for complaining customers. When customers are frustrated this often leads to sales attacking the product organization with specific individual account requirements which in effect distracts product teams from actually focusing on delivering innovation and understanding the impact of each increment which directly leads to a low-quality product. It’s a virtuous cycle that is hard to break and the most costly way to fail for the company over a long period of time. If you do not take my word for it I am encouraging you to spend a few minutes and hear how software and usability expert Jared Spool explains the same idea.

Instead, companies need to focus on having a strong product strategy and measure the impact of every feature increment while trying to succeed with the product goals. One of the ways to do this is by shifting the focus from features to outcomes by adopting OKR’s (Objectives and Key Results) goals-based planning framework.

As a product organization, focus on the hard things instead

Your company can spend countless days improving the Scrum process, building and enhancing Jira dashboards, drilling detailed feature-focused roadmaps, and strengthening those with epic level burnup charts. All of this, to some extent, is also important, but people tend to spend too much time optimizing for better delivery because this is relatively easy to do.

Instead of focusing on mostly irrelevant measures of progress, your organization needs to focus more on extracting customer and business value. If the company is not ready for this now — prioritize fixing this first by slowly shifting the company towards outcomes-focused thinking. Rather than appointing features to development teams and measuring success in terms of how fast they can go, desired outcomes would need to be allocated instead and success should be realized by measuring if the goals were achieved or not.

If everybody in the organization is used to just churning features, this new way of work is going to be very uncomfortable for the company at first, though this is the most effective way to break the tragic virtuous cycle described before. Organizations need to embrace and encourage a lot of small failures over a big span of time from which the company will be able to easily recover. Moving towards outcomes-based delivery is a big organizational shift in terms of the mindset it requires. All of this is sure, is going to be hard to achieve, but this is exactly why your company should commit to doing this.