How Not to Prioritise Growth Experiments
By: Matt H. Lerner
Tags: Startup, Marketing, Lean Startup

How Not to Prioritise Growth Experiments

Product vs Growth Sprints

The Startup Growth Process differs from Lean Product Process in some important ways. A big one: How you prioritise work. In Lean Product, you traditionally prioritise based on I.C.E. (Impact, Chance of success, Effort). But on the growth side, that’s actually quite a bad idea.

From First Principles

Let’s think this through from the very definition of breakout growth: Can you find a way to grow really big really fast ahead of all competitors? If you do the same stuff as everyone else, you cannot expect different results by dint of a few late nights or better execution at the margins. You need to find a game-changing approach.

Every great startup case study has found a special “hack” or three. (PayPal, AirBnB, Facebook, Instagram, Hubspot, etc.) And every one used the some variant of metrics-driven process of rapid experimentation. Most experiments will fail. But occasionally one will have a huge impact. So you need to test lots, and you need to test fast. And you need to test stuff that can be huge.

You’re always constrained in the number of tests you can run — by traffic, engineering, cash, etc. So each test cycle is precious.

The difference between the 1% of startups that have huge success and the 99% who do not, comes down to which stuff they choose to try. What’s the best way to decide?

First, Don’t Run a Democracy!

Since this is a very important decision… avoid voting. (Surely you see the sorts of leaders we get when we let everyone vote?) Instead, the decision should be made by an expert who is personally accountable for the results. (Note: That expert is not necessarily the ranking executive! I’m not advocating HiPPO). Instead, find someone close to the customer experience, the numbers, and the resource commitments required to deliver. She could be a founder, head of growth, with a product, engineering, analytics or marketing background. (For more thoughts on how an organisation can make the best possible decisions, Ray Dalio’s Principles is the best book I know. And experiment priority really is an investment decision, nothing more, nothing less).

The Problem with Effort vs. Impact

If you take a list, score each item from 1–5 on effort and impact, you end up testing a bunch of things that are “easy” (effort = 1) and moderate impact. Because most of the “big” impact ideas are “big” effort, you end up doing a bunch of small and medium stuff, and wishing you had the resources to do the big stuff.

If you only test small things, you cannot have big results. And you can only grow insanely fast if you try high-impact stuff.

No great startup was built on a handful of incremental improvements around the margins. So you must do the high-impact experiments with huge binary outcomes. And if you only look at “low effort” ideas, you will ignore most of the big ideas, and squander your extremely limited resources on medium-sized stuff.

A Better Approach

When I work with teams, we score on “effort” and “impact.” But we start with the biggest ideas (regardless of effort) and dream up ways to test them with small quick experiments. That’s the secret — small tests of big ideas.

To be clear, it’s fine to score growth ideas based on effort and impact. And it’s fine to run some “low effort medium impact” ideas as resources allow. But do not disregard the “high effort” ideas. Instead, find a quick cheap easy way to test them!

For Example

The AirBnB founders thought better photography would improve rental rates. But sending professional photographers to every unit on their site would have been too complex and expensive. Instead, they grabbed their cameras and spent a weekend photographing only the listed flats near their base in New York City, to see if that helped conversion, relative to other properties. (It sure did!)

The tricky important bit is to design great experiments: First, for each idea, you need to identify “what are the riskiest assumptions?” Next, what is the quickest easiest way to test each of those assumptions? That’s it — take all your biggest ideas and find ways to test them quickly.

Want more experiment ideas and examples? Grab my free Playbook PDF and you’ll get the option to subscribe to my weekly email. It’s short and helpful, like me. 😏

What about “Chance of Success”?

Chance of success… for the big ideas, the chance of success is pretty much always low (even though it doesn’t seem like it to the person who thought of the idea!). And the only way to really gauge the chance of success is to run an experiment!

Concrete Next Action For You

Go back through all your old ideas and find the ones that seemed like they could be huge, but were too hard, slow or expensive. Pick the ones you’re most excited about, and dream up some quick simple ways to try them out. Then, commit to run 3 experiments in the next two weeks.

I wish you the best of luck!