Identifying CRO Opportunities and Prioritizing A/B Testing Ideas — Review
By: Poli Dey Bhavsar
Tags: Google Analytics, Conversion Optimization, A B Testing, User Research, Heuristic Evaluation

Are you an optimizer wondering “what should I test?” Well, you’re not alone in this. Many people begin with this question when it comes to conversion optimization. Good you landed on this page because you won’t ask this wrong question again and as this blog would tell you exactly how to begin with your optimization efforts.

Image: Pondering, “What Should I Test?”. Not anymore with CRO minidegree at CXL

In this blog, I am going to share (on a high level) my learnings from the Conversion Optimization minidegree at the CXL Institute about how to identify conversion optimization opportunities, prioritize them and then do A/B testing.

So, there are basically three parts for us to discuss here, research, analysis and testing. Research can help you form a strong foundation for your experimentation. As Peep Laja aptly says,

“CRO is 80% research and 20% experimentation. You need conversion research to get sustainable results over time. Learn this or fail.”

Collect data & stop guess work

Therefore, it’s time you stopped guessing and started digging into your Google Analytics, Google Tag Manager to identify areas on your site that are leaking conversions. In case your GA account is not properly set or you’re not sure if you’re getting accurate data from it then I would recommend you to do yourself a favour and enrol in CXL’s The Google Analytics Audit course by Fred Pike, Managing Director at Northwoods. This can be complemented with a qualitative user research to discover the areas in your site that you can improve.

Analyze your CRO opportunities

Now, let’s move on to the analysis part. Here, I would like to share with you my learnings from the course Heuristic Analysis Frameworks for Conversion Optimization Audits by Andre Morys, CEO and Founder at KonversionsKRAFT.

In this course, you would learn about how to conduct proper heuristic evaluation in less than an hour. And you can enhance the benefits of this evaluation with persona development. Wondering how would personas facilitate heuristic evaluation? Well, they’ll make it easier for you to put yourself in your ideal customers’ shoes and see the website (subject to analysis) through the customers’ eyes.

About the framework

This 7-level framework by Andre Morys when compared to existing models not just tells you the why behind the data but also helps you focus on subconscious mechanisms. Yes, “data you measure on your website is the result of user behaviour,” says Andre Morys. “So, if you understand user behaviour, then you can understand the data.”

Image: Heuristic evaluation framework by Andre Morys

Cognitive biases

He explains many cognitive biases that if we understand them it would be easier for us to understand why users behave the way they do. Some of the cognitive biases you learn in this course are confirmation bias, decoy effect, contrast effect, social desirability bias and more. Find the exhaustive list of cognitive biases here.

The ROI pyramid

He also shows us the underlying mechanism that makes a difference between mediocre and great products.

Image: Increase the ROI of your website by focusing on user motivation.

The must-haves in the image are the hygiene factors that people expect in your product or website. They are not fascinated by them. They’re supposed to be there anyway.

So, as a product owner if you want to create great products that users would feel motivated to buy then you must focus on user motivations instead of the hygiene factors or the must-haves.

Competitive benchmarking

One of the best things that you learn from the course is how to get buy-in from the C-suite executives in your company. You can use the framework to analyze your competitors website also along with yours.

Then show your findings in a form of graph such as why they are better than you and how you can also reach there. Even, instead of pointing out designers or developers, etc. what they’ve not done right, you can simply show them how competitors are doing it better.

Bring more credibility to your evaluation

You can work with different raters, say in a team of three raters. Thus, you can raise the validity of an expert evaluation.

Now, from this evaluation as soon as you have many hypotheses, start prioritizing them.

Next, let’s move on to testing.

Why testing?

“Testing is for validation and learning”. So that we no longer rely on gut feeling or intuition.

A significant fact about A/B testing is that you don’t know why you’ve a winning test. So, don’t fall in the trap of your conscious mind or rational mind to create a story behind the success of your winning test.

You can use the PXL framework by CXL as it offers the most objective way of prioritization. For A/B testing statistics you can use the A/B testing calculator by CXL. This would help you not fall prey to myths about sample size, statistical significance, etc.

2 Important criteria to keep in mind while giving a score:

Ease of implementation Opportunity of getting a big lift.

In the minidegree under the Testing track there is a comprehensive lesson on “How to Run Tests?” I would suggest you enrol for it so that you never ever get stuck with the question, “What Should I Test”.

Now, let’s put a quick glance on testing strategies!

Peep Laja gives a very straightforward answer to the eternal question “What to test?”

“Identify problem not ideas”

So, he says,

Perform conversion research to identify problems

Determine urgent user problems

Test possible solutions

Next he answers your other question in the course, “How many changes per test?”

Strategy #1 — Making many changes. Peep recommends one thing at a time unless you have millions of visitors. Just make sure each change that you’re making is directly connected to an identified problem.

Strategy #2 — Making many changes. Suppose your hypothesis says “Clarity is obstructing optimization” then you make say 75 changes across the website that would improve clarity. So, each change on your website supports the same hypothesis.

Let’s have a look at a few types of testing.

A/B testing vs. Multivariate testing

In a nutshell, A/B testing is ideal for testing dramatic changes whereas multivariate is good for testing interaction effects provided you have more that 100,000 visitors per month.

Iterative testing

You can do iterative testing when you want to attribute a specific change to a result. It’s easy to start as it’s low budget. It can help you build momentum for a A/B testing program. Moreover, you can test any random idea, suppose one that comes from the C-suite and you can prove that it might not work all the time by testing. As it’s low budget you can build the test multiple times and learn from hits and misses of each iteration.

Innovative testing

When iterative testing is not working, you make larger changes (more likely to have an impact on user behavior) on a portion of your website to move the needle, i.e., have an impact on the conversion rate.