Conversion Research in a Nutshell
Tags: Marketing, Conversion, Google Analytics, Growth Marketing, Conversion Optimization

Review of key steps & essential statistics for A/B testing

What does conversion optimisation start from?

CRO is a data-driven process, which means that we need data in the first place.

How do we collect this data?

There are a few approaches you can choose from. But I’ll focus on the ResearchXL Framework, which I discovered in one of the courses at the CXL Institute.

There are 6 key steps in this approach. I’ll give an overview of each so that you can see if it works for you.

Step 1. Technical Analysis.

You can never be sure that your website or app performs perfectly fine on all kinds of browsers and devices. That’s why cross-browser and cross-device testing is a good way to begin your research.

Create custom reports in Google Analytics (or on any other analytics platform you’re using) to see:

Conversion rates per device (mobile, desktop, tablet)

(mobile, desktop, tablet) Conversion rates per browser (type as well as version)

Custom reports will show where the problem is and then you can find out what actually the problem is. It’s also important to do speed analysis, that is check how much time it takes for your website to load.

Step 2. Heuristic Analysis.

When auditing a website for user experience, there are four main things to look at on each page:

Clarity

Relevancy

Friction & anxiety

Distraction

Motivation & incentives

Clarity includes 2 components: content and design.

Content clarity allows users to understand what the page is about, what actions they can take on it, why it is useful, how it works, what the product is, and what they should do next.

Design clarity comes down to the visual hierarchy on the page, the relevance of the visuals, the degree to which they support the content and call to action, the clarity of the eye path, and ease of visual navigation.

Relevancy deals with user expectations. Here we evaluate whether and how the page meets users’ expectations and needs contentwise and designwise.

Friction & anxiety are about all kinds of doubts, concerns and insecurities users might have on certain pages. There might be various sources of friction such as long and complex processes, slow page load, poor website design, requests for sensitive information, technical errors, usability issues and others. This is something you’ll need to find out during your site walkthroughs.

ONE PAGE –> ONE GOAL. It’s crucial to define one most wanted action for each page. If there are more, users will get distracted. Distraction on a given page can be caused by moving and blinking elements, items irrelevant for the call to action, extra, unnecessary elements, the wrong hierarchy of visuals and many others. Try to get rid of all distraction elements.

Evaluating a page for motivation & incentives focuses on the value proposition, its clarity, features translated into benefits, sufficient information about the product, and persuasive copy.

Step 3. Digital Analytics.

Time to look at your analytics data! Inspect your funnel, identify drop-off points causing leaks, and try to correlate user behaviours with outcomes. Your analytics is a really powerful tool!

Important! Before drawing insights from the data, you should verify it. Make sure your Google Analytics is set up correctly. Here is a Google Analytics health checklist.

Step 4. Qualitative research.

Once you know the numbers, you can turn to qualitative data collection. Depending on your problems and needs, you can go for on-site polls, follow-up surveys, and live chat transcripts.

You can learn more about qualitative research for conversions here.

Step 5. User testing.

User testing is a great way to discover how users interact with your website. You can observe how people understand what your site is about, their journey on the website, the ease of navigation, and the issues they encounter in the process. If you want more details, here is a great guide on usability testing.

Step 6. Mouse tracking analysis.

Mouse tracking provides information on where people click & where they don’t as well as how far down they scroll on a page.

When you collect this data, remember about the sample size. You need to have 2,000 visitors per analysed page (300 is the absolute minimum), 5,000 is even better.

Learn how to create click and scroll maps here.

So, you collected your data, identified problems, came up with hypotheses and ideas for testing. What’s next?

You run tests.

But before you do that, it’s crucial to know statistics fundamentals to reduce error and increase the validity of results.

If you need a comprehensive easy-to-understand guide into A/B testing statistics, check this one. I’ll outline only my main takeaways from the course I took this week.

Here are a few things to remember whenever you do A/B testing:

Don’t stop your test too early. Ensure the minimum duration of the test to avoid sampling error. Try not to test too many variants as low error probability is valid only in the case of one test variant. Of course, you can go for multiple variants but you’ll need to do additional calculations. Prioritise main KPIs BEFORE you start testing as there is a connection between the type of KPI and the required test size.

If you’re particularly interested in conversion rate optimisation, you can also try this free course on CRO foundations.

Clearly, the courses I took provided me with much more content than described here. There was lots of practical material that can’t be briefly presented in a blogpost. My journey in Growth Marketing at CXL is getting a bit more challenging every week as courses are comprehensive and require hours of diligent study and concentration.

What I really like is all the spreadsheets and cool digital marketing tools I learn about in the courses and can immediately apply at work. You aren’t fed with pure theory. They actually show you on real cases how to implement the material and the digital tools covered in the course. If you got interested, here are free marketing courses you can look at.

Busy days for me. But I’m moving onwards!