Our A/B Testing Formula (The Easiest Way To Improve Performance By 2x Or More)
By: Josua Fagerholm
Tags: Digital Advertising, Advertising, Marketing, Digital Marketing, Facebook Marketing

Why Test?

The majority of ads will fail. So unless you’re expecting a neverending streak of luck, you’ll need a process for separating the losers from the winners.

That’s where A/B testing comes in. It’s simply the process of testing two or more ad variations against each other, analyzing the results, and doing less of what doesn’t work & more of what works.

It’s hard to overstate how important this is. We frequently see ads perform 2x, 5x or even 10x better than others. And the first ads are almost never among the top performers.

So if you’re not running at least one test at any given time, you’re leaving money on the table.

What To Test

First, you’ll need a bunch of copy angles and creatives. This is a huge topic in and of itself, so we won’t get into it here. Let’s just assume you have them.

Great! Where do you start?

You’ll want to go as BIG as possible with the first test. An example would be testing a professionally shot studio photo vs an unedited UGC (User Generated Content) video, or short CTA-focused ad copy vs long-form storytelling copy. The more contrast you add, the easier it will be to analyze the results and hone in one the winning angles.

We like to start things off with a 2 x 2: two creatives and two copy. We find that this strikes a good balance between simplicity and effectiveness.

2 x 2 Is An Easy Way To Get Started

The results from the first test determine what we do next. For instance, let’s say we find that there was a huge difference in performance between the creatives but no real difference between the copy variations. We would then isolate that variable, i.e. test a handful of creatives with the same copy.

If you’ve got a large budget and want to be as hands-off as possible, you can use Facebook’s Dynamic Creative. It does work and we do use it, but most times we prefer to have more control.

When To Test

When it comes to testing, we believe that frequency is more important than volume. It’s better to run multiple tests with a few variations, then a few tests with multiple variations.

That’s why we use a two-day testing cycle. On any given ad account we’re analyzing and implementing new ad copy and creative up to three times per week (Monday, Wednesday, Friday).

Usually, the number of tests you can run is limited by the budget. But even on the accounts where we’re not able to test new variations every two days, we still analyze and monitor performance.

Note: Adding a new ad to an existing ad set will reset the learning phase. It may make sense to use a designated campaign for testing.

How To Test

Analyzing the results is maybe the most challenging part.

There’s a lot that goes into the analysis, but there are a couple of simple tools that can do most of the heavy lifting. Having a simple process is extremely helpful for removing emotion and making informed decisions quickly.

We like to use this decision tree (credit: https://commonthreadco.com) as a guide.

Ad Kill Decision Tree

In order to be able to use the decision tree, you’ll need to have target CPAs for all correlated variables. Here’s a document that will help you with that.

When needed, we also use a Significance Calculator.

A/B Testing Like A Scientist

You simply plug in the numbers and the calculator tells you how confident you can be that any difference in performance is real and not due to random variation.

Since the whole point of Facebook’s targeting is to move away from random sampling, this obviously isn’t a perfect tool. But it’s useful as a reality check.

After The Test

The results from the test are documented and handed over to the creative team. We use a simple scoring scale (Poor, Okay, Good) and qualitative comments.

Scoring the ads can be more art than science, given the many factors involved. That’s why it’s best done by a media buyer who spends a lot of time in the ad account.

In Summary

By the time you’re reading this, we may have made a few (or many) changes to our A/B testing process. Nevertheless, the general principles and concepts apply.

Good luck!