A/B testing (a.k.a. split-run testing) in Ad Tech— why, when & what?
By: Charith Peris
Tags: Optimization, Advertising, Ab Testing, Data Science, Adtech

What to do before an A/B test…

One fundamental requirement of an A/B test is to ensure your two test flights are both targeting unbiased audiences. If the audience-splitting process is biased your A/B test could yield invalid results.

dataxu’s A/B testing framework enables audience splitting on a range of identifiers. With our OneView™ and graph technologies, we are able to split audiences on:

Households (which should be the default choice for all Connected TV inventory)

(which should be the default choice for all Connected TV inventory) Cookies

Devices — Google_ad_ids; IDFAs; Generic_IFAs

Google_ad_ids; IDFAs; Generic_IFAs People — LiveRamp_IDLs

dataxu’s framework uses a hashing function applied on a selected identifier (one of the above) to separate incoming bids into treatment and control groups. This hashing function is applied in real-time on bids arriving from exchange enabling us to do real-time audience-splitting. The hashing mechanism also ensures that the identifiers that arrive in our bid-stream, and are assigned to either treatment or control, stay within the assigned group for the duration of the test. This ensures no overlap in groups and thus a fair test.

Here is an example setup:

Let’s say you have a new creative and wanted to test how well it would perform in comparison with an existing creative. The summary steps to execute are as follows:

Create a copy of the existing flight with the exact same settings. Setup the new flight with the new creative that you want to test. Launch the flights to two audiences which are split using the hashing method described above.

However, there are a few things to check off before you begin this test:

1. Identify test variable

Each flight contains many variables that can be tested. Identify which one you would like to test.

An A/B test should only test one variable at a time.

In the example above, the variable is the creative.

2. Identify KPI

Identify which KPI you would like to measure ahead of setting the test. In our test involving the two creatives, we can consider using action-through-rate (ATR) as the KPI. ATR is the number of converters divided by the number of impressions served. It can reveal which creative is driving more conversions, therefore, is a good KPI to measure.

3. Setup a control and a treatment flight

All settings in both flights should be equal except for the variable you would like to test.

4. Determine your budget (sample size)

In advertising, your budget determines the number of impressions you receive and thus your sample size for the A/B test. It is important that your budget is of sufficient size to allow detection of an actual difference in the KPI between the A and B flights.

The time taken to complete the test would depend on the time taken to spend this budget.

This time can be estimated by using features such as impression-pacing and spend-forecasting, available on the platform.

In order to avoid bias due to the effects of seasonality, spend can be further regulated so that the time span would be a whole number of cycles (typically a week or multiple weeks).

5. Decide the split proportions and how significant your results need to be

First, decide how much of a change you need to see.

Next, decide the level of significance that needs to be achieved in order to consider the results valid.

For our creative example above, let’s assume that any change less than a 10% difference between treatment and control ATRs is not of use to us. Let’s also assume that you need to be 95% positive that one creative is working better than the other creative. In this case, the test should be set up with the expectation of seeing at least a 10% improvement, at 95% significance (p-value < 0.05). At dataxu, we provide a handy tool which enables you to tune your budget and split it in order to increase your probability to see the desired result, given the usual KPI you expect to see.

6. Make sure that only one test is running at a given time on any campaign

Tests can easily yield inaccurate results if there are multiple variables being changed. Make sure you are only running one test at a time

Now we have all the information that is needed to setup and run the A/B test!

What to do after an A/B test…

Once the test is complete, you need to collect and analyze your results. In the case of dataxu, the use of a reproducible deterministic hash on the identifier to panel to A/B makes it easy to split the results for analysis. Recall the KPI that you chose prior to the test and measure its difference across the two groups. Measure the significance of your result by calculating a p-value [4] for the test. Finally, if the results of the test are significant, the most important step is to make a choice. Now that you are armed with results based on an A/B test, you can make a scientific choice on how to optimize the variable you are testing. And don’t stop there. Pick the next variable you want to optimize and continue to test!

Multivariate testing

You can also set up multivariate testing which allows for many tests to run at the same time. However, please note that this will significantly lower the potential of achieving statistical significance and might not always help with the decision-making process. Even when there is a need to test multiple changes, we recommend testing a single change at a time.