Think Designing an Effective A/B Test is Easy? Think Again
By: Andre Ye
Tags: Machine Learning, Marketing, Business, AI, Data Science

Let’s begin by considering the question again.

Lookout is aiming to increase the number of users on its platform. Should it advertise on TV?

Lookout wants to increase the number of users on its platform. It doesn’t seem that advertising on TV would hurt the number of acquired users. (Perhaps this is not the case, but we’ll take it as an assumption. In reality you’d need to test this assertion out.)

The obvious answer, then, without any testing, is yes, advertise on TV!

Regardless the cost, Lookout should advertise on TV if it’s looking to increase users. Surely paying for exposure, at any price, shouldn’t hurt.

However, this is obviously not the intention of the question. Let’s rephrase it:

Lookout is aiming to increase the number of users on its platform. Is it worth it to advertise on TV?

Cost is now introduced. If it costs several million dollars to run TV ads but only a small handful of users joined Lookout, it’s not worth it, even if the service did increase the number of users on its platform.

But as with most things, grounding our problem in reality also made it more complicated. We now need to not only think about cost, but also add some aspect of comparison. Is running TV ads worth it, compared to what?

To start, we might consider something like “cost per user” and compare it to other advertising mediums, like Google search ads or Facebook ads. Finally, we can add some more clarity to our question:

Lookout is aiming to increase the number of users on its platform. If we have $x to spend on advertising, should it be spent on TV or some other ad medium?

Great! So we are aiming to find the cost per user. A simple approach towards calculating this can be as follows:

This definition of the metric is inherently a bit flawed. It essentially treats all users the same; that is, if ad A only appeals to 30% of the audience and ad B only appeals to 50% of the audience, we must choose ad B. On the other hand, it may be true that ads A and B appeal to different segments of the audience, and that together the ads can appeal to 80% of them.

Instead, we may try to come up with some model M like this:

This more dynamic model can be used for better comparison. At the same time, we’ve made the whole situation a lot more complex. If we’re going to train a model to identify the cost to acquire a user, we need to have training data that looks something like this:

We still need to determine how to acquire the cost of the user for this model to work. We’ve wound ourselves into a bit of a loop: to find the cost to acquire a user, we need to find the cost to acquire a user. There are probably ways around this, but there’s too complex to cover in just one article.

In the meantime, let’s just return to our original simpler definition of cost to acquire a user, acknowledging that it has weaknesses. We still have an important question to consider that we’ve been pushing back a bit: how does one determine is a user joined Lookout from television?

There’s a fair bit of obscurity around what constitutes ‘from’.

For example, what if a user clicks a Facebook add for Lookout and signs up for an ad, only because they saw a Lookout ad on TV? In this case, the television ad was the determining factor. There are further questions of overlap that go in the other direction: even if someone goes to Lookout directly after watching a TV advertisement, what role do previous Lookout ads play in exposure and brand?

Again, we’re going to tread over these problems for now, although in real experimentation they need to be considered. Noise between interfering sources of variation is a real problem that exists in most A/B tests. It’s not enough to just say that variation is accounted for on both sides; it causes inaccuracies and precision errors.

Regardless, we may design a test that looks something like this. We take two groups of DMAs, or Designated Marketing Areas (discrete marketing regions that can include cities and/or rural areas)— that are very similar to each other.

We run exclusively TV ads in one DMA group and Facebook ads in another DMA group (assuming we’re only comparing these two mediums). Then, we measure the number of new users from each of these areas.

If there is a substantial difference between the groups, we can attribute it to the type of advertisement. We do need to be careful, though; because this type of test has such a large error range, we likely cannot take away much unless the different is truly significant.

What’s nice about this is that it embodies the spirit of A/B tests, and it makes the problem a bit simpler. Factors other than the presence of a certain type of ad will be cancelled out (at least in theory). We can make a conclusions about the cost-per-user for TV ad and Facebook ad campaigns, and compare them.

Finally, we can make our boss happy!

Not so fast. The question of finding these DMA groups, is extraordinarily difficult. The two DMA groups needs to be representative of each other, on various levels, such that they can be properly compared.

The people in the DMA groups must be similarly receptive — effective — to certain types of ad campaigns. That is, if the audience in group A is 30% receptive to Facebook advertisements, the audience in group B should be at least between 25% and 35% receptive. This way, we can properly compare the users acquired.

The accessibility to advertisement mediums must be similar across DMA groups. If few people in DMA group A turn on their TVs and many people in DMA group B don’t use Facebook, we can’t expect to compare the advertising mediums properly. (This is separate from ‘repective-ness’, which assumes equal access to advertisement and concerns user choice.)

Other factors that may impact the results, like population sizes, need to be very similar.

The reason why we choose DMA groups is because it allows us to create two similar groups without DMAs that need to be identical, which would be pretty much impossible.

Although this is difficult, it’s totally possible. Data about accessibility to advertisement mediums is available. Regional TV stations have viewer data, and Facebook provides reach data to advertisers. We can build a probabilistic model determining if a user will see an ad or not, and apply it to the entire population of a DMA to find how receptive it is over all. In the end, our data may look something like this:

Then, we can set some constraints and set a program to explore possible splits of DMA groups. Machine learning certainly can be used for this, but it might be overkill, depending on the number of DMAs we have data for.

So, finally, we’ve arrived at a solution — kind of; we’ve explored plenty of issues with it. We have generated many more questions than answers, but hopefully it illustrates just how much thought and problem-solving is required to engineer a meaningful and effective A/B test in the real world.