Causal Inference with A/B Testing
By: Trent Hommeyer
Tags: Simulation, P Value, Python, Ab Testing, Causal Inference

A/B testing fundamentally looks to evaluate the causal relationship between a treatment and an outcome, but before we dive into the specifics, we must understand why it’s so hard to establish a causal relationship and why it’s so important.

The fundamental problem with causal inference is that in any given scenario, we can only observe what happens, not what would have happened had we done something else. While this seems obvious, it also means that we can never truly be certain about what caused what.

So if we can never know what leads to what, what can we do? Instead of sitting in a dark room all day because nothing is for certain, we can collect information and begin to make judgments based on what we see. This is done constantly in our brains — when we notice that the sun tends to rise in the morning, we slowly begin to trust that the sun will rise again tomorrow morning. While the kind of evaluation done in our brains is imperfect and can lead to overgeneralizations and oversimplifications, it’s the basis of inductive inference.

How can we make valid inferences?

In the business world, we’re always attempting to perfect our value offering, finding the perfect price for our product in various channels, finding the most effective product to advertise to our customers, or finding the perfect color scheme for our website to make our customers buy more.

In all of these cases, we’re trying to understand the impact of a given intervention (D), on an outcome (Y) — to do this, we must attempt to establish a relationship between the two variables.

Historical Approaches vs. A/B Testing

Imagine you’re a company that sells work boots and you’re trying to figure out which kind of boots you should advertise to users when they visit your website. In this case, your intervention is the kind of boot you advertise, and your outcome variable is whether or not the customer makes a purchase. While you may be tempted to look at historical data and count the number of customers that bought each kind of boot in the past, you would only be finding a correlation between the variables. Your past data may be plagued with non-controlled issues like periods where certain boots were sold out and unable to be purchased, periods where certain kinds of boots were being advertised more heavily, and many other confounding variables such as certain boots being in/out of fashion in varying periods. If you simply counted sales, you’d not only be finding simply a correlational relationship, but you’d also be solving for a different problem than the one you are presented with. Instead of learning what kind of boot would be the best to advertise to users on visiting your website, you’re learning what boots sold the best in the past, which is related to your outcome variable but isn’t the same.

To learn what will be the most effective strategy, we must actively test each treatment. Instead of using this past data, which is marred with hundreds of confounding variables that cloud our insight, we’ll test which one sells better. Statisticians like to call these kinds of tests RCT’s (Randomized Controlled Trials), but in the business world, they’re known as A/B tests.

When a user visits our website, we randomly select one of three kinds of boots as their front page and track the percent of users that make a purchase.

While many companies may simply look at which boot ad drove the most sales and make a decision, they’re forgetting about the fundamental problem of causal inference, we don’t know what each person would have done given another treatment. We could have simply happened to advertise one boot to people who wanted to buy boots.

The Solution

To truly make a causal inference, we can compare our observed results to a potential world where the treatments are known to have no impact, and see how likely our results would be if we *knew* that the treatments don’t make a difference. If we can conclude that it’s very unlikely that the differences observed in our treatment groups were due to variables outside of our control, we can conclude that the variables in our control (the treatment variables) truly made a difference on our outcome.

Before we continue, let’s nail down some terms and assumptions.

Key Terms

Treatment

Treatment is our independent variable or the thing we are changing in a given trial to test its impact on what we care about.

Our treatments, T_i, can be understood as a variable denoting which treatment is being used on which person. If the value of T_i is 1, we know that we used treatment 1 on customer i.

Individuals who received the same treatment are part of a treatment group.

Observed Outcome

Our observed outcome is the outcome we see for a given person. It’s what we care about most, and are trying to maximize.

Our observed outcome, Y_i, can be understood as the outcome we saw for a given individual. If the value for T_i is 0, we know that customer i ended up doing action 0. Often, there are only two actions, 0 and 1, with the customer doing something such as purchasing or not purchasing, but the observed outcome can also be a continuous variable such as the total amount spent during the website visit. It’s critical that the outcome variable selected is tied directly to success, and that maximizing it doesn’t have any major negative impacts.

Key Assumptions

Independence:

First, we need to know that the outcome of each trial is independent of the other trials. Practically, this means we need to know that one person’s purchase isn’t dependent on the treatment we give to someone else.

No Hidden Variation:

Second, we must know that there’s no hidden variation in our treatments. Practically, this means that we must know that each treatment would be executed the same way for a given individual. This also means that all individuals must be equally likely to receive each treatment. This ensures that our treatment variable that the only variable that distinguishes our treatment groups.

A Practical Example

While many companies simply rely on collecting a massive amount of data each time they run an A/B test so they naturally converge to an accurate picture of reality due to the law of large numbers, inference can be done much more methodically and pragmatically.

Consider an A/B tests run at scale, instead of these very simple tests of two treatments, a company such as Macy’s may be running an A/B test with hundreds of treatments and hundreds of potential demographics that must be tested separately. This kind of segmentation is necessary as it helps Macy’s advertising different products to different customers, but creates the issue of having extremely small sample sizes for their tests. Instead of being able to compare 500,000 outcomes of one treatment with 500,000 outcomes of another treatment (one million site visits), we now must evaluate the impact of 100 treatments on 100 demographics, meaning we the average number of samples we can collect from our dataset is only 100. If one in twenty people make a purchase, we have an average of 5 purchases, and can’t know if these 100 people happened to like shopping more, or if the treatment was truly effective.

Let’s evaluate this scenario.

Simple Case Study

Consider a scenario where we found that when 100 people were given each treatment:

Treatment group 1

Out of 100 real customers who were exposed to treatment 1, 7 made a purchase, and 93 did not.

Treatment group 2

Out of 100 different real customers who were exposed to treatment 2, 3 made a purchase, and 97 did not.

This equates to a conversion rate of 7% with treatment group 1 and a conversion rate of 3% with treatment group 2. At this point, you may be tempted to simply go with treatment 1, as in our sample, the purchase rate seen in treatment group 1 was double that of treatment group 2.

How can we know whether or not there’s a real difference between our treatments or if the difference is due strictly to randomness outside of our control?

We know that on average, our customers made purchases 5% of the time (10/200 people made a purchase), so let’s, very quickly, simulate a scenario where each customer, regardless of their treatment, has a 5% chance of making a purchase, and evaluate how often we see results like ours.

Very Simple Python Monte-Carlo Simulation (Repository found here: https://github.com/trentinsf/AB_causal_inf/blob/master/A_B_Testing_Done_Right.ipynb)

Evaluation of Results

Our model’s results are 12.38%, meaning that even in this simulated world where we know that there’s no difference between the treatment groups, about 12% of the time, due to randomness outside of our control, 4 or more people purchase when given treatment 1 than treatment 2. The earlier statement that “treatment 1 is two times more effective than treatment 2” seems a bit ridiculous now, given the fact that even if they were exactly the same, we’d see the results we did 12.38% of the time. This is why considering causality is so crucial.

As we stated earlier, to claim that one treatment group is actually more effective, we must be able to conclude that our results were very unlikely to be generated randomly (and not due to our treatments).

In our case, we can’t infer that treatment 1 is superior to treatment 2 because we can’t infer from our data that the difference between the groups was not due to random chance.

Different contexts call for different thresholds to claim there is a causal connection. In most modern research contexts, a researcher needs to demonstrate that there’s a less than 1% chance that their results would have been seen completely randomly (unrelated to their treatments). And even with those strict guidelines, many studies still end up falsely claiming that a causal relationship is present.

This percent chance that our results were generated randomly is called a P-value, and in business contexts, a P-value of .05 (5%) is required. Meaning a team must demonstrate that their results had a less than 5% of being due to random chance.

In our context, no matter what threshold we use, and considering that we have 100 treatments and 100 demographics, meaning we’re running 10,000 smaller experiments of this type, we don’t even meet the most lenient threshold of .05, and if we trusted our initial instinct, we would be acting based on wrong information very consistently.

While there are many mathematical approaches out there, such as the ANOVA test, or the T-Test, the essence of both is the same as this simulated test. They all evaluate whether or not we can assert that our observed outcome was NOT due to other variables outside of our control by computing the probability of getting our observed data in a world where we know our treatments have no causal impact. The ANOVA test and T-Tests make something that’s relatively simple in concept into a confusing mathematical calculation, obscuring away the true meaning behind why you’re doing the test in the first place. Hopefully, this very simple demonstration was a helpful first look into the power and utility of Monte-Carlo simulation in Python.

Choosing a required P-Value

While we have already pointed out the problems will simply trusting the outcome of your A/B test without understanding the possibility of it being due entirely to randomness, it’s equally important to discuss the modern discussion taking place about using a P-value that’s low enough.

Reminder: A P-value represents the likelihood that our results, (or results more extreme) would be seen in a world where we know that our treatments make no difference.

A P-value of .05 does not tell us that there’s a 95% chance that our treatment has a causal impact on our outcome. This distinction is often overlooked but is very crucial as it’s been empirically found that when the P-value is .05, there’s only a 70–80% chance that there’s a causal impact. Meaning that with a P value of .05, (the standard in business contexts), your test will lead you to take a misled action 20–30% of the time (Johnson et al., 2013).

This same study, along with more recent research from more than 100 leading researchers suggests that a P value of .005 should be required for any meaningful argument to be made, and a P-value of .001 is required for a significant connection to be made. (Benjamin et al., 2018) While of course, research contexts are much more high-stakes than business contexts and require more certainty, the errors very quickly add up.

Take this into consideration next time you interpret your A/B test values: even the companies that being diligent and using P-values instead of eye-balling are still acting on misleading data up to 30% of the time.

Having worked in the data-science world for years, it’s all too clear to me that we need to stop trusting averages and start thinking in terms of certainty and P Values if we want to be making consistently good decisions.

This is the reason data literacy, and an understanding of how to interpret the information provided by these test is more important than knowledge about how to execute these tests. It’s the reason that this tutorial for implementing AB tests has 11 lines of code but over 2000 words. This very simple simulation is extremely powerful and provides extremely intuitive results to even a nontechnical audience, of course, this kind of simulation using basic distributions can be scaled to more complicated kinds of tests, but without understanding the most basic example of an A/B test, those tests’ results will be misused.

References:

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., … & Cesarini, D. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6.)

Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313–19317. https://www.pnas.org/content/110/48/19313.abstract