Having issues with under-powered A/B tests? Try controlling your variability.
By: Stas Sajin
Tags: Machine Learning, Data Science, Ab Testing, Experiment

Having issues with under-powered A/B tests? Try controlling your variability.

An overview of variance reduction techniques for improving the power of your A/B tests

In graduate school, I attended a session on reinforcement learning behavior, specifically on topics related to laboratory rats and experimentation. One thing that struck me about the results of the talks is that there was almost no variability in experimental conditions. The rats in these experiments were inbred to have identical genetic makeup, so any variability would be an anomaly caused by not controlling the environment correctly (e.g., exposing rats to a stimulus accidentally during transportation to the lab) or by the actual experimental treatment. Hence, given that the researchers had control over the genetic makeup and the environment in which the lab rats grew, they were able to carry out experiments where the concept of confidence intervals and error bars was meaningless. This is an extreme example of controlling your variability when running experiments, one that most of us can’t afford in practice.

If you take a basic statistics course, the topic of controlling variance is likely to show up several times. For example, you might learn that it is better to design a within-subjects rather than a between-subjects experiment. You might also learn that a lot of longitudinal studies that want to explore the effects of gene-environment interactions pay particular attention to twin studies. In more advanced classes you might learn about Linear Mixed Effect models and how to model some of the variance caused by random effects in your experiments, or how to model the hierarchical nature of the experimental designs when performing stratified sampling. All of these topics focus on the same problem: how to model and control your variability so that you can be sure that your study is able to detect a real effect when it is there.

When doing power analyses, you’ll quickly learn that the power of your experiment scales with the inverse of the square root of N subjects you have in the study, which means that it does not scale well at all! Because experiments involve testing conditions that might be worse than the status quo, it becomes very impractical to rely on group size as a parameter to increase your power. Instead, you have two other options in order to increase your power: 1) come up with better experiments that have larger effect sizes or 2) figure out a way to decrease the variance in your experiment without increasing the number of users. This post will focus on specific techniques related to the latter and will showcase how you could double the available traffic or even triple it for the purposes of running experiments.

Technique 1: Linear Model Covariate Adjustment

The most common option for controlling your variance is to specify the non-treatment effects as covariates in a linear model (If you’re using LMM model variants, you can also specify these as random effects). For example, let’s assume that I’m trying to estimate how quickly users can complete their taxes. Moreover, let’s assume that we have two application formats, and we think that the new format we developed is better than the old one. When we design our experiments, we randomly assign our users to two groups: new vs old. Moreover, let’s assume that we collect for each user information about their income and that we expect individuals with higher income to take longer to file their taxes. This is how the structure of the data looks like:

In our example, the application type is the treatment effect, minutes is the outcome, and income is a covariate that we were lucky enough to measure. Note that here we assume that all users who start the tax filing application, end up finishing it. If that is not the case, you would have to collect covariates that are present before the experimental assignment is performed. The general form of our equation then becomes like this:

Y represents the target outcome (minutes)

Z is the treatment variable (application type)

X is the matrix of covariates (we only have income, so it is a vector)

is the matrix of covariates (we only have income, so it is a vector) δ the effect size we are trying to estimate (how much faster is the new application format)

θ and θ_0 are the parameters for intercept and covariates

Let’s assume that we have the following conditions:

100 users in each group

the expected lift is 2 minutes. In other words, we would like to design a powerful enough experiment to detect an improvement of 2 minutes.

If we run a simple OLS only on the treatment effect, we get the following results (note that these results are identical to having a between-samples t-test).

These results are from a single run, so they are not very informative. If we bootstrap our experiments, we should expect the shape of our effect to look like this:

The blue line represents the expected effect size. The orange line is when there is no effect.

We can immediately notice that in a lot of our bootstrapped experiments, we failed to detect an effect when it is there. What happens when we add income as a covariate to the model? The figure below shows that having income as a covariate helped reduce the variance and thus increase the power of your experiment.

Although this might be good news, in practice I find that linear models are very challenging to build correctly due to the many assumptions you are required to fulfill. For example, if your outcome data is not linear, the covariate might not reduce the variance uniformly across the range of the output variable. For example, the actual relationship between income and minutes that I modeled is not linear, but the formula specified in the OLS models does not know this. This will cause the variance in some places to be underestimated and in other places to be overestimated making it more challenging to draw relevant conclusions.

Technique 2: Controlled-experiment using Pre-Experiment Data (CUPED)

Another option for controlling variance is described in a paper by Deng et. al (2013) and is referred to as CUPED. They borrow this idea from control covariate used in Monte Carlo simulations. In this method, we can notice the following relationship:

Y_hat represents an unbiased estimator for E(Y). −θE(X) + θE(X) should cancel each other out given an appropriate theta, because we assume that before experimental treatment the effect of a covariate between groups is the same. The equation basically says that this new control variate outcome has the same expected value as the actual outcome. Next, we compute the variance of the control variate.

The variance equation allows us to see that the variance of this unbiased control variate will be minimized when θ is the following:

All this can be codified in a few very simple lines of code:

And here are the results:

In our experimental setup, CUPED performs about the same as the linear covariate method. This method is very elegant though it is not entirely clear how it performs when you have multiple covariates. If we have multiple correlated covariates, we would have to generate a one-dimensional summary of those covariates, which might result in loss of information. Moreover, the assumption of linearity between the outcome and covariate still needs to be maintained.

Technique 3: Non-parametric model covariate adjustments

This technique is somewhat a continuation of Technique 1. This is not a technique I fully explored and seen elsewhere, so suggestions are welcome. You can notice that all we did so far is engage in an exercise of measuring variance using different methods and then performing an adjustment to the total variance. The natural question is why should we use parametric models to do that given that in the real world we have the following scenarios:

We can have multiple covariates

We can have interactions among covariates

We can have non-linear data

We can have covariates with missing data

We know that non-parametric models will be much better at reducing the bias in the data, so one technique I experimented with is replacing the OLS models with a RandomForest model. The only change I made to get unbiased predictions is to use out of bag scores from my model. See the code snippet below.

The output of bootstrapped experiments can be found below:

Using the random forest model to remove the variance caused by covariates does seem to work, though the effect is slightly smaller than other methods. My intuition suggests that this might be because OOB estimates contain a lot of variance because the sample size is very small (200 observations in total from two groups), so it is hard to build non-parametric models that generalize well in their OOB score.

Results Recap

My experiment setup is a bit trivial since I picked a covariate that has a high correlation with the outcome. Because of that, we will see a lot of variance reduction. Specifically, I’m seeing the following in variance reduction and subject requirements for power of 0.8 (I’m using the average of bootstrapped variance for power calculations).

Conclusions and Caveats

In a data-driven organization, performing experiments in a way that minimizes traffic requirements is critical. Any of the methods I described above can be used to effectively double or triple the number of experiments that can be run while having the same amount of traffic. Nonetheless, each method does require knowing how your covariates behave with respect to the outcome of your experiment, and some methods might not work when the available experimental data is just too small and a model can’t be used to correctly eliminate the variance.