8 Common Pitfalls of Running A/B Tests | by Leihua Ye, PhD | Towards Data Science

1
2
3
4
5
6
7
8
9
10
Powered by Play.ht
Create audio with Play.ht
Powered by Play.ht
Online experimentation has become the industry standard for product innovation and decision-making. With well-designed A/B tests, tech companies can iterate their product lines quicker and provide better user experiences. Among FAANG, Netflix is the company most open about its experimental approach. In a series of posts, Netflix has introduced how to improve experimentation efficiency, reduce variance, quasi-experiments, key challenges, and more.
Indeed, online controlled experiments offer a high level of internal validity after controlling for all other external factors and only allow for one factor (the treatment condition) to vary. Unlike other statistical tools (e.g., regressions), experiments can derive causal effects and offer the best solution of separating causation from correlation.
However, some folks hastily jump to the conclusion without checking anomaly, end the experiment prematurely due to the primacy and novelty effects, and roll out products that hurt their revenue after confusing negative with positive outcomes. The list goes on and on.
Bad test design is worse than no design!
Conducting scientistic research takes discipline, and we need to understand what is appropriate and what isn’t. In today’s post, I list 8 common pitfalls that industry folks frequently make and how to address each scenario, respectively. My original intention is to write an end-to-end practical manual on how to run A/B tests correctly. I’ll hold your hands through common mistakes that happen at the pre-, at-, and post-experiment stages.
Not enough sample size
End prematurely
There are thousands of ways of failing your online experiments, and having an insufficient sample size is the primary way of test failure. I’ve seen companies roll out an A/B test and immediately end the experiment after seeing a better metric in the treatment group.
Have you done a power analysis to decide the sample size?
Have you ended your experiments too soon?
Companies without sophisticated Data Manpower may find these two questions foreign and daunting to answer. The underlying statistical support for A/B tests is called Hypothesis Testing: we try to attribute the difference in the metric, if any, to the presence of the treatment instead of random noise (randomness).
To that end, we need a ‘large enough’ sample size to reject the Null Hypothesis. Here, I put quotation marks because the sample size varies for different business scenarios.
The sample size depends on three parameters:
Significance Level, i.e., the probability of False Positive. Typically, we settle at 0.05.Statistical Power, i.e., the probability of correctly identifying the effect when there is indeed an effect. Also, Power = 1 — Type II Error. Typically, we set power at 0.8.The Minimum Effect. This is a business question. After conducting cost and benefit analysis, what is the smallest acceptable difference between the treatment and control groups? Hypothetically, there is a statistically significant improvement of 0.000000001% between the experimental groups. Would you roll out the new version? No. It would be better off if we stick with the old version and not roll out the change, considering the business costs to update (e.g., engineering time, business cycle, bad UX, etc.). Talk to your business partners, particularly the Product Manager, about the tradeoff and set the minimum threshold.
Significance Level, i.e., the probability of False Positive. Typically, we settle at 0.05.
Statistical Power, i.e., the probability of correctly identifying the effect when there is indeed an effect. Also, Power = 1 — Type II Error. Typically, we set power at 0.8.
The Minimum Effect. This is a business question. After conducting cost and benefit analysis, what is the smallest acceptable difference between the treatment and control groups? Hypothetically, there is a statistically significant improvement of 0.000000001% between the experimental groups. Would you roll out the new version? No. It would be better off if we stick with the old version and not roll out the change, considering the business costs to update (e.g., engineering time, business cycle, bad UX, etc.). Talk to your business partners, particularly the Product Manager, about the tradeoff and set the minimum threshold.
Setting the normal significance level (0.05) and statistical power (0.8), we have the following formula:
where:
σ2: sample variance (the variance of the Overall Evaluation Criterion, aka. the metric of interest)
δ: the difference between the treatment and control groups
To obtain the sample variance, we can run an A/A test before rolling out the A/B test. Here are the relationships among sample size and the parameters:
Significance Level decreases → Larger Sample Size
Statistical Power increases → Larger Sample Size
The Minimum Effect decreases → Larger Sample Size
Pitfall 0: Don’t understand power analysis and end the experiment prematurely.
Solution: We can use Online Calculator or Python to calculate the necessary sample size. Let’s say you need to collect a sample size of 5,000 for each group after conducting a Power Analysis. The daily website visit is 500, which is divided into 2 groups, 500/2=250, for each group. So, you need to keep the test running for 5,000/250 = 20 days, roughly 3 weeks. This falls into the reasonable range of timeframe. Don’t end the experiment prematurely because you have seen a positive uptick in the treatment group in the first few days. Wait until the end of Week 3, when you have a large enough sample size to make informed decisions.
Level of random assignment
Heavy VS light users
A/A tests
The success of online experimentations hinges on the random assignment of the treatment condition. We have to make sure that the assignment is truly random. However, ensuring a random assignment in the industry is easier said than done. Not every company has the required Data Manpower to do it correctly.
To administer randomization, companies use hashing and a ‘cluster’ system to randomly assign users into the treatment and control groups. The underlying idea is to create a hashing value for each visitor and assign him/her directly to the treatment condition or through a ‘cluster’ that is exposed to the treatment condition.
The challenging part is they can’t guarantee the consistent assignment of users to the same treatment condition. Visitors may be exposed to the treatment condition differently if they visit the website multiple times. For example, an online video-streaming company is testing out a new content recommendation algorithm, and its experimentation platform adopts the ‘cluster’ system. Each time, the company randomly assigns users to the two variations and record their streaming hours (the metric of interest), respectively.
Any problem?
Yes, a huge problem. If a user visits the website multiple times a day, he/she may be exposed to both the treatment and control conditions, which invalidates any causal conclusions.
Pitfall 1: Companies can’t guarantee the consistent cluster assignment and thus not a truly random process.
Solution: The most common solution is to adopt the ‘cluster’ system and ensure a consistent assignment. First, the engineering team assigns users to different clusters. Second, randomize at the cluster level, not the user level. This method can largely address the inconsistent assignment problem identified above. Unfortunately, not every company has the Engineering Staffing and sophistication to design such a system as Netflix and Google. Besides, some business models are not fit for the ‘cluster’ system.
The next pitfall is also about randomization. For consumer-facing companies, there is a division between heavy and light users. Heavy users usually have a small percentage of the total population but contribute the most traffic. A simple random assignment can’t ensure the equal distribution of heavy users for small sample sizes, resulting in large variances.
As a side note, this also explains the low statistical power of traditional A/B tests with small treatment effects. I will explain how to increase statistical power through variance reduction techniques in another post.
Here is a hypothetical example. After conducting a Power Analysis, we need 1,000 samples for each variant. Within the target population, heavy users roughly take up 5% but contribute 60% of website traffic. Particularly when the sample size is relatively small (e.g., 1,000), a random assignment may result in more heavy users falling in one group over another.
Heavy users are the most loyal customers who respond more actively to the stimulus than light users. We will overestimate the treatment effect if we attribute the final differences between the experimental groups to the treatment condition alone.
Pitfall 2: A random assignment fails to distribute heavy users equally.
This pitfall concerns me the most: people don’t realize it is a problem. They naively believe a random assignment would create two comparable groups, which holds theoretically but rarely in practice. If there is a small percentage of heavy users, a random assignment does not guarantee an equal distribution of heavy users.
There are dozens of ways of fumbling your experiments, but there is only one way of doing it right.
Solution: Like stratified sampling, we set the threshold for heavy users and use a dummy variable to randomly assign them to the treatment and control groups. In so doing, the heavy VS light users ratio balance out in the experimental groups. Data Scientists need to team up with the Product team to set the threshold. After randomization, we need to check the distribution of critical covariates between the experimental groups.
Another solution that companies often do not aware of is to conduct an A/A test.
A/B tests for version differences; A/A tests for version stability and consistency.
The engineering team has solved all the technical difficulties and created two comparable groups. Now, let’s administer the same treatment condition to both groups, aka an A/A test. Since the experimental groups share the same attributes and face the same treatment condition, we should observe no difference.
Statistically speaking, we should fail to reject the Null Hypothesis and observe no difference between these groups 95% of the time, setting the α level at 5%. Just keep in mind, there is still a 5% chance of detecting False Positives even when the experimental groups are the same.
Ceteris Paribus
The spillover effect
Let’s say we have collected a sufficiently large sample size and comparable experimental groups after a true randomization process.
Can we trust the treatment effect now?
Not until we check these two additional factors: Ceteris Paribus and the spillover effect.
First, is the Ceteris Paribus assumption valid? That is, it’s likely to introduce additional variations in one experiment group but not the other. Here is a real case study at Microsoft (Crook et al., 2009). Following the same A/B test logic, they administer the treatment condition to one group, not to the other. However, the treatment condition needs to run JavaScript to load its content, which creates a response delay that affects the outcome variable. No delay in the control group.
Pitfall 3: Fail to check the Ceteris Paribus assumption.
If that happens, the difference between the treatment and control group is attributable to the true treatment effect (version change) and the extra response delay.
The overall difference = the true treatment effect + the delay effect
The overall difference is a biased estimate of the treatment effect, which underestimates the treatment effect. Let’s say the new version is indeed better than the old version. Due to the delay effect introduced in the treatment group, we may not identify the actual effect and decide not to update. Economically speaking, companies lose out in the competition and incur tremendous opportunity costs.
Solution: For experiments to succeed, we have to do the balance check before, at-, and after the assignment. If the intervention introduces extra delay, we should introduce the same amount of delay time to the control group. In so doing, we eliminate the influence of extraneous factors.
Besides, we have to check the spillover effect between the experimental groups. Does your treatment group interact with the control? Are they friends on LinkedIn, Facebook, or Twitter? Do they talk about the treatment online? If the answers to these questions are ‘Yes,’ we are in a difficult spot of cross-contamination and diluted treatment effects.
Another source of the spillover effect comes from the same geographic location. Here is a quick example. For ride-hailing companies like Uber and Lyft, if they try to evaluate two matching algorithms’ performance and assign drivers in Los Angeles randomly to the treatment and control groups. Regardless of which algorithm is better, drivers in the treatment group cannibal the total share of requests for the control group’s drivers. Again, this causes a biased estimate.
Pitfall 4: Cross-contamination between the treatment and control groups.
Solution: we can address the cross-contamination effect from space and time dimensions, respectively. Spatially, a pretty standard way is to administer the randomization process at a macro level (e.g., city) instead of an individual level (e.g., driver) and pick two geographically remote cases. Check out this case study and learn how Airbnb addresses this problem (link).
The two-sample comparison is a valid design for companies with large user bases, where it’s possible to pick control and treatment cases. In contrast, most companies do not have such luxury and only have a limited user base. If that is the case, we can solve the problem from the time dimension. We can still pull some strings and artificially create a control and treatment group.
Valid causal inference originates from two dimensions: space and time.
To compare two versions, we can administer one version to the entire population in Week 1 and check if the metric decreases or increases. Then, we take the treatment away and re-record the metric in Week 2, checking if it returns to the normal level. We treat the treatment condition as a switch toggle: turn it on and off multiple times and see any jump in the indicator for a significant effect.
For example, we are interested in evaluating the effects of a new version on the Click-Through Rate (CTR). For some unknown reason, there is no way of dividing the user population into a treatment and a control pair.
At Time Point A, we turn on the switch toggle and roll out the new update to all users. After a short period, we turn it off at Time Point B. Repeat the on-and-off process at Time Points C and D. We observe a spike in the metric every time the switch is on and a regression back to the mean after switching off. This approach provides a pretty good directional take on the treatment effect, e.g., the new version has a positive impact in the above case. However, the downside is it is difficult to give a precise estimate due to the lingering effects from past weeks.
Well, a directional take is good enough for decision-making.
Multiple Comparisons
e.g., Google’s 41 shades of blue
OK, we have done everything right before and during the experiment. Our statistical results suggest a small p value to reject the Null Hypothesis, and shall we roll out the new version and call it a day?
No! You shall not.
For multiple comparisons, the cumulative alpha level increases, and it’s easy to get False Positives. Here is a real case. Google wants to pick the most engaging color for its product and tests out 41 shades of blue.
Here is the detailed mathematical breakdown for multiple comparisons:
- Set α level to 5%-  P(False Positive) = 5%: no effect but falsely concludes there is an effect- P(Rejecting a False H0) = 1 — 5% = 95%- For 41 comparisons, P(at least one significant result) = 1 — (95%)41 = 88%
For 41 comparisons, the probability of falsely observing one significant result is 88%, even if there are no statistically significant differences. The problem is the increased accumulated alpha level, which adds up and poses a threat to the test validity. Without a proper adjustment, it is so easy to observe False Positives for multiple comparisons.
Pitfall 5: Get fouled by False Positives while making multiple comparisons.
Solution: We can use the Bonferroni correction, i.e., dividing the alpha level by the number of comparisons, α/n, and check for statistical significance at the divided level. In the previous example, we re-set the significance level at 0.05/41 = 0.0012 and only reject H0 if the p value is smaller than the adjusted threshold.
However, some critics argue that the Bonferroni correction is too conservative, meaning it’s more difficult to reject H0. In Google’s example,
P(at least one significant effect)
= 1 — P(no significant effects)
= 1 — (1 — 0.0012)41
= 0.048
The result (0.048) is smaller than the traditional 5%, or 0.05. (Check out this handout from Berkeley).
Simpson’s Paradox
A typical approach of rolling out experiments is to administer the experiment to a small percentage of subjects and then gradually ramp up to a wider audience. The reason for the cautious ramp-up approach is to end the experiment immediately if it shows a negative impact on key business metrics.
The following is a real example. For an experiment, Microsoft adopts a 99% (control) to 1% (treatment) ratio and observes a higher conversion rate in the treatment group on Day 1. They bump the ratio to 50% VS 50% and observe a higher conversion rate in the treatment group on Day 2 (Crook et al. 2009).
Here is the breakdown.
On Day 1 and Day 2, the treatment group performs better. Surprisingly, the pattern reverses at the aggregated level: the control group has a higher conversion rate (1.68%). Mathematically, it is possible to reverse the pattern before and after data aggregation. We make different decisions at the individual and collective levels. I’ve discussed Simpson’s paradox in another post.
Pitfall 6: Individual and aggregated patterns look different for ramp-up experiments.
Solution: There are three solutions (Crook et al., 2009). First, we can adopt paired t-tests on data when the proportions are stable, i.e., comparing the treatment in Day 1 to the control in Day 1 and the treatment in Day 2 to the control in Day 2. Second, we can use the weighted sum to adjust for the different ratios of Day 1 and 2. Third, we can throw away the data from the ramp-up period, which is shorter relative to the experiment. Another benefit of throwing away the initial data is it’s too messy and noisy due to the Primacy and Novelty Effects.
The Primacy and Novelty Effects
The primacy effect refers to the experienced users adjust to the new version. For example, we are interested in learning how a new version update would affect the CTR. The new version is so different from the old version, making experienced users confused and click (open) multiple links.
The novelty effect refers to existing users want to try out all new functions, which leads to an increase in the metrics. Any increase or decrease in the metric due to the primacy and novelty effects quickly dies out in days.
It’s likely that these two psychological effects mess up the initial treatment data and inflates or dilutes the treatment effect.
Pitfall 7: Fail to check the primacy and novelty effects, biasing the treatment effect.
Solution: The best way of teasing out both effects is to resort to new users who have not been exposed to the old version. In other words, the treatment effect on new users should be the treatment effect. Certainly, we can compare the treatment effect on new users and existing users to check for the existence of both effects.
Pitfall 0: Don’t understand power analysis and end the experiment prematurely.
Pitfall 1: Companies can’t guarantee the consistent cluster assignment and thus not a truly random process.
Pitfall 2: A random assignment fails to distribute heavy users equally.
Pitfall 3: Fail to check the Ceteris Paribus assumption.
Pitfall 4: Cross-contamination between the treatment and control groups.
Pitfall 5: Get fouled by False Positives while making multiple comparisons.
Pitfall 6: Individual and aggregated patterns look different for ramp-up experiments.
Pitfall 7: Fail to check the primacy and novelty effects, biasing the treatment effect.
Medium recently evolved its Writer Partner Program, which supports ordinary writers like myself. If you are not a subscriber yet and sign up via the following link, I’ll receive a portion of the membership fees.
leihua-ye.medium.com
Crook, T., Frasca, B., Kohavi, R., and Longbotham, R., 2009, June. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1105–1114).
Kohavi, R., Tang, D., and Xu, Y., 2020. Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.
Eryk Lewinson, Introduction to Power Analysis in Python
Emma Ding, 7 A/B Testing Questions and Answers in Data Science Interviews
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Please find me on LinkedIn and Youtube.
Also, check my other posts on Artificial Intelligence and Machine Learning.