Analyzing A/B Test Results with Python Code | by Sujeewa Kumaratunga PhD | Towards Data Science

The seven steps of A/B testing and how to pick the correct Python library that will give you the p-value.
So you’ve designed your A/B test. And now you run it, with the sample size (number of customers) and the number of days that you determined from the previous step.
Now here is the moment of truth. The sales team and the web design team are both eagerly waiting for your answer : your company introduced a new feature to the website; did this feature decrease sales like the sales team had predicted or did it keep the sales constant like the web design team had predicted?
Basically did the sales decrease or not?
They could have also asked “did the new feature on the website decrease the sales or increase the sales?”. There is a difference — the first asks if there was a change and the second asks if the change was positive or negative. We will come to this second question later.
We talked about the p-value in a previous post. In that post we also looked in to the 68–95–99.7 rule . The p-value together with the 68–95–99.7 rule, answers the above questions with a certain confidence level.
Determine the number of days you will run the testDetermine the sample size — how many customers in each dayDecide your null hypothesisDecide the confidence level you want to haveRun the test fully for 1. and 2. aboveGet the p-valueMake a claim on the the null hypothesis: either reject or fail to reject
Determine the number of days you will run the test
Determine the sample size — how many customers in each day
Decide your null hypothesis
Decide the confidence level you want to have
Run the test fully for 1. and 2. above
Get the p-value
Make a claim on the the null hypothesis: either reject or fail to reject
Let’s look at the same example as in the previous posts. Our customer base on average buys at a $170 value. The web design team wants to introduce a new feature, but the sales team is apprehensive. You run an A/B test to see if the new feature will change the average order value at all. You will have a control group who will be shown the old website and a target group who will be shown the new website with the new feature.
We discussed this in a previous post. Let’s assume we will run the test for 14 days.
We discussed this also in the previous post. Let’s assume we will have 10,000 customers in our sample.
In this case the null hypothesis is that the new feature will not change the average order value. That is the control group and the target group both will have an average order value of $170. In yet other words, control group and target group will be drawn from the same population, as if they were shown the same website. This is the hypothesis we will test.
Often the business will tell you what confidence level they want in the results. In general most businesses take a 95% confidence interval. That is you want to be 95% certain that the control and target groups come from the same population; i.e. it is ok if the control and target group don’t look like they come from the same population 5% of the time. So our critical p-value will be 0.05.
Now sit back and run the test for the parameters decided in 1. and 2. No peaking. If you need to peak, we need to have designed a different A/B test, like a Bandit test. An A/B test first explores, then after the test ends exploits. A Bandit explores and exploits, or learns and earns, at the same time.
We are on an A/B test, so let’s explore for 14 full days. Each day we show the control group the old website, and at the end of the test we would have 14 average order values. And each of those 14 days the customers in the target group get shown the new website, and at the end of the test we would have 14 of their average order values as well.
Like this:
control means = cm = [169.84, 166.66, 170.5, 168.09, 168.36, 171.28, 169.53, 169.25, 171.66, 171.03, 168.89, 171.04, 171.20, 170.22]target means = tm = [174.54, 171.95, 177.06, 174.33, 174.85, 174.34, 176.89, 176.71, 172.81, 176.22, 177.45, 177.10, 175.23, 177.09]
There are many ways to get a p-value from two samples.
In this case we will be comparing two samples to see if they are similar. The Kolmogorov–Smirnov test is one of the simplest ways to do so, also implemented in Python .
import scipy.stats as statsstats.ks_2samp(cm, tm)
gives you:
Ks_2sampResult(statistic=1.0, pvalue=4.985467362637908e-08)
What does this p-value mean? It means that given the control sample mean distribution given above (the array of cm), the probability that we obtain the above target sample mean distribution (given by array tm), is 5.0x10-6 %(comes from multiplying the above pvalue of 4.985467362637908e-08 by 100). That is, the probability this target samples came from the same population that the control samples came from is very very low at about 50 in a billion chance.
This p-value of 5.0x10-6 % is much smaller than the allowance we made in step 4 above which was 5%. And so it seems we have to reject the hypothesis that the target sample and the control sample came from the same population.
Let’s actually plot the control and target means given by cm and tm above to see what it looks like:
So the target mean is consistently different than the control mean. And that is what our p-value suggested.
Notice the target mean is actually higher than the control mean all the time, but that was not what we sought to discover in step 3; we sought to discover if the control and target samples were the same or different, not higher or lower. So we can not make a concrete conclusion about that. If that is an actual question we will have to run another A/B test to find out. No moving the goal posts after the game has started!
Let’s look at a second example:
cm = [169.84, 166.66, 170.5, 168.09, 168.36, 171.28, 169.53, 169.25, 171.66, 171.03, 168.89, 171.04, 171.20, 170.22]tm = [169.66, 167.14, 172.11, 169.44, 169.95, 169.46, 171.94, 171.77, 167.97, 171.29, 172.49, 172.14, 170.33, 172.13]stats.ks_2samp(cm, tm)Ks_2sampResult(statistic=0.42857142857142855, pvalue=0.15493471530488623)
In this case since the KS-test p-value of 0.15 is larger than the critical p-value of 0.05 we say we fail to reject the null hypothesis; we fail to reject that the control and target samples are coming from the same distribution.
Given that ‘fail to reject’ is a double negative, doesn’t it mean we accept the null hypothesis? No it doesn’t. We can never accept the null hypothesis or say that the two samples are coming from the same distributions with a 100% certainty. Why? Because we set ourselves a goal of being 95% confident at the very beginning. Again, no moving the goal post once the game has begun.
You may ask then could we not have set our confidence level at 100%? If we did that our tail allowance will be 0.0 (from 100%–100%; earlier when we asked for a 95% confidence level our allowance was 5%). That is our critical p-value is 0.0, and whatever we do, the experiment’s p-value will always be above 0.0 and we would have to reject the null hypothesis at all times.
Python has many libraries to calculate p-values. You can use any of them with the same syntax as above.
Which one should we use? Here is a loose guideline:
Two-sample t-test (Student t-test)If you are confident that the control and target sample means are normally distributedorIf you are testing for over 30 days (because over 30 days Central Limit Theorem kicks in and assures normalcy — 30 is not a magic number, but widely accepted in the industry.)
Kolmogorov-Smirnov test (KS test)You can not guarantee that the control and target means are normally distributedorYou don’t have many days in your test (About 30 days is industrially accepted as enough, though there is no magic behind this number. In any case no business will allow you to run an A/B test for 30 days.)orYou think the mean, variance and the shape of the distribution is the same for the two samples. orIf you want to capture the effect of the day that the control and target samples differed mostly, use this test. KS test is sensitive to the largest discrepancy of your test.
Mann-Whitney rank testYou can not guarantee that the control and target means are normally distributedorYou don’t have many days in your test (About 30 days is industrially accepted as enough, though there is no magic behind this number. In any case no business will allow you to run an A/B test for 30 days.)orYou think the mean, variance or shape of the distribution are different for the two samples.
If you are unsure which one to use, a good rule of thumb is to use the Mann-Whitney rank test.
if test p-value < critical p-value reject the null hypothesisif test p-value > critical p-value fail to reject the null hypothesis
if test p-value < critical p-value reject the null hypothesis
if test p-value > critical p-value fail to reject the null hypothesis
The p-value gives us the probability of observing what we observed, given the null hypothesis is true. It does not tell us the probability that the null hypothesis is true.
The p-value gives us the probability of observing what we observed, given the null hypothesis is true. It does not tell us the probability that the null hypothesis is true.
So in our first example the null hypothesis is that the target and control samples both come from the same population. Then we can say, the probability of observing the target and control sample means, tm and cm, given the hypothesis that they both come from the same population is 5.0x10-6 %. A 50 in a billion chance, a very small probability indeed. So we reject the null hypothesis: we reject the hypothesis that the two samples are coming from the same distribution.
It tells us nothing about the probability that the null hypothesis itself is true.
It is a difficult lesson to drive home. But in general, in life, we are not able to tell the probability that any hypothesis is true — all we can say is something about the probability of an observation given someone’s hypothesis.
2. With the p-value we will either reject the null hypothesis or fail to reject the null hypothesis. You can never accept a null hypothesis.Failing to reject the null hypothesis does not mean accepting the null hypothesis. This is because when we set the confidence level we are always setting ourselves at less than 100%, which means we can never be 100% sure that the null hypothesis is true.