The ultimate guide to A/B testing. Part 4: non-parametric tests | by Maria Paskevich | Towards Data Science

A/B testing is a very popular technique for checking granular changes in a product without mistakenly taking into account changes that were caused by outside factors. In this series of articles, I will try to give an easy hands-on manual on how to design, run and estimate results of a/b tests, so you are ready to go and get these amazing statistically significant results!
In parts one and two, we’ve discussed estimation of samples’ sizes before running an a/b test and the most common probability distributions, that actually describe how your data is shaped. And in part three we talked about two types of tests (parametric and non-parametric) and when and how to use parametric tests.
If you read the previous part (part three), you may remember that we were investigating how the new co-op mode affected the game’s metrics and found out that a 1-minute increase in session length was statistically significant (using Student’s T-test).
So now it’s time to check if we have a decrease in monetization metrics, and if that 5% retention growth is significant?
Just a quick reminder of the A/B test set up:
We’ve been developing this amazing arcade game for the last couple of years, and things seem to be going pretty well. But at some point community started to ask for a co-op mode. After some discussions, game team decided to develop the new mode and run an A/B test to check, how it affects the metrics.
We ran an A/B test for the test group of 2500 users (the same size as the control group) and got these results:
- Average session length increased from 8 to 9 minutes (statistically significant result)
- Day-1 retention increased from 40% to 45%
- At the same time, conversion lowered from 2% to 1.7%
So while the team is celebrating the growth in an average session, it’s time to figure out if our retention and conversion have actually changed or these are just fluctuations in data.
But unlike the average session length, these two metrics aren’t distributed normally, thus require using non-parametric tests.
Non-parametric statistical tests are used when continuous data is not normally distributed or when data is discrete, so we can’t apply usual parametric methods such as T-test or ANOVA.
The most popular non-parametric tests are Pearson’s chi-squared, Fisher’s exact tests and Mann–Whitney U-test.
For our example let’s use chi-squared to check retention and Fisher’s exact tests for conversion (cause for this metric data is highly imbalanced).
This test works only for categorical data when you have two or more categories and want to check if there is a significant difference between them.
Let’s apply this test to check if the 5% increase in retention was statistically significant.
In our example, there are two categories: churned and returned players, and the null hypothesis claims that there is no dependency between the number of returned players and the new co-op mode, therefore the 5% difference in the percentage of returned players is just a matter of chance. The alternative hypothesis is more optimistic: we have increased retention by adding a new game mode.
Firstly, let’s put our data in a table to see which categories are there:
The chi-square test is based on the difference between observed and expected values in each of the cells of the table:
Observed values are quite easy - these are the numbers we already have in the table
Expected values are that we would have if the null hypothesis is right about no difference between control and test groups
So the second step is to calculate the expected values for each cell using following formula:
In our case calculations look like this:
Note: Because in our case the totals for control and test groups are equal (2500 each), expected values are equal as well, but it won’t be the case when a/b test groups slightly differ in size.
Let’s put results into the table together with observed values:
After calculating the expected value for each cell, we finally get the value of the Chi-Square test:
Here R - number of rows, C - number of columns in our table, and Oij and Eij are observed and expected values of each cell.
After applying the formula, we get Chi-Square equal to 12.79:
The last step will be to check the critical value for chi-square, but we need to know degrees of freedom for the data: each chi-square distribution has a different number of degrees of freedom and thus has different critical values. We can calculate it as (R — 1)(C — 1), which in our case is 1.
So all in all, the critical value for α = 0.05 is 3.841, which is lower than our 12.79, and we can reject the null hypothesis and happily report that the changes in gameplay have increased our retention by 5%!
Test case scores one more goal! 2–0! At this point, we can conclude that the new co-op mode engages players and positively affects the game.
But with all this enthusiasm let’s not forget about monetization! As we see, it went down from 2% to 1.7% in the test group, which makes the board of directors quite upset.
Fisher’s exact test is a non-parametric test that is often used as a substitute for chi-square when the data set is small or categories are imbalanced.
In our case with the paying conversion, the categories are highly imbalanced: there are only 1.7–2% of paying players within test and control groups and the rest 98% are non-payers, so we can apply Fisher’s to check if the drop in conversion from 2% to 1.7% is actually meaningful.
Just as chi-square, Fisher’s test uses tables for all possible test values, but it calculates the exact probability of observing the distribution seen in the table(aka p-value) or even more extreme distribution, that’s why it is an “exact” test.
As in the previous test, let’s start with the table:
In this case, the null hypothesis is that the number of payers among players in the test group is the same as in the control, and the alternative hypothesis is that there are fewer payers in the test group.
Fisher’s test calculates the probability of results being at least as extreme as in the data, or even more extreme, which means even fewer payers in the test group.
The formula to calculate the probability is:
Where Cij is a cell in the ith row and jth column, n — total number of observations in the table.
With our data the formula will look like this:
Although 5000! looks terrifying, it’s easy to compute using Python and Scipy library:
from scipy.stats import fisher_exactoddsratio, pvalue = fisher_exact([[50, 2450], [42, 2458]])
The p-value is equal to 0.86, which is higher than critical 0.05, so we can’t reject the null hypothesis and happily report that the drop in conversion by 0.3% is not statistically significant.
We can conclude that the results of the experiment are quite good: from one point we have growth in engagement, and from another one — decrease in monetization wasn’t significant, which can be crucial for the business side, so the feature can be rolled out to the whole audience.
So this being said, we have discussed the most popular parametric and non-parametric statistical methods, and it’s time to get a bit more probabilistic... So next time we’ll talk about Bayesian tests. Stay tuned!