The essential guide to Sample Ratio Mismatch for your A/B tests | by Iqbal Ali | Towards Data Science

Lack of data integrity is a common issue for experiments. They’re especially common when dealing with redirect tests, single-page applications, or other complex setups. It doesn’t matter how long you’ve been building tests; these problems just happen.
So, given this fact, what can we do about it? First (and most importantly), we need to be tracking the data integrity of our experiments very closely. So, if there is an issue, we know about it sooner rather than later.
One essential (and simple) check anyone can perform is the Sample Ratio Mismatch (SRM) check. You don’t have to be an analyst or data scientist to do this. In fact, many of the problems I’ve come across with tests were identified by product owners.
In this article, I’ll give you a practical overview of what SRM is and the various ways to check it. With this knowledge, you can go away and integrate it into your own experiment process.
Before we start, I must stress one point: while SRM catches many problems, it doesn’t highlight every possible problem with your setup. This is just the minimum you should be doing.
Suppose you have an A/B test where the expectation is a 50/50 split...
But suppose what you actually see is something like this...
Note that the sample sizes don’t match. This is what we mean by “sample ratio mismatch” — i.e. the ratios of samples don’t match our expectation (a 50/50 split). Having a skew like this can invalidate your test.
Before we go further, we need to identify a couple of rules to follow.
The first of these rules is to prioritise SRM checks with “users” rather “visits”. That’s because it’s “users” who are assigned to experiments. In comparison, “visits” are the number of “sessions” these users have made.
We might actually expect a skew in visits if a variation encourages a user to return more (or less) often.
So, does this mean that SRM with “visits” is always okay? Not necessarily. I’ve come across scenarios where defects caused users to drop visits in one variation more than another. It’s just that SRM with “visits” isn’t the “smoking gun” evidence of a problem that SRM with “users” is.
Anyway, onto our second rule for checking Sample Ratio Mismatch. And that’s this: we need to be looking for problems frequently. Checking for SRM is not a one and done activity. We need to be checking our tests pretty much as soon as they launch, and then we should be checking regularly.
New experiments should be treated like intensive care patients for at least the first week of launch.
This rule immediately gives us a problem to overcome because the traffic volumes being interrogated could be really low. But there are ways to increase our certainty levels. We’ll cover this a little later in the article. First, let’s look at the easiest ways to identify a problem.
The first and easiest way to identify a problem is by looking at the test assignment numbers.
Some problems are so glaring; you can raise the alarm straight away without needing to do any math. For instance, if you see 1,000 users in one group and 100 in the other, you know there’s a problem.
It might seem an obvious thing to point out, but it’s important not to assume that checking SRM is some sort of major activity. Just being able to easily see the numbers quickly could save you some major time.
What’s more, once you develop a habit of regularly looking at these sorts of numbers, you develop an eye for spotting mismatches — especially the big ones.
But suppose it’s closer? Suppose you have 10,000 users in the control group and 9,500 users in the variation. What do you do then?
We can run a simple calculation to find the sample ratios.
First, get the total sum of users assigned to the experiment...
total_users_in_test = users_in_control + users_in_variation
...and then work out the percentage of users in each group.
control = users_in_control / total_users_in_testvariation = users_variation / total_users_in_test
You don’t have to work out both control and variation. Just one will do.
For our example, the calculation:
control = 10,000 / (10,000 + 9,000) = 0.5263variation = 9,500 / (10,000 + 9,000) = 0.4737
So, 52.63% of users are in the control group, and 47.37% are in the variation. Now, we might expect to see a certain amount of mismatch during the early days of an experiment, but the above looks pretty suspicious.
How can we improve our level of certainty? It’s time to employ some statistics...
Statistics are ways of describing data in useful ways.
We have two samples: the traffic volumes for control and variation groups. It would be useful to know the likelihood that the differences between the numbers are outside of normal chance.
We can do this using the chi-squared test of independence. This test tells us that, given two samples, the probability that the samples are independent.
You actually don’t need to know the formula for calculating Chi (though I’ll go through it a little later), as it’s super simple using python:
observed = [ 170471, 171662 ]total_traffic= sum(observed)expected = [ total_traffic/2, total_traffic/2 ]from scipy.stats import chisquarechi = chisquare(observed, f_exp=expected)print(chi)
The “observed” variable lists two values: number of users in control and number of users in variation.
The “expected” variable is also a list of two values: the number of users we expect for each group. The volume we’re expecting is half of the total traffic in each. That’s the total traffic, divided by two.
We need the “chi-square” module from scipy.stats. After that, we just feed the numbers in. The output we get is:
Power_divergenceResult(statistic=4.145992932573005, pvalue=0.041733172643879435)
This is a tuple, where the first item is the chi-squared statistic, and the second item is the p-value. Normally, one would look for a p-value of 0.05 or less to determine independence (and, in our case, proof of SRM).
The problem with 0.05 is that it’s not strict enough for our purposes. Using this might give us a false signal of a problem. Michael Lindon (part of the Optimizely team) goes into detail about this in the following article.
What we need is to be stricter for our test of independence. A value below 0.01 should be enough. With our Python example, we can write up a conditional statement to make it easier to read:
if chi[1] < 0.01:    print('Warning. SRM may be present.')else:    print('Probably no SRM.')
By the way, chi[1] means we access the p-value from the tuple. Our full script looks like this.
Using our example, we don’t have hard evidence of SRM. One more thing to add is that I like to look at cumulative views. The more traffic we have, the closer the two groups should align in terms of the sample split:
Above is a pretty typical view of sample ratios. These cumulative views can be useful for determining when SRM began occurring if a defect is introduced partway through the runtime of an experiment.
I realise that many reading this may not be familiar with python. So, here’s an example using a spreadsheet (works in Google Sheets or Excel):
Using the CHITEST formula, we can pass the two sets of values: observed, and expected.
=CHITEST(observed_cell_range,expected_cell_range)
The first range of numbers is the “observed” values for control and variation. The second range is for the “expected” (total divided by two).
The output of the formula is the P-value. After that, the rules are the same as our python example: less than 0.01 indicates a possible SRM.
For those who really want to dig into the Chi-Test formula, here it is:
The chi statistic is the sum of each of the observed values minus the expected values squared, divided by the expected.
Huh? Let’s use a spreadsheet to break this down:
The columns:
Observed: the control and variation traffic volumes, respectively.
Expected: the expected values for each— i.e. the observed total divided by 2.
Difference: the observed value minus the expected
Difference Squared: the difference multiplied by itself
Difference Squared / Expected: difference squared divided by the expected.
Each row is a variation group. Eg. control and variation (A/B).
After this, we’re ready to find the Chi Statistic, which is just a sum of the Diff Squared/Expected (as in the formula above):
How do we get a p-value from this? Well, there’s one more thing we need first: that’s the degree of freedom, which is calculated as:
Degree of Freedom = (rows − 1) × (columns − 1)
In our case, our degree of freedom is 1 (two rows for test and control and two columns for observed and expected).
We can then use a p-value table to locate our score and find the associated p-value:
Or use a spreadsheet function:
=CHISQ.DIST.RT( chi_statistic, degrees_of_freedom)
This function gives us the precise p-value. Our final spreadsheet looks like this:
Of course, you can circumvent all of this by using the CHITEST function as shown earlier.
Checking the validity of the traffic is super easy using Chi. There’s really no excuse not to do it. Besides, you could really save yourself a lot of time by doing this simple check.
Once you’re doing this regularly, you can start going further by adding more checks for data validation. This is really the beginning. But it’s an essential beginning.
Having said all of that, be wary of crying wolf. That can be as damaging to your experimentation process as data validity issues themselves. I’m usually wary of declaring a problem during the first day of the experiment launch unless there’s a glaring issue.
Do this often, and you’ll become an expert at determining SRM issues early.
I’m Iqbal Ali, writer of comics, former Head of Optimisation at Trainline, creator of the A/B Decisions tool, and a freelance CRO specialist and consultant.
I help companies with their experimentation programs, designing, developing, training, and setting up experiment processes.
Here’s my LinkedIn if you want to connect. Or follow me here on Medium.