Bayesian A/B Testing in PyMC3 | by Dr. Robert Kübler | Towards Data Science

From time to time, you have to choose between two options. This might be in completely uninformed situations where you can pick the better option with a mere probability of 50%. In some of these uninformed cases, you can even boost this probability with a simple trick, as demonstrated in my other article.
However, usually, you are able to gather some information that helps you pick the better option with high probability. One easy, yet smart method to do this is A/B testing that you have probably heard of or even used already.
In this article, I will briefly explain the motivation behind A/B testing and outline the logic behind it. The problem with this approach is that it uses p-values that are easy to misinterpret.
Thus, we will take our time to refine this approach with a pinch of Bayesian magic, yielding Bayesian A/B testing. To top it off, we will implement a simple Bayesian A/B test using the wonderful PyMC3 library in Python.
If you are not entirely sure what Bayesian statistics are or you have never used PyMC3 before, check out my entry-level articles about this topic:
A gentle Introduction to Bayesian Inference (introduction)
Beginner-friendly Bayesian Inference (a simple problem solved via Bayes without programming)
Conducting Bayesian Inference in Python using PyMC3 (the aforementioned simple problem solved via Bayes in PyMC3)
Imagine that you operate a moderately successful online store with around 10,000 visitors every day. Around 100 of these 10,000 people actually buy something from your shop — your so-called conversion rate is about 100/10,000 = 1%, which is rather small.
Most online shops achieve conversion rates of around 1–3%, Amazon more than 10% even.
Since many people reach your shop anyway, yet only an underwhelming percentage of them become customers, you conclude to increase your shop’s conversion rate as this is the biggest lever for higher revenues. You think:
What keeps people from buying my awesome products?
You can’t come up with a good explanation, so you ask some friends for inspiration. They tell you that they no fans of the color blue of your buy button. Perhaps a red would be more inviting? This is something you can test!
The method is the following:
Randomly assign new visitors to your shop to either team blue or team red with a 50% probability. People in team blue will see the blue button, team red the red one. After running this process for some time, you check which team achieved a higher conversion rate.
Terminology: You call the blue team control group and the red team treatment group or test group. The control group gives you an indication of what happens if you leave everything as is, the treatment group tells you what happens if you do some kind of intervention, such as changing the button color.
Note that this kind of randomization is important if you want to evaluate the experiment easily. Things that you should not do include:
Assigning all men to team red, all women to team blue, because even if one team’s conversion rate is much higher, you cannot know if this is due to the button color or merely the gender.
Assign everyone to team red this week and to team blue the next week, because even if one team’s conversion rate is much higher, you cannot know if this is due to the button color or if people just bought more stuff in one week, regardless of the color. Maybe one of the weeks included holidays like Black Friday or Christmas, or other, more subtle seasonal patterns.
...
You get the idea. In order to distill the button color effect, there should be no other confounding effects that could explain the different conversion rates in both teams as well.
Let’s assume that you conducted a proper data collection for your A/B test for one day and ended up with some data. You had exactly 10,000 visitors which you randomly grouped into team blue (control group) and red (treatment group). You took notes about if this visitor bought something (encoded as 1) or not (encoded as 0).
We simulate the outcome using this code:
import numpy as npnp.random.seed(0)blue_conversions = np.random.binomial(1, 0.01, size=4800)red_conversions = np.random.binomial(1, 0.012, size=5200)
You can see that I put 1% as the control group conversion rate and 1.2% as the treatment group conversion rate. So you already know that the red button should perform better if the sampling did nothing too weird! But let us pretend from now that we do not know how the data was generated because that is what we always face in reality.
Also, note that the groups have different sizes, which usually happens with random assignments. They are around 10,000/2=5,000, but it’s unlikely to get such a perfect 50:50 split. This is fine, however.
Both NumPy arrays consist of ones and zeroes, with most of the entries being zero.
print(blue_conversions)# output: [0 0 0 ... 0 0 0]print(red_conversions)# output: [0 0 0 ... 0 0 0]
The next thing we can check is the share of ones (i.e. the visitors who bought something) is.
print(f'Blue: {blue_conversions.mean():.3%}')print(f'Red: {red_conversions.mean():.3%}')# output: Blue: 0.854%, Red: 1.135%
This indicates that the red button might perform better, but we can’t really know at this point. Even if the conversion rates across the different groups were exactly the same, one group would end up with more conversions than the other one with a high probability.
We still have to rule out that this happened only due to mere chance and this is exactly what an A/B test is for.
Let us now see how we can come up with a better explanation of what is going on. Is the blue or the red button better? At first, we will reproduce the traditional way, and afterward, we will do it Bayesian style.
I will not go into depth here because there are plenty of resources that perfectly explain how to conduct a normal A/B test. You want to know which of the two following hypotheses are true:
Basically, you compute some kind of test statistics such as in Fisher’s exact test or Welch’s t-test. Then you compute a p-value and check if this is under some arbitrary bound such as 5%. Let’s go for Welch’s t-test.
from scipy.stats import ttest_indprint(f'p-value: {ttest_ind(blue_conversions, red_conversions, equal_var=False, alternative="less").pvalue:.1%}')# output: p-value: 7.8%
Because some people struggle with p-values, here is how you can read it:
Given that H0 is true, the chance that we get our observed or an even more extreme outcome is at most 7.8%.
Since 7.8% > 5%, we keep the null hypothesis. It’s not evident that the red button is significantly better, so we just leave the blue button.
I think that the definition of a p-value is rather unintuitive — and everyone who ever misinterpreted p-values proves this. The most common misinterpretation is the following:
The probability that blue is better is 7.8%. (WRONG!!!)
While this is not what a p-value tells you, being able to formulate such a clear and short statement is desirable, isn’t it? Well then, Bayes to the rescue.
Bayesian A/B testing has the following advantages:
It enables you to make statements about one version being better than the other one with a certain probability. Just what we want.You don’t have to know all the statistical tests. You can just make up a proper generative model and press the Bayesian Inference ButtonTM.
It enables you to make statements about one version being better than the other one with a certain probability. Just what we want.
You don’t have to know all the statistical tests. You can just make up a proper generative model and press the Bayesian Inference ButtonTM.
I assume that you have some prior knowledge about using PyMC3 now, otherwise, check my linked article above.
To make these advantages more clear, let’s analyze our problem the Bayesian way with the help of PyMC3. First, we need to think about which parameters we want to infer. This is easy, there are two unknown parameters:
the conversion rate of the blue button and
the conversion rate of the red button.
We now need to decide for a prior distribution for both parameters. As conversion rates can be between 0 and 1, a Beta distribution makes sense. Beta distributions have two parameters a and b, and you can create diverse distributions by varying them.
Because we know that the conversion rates are rather small, good priors for both of them are Beta(1, 99), for example.
Now, we have to model how the outcomes that we observed (arrays containing zeroes and ones) are connected to these parameters. The straightforward way is to use Bernoulli variables, as they can take only the values 0 and 1 and use a single probability parameter.
Long story short, we can write the following short program:
import pymc3 as pmwith pm.Model():    blue_rate = pm.Beta('blue_rate', 1, 99)    red_rate = pm.Beta('red_rate', 1, 99)        blue_obs = pm.Bernoulli('blue_obs', blue_rate, observed=blue_conversions)    red_obs = pm.Bernoulli('red_obs', red_rate, observed=red_conversions)        trace = pm.sample(return_inferencedata=True)
Together with the explanations, the program should make sense to you. In the top first two lines of the model, we define the priors. Afterwards we design the output of the model (Bernoulli variables) and additionally give it the observations from our A/B test preparation using the observed parameter. The last line is the PyMC3 version of the famous Bayesian Inference ButtonTM.
With the help of thetrace object you can reconstruct the posterior distributions of the conversion rates.
import arviz as azaz.plot_posterior(trace)
We find something close to our maximum likelihood estimated conversion rates of around 0.854% for blue and around 1.135% for red, and even a credible interval for these estimates. For example, the blue rate is between 0.62% and 1.1% with a 94% probability.
Neat, right? But this does not answer our question. What is the probability that the conversion rate of the red button is higher than the one of the blue button? To answer this, we can just sample both posterior distributions and see, how often the red rate is higher than the blue rate. Luckily, the samples are stored within the trace object already. We just have to see how often the red rate samples are larger than the blue rate samples.
blue_rate_samples = trace.posterior['blue_rate'].valuesred_rate_samples = trace.posterior['red_rate'].valuesprint(f'Probability that red is better: {(red_rate_samples > blue_rate_samples).mean():.1%}.')# output (for me): Probability that red is better: 91.7%.
This is something we can work with! It’s easy for us, but also everyone else — I’m especially looking to the business department — to understand. The red button is better with a probability of around 92%?
“Perfect!”, you think and the red version in your shop, hoping that it will increase your conversion rate.
And now imagine: This is only a tiny change that has the potential to increase your sales. There are probably various other smaller and larger things you can tinker with that leave your shop better off. The game has just begun.
There are many circumstances in which we have to choose between two options. This can simply be the color of a button, but also the position of an object on the website, the text on a button or image, anything that could potentially be changed.
One way to choose between two versions is by running an A/B test. The idea is simple: You group your customers into two groups that only differ in this very thing you want to change, i.e. a button color. This way you can be sure that only the thing you changed caused a change in the outcome, and nothing else. Not the age or gender of people, not the weekday when you collected the data.
Usually, people use a classical A/B test using p-values. While this is a familiar concept for statisticians, normal people often get confused statements involving p-values. That’s why we turned to Bayesian A/B testing, which allows for results that everyone can easily grasp. In the end, we could even formulate such a result using PyMC3 without much code.
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!