Why you should use Bayesian approaches in your Offline A/B tests | by Alain Tanguy | Towards Data Science

A first introduction to the importance of Bayesian approaches in classic statistics can be found here, and a more advanced application there!
Hypothesis testing is a central topic in statistics with a wide range of applications which exceed the limits of mathematics and extend to most scientific domains, if not all.
We are going to highlitght the limitations of regular statistical frequentists approches and present a relevant bayesian framework to solve one of the most common situations data scientists working in retail or marketing may encounter: A/B testing.
A/B testing (also known as bucket tests or split-run testing) is a randomized experiment with two variants, A and B. It is a way to compare two versions of a single variable, typically by testing a subject’s response to variant A against variant B, and determining which of the two variants is more effective.[Wikipedia]
Maybe we want to test a new design for our website, new features, or even new strategies for targeting our customers (mailing, RTB, etc.), in order to measure which one would lead to the highest ROI. For the sake of clarity, we’ll consider the case of an advertiser trying to improve the conversion rate using one of two creatives A and B: each interaction of an ad with a user can be seen as a Bernoulli Trial with exactly two possible outcomes, “conversion” and “no conversion”, depending on the user buying our product after being shown the ad or not.
Two designs can be considered to run this experiment:
Offline: The experiment is run, the analysis and decisions are only made once it is over. (we will only consider this design in this story)
Online: The Analysis and decisions are made along the course of the experiment.
For this story only the Offline case will be adressed as the Onlive A/B testing problems requires drastically different solutions.
We consider the case of a test being run for a weak, with each user being shown creative A or B with a probability of 0.5, and directly converting with an unknown proba p_A=0.04 or p_B=0.05.
A first difficulty quickly arises: what test should we use in order to determine if a creative is significantly better than the other? Common tests are summarized in the following table, and it is up to us to pick the one that suits our problem the best.
In our case, Fisher’s exact test and Barnard’s test seem to be the most relevant as we focus on conversion rates obtained from a Binomial Distribution, though a Chi-Squared test could have been used given the quantity of observations we will have access to in the experiment.
We will use Fisher test which has the characteristic of belonging to the class of exact tests, so called because its p-value can be calculated exactly, with
H0: Creatives performs equivalently (p_A = p_B)
H1: One creative performs better than the other
Good practice wants us to determine the statistical power of our test. It can be computed through simulation given how specifically our environment has been designed. Our target is 0.8, and the amount of observations we will use will be calibrated in order to reach that threshold. For convenience, we will only consider an alternative hypothesis “(H1) “alternative hypothesis p_A=0.04 or p_B=0.05”, which will allow us to easily compute the power of our test.
Usually α= 0.05 is the limit accepted for the rejection region for the p-value, or in other words, the False Positive Rate. For this threshold a minimum average of 13500 observations will be required.
We face a similar situation as the one we did Here, asking ourselves similar questions. What can we conclude from our test results and our p-value?
Do we have a way of estimating the probability of creative A to be Better than creative B?If so can we estimate the improvement between them?
Do we have a way of estimating the probability of creative A to be Better than creative B?
If so can we estimate the improvement between them?
I will not keep you in suspense any longer. The answer is No for both questions. We already understood that p-values were merely a probability of observing results at least as extreme under the hypothesis (H0). But this also highlights maybe the most important limitations of p-values:
p-values are not measures of effect sizes! It doesn’t give us any information on how much better creative A performed over B, even in case of a positive result!
In order to extract knowledge about the effect size, frequentists can use another and more powerful tool: Confidence Intervals.
Confidence Intervals represent the range of plausible values for an unknown parameter associated with confidence level that the true parameter is in the proposed range.
As such CI give us much more understandable and rich insights, providing us with knowledge about the probability and uncertainty of our estimate to. Yet they carry their own share of drawbacks:
We need to choose between different approaches to define these intervals, depending on a number of hypothesis: Confidence intervals for binomial distributions.Confidence intervals for the values of p_A and p_B don’t translate directly to confidence intervals on the difference between p_A and p_B! Fortunately there exists ways to compute directly confidence interval on the difference, yet again, we need to pick one among dozens of methods.
We need to choose between different approaches to define these intervals, depending on a number of hypothesis: Confidence intervals for binomial distributions.
Confidence intervals for the values of p_A and p_B don’t translate directly to confidence intervals on the difference between p_A and p_B! Fortunately there exists ways to compute directly confidence interval on the difference, yet again, we need to pick one among dozens of methods.
We could for example use the “simplest”, namely the Wald method without continuity correction:
With p̂ the empirical estimates for p_A et p_B, and z corresponding to the α/2-percentile of the normal distribution. Thus, for a 95% CI (ie, α=0.05), the corresponding z value would be 1.96.
A numerical application would give us the interval [0.0009 ; 0.0171] centered around 0.009 which doesnt overlap with 0 as we hoped for!
For this part, we are going to use the PyMC3 library for python, allowing us to easily build Bayesian nonparametric models.
It is important to remember that our objective as statisticians or data scientists is obviously to conceive accurate and relevant results, but also and just as importantly to generate KPIs that can be both shared and understood by any other stakeholders, even non-scientists ones, and that would carry as much information as possible. Bayesian approach gives us the tools to do so by allowing us to compute exactly what we are looking for: the posterior distributions of p_A and p_B, namely P(p_A|X) and P(p_B|X) as well as P(p_A — p_B > 0| X), ie. the probability of creative A to yield more conversions than creative B.
Here the Bayesian modeling is once again pretty straightforward, since we already mathematically designed the environment, we just need to replicate it using bayesian language.
The question of priors choice inevitably arises. We will pick uninformative uniform priors for p_A and p_B.
We can compute a graphical representation of our Bayesian network as well:
And as simply as that our work is over! All that remains for us is running an MCMC algorithm through the network to compute posterior distributions. Those will give us much more insights than the previous methods, since now we can yield confidence intervals for any α level directly by sampling from the posteriors.
(it is important to notice that delta cannot be computed from the posterior of p_A and p_B since p_A and p_B could be dependant)
We can then simply compute the posterior probability of delta being inferior to 0:
np.mean(delta_samples<0)
Here we get 0.956, which translate to a probability of 96% forcreative A to generate more conversions than creative B!
But we can go further. Imagine shifting from design B to A is expensive, and it is establish that only an improvement of performances of 5% at least would be profitable. We have a way to compute this! Just insert a new deterministic variable Tau = p_A / p_B in our network and sample the posterior distribution.
tau = pm.Deterministic(“tau”, p_estimates[0] / p_estimates[1])
np.mean(tau_samples>1.05)
This time we get 0.91... Not as decisive as the 95% we usually hope for. If we need more confidence, we would just need to run the A/B test longer.
The Bayesian framework provides an easy to perform and easy to read alternative to classic approaches of A/B testing, and allow us to test any hypothesis by simply computing posterior distributions.
Bayesian approaches enable us to achieve more efficient offline decision-making in the case of A/B test, as well as more efficient online decision-making, as will be shown in another story.
For the frequentist approach, we had to choose a specific test and a specific way of calculating our Confidence Intervals, other tests and intervals computations would have given different values ; exactly the same way that different priors would have given us different posteriors.
Bayesian Networks applications are numerous and A/B testing was merely the tip of the iceberg. It can be used for modeling and machine learning when it proves to be an extremely powerful tool, as you can read about in the article:
towardsdatascience.com
Cameron Davidson-Pilon. (2015) Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference