A/B testing with probabilistic programming and PyMC3 (part I) | by Tung T. Nguyen | Towards Data Science

Disclaimer: This article is strongly influenced by the book Probabilistic-Programming-and-Bayesian-Methods-for-Hackers. Our main contribution is, perhaps, the expository style-which we hope will benefit the beginners to either probabilistic programming or A/B testing or maybe both. We will also reflect on some of our experiences working with these topics while being an intern at Kibo Commerce.
This is the second post in a series of articles on applications of probabilistic programming in general and of PyMC3 in particular. In this article, I will describe how to use PyMC3 to do A/B testing for discrete variables. As it is already long, I will discuss A/B testing with continuous variables in a sequel article. The current article is structured as follows.
1. What is A/B testing and why it is important for business success?
2. Some drawbacks of classical A/B testing, the need for more informative A/B testing, and how PyMC3 can help.
3. An example of A/B testing with discrete variables.
Let’s dive into the details.
Part 1: What is A/B testing and why it is important for business success?
First, let us briefly explain what A/B testing is by a concrete example. We note that there are variations when it comes to implementing and deploying A/B testing, so while our example is rather typical it is not uniquely representative.
Suppose, a campaign organization suspects that by having a longer subject line, it would attract more people to read the emails and then donate. Of course, there is no way for the campaign organizers to verify this beforehand. In other words, they just form a hypothesis that needs to be tested with data; otherwise, the campaign might lose donors if their hypothesis turns out to be wrong. For simplicity, let say that, the campaign organizers plan to send campaign emails to 50,000 potential donors. A simple procedure would be as follow.
1. Randomly divide the potential donors into two sets: the test set and the holdout set.
2. As the name suggests, we will use the test set to test our hypothesis. This part can be done as follow: we randomly divide the test set into two groups: the control group and the experiment group. We will then apply the old campaign strategy to the control group and the new one to the experiment group. We will wait until we collect enough data from these two groups to compare the two strategies.
3. We can then apply the better strategy verified by the second step to the holdout set.
This simple procedure is simple enough to implement and in fact, it has proved to be successful: see here for some real-world examples.
Part 2: Some drawbacks of classical A/B testing, the need for more informative A/B testing, and how PyMC3 can help.
Our main concern in this article is the second step. Typically, to compare the two strategies, we often use the t-test. While this test is relatively easy to implement, there are some drawbacks. Some of them are:
(i) It does not provide full information about the difference between the two means. In particular, it does not really provide a certain level of confidence in our final decision.
(ii) It requires us to fix a sample size beforehand.
(iii) The traditional t-test uses p-values to accept/reject the null hypothesis. From my conversations with my mentor at Kibo Commerce, Austin Rochford, I learned that marketers often misinterpret p-values as the probability that the experiment group performs better than the control group. This misunderstanding could lead to poor business decisions.
(iv) It is not easy to use to t-test to answer more complicated questions such as quantifications of the difference. For example, sometimes we are not just interested in whether the experiment group performs better than the control group but also how much better. This is important because it often costs resources such as time and money to implement a new strategy; so we have to be strategic to decide whether it is worth doing so.
These reasons among others are the motivations to find more informative tests. The approach that we take in this article is Bayesian A/B testing. Roughly speaking, the procedure to do A/B testing in the Bayesian/probabilistic programming framework is as follows.
Step 1: We form our prior belief about what should be true about the data.
Step 2: From the evidence/observed data, we update our beliefs; in other words, we (numerically) compute the posterior distributions.
Step 3: We sample from the posterior distributions and use these samples do make statistical inferences.
As we explain in our first article, it is often challenging to compute the exact forms of the posterior distributions especially when conjugate priors are not available. Luckily, we have PyMC3 to magically help us with that.
3. An example of A/B testing with discrete variables
An important metric for the A/B testing problem discussed in the first section is the conversion rate: that is the probability of a potential donor to donate to the campaign.
We will use simulated data as shown in the picture in the first section. More precisely, suppose we run the experiment and the result is given by:
donors_A=1300donors_B=1500conversions_from_A=273conversions_from_B=570
Note that, unlike the picture, we do not have equal sample sizes for group A and group B. This is actually not a rare situation in practice: it is often challenging to strictly follow the A/B testing procedure described above (for example, I observed that this happens rather regularly with Kibo’s data.) However, as we can see, the sample sizes are almost equal. Furthermore, in the Bayesian framework, un-equaled sample sizes are not a big problem.
We can numerically approximate the posterior distributions of conversion rates for these two groups by the following lines of code.
#group Awith pm.Model() as model_A:    #set the prior distribution of $p_A$.    p_A=pm.Uniform('p_A', lower=0, upper=1)          #fit the observed data to our model     obs=pm.Binomial("obs", n=donors_A,                     p=p_A, observed=conversions_from_A)    #MCMC algorithm     step=pm.Metropolis()    #sampling from the posterior distriubtion.     trace_A=pm.sample(30000, step=step)    burned_trace_A=trace_A[1000:]#group Bwith pm.Model() as model_B:    #set the prior distribution of $p_B$.    p_B=pm.Uniform('p_B', lower=0, upper=1)          #fit the observed data to our model     obs=pm.Binomial("obs", n=donors_B,                     p=p_B, observed=conversions_from_B)        #MCMC algorithm     step=pm.Metropolis()    #sampling from the posterior distriubtion.     trace_B=pm.sample(30000, step=step)    burned_trace_B=trace_B[1000:]
A few words about the codes. As our observed data is the number of conversions, we use the Binomial distribution to describe our observations. If we have point-wise data (i.e the conversion for each potential donor), we can use the Bernoulli distribution as described in our previous article. Of course, the posterior distributions will be the same regardless of the distributions we use. Mathematically speaking, the total conversion is sufficient statistics for our problem.
We also remark that this is not the only way to compute the posterior distributions. In this particular case, we can explicitly compute them, see our previous article for details. More precisely, we can show that the posterior distribution for p_A is Beta(274,1028) and the posterior distribution for p_B is Beta(571,931).
We plot the histogram of the samples from group A as well as its exact posterior distribution Beta(274,1028). We see that PyMC3 performs pretty well in approximating the posterior distribution of pA.
#histogram for samples from the posterior distribution of  $p_A$figsize(12.5, 4)alpha_prior=beta_prior=1a=alpha_prior+conversions_from_Ab=beta_prior+donors_A-conversions_from_Acoef=beta(a,b)plt.title(r"Posterior distribution of $p_A$")plt.hist(burned_trace_A["p_A"], bins=50, histtype='stepfilled', density=True, color='#348ABD')x=np.arange(0,1.04,0.001)y= beta.pdf(x, a, b) plt.plot(x, y, color='black', label='True posterior distribution of $p_A$')plt.xlim(0.15, 0.3)plt.legend()plt.show()
We can give an estimate for the conversion rate from group A by taking the sample mean.
#the sample mean-an estimate for conversion rate of group A. samples_posterior_A=burned_trace_A['p_A']samples_posterior_A.mean()0.21050251102125828
We see that this estimation is pretty close to the frequentist estimation.
Similarly, we can do the same for p_B.
#histogram for samples from the posterior distribution of  $p_A$figsize(12.5, 4)alpha_prior=beta_prior=1a=alpha_prior+conversions_from_Bb=beta_prior+donors_B-conversions_from_Bplt.title(r"Posterior distribution of $p_B$")plt.hist(burned_trace_B["p_B"], bins=50, histtype='stepfilled', density=True, color='#348ABD')x=np.arange(0,1.04,0.001)y= beta.pdf(x, a, b) plt.plot(x, y, color='black', label='True posterior distribution of $p_A$')plt.xlim(0.3, 0.45)plt.show()
Now, as we having enough samples from the posterior distributions of pA and pB, we can start making statistical inferences.
We can plot the two histograms together.
figsize(12.5, 4)plt.hist(samples_posterior_A, bins=40, label='posterior of p_A', density=True)plt.hist(samples_posterior_B, bins=40, label='posterior of p_B', density=True)plt.xlabel('Value')plt.ylabel('Density')plt.title("Posterior distributions of the conversion rates of group $A$ and group $B$")plt.legend()plt.show()
We clearly see that group B performs better than group A.
But how much better? There are two metrics we can use to answer this question:
(a) The absolute difference between the two means.
(b) The relative difference between the two means.
These are examples of quantification questions that we discussed earlier. Both of them can be solved in the Bayesian framework. First, let us answer the first question. To do so, we can plot the difference between the samples from group A and group B.
difference=samples_posterior_B-samples_posterior_Afigsize(12.5, 4)plt.hist(difference, bins=40, density=True)plt.vlines(0.14, 0, 25, linestyle='--', color='red')plt.title('Posterior distribution of the difference of the two means')plt.show()
As we can clearly see, this distribution is far from 0. How far? For example, are we confident in saying that the conversion rate of group B is 14% higher than the conversion rate from group A? Well, we can approximate this probability by the following lines of code.
#probability of 14% increase by group B100*len(difference[difference>0.14])*1.0/len(difference)96.23620689655172
This is pretty close to 100; so we are really confident about the above statement. What about a conversion rate from group B is 16% higher than the conversion rate from group A?
#probability of 16% increase by group B100*len(difference[difference>0.16])*1.0/len(difference)71.44913793103449
Now, we are less confident about the second statement.
Next, we answer the second question. As above, we can plot the relative difference between the conversion rates of the two groups.
rel_difference=100*(samples_posterior_B-samples_posterior_A)/samples_posterior_Afigsize(12.5, 4)plt.hist(rel_difference, bins=40, density=True)plt.vlines(60, 0, 0.04, linestyle='--', color='red')plt.title('Posterior distribution of the relative difference of the two means')plt.xlabel("percentage")plt.show()
Let us compute the probability that group B increases conversion rate by at least 60%.
100*len(rel_difference[rel_difference>60])*1.0/len(rel_difference)97.6301724137931
Well, we are confident that group B does increase the conversion rate by 60%. What about 80%?
100*len(rel_difference[rel_difference>80])*1.0/len(rel_difference)52.0051724137931
As we can see, this probability is not much different from the probability of getting a head by tossing a fair coin, in other words, it is mostly random and we should not believe in an 80% increase.
Conclusion
As demonstrated above, the Bayesian framework is able to overcome many drawbacks of the classical t-test. Furthermore, PyMC3 makes it pretty simple to implement Bayesian A/B testing in the case of discrete variables.
References
[1] Cameron Davidson-Pilon, Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
[2] Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin, Bayesian data analysis.