Conducting Bayesian Inference in Python using PyMC3 | by Dr. Robert Kübler | Towards Data Science

We have covered the intuition and basics of Bayesian inference in my article A gentle Introduction to Bayesian Inference. We then moved on to actually conducting Bayesian inference by hand using a coin example in my article Beginner-friendly Bayesian Inference.
If you still feel unsure about this topic, take a look at them, and have fun! 😊
Let us recall the problem definition: Imagine that you flip a coin with 0 on one and 1 on the other side a hundred times and write down the results. You end up with
1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0
These are 57 ones and 43 zeroes. We want to answer the following question:
What is the probability θ of receiving 1?
Again, a simple (and the maximum likelihood) answer is 57/100=57%. This approach yields a single number, but we also want to be able to describe the uncertainty of this result. Is this 57% quite safe, or if it is also likely that θ might be even 65% or 45%? That’s what Bayesian inference is for: we get estimates, including free predictions about the uncertainty of the estimate.
Let us reiterate a few mathematical details swiftly because it makes following the PyMC3 code much easier.
In math, we want to find the posterior distribution of our parameter:
To do Bayesian inference, we always need
the likelihood and
the prior.
Without these two ingredients, we can’t proceed. So, let us quickly revise what these quantities are.
The likelihood for a single coin flip (i.e. data = one coin flip) is given by
We say that the result of a coin flip such as above, which we call C now, is Bernoulli distributed, often denoted as C~Ber(θ). This is all we need to compute the likelihood term in the Bayes formula.
Regarding the prior, let us choose the one from the last article, i.e. a Beta(2, 2) prior. We have
and denote this as θ~Beta(2, 2). The 6 is only there to normalize this density, i.e. the integral of f between 0 and 1 is one.
Alright, everybody, enough with the formulas! Let us finally work with PyMC3 to solve the initial problem without manual calculations, but with a little bit of programming.
Let us first explain why we even need PyMC3, what the output is, and how it helps us solve our Bayesian inference problem. Then, we will dive right into the code!
So far, we could easily calculate everything by hand, because the Bernoulli and Beta distributions work together nicely (they are conjugated), meaning that their product is a Beta distribution again. This makes manual computations easy, with no problems at all.
We have implicitly omitted to deal with the little part in the denominator of the Bayes formula: p(data). However, this one is a serious source of trouble, in general. This term looks simple but it is extremely hard to calculate for
complicated likelihoods and priors, or even
for simple likelihoods and priors if they don’t play well together.
Anyway, that’s what PyMC3 is there for: Internally, it uses MCMC algorithms or Variational Inference to help us out. Please, don’t be discouraged when you read the Wikipedia articles. It’s difficult.
In simple terms, PyMC3 can’t give us a formula for the posterior distribution. However, it can give us as many samples as we like from the posterior distribution without explicitly calculating the denominator.
If we can get samples from a distribution, it’s nearly as good as having a formula for it. Why? Because we can estimate a lot of interesting statistics from these samples, such as the mean (take the average of all elements), median, standard deviation, quantiles, ... and even visualize it by plotting a histogram or density plot.
The moment you have waited for. First, install PyMC3 via a simple
pip install pymc3 .
Inside the programming environment of your choice, prepare the imports and the coin toss data.
import pymc3 as pmimport arviz as aztosses = [    1, 1, 0, 0, 0, 1, 1, 1, 1, 1,    0, 1, 0, 1, 0, 1, 0, 1, 1, 0,    1, 0, 1, 1, 1, 1, 1, 1, 1, 1,    1, 1, 1, 1, 1, 0, 0, 0, 1, 1,    0, 1, 0, 1, 1, 1, 0, 0, 1, 0,    0, 1, 1, 0, 1, 1, 1, 0, 0, 0,    1, 0, 1, 0, 0, 0, 0, 0, 0, 1,    1, 1, 1, 1, 0, 0, 0, 0, 0, 1,    0, 1, 1, 1, 0, 0, 1, 0, 1, 1,    1, 1, 1, 0, 0, 0, 1, 0, 1, 0]
So far, so good. The actual model is even shorter than this piece of code, so I will just present it to you and explain it afterward.
with pm.Model() as model:    # define the prior    theta = pm.Beta('theta', 2, 2)        # define the likelihood    data = pm.Bernoulli('data', theta, observed=tosses)        # get the samples    trace = pm.sample(return_inferencedata=True)
Basically, that’s it, the entire Bayesian inference. Probably you expected more, but this speaks even more for the awesomeness of PyMC3. Don’t be fooled, however: Under the hood, a lot of processes are running, more than I can explain.
Anyway, we end up with this variable trace which contains 500 * #cores_of_your_machine samples, 2000 for me. It involves the number of cores because PyMC3 generates samples in parallel, without us having to specify anything. Reasonable defaults, I love it!
If you followed this and my older articles that I mentioned in the beginning closely, the code should make kind of sense for you.
The only weird part should be the context manager in the first line, with pm.Model() as model: . It’s just a programmatic design choice that the PyMC3 people have made. You open up a model (like you open a file in plain Python) and do things inside this context. In our case, we define distributions and sample.
We then start defining our prior θ~Beta(2, 2), which in PyMC3 language is
theta = pm.Beta('theta', 2, 2)
PyMC3 distributions always want a name, that’s always the first parameter you have to specify. Usually, I just use the variable name again. Then the two parameters for the Beta distribution follow, in our case a=2, b=2. Done with the prior!
Then we have to tell the model how the likelihood works. We already established that a coin flip just follows a Bernoulli distribution Ber(θ). The code for this is
data = pm.Bernoulli('data', theta, observed=tosses)
This way, the model knows that each coin flip is a Bernoulli variable with the parameter θ. In addition, we easily pass the observed data of 0/1 tosses as a list.
What confused me when using PyMC3 for the first time: You don’t have to specify the complete likelihood p(data | θ) for all observed data. A single coin flip is enough, as PyMC3 multiplies these small single coin flip likelihoods together for you, i.e. you specify p(single toss| θ) for each toss and PyMC3 computes
This is done because we always expect the tosses to be stochastically independent of each other.
At last, let the sampling begin!
trace = pm.sample()
This step can take quite long in general, but for this example, even a weak computer should finish in a few seconds. We obtain some trace object, but just imagine that it contains a lot of numbers from the posterior distribution. Wanna peek inside? Use trace.get_values('theta') . You will receive a numpy array containing the samples you can then play around with.
You can use the tool arviz to visualize the posterior distribution, without the need to import matplotlib, seaborn, or pandas. If you type
az.plot_posterior(trace, hdi_prob=0.99)
you should receive something like
You can see that the mean and also the mode of the posterior distribution is around 57%. The 99% HPD (highest posterior density) is a so-called credible interval containing θ with a probability of 99%. The HDP especially is the shortest interval with this property. Looking back to the old article, we can see that we got the same numbers when calculating everything by hand. Nice!
So, we can answer the question again:
If we are expected to give a single number, we would go for 57%. Otherwise, we are able to say that it’s somewhere between 45% and 69% with a probability of 99%.
IMPORTANT: This is a completely different statement than what you can obtain from confidence intervals when tackling this problem the frequentist way. Confidence intervals let you make statements like: “If you compute this interval a lot of times for different data samples from the same distribution, in 99% of the cases the true (fixed) parameter θ lies within the interval.”. This statement is hard to wrap your head around, and actually not what we want to state. But this is a different story. Just remember that a credible interval has an easy inerpretation.
In this article, we have yet again analyzed the coin problem which we have solved analytically already in the past. This time, however, we used PyMC3 to solve it for us. The code is extremely short and simple (so is the problem), but yields the same results we expected, which is a nice sanity check.
But the real fun starts now: You can start building more complicated models, such as Bayesian linear regression, where each weight is a parameter, for example with a normally distributed prior for each parameter. Doing this, we have credible intervals around each parameter and even around predictions, which is a great thing to have. You could also implement Latent Dirichlet allocation for unsupervised topic modeling for NLP problems in PyMC3 or Facebook’s Prophet for time series forecasting.
But you can start smaller. How about you have data that is binomially distributed with parameter n and p, which you want to estimate. You can simulate it via np.random.binomial(n=23, p=0.1, size=100) using numpy, for example. p is a probability again, you know which prior works well for that. n is an integer. Distributions over integers are another binomial distribution, or the Poisson distribution, among others. Just play around a bit!
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!