Bayesian Linear Regression in Python via PyMC3 | by Dr. Robert Kübler | Towards Data Science

In this article, we will see how to conduct Bayesian linear regression with PyMC3. If you got here without knowing what Bayes or PyMC3 is, don’t worry! You can use my articles as a primer
A gentle Introduction to Bayesian Inference
Conducting Bayesian Inference in Python using PyMC3
You can view Bayesian linear regression as a more verbose version of standard linear regression. Linear regression gives you single values, for the model parameters as well as the predictions. Bayesian linear regression, in turn, gives you distributions.
We will see what this exactly means in a second. Let us quickly introduce a simple dataset to be able to compare both linear regression approaches.
We have done it all several times: Grabbing a dataset containing features and continuous labels, then shoving a line through the data, and call it a day. As a running example for this article, let us use the following dataset:
x = [   -1.64934805,  0.52925273,  1.10100092,  0.38566793, -1.56768245,    1.26195686,  0.92613986, -0.23942803,  0.33933045,  1.14390657,    0.65466195, -1.36229805, -0.32393554, -0.23258941,  0.17688024,    1.60774334, -0.22801156,  1.53008133, -1.31431042, -0.27699609] # inputsy = [   -3.67385666,  3.37543275,  6.25390538,  1.41569973, -2.08413872,    6.71560158,  6.32344159,  2.40651236,  4.54217349,  6.25778739,    4.98933806, -2.69713137,  1.45705571, -0.49772953,  1.50502898,    7.27228263,  1.6267433 ,  6.43580518, -0.50291509,  0.65674682] # outputs
We can’t see much by looking at the raw numbers, so let’s visualize it.
Look about linear. Now, whatever linear regression tool we use (statsmodels, scikit-learn, R’s lm method, ...), we will end up with:
We see that the algorithm computes a slope of around 3.2, an intercept of around 2.19, and hence outputs the model y = 3.2x + 2.19. So far, so good. If desired, we can even check the p-value for the slope and determine if this apparent positive connection between x and y is significant.
If you are into this: The t-statistic is around 13.8, the p-value is around 0.0 and a 95% confidence interval is [2.7, 3.7]. In short, this means that x has a significant positive influence on y. Alright, enough of this. Let us get to one of the main drawbacks of linear regression.
The standard linear regression model outputs a single y value for each input x. Do you want to know the prediction for x = 0? Well, it is y = 2.19, take it or leave it.
To understand why this is dangerous, ask yourself the following question:
You have to use a standard linear regression model that was trained on only 5 data points, and has a decent r2 score. Are you really feeling safe about the predictions made by the model for a new data point?
Sure, the model outputs a number like 2.19, but could it be possible that the prediction is 1.1? Or even -17.3? Or is the model certain that it is really closely around 2.19?
I will not lie to you: there are ways to address this issue not only for linear regression but for any regression algorithm via bootstrapping. In short:
Subsample your data, fit a model on this subsample and use it to make a prediction. Repeat this 1000 times and you obtain 1000 different predictions.
Of course, 1000 should just represent a large number.
You can compute the mean μ, the standard deviation σ, quantiles, anything you want of these 1000 samples. If you still want to output a single number, just take the mean. However, now you also have the choice to output intervals, such as [μ-3σ, μ+3σ].
This approach works well, but it’s extremely computationally expensive since we have to fit a model 1000 times. Let us, therefore, take another road and upgrade the standard linear regression approach instead.
First, we have to explore the theory of Bayesian linear regression to then be able to understand the code in PyMC3.
In Bayesian statistics, we deal with distribution. This makes it necessary to specify linear regression as a distribution as well. One assumption commonly used for the standard linear regression approach is the following:
or equivalently
This means that y is normally distributed with mean ax+b (the quantity we always predict) and some standard deviation σ.
In Bayesian statistics, we don’t treat a and b as fixed variables, but also as coming from distributions. This is usually the hardest part: How could they be distributed? Let’s just say
This tells the model that without any data, we expect the slope and intercept of the line to be around zero with a standard deviation of four, i.e. we expect a and b to be around -12 and 12. The prior is something we have to play around with if things don’t work out. This means that PyMC3 does not give a proper result, or the prediction performance of the model is bad.
But we missed a parameter: what about σ? We could give it the same normal distributions as a and b. What we do instead is modeling σ2 directly because we never need σ directly, but only σ2 for the variance of y. Since σ2 is a positive number, we choose a prior distribution that yields exclusively positive values as well, such as the Exponential distribution, the Gamma distribution, or the Half-normal distribution. Let us go for the exponential distribution.
Examples of exponential distributions are the following:
Note that the density of the exponential distribution is always decreasing. Using this distribution as a prior tells the model that we expect a small, but also positive value for σ2. That’s all we need! let’s switch from math to PyMC3.
We can program the model described before the following way and plot the posterior distributions.
import pymc3 as pmimport arviz as azwith pm.Model() as model:    # define priors    a = pm.Normal('slope', 0, 16)    b = pm.Normal('intercept', 0, 16)    s = pm.Exponential('error', 1)        # predictions    obs = pm.Normal('observation', a*x + b, s, observed=y)        # use MCMC to sample    trace = pm.sample(return_inferencedata=True)    az.plot_posterior(trace)
We can see that the model comes to the same conclusion as the standard linear regression: A slope of around 3.2 and an intercept of around 2.2. We even see that the standard error is about 1.1. But here, we also get credible intervals for free!
Looking at these figures, it seems like the model is still unsure about the parameters. Look at the slope, for example. With a probability of 94%, the slope a is between 2.7 and 3.7, which is still a rather wide interval. The same goes for the other parameters we estimated.
The great thing about Bayesian reasoning: All of these distributions narrow down further, i.e. the model gets more certain the more data we provide.
This is all good, but the goal was not to estimate the model’s parameters. Don’t get me wrong, it’s nice to have them, but actually, we are interested in getting predictions for new values of x.
Unfortunately, this is not as straightforward as in scikit-learns fit-predict paradigm. It’s still easy, and you can create a model with changeable inputs as follows:
with pm.Model() as predictive_model:    a = pm.Normal('slope', 0, 16)    b = pm.Normal('intercept', 0, 16)    s = pm.Exponential('error', 1)        x_ = pm.Data('features', x) # a data container, can be changed        obs = pm.Normal('observation', a*x_ + b, s, observed=y)        trace = pm.sample()
I have marked the changes for you in bold. It’s basically telling the model to use a placeholder x_ which was initially filled with our training data x. We then train the model, i.e. get posterior distributions for all of the parameters. We can pass the model new data via
x_new = np.linspace(-3, 3, 50) # 50 input values between -3 and 3with predictive_model:    pm.set_data({'features': x_new})    posterior = pm.sample_posterior_predictive(trace)
This is basically all you have to do! To have a better understanding, let us check what’s inside posterior : It’s a dictionary that contains the new observations. Let’s grab them and store them as y_pred .
y_pred = posterior['observation']
The variable y_pred is a numpy array containing 4000 observations for each of the 50 inputs in x_new , hence its dimensions are 4000x50. You can also inspect this via y_pred.shape . You can visualize this array as:
For example, y_pred[:, 0] gives us 50 predictions, one for each entry in x_new , while y_pred[0, :] gives us 4000 predictions for the first element in x_new , namely -3.
Using this table, we can get the mean and the standard deviation across the 4000 samples for each of the 50 new inputs and visualize the result.
y_mean = y_pred.mean(axis=0)y_std = y_pred.std(axis=0)plt.figure(figsize=(16, 8))plt.scatter(x, y, c='k', zorder=10, label='Data')plt.plot(x_new, y_mean, label='Prediction Mean')plt.fill_between(x_new, y_mean - 3*y_std, y_mean + 3*y_std, alpha=0.33, label='Uncertainty Interval ($\mu\pm3\sigma$)')plt.xlabel('$x$')plt.ylabel('$y$')plt.ylim(-14, 16)plt.legend(loc='upper left')
Nice, right? That’s just what we wanted! However, it feels like a bit much code, right? Especially if we merely want to do a really simple Bayesian linear regression and nothing fancy. Let’s see how we can achieve the same results faster.
Similar to the R syntax, you rewrite our first simple model as
with pm.Model() as simpler_model:    pm.glm.GLM.from_formula('y ~ x', dict(x=x, y=y))        trace = pm.sample()
This is a nice feature you should know! Another great property about this submodule is that it integrates with pandas data frames very well. And you can even specify priors if you don’t like the defaults (normal for all coefficients, half-Cauchy for the error as far as I know). Take a look at the following code:
import pandas as pddf = pd.DataFrame({'x': x, 'y': y})my_priors = {    'Intercept': pm.Cauchy.dist(0, 10),    'x': pm.Cauchy.dist(0, 10)}with pm.Model() as model:    pm.glm.GLM.from_formula('y ~ x', df, priors=my_priors)        trace = pm.sample()
In this article, you have learned the difference between regular linear regression and its Bayesian counterpart. Linear regression models output single values as predictions, while Bayesian linear regression models can output distributions as predictions.
This has the advantage that we can quantify the uncertainty for our predictions, and be careful when the prediction interval is too large. In our example, we could see that the slope is clearly positive. Maybe it is not 3.2, but with high probability, it is between 1.7 and 3.7. In fact, I used 3 as a slope and 2 as an intercept. So, if it is better to underestimate the true y, let us use 1.7 as the slope. If overestimating is better, we could use 2.7.
We have used PyMC3 to implement Bayesian linear regression for one variable. Of course, you can add as many variables as you wish (and your machine can handle). You just have to declare more variables. Either explicitly like
a = pm.Normal('slope', 0, 16)a2 = pm.Normal('slope2', 0, 16)a3 = pm.Normal('slope3', 0, 16)
or more compact and scalable as
a = pm.Normal('slopes', 0, 16, shape=3)
Now, you should try it out on your own! It’s fun to play around with, especially if you leave the path of very simple models. You can use different distributions than the normal distribution for y, for example, the Student’s t-distribution or the Cauchy distribution.
The tails of these two distributions are wider than the ones of the normal distribution. This has the effect that outliers don’t surprise, and hence influence the model as much as in the case of the normal distribution.
Maybe we don’t want to predict continuous values at all but counts instead. In this case, we could use the Poisson distribution via
You can do it manually, or provide a family argument to the glm submodule, as in
with pm.Model() as model:    pm.glm.GLM.from_formula('y~x', df, family=pm.families.Poisson())        trace = pm.sample()
However you do it, the possibilities are endless. Check your data, think about which distribution makes sense, or just test a lot of different distributions in the worst case! You can even create more complex models, such as Bayesian neural networks if you want. Just start with your own projects, and you will get more experience with this whole topic and become a Bayesian Master.
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!