Predictive Analytics: Bayesian Linear Regression in Python | by Niousha Rasifaghihi | Towards Data Science

In this post, I would like to focus more on the Bayesian Linear Regression theory and implement the modelling in Python for a data science project. The whole project is about forecasting urban water consumption under the impact of climate change in the next three decades. The complete version of the code is available as a Jupyter Notebook on GitHub and I encourage you to check it out. Also, the result is published in Sustainable Cities and Societies.
Statistics have two main branches, descriptive statistics and inferential statistics. Descriptive statistics provide a concise summary of data, while inferential statistics find patterns and make inferences about the data. But why am I talking about these two terms? 😀 Well, there are two general “philosophies” in inferential statistics that are important for us; frequentism and Bayesianism.
Frequentist statistics and Bayesian statistics divide the statistics world by two different approaches to define probability. In Frequentist statistics, the probability is related to the frequencies of the repeated events, whereas, in Bayesian statistics, the probability is related to our own certainty and uncertainty. Therefore, the consequence is that frequentists believe models are fixed, and data vary around them, but Bayesians talk about observed data being fixed, and the models can vary around them.
So the answer to this question “why Bayesian Statistics?” is, with Bayesian statistics, we can use probabilities to represent the uncertainty in any event or hypothesis.
Bayes theorem is the key rule in Bayesian statistics [1]. If H is a hypothesis and D is observed data, the Bayes theorem is as follows
where P(H|D) represents the posterior probability of the hypothesis given the condition that D occurs. P(H) is the prior probability of the hypothesis, P(D) is the marginal (total) probability of observed data and is effectively a normalizing constant, and P(D|H) is the probability (likelihood) of data given hypothesis.
The city of Brossard, Quebec, Canada, is chosen as a study site. I obtained a time series of daily urban water consumption, q, over the time period of January 2011 to October 2015. For the same time period, I also obtained time series of climatic variables: daily minimum temperature θ, daily maximum temperature t, and total precipitation p. Measurements of these climatic variables were made from an Environment Canada station in the Pierre Elliott Trudeau International Airport (YUL).
Bayesian statistics is a powerful technique for probabilistic modelling that has been adopted in a wide range of statistical modelling, including Linear Regression models to make a prediction about a system [2,3,4,5]. A Linear Regression model is expressed as
where q is the dependent (target) variable, β_0 is the intercept; β_1, β_2, and β_3 are the weights (known as the model parameters); θ, t and p are called the predictor variables; and ε is an error term representing random noise or the effect of variables not included in the model equation. The equation can be rewritten as
where β= (β_0, β_1, β_2, β_3); and X = (1, q, Q, p) [2,6]. The Linear Regression is formulated using probability distributions rather than point estimates to predict q. Therefore, q is not estimated as a single value but is assumed to be drawn from a probability distribution. The model for Bayesian Linear Regression with the response sampled from a normal (Gaussian) distribution N is
The output, q, is generated from a normal distribution characterized by a mean and variance. The mean for the normal distribution is the regression coefficient matrix (β) multiplied by the predictor matrix (X). The variance is the square of the standard deviation, σ.
The Bayesian Linear Regression model provides the representation of the uncertainties in predictor variables and determines the posterior distribution for the model parameters. Not only is the response generated from a probability distribution, but the model parameters are assumed to come from distribution as well. The posterior probability of the model parameters comes from Bayes theorem:
where P(β|q,X) is the posterior probability distribution of the model parameter given the predictors and the independent variable; P(q|β,X) is the likelihood of data; P(β |X) is the prior probability of parameters and the denominator of the equation is the marginal probability that can be found as per the law of total probability.
In practice, it is difficult to compute the marginal likelihood for continuous values; it is intractable to calculate the exact posterior distribution. As a solution, a sampling method Markov Chain Monte Carlo is involved in approximating the posterior without computing the marginal likelihood [3,5,7]. Monte Carlo is a general technique of drawing random samples, and Markov Chain means the next sample drawn is based only on the previous sample value. By bringing more samples, the approximation of the posterior will eventually converge on the actual posterior distribution for β_1, β_2, and β_3.
As the starting point in applying Markov Chain Monte Carlo, define parameter space covering all the possible values for β_1, β_2, and β_3. Then, define the prior probability for each of the parameters as a normal distribution. Next, compute the likelihood for each possible parameter. Last, compute prior × likelihood for any given point in parameter space.
Markov Chain Monte Carlo is implemented using the No-U-Turn algorithm [8]. This algorithm is efficient when the variables involved are continuous and there is no need to set the number of steps. This is an advantage over Hamiltonian Monte Carlo, which takes a series of steps informed by first-order gradient information, and is sensitive to the number of steps.
A set of parameter values for accepted moves are generated (if the proposed move is not accepted, the previous value is repeated) and after many repetitions, the empirical approximation of the distribution is found. Eventually, the result of performing Bayesian Linear Regression is the probability density function of possible model parameters based on the data and the prior.
The entire dataset is split into training and testing sets. The training set contains 75% of data and used to build the model; while, the testing set contains 25% and is utilized for verifying the accuracy of prediction. Also, two goodness-of-fit measures are used to evaluate the performance of models developed in this study. These measures are Mean Absolute Error and Root Mean Squared Error. Mean Absolute Error is the average of the absolute value of the difference between predictions and the actual values, and Root Mean Squared Error is the square root of the average of the squared differences between the predictions and the actual values.
As I mentioned before, my focus in this post is on the Bayesian Linear Regression model. Therefore, I skip the data pre-processing step. The input dataset to the Bayesian Linear Regression model is as follow
UWC: Urban water consumption
Max_T: Maximum temperature
Min_T: Minimum temperature
T_P: Total precipitation
⚠️ Prior to model creation, make sure you have handled missing values because the Bayesian Linear Regression model does not take data with missing points.
First, I use sklearn library to split the pre-processed dataset (df) as 75% training and 25% testing.
Note: X_train and X_test include the target variable, UWC.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(df, df['UWC'],test_size = 0.25,random_state = 42)
In this project, I create the Bayesian Linear Regression Model in PyMC3. Let a Markov Chain Monte Carlo algorithm draw samples from the posterior to approximate the posterior for each of the model parameters.
To create the model, I pass the Linear Regression formula, the family which is the prior for the parameters and the data to GLM. To perform Markov Chain Monte Carlo sampling, I specify the number of samples, 2000, the number of chains, 2, and the number of tuning steps, 500.
import pymc3 as pmfrom pymc3 import traceplot# Formula for Bayesian Linear Regression formula = ‘UWC ~ ‘ + ‘ + ‘.join([‘%s’ % variable for variable in X_train.columns[1:]])print(formula)# Context for the modelwith pm.Model() as normal_model:            # The prior for the model parameters will be a normal distribution    family = pm.glm.families.Normal()# Creating the model requires a formula and data (and optionally a family)    pm.GLM.from_formula(formula, data = X_train, family = family)# Perform Markov Chain Monte Carlo sampling    normal_trace = pm.sample(draws=2000, chains = 2, tune = 500)
So, UWC is a function of maximum temperature, minimum temperature and total precipitation.
pm.plot_posterior(normal_trace)
The results are illustrated as posterior histograms for β_0, β_1, β_2, β_3, and σ. The histograms show 95% Highest Posterior Density (HPD), which is a credible interval for the parameters. A credible interval in Bayesian statistics is an equivalent of a confidence interval. For example, the probability of 0.034 <σ< 0.037 is 95%.
To calculate MAE and RMSE, I take the median of the probability distribution for UWC and give it as the prediction to evaluate_prediction function.
# Define a function to calculate MAE and RMSEdef evaluate_prediction(prediction, true):    mae = np.mean(abs(predictions - true))    rmse = np.sqrt(np.mean((predictions - true) ** 2))        return mae, rmsemedian_pred = X_train['UWC'].median()median_preds = [median_pred for _ in range(len(X_test))]true = X_test['UWC']# Display mae and rmsemae, rmse = evaluate_prediction(median_preds, true)print('Mean Absolute Error: {:.4f}'.format(mae))print('Root Mean Square Error: {:.4f}'.format(rmse))
The assessment of model performance shows that the predictor model has good accuracy.
This post was a very brief overview of Bayesian Linear Regression for a data science project. I hope it helped you to better understand Bayesian Linear Regression fundamentals.😊
Your feedback is greatly appreciated. You can reach me on LinkedIn.
[1] Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning Bayesian Networks: The Combination of Knowledge and Statistical Data. Machine Learning, 20(3), 197–243. https://doi.org/10.1023/A:1022623210503
[2] Han, Jiawei and Pei, Jian and Kamber, M. (2011). Data mining: concepts and techniques. Retrieved from http://myweb.sabanciuniv.edu/rdehkharghani/files/2016/02/The-Morgan-Kaufmann-Series-in-Data-Management-Systems-Jiawei-Han-Micheline-Kamber-Jian-Pei-Data-Mining.-Concepts-and-Techniques-3rd-Edition-Morgan-Kaufmann-2011.pdf
[3] Mudgal, A., Hallmark, S., Carriquiry, A., & Gkritza, K. (2014). Driving behavior at a roundabout: A hierarchical Bayesian regression analysis. Transportation Research Part D: Transport and Environment, 26, 20–26. https://doi.org/10.1016/j.trd.2013.10.003
[4] Raftery, A. E., Madigan, D., & Hoeting, J. A. (1997). Bayesian Model Averaging for Linear Regression Models. Journal of the American Statistical Association, 92(437), 179–191. Retrieved from https://www.tandfonline.com/doi/abs/10.1080/01621459.1997.10473615
[5] Yuan, X. C., Sun, X., Zhao, W., Mi, Z., Wang, B., & Wei, Y. M. (2017). Forecasting China’s regional energy demand by 2030: A Bayesian approach. Resources, Conservation and Recycling, 127(May), 85–95. https://doi.org/10.1016/j.resconrec.2017.08.016
[6] Taleb, T., & Kaddour, M. (2017). Hierarchical agglomerative clustering schemes for energy-efficiency in wireless sensor networks. Transport and Telecommunication, 18(2), 128–138. https://doi.org/10.1515/ttj-2017-0012
[7] Godsill, S. J. (2001). On the relationship between markov chain monte carlo methods for model uncertainty. Journal of Computational and Graphical Statistics, 10(2), 230–248. https://doi.org/10.1198/10618600152627924
[8] Hoffman, M. D., & Gelman, A. (2011). The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. (2008), 1–30. Retrieved from http://arxiv.org/abs/1111.4246