Bayesian logistic regression with PyMC3 | by Tung T. Nguyen | Towards Data Science

This is another article in a series of articles (see here and here for the other relevant articles) on probabilistic programming in general and PyMC3 in particular. In our previous articles, we explained how PyMC3 helps with statistical inference. In this article, we will solve a classification problem from end to end with PyMC3. More precisely, we will use PyMC3 to do Bayesian logistic regression using the following public dataset:
https://archive.ics.uci.edu/ml/datasets/Occupancy+Detection+
The dataset contains several variables such as light, temperature, humidity, and CO2 levels. The goal is to detect a room’s occupancy from these variables.
First, we will need to load several relevant packages.
import arviz as azimport matplotlib.pyplot as pltimport numpy as npimport pandas as pdimport pymc3 as pmimport seabornimport theano.tensor as ttimport warningsfrom IPython.core.pylabtools import figsizeimport seaborn as snsfrom sklearn.metrics import (roc_curve, roc_auc_score, confusion_matrix, accuracy_score, f1_score,                              precision_recall_curve) from sklearn.metrics import confusion_matrix
Next, we load the dataset.
df=pd.read_csv('datatest.txt')df.sample(5)
For convenience, we convert the date variable into a DateTime object.
df['date']=pd.to_datetime(df['date'])
First, let us take an overview of the dataset.
df.describe()
We see that there are 2655 samples in this dataset. Furthermore, there are no missing values. Let us also take a look at the timeframe of this dataset.
df['date'].describe()count                    2665unique                   2665top       2015-02-03 07:25:59freq                        1first     2015-02-02 14:19:00last      2015-02-04 10:43:00Name: date, dtype: object
So our data are collected in only three days. Next, we will explore our variables and their relationship. First, let us plot the temperature variable.
figsize(12.5, 4)plt.hist(df['Temperature'], bins=40, density=True, label='Temperature')plt.xlabel('Temperature')plt.title('Distribution of temperature')plt.show()
The plot shows that the temperature has a heavy tail distribution. What about the Humidity variable?
figsize(12.5, 4)plt.hist(df['Humidity'], bins=50, density=True, label='Humidity')plt.xlabel('Humidity')plt.title('Distribution of Humidity')plt.show()
It is interesting to see that there are two peaks around 22.5 and 25. We are also interested in the Light variable during different days.
figsize(12.5, 4)sns.boxplot(x=df['date'].dt.day,y=df['Light'], orient='v')plt.xlabel('Day')plt.title('Boxplot for Light during different days')plt.show()
We see that the distributions of light are almost identical during these three days. Next, let us take a look at the CO2 level.
figsize(12.5, 4)sns.boxplot(x=df['date'].dt.day,y=df['CO2'], orient='v')plt.xlabel('Day')plt.title('Boxplot for CO2 level during different days')plt.show()
These distributions are significantly different. There are lots of outliers on 2/4/2015. Finally, we will dive into the Humidity Ratio variable.
figsize(12.5, 4)sns.boxplot(x=df['date'].dt.day,y=df['HumidityRatio'], orient='v')plt.xlabel('Day')plt.title('Boxplot for Humidity Ratio level during different days')plt.show()
This looks pretty much similar to the boxplot for the CO2 level. Maybe, there is a strong correlation between the CO2 level and Humidity Ratio. We can check that by using a scatter plot for these two variables.
ax=sns.scatterplot(df['CO2'], df['HumidityRatio'], style=df['date'].dt.day)
Indeed, there is a strong linear relationship between the CO2 level and the Humidity Ratio. Let us take an overview of the relation between our variables. This can be done by the pair plot function with seaborn.
ax=seaborn.pairplot(df)
This visually shows that there is a strong linear relationship between the following pairs: CO2 and temperature, CO2 and Humidity, Humidity and Humidity Ratio, Humidity Ration, and CO2. We can even quantify these relations by plotting the heatmap.
corr=df.iloc[:, 1:-1].corr()# Generate a mask for the upper trianglemask = np.triu(np.ones_like(corr, dtype=np.bool))# Set up the matplotlib figuref, ax = plt.subplots(figsize=(11, 9))# Draw the heatmap with the mask and correct aspect ratioax=sns.heatmap(corr, mask=mask,            square=True, linewidths=.5, cbar_kws={"shrink": .5})
It is clearly shown that the two pairs Humidity-Humidity Ration and Humidity Ratio-CO2 express the strongest linear relationship.
We will build several machine learning models to classify Occupancy based on other variables.
Recall that we have a binary decision problem. In other words, our target variable is assumed to follow a Bernoulli random variable with p given by:
where var is the set of all variables that we use in our model and logit is the logistic function.
To build a Bayesian logistic regression model, we first have to put a prior distribution on each parameter. The choice of these priors will affect the outcome (though with more data, they probably will “converge” to the same distribution.)
Once our priors are specified, PyMC3 will numerically approximate the posterior distributions using Markov Chain Monte Carlo simulations and its generalizations. We can then use samples from these posteriors to make inferences.
Since we have no prior knowledge about these parameters, we can assume that they could be anything. In other words, we assume that all β_var follows a uniform distribution with large lower and upper bounds. In order to catch a big net, we use big lower and upper bounds for our uniform distributions.
lower=-10**6higher=10**6with pm.Model() as first_model:    #priors on parameters    beta_0=pm.Uniform('beta_0', lower=lower, upper= higher)    beta_temp=pm.Uniform('beta_temp', lower, higher)    beta_humid=pm.Uniform('beta_humid', lower, higher)    beta_light=pm.Uniform('beta_light', lower, higher)    beta_co2=pm.Uniform('beta_co2', lower, higher)    beta_humid_ratio=pm.Uniform('beta_humid_ration', lower, higher)        #the probability of belonging to class 1    p = pm.Deterministic('p', pm.math.sigmoid(beta_0+beta_temp*df['Temperature']+                               beta_humid*df['Humidity']+                               beta_light*df['Light']+                               beta_co2*df['CO2']+                               beta_humid_ratio*df['HumidityRatio']))with first_model:    #fit the data     observed=pm.Bernoulli("occupancy", p, observed=df['Occupancy'])    start=pm.find_MAP()    step=pm.Metropolis()        #samples from posterior distribution     trace=pm.sample(25000, step=step, start=start)    burned_trace=trace[15000:]
This might take a while to run. Once it is done, we can plot the samples.
pm.traceplot(burned_trace)plt.show()
We conclude that our algorithm does converge. We can compute the mean of these posterior distributions.
coeffs=['beta_0', 'beta_temp', 'beta_humid', 'beta_light', 'beta_co2', 'beta_humid_ration']d=dict()for item in coeffs:    d[item]=[burned_trace[item].mean()]    result_coeffs=pd.DataFrame.from_dict(d)    result_coeffs#coeff_result=pd.DataFrame(d)    #coeff_result
An advantage of Bayesian statistics in comparison with frequentist statistics is that we have a lot more than just a mean value. In particular, we can compute the 95% High-Density Interval for those parameters.
pm.stats.hpd(burned_trace['beta_0'])coeffs=['beta_0', 'beta_temp', 'beta_humid', 'beta_light', 'beta_co2', 'beta_humid_ration']interval=dict()for item in coeffs:    interval[item]=pm.stats.hpd(burned_trace[item]) #compute 95% high density interval    result_coeffs=pd.DataFrame.from_dict(interval).rename(index={0: 'lower', 1: 'upper'})result_coeffs
Note that, the coefficients of the Humidity Ratio are significantly bigger than other coefficients. That does not necessarily mean that this variable is more important. If we take a close look at the data, we observe that this variable takes very small values.
Furthermore, we can explore the relationship between different parameters. For example, let us take a look at the beta_co2 and beta_humid_ratio coefficients.
figsize(12.5, 12.5)seaborn.jointplot(burned_trace['beta_co2'], burned_trace['beta_humid_ration'], kind="hex") #color="#4CB391")plt.xlabel("beta_co2")plt.ylabel("beta_humid_ratio");
The plots show that these two coefficients are negatively correlated. Note that the CO2 level and Humidity Ratio are positively correlated.
Recall that in classical logistic regression, we find for the best parameters by maximum a posteriori estimation (MAP solution). In other words, the best-fitted parameters are given by
where p(θ|D)p(θ|D) is the posterior distribution of θ given the data, p(D|θ) is the likelihood function, and p(θ) is the prior distribution of θ.
Note that since we use uniform distributions on our first model, we can expect that our MAP solution should coincide with the MLE solution (maximum likelihood estimation) which corresponds to frequentist logistic regression. We can use the Scikit-Learn library to test this statement. First, we compute the coefficients using MAP.
coeffs=['beta_0', 'beta_temp', 'beta_humid', 'beta_light', 'beta_co2', 'beta_humid_ration']d=dict()for item in coeffs:    d[item]=[float(start[item])]    map_coeffs=pd.DataFrame.from_dict(d)    map_coeffs
Next, we compute the beta coefficients using classical logistic regression.
from sklearn.linear_model import LogisticRegressionX=df.iloc[:, 1: -1]y=df['Occupancy']logit=LogisticRegression()logit_model=logit.fit(X,y)pd.DataFrame(logit_model.coef_, columns=X.columns)
YES! The coefficients for the two methods are almost identical.
Next, we discuss the prediction power of our model and compare it with the classical logistic regression. We record the prediction using the classical method.
logit_prediction=logit_model.predict(X)
To make predictions with our Bayesian logistic model, we compute the y_score by averaging over our sample values.
#compute the average probability of predicting 1. y_score = np.mean(burned_trace['p'], axis=0)#histogram of the distributionfigsize(12.5,4)plt.hist(y_score, bins=40, density=True)plt.xlabel('Probability')plt.ylabel('Frequency')plt.title('Distribution of $y_score$')plt.show()
It is interesting to see that the majority of p is concentrated near 0 and 1. We can use y_score to make a prediction as well.
first_model_prediction=[1 if x >0.5 else 0 for x in y_score]
Let us evaluate the performance of our model by computing the confusion matrix.
first_model_confussion_matrix =confusion_matrix(df['Occupancy'], first_model_prediction)first_model_confussion_matrixarray([[1639,   54],       [   3,  969]])
This is pretty good. We can even quantify the performance by other metrics as well.
import sklearnfrom sklearn.metrics import classification_reportprint(sklearn.metrics.classification_report(y, first_model_prediction))precision    recall  f1-score   support           0       1.00      0.97      0.98      1693           1       0.95      1.00      0.97       972    accuracy                           0.98      2665   macro avg       0.97      0.98      0.98      2665weighted avg       0.98      0.98      0.98      2665
We can also compute the area under the curve.
pred_scores = dict(y_true=df['Occupancy'],y_score=y_score)roc_auc_score(**pred_scores)0.99358530283253
So, our model performs pretty well. Let’s compare it with the classical logistic regression.
print(sklearn.metrics.classification_report(y, logit_prediction))precision    recall  f1-score   support           0       1.00      0.97      0.98      1693           1       0.95      1.00      0.97       972    accuracy                           0.98      2665   macro avg       0.97      0.98      0.98      2665weighted avg       0.98      0.98      0.98      2665
They are the same! However, with the Bayesian model, we gain more information and hence we are more confident about our estimations.
Now, let us train our model using a different set of priors. For example, we can assume that the coefficients follow normal distributions.
with pm.Model() as second_model:     #priors with normal distribution    beta_0=pm.Normal('beta_0', mu=0, sd=10**4)    beta_temp=pm.Normal('beta_temp', mu=0, sd=10**4)    beta_humid=pm.Normal('beta_humid', mu=0, sd=10**4)    beta_light=pm.Normal('beta_light', mu=0, sd=10**4)    beta_co2=pm.Normal('beta_co2', mu=0, sd=10**4)    beta_humid_ratio=pm.Normal('beta_humid_ration', mu=0, sd=10**4)    #probability of belonging to class 1    p = pm.Deterministic('p', pm.math.sigmoid(beta_0+beta_temp*df['Temperature']+                               beta_humid*df['Humidity']+                               beta_light*df['Light']+                               beta_co2*df['CO2']+                               beta_humid_ratio*df['HumidityRatio']))#fit observed data into the modelwith second_model:    observed=pm.Bernoulli("occupancy", p, observed=df['Occupancy'])    start=pm.find_MAP()    step=pm.Metropolis()    second_trace=pm.sample(25000, step=step, start=start)    second_burned_trace=second_trace[15000:]pm.traceplot(second_burned_trace)plt.show()
Again, we see that the algorithm does converge. Let us compute the beta coefficients for the MAP solution.
coeffs=['beta_0', 'beta_temp', 'beta_humid', 'beta_light', 'beta_co2', 'beta_humid_ration']d=dict()for item in coeffs:    d[item]=[float(start[item])]    second_map_coeffs=pd.DataFrame.from_dict(d)    second_map_coeffs
They are very closed to the ones that we got in the first model. To go further, as we are in a Bayesian framework, we can even compare the posterior distributions across the two models. For example, let take a look at the intercept variable.
figsize(12.5,4)plt.hist(burned_trace['beta_0']-second_burned_trace['beta_0'], bins=40, density=True)plt.title('Distribution of the difference between beta_0')plt.legend()plt.show()
While the MAP solutions give the same estimation for β_0, we see that the two posteriors are rather different. Let us also compare the posterior distributions of β_temp between these two models.
figsize(12.5,4)plt.hist(burned_trace['beta_temp'], label='First model', bins=40, density=True)plt.hist(second_burned_trace['beta_temp'], bins=40, label='Second model', density=True)plt.title('Distribution of of beta_temp')plt.legend()plt.show()
The difference is in fact small. Next, let us compute the prediction power of the second model.
second_y_score = np.mean(second_burned_trace['p'], axis=0)second_model_prediction=[1 if x >0.5 else 0 for x in second_y_score]second_model_confussion_matrix =confusion_matrix(df['Occupancy'], second_model_prediction)second_model_confussion_matrixarray([[1639,   54],       [   3,  969]])
This is identical to the result that we got from the first model. We can check that the y_score and second_y_score are almost the same.
figsize(12.5,4)plt.hist(y_score-second_y_score, bins=40)plt.title('Distribution of the difference between y_score and second_y_score')plt.ylabel('Frequency')plt.show()
In the previous sections, we use a hands-on approach to build our models. This is rather easy because we only have a few variables. When the number of variables is very large, it will not be very practical. Fortunately, PyMC3 has a built-in generalized linear model in which everything will be automated. Let us use this built-in model to fit our data.
with pm.Model() as third_model:    pm.glm.GLM.from_formula('Occupancy ~ Temperature + Humidity + Light + CO2 + HumidityRatio',                            df,                            family=pm.glm.families.Binomial())    third_trace = pm.sample(25000, tune=10000, init='adapt_diag')pm.traceplot(third_trace)plt.show()
Unlike the previous models, the posterior distributions of our parameters are unimodal in this case.
Let us get a summary of these posterior distributions.
pm.summary(third_trace)
Instead of looking at summarized statistics, we can also look at the MAP solution.
with third_model:    map_solution=pm.find_MAP()d=dict()for item in map_solution.keys():    d[item]=[float(map_solution[item])]    third_map_coeffs=pd.DataFrame.from_dict(d)    third_map_coeffs
We see that there is a significant difference between the MAP solutions between the second and the third model. What about prediction?
with third_model:    ppc = pm.sample_posterior_predictive(third_trace, samples=15000)#compute y_score with third_model:    third_y_score = np.mean(ppc['y'], axis=0)#convert y_score into binary decisions    third_model_prediction=[1 if x >0.5 else 0 for x in third_y_score]#compute confussion matrix third_model_confussion_matrix =confusion_matrix(df['Occupancy'], third_model_prediction)third_model_confussion_matrixarray([[1639,   54],       [   3,  969]])
This confusion matrix is identical to the ones from the first two models. What about the distributions of y_scores for the second and the third model?
figsize(12.5,4)plt.hist(third_y_score-second_y_score, bins=40)plt.title('Distribution of the difference between y_score and second_y_score')plt.ylabel('Frequency')plt.show(
This distribution concentrates around 0. In other words, the distribution of y_scores is almost the same across different models. What about the coefficients, for example, the coefficients for temperature?
figsize(12.5,4)plt.hist(third_trace['Temperature'][-40000:]-second_burned_trace['beta_temp'], bins=40, density=True)plt.title('Difference between the temperature coefficients for the second and the third model')plt.show()
The difference approximately follows a normal distribution with a small mean.
Let us also check the difference between the humidity coefficients.
figsize(12.5,4)plt.boxplot(third_trace['Humidity'][-40000:]-second_burned_trace['beta_humid'])plt.title('Difference between the humidity coefficients for the second and the third model')plt.show()
Again, that difference is small.
We see that even though our models use different priors, the prediction performances are similar. This confirms our belief that as our dataset gets bigger, they should converge to the same solution.
We expect that our project will beginners to PyMC3 learn its syntaxes. We find PyMC3’s codes rather intuitive and we hope that our codes clearly demonstrate this point.
[1] https://docs.pymc.io/notebooks/GLM-logistic.html Official PyMC3 documentation