Building and Evaluating Your Bayesian Statistical Model | by Juan Nathaniel | Towards Data Science

Even for a nondata-scientist, the term Bayesian statistics has been popular. You might have learned during your university days as one of the compulsory classes to take, not realizing how important Bayesian statistics is. In fact, Bayesian statistics is not just a particular method or even a class of methods; it is an entirely different paradigm for doing statistical analysis.
Bayesian statistics provides you with the tools to update your beliefs in the evidence of new data, which is a notion that is common in many real-world scenarios, such as for tracking pandemics, forecasting economic trends, or predicting climate change. They are also the backbone of many of the more well-known statistical models, such as the Gaussian Process.
towardsdatascience.com
More importantly, learning the principles of Bayesian statistics can be a valuable asset for you as a data scientist because it gives you a fresh perspective to solving novel problems with dynamic sources of real-world data.
Note: This is the second post out of a planned 5-part series covering the topic of Bayesian Statistical Programming.
The first post covered the basic theories of Bayesian statistics and how to implement a simple one in Python.
The first post covered the basic theories of Bayesian statistics and how to implement a simple one in Python.
towardsdatascience.com
2. This will introduce you to Bayesian inference and how to perform basic model evaluations.
3. The third part will cover a special type of Bayesian algorithm called Monte Carlo Markov Chain (MCMC).
towardsdatascience.com
4. The fourth part will build upon previous posts to build a more complex Bayesian model.
5. The fifth part will introduce you to the advanced concepts of the Bayesian model’s checking and evaluation.
RecapInference and PredictionModel EvaluationSensitivity Analysis
Recap
Inference and Prediction
Model Evaluation
Sensitivity Analysis
Without further ado, let’s begin the introduction to the Bayesian Statistical Programming post.
In the first post here, I have discussed the basic principle of Bayesian statistics, the key terms, and how to implement a simple model using PyMC3. We use the Radon concentration (toxic gas) example to illustrate how Bayesian programming works in the real-world scenario.
We will get you up to speed from where we left off. So, if you like to follow these steps in greater detail, please refer to my first post before proceeding further.
Install dependencies (recommended to use conda)
Install dependencies (recommended to use conda)
conda install -c conda-forge pymc3
2. Download our data
!wget "https://raw.githubusercontent.com/fonnesbeck/mcmc_pydata_london_2019/master/data/radon.csv"
3. Get the column of interest (log_radon)
import pandas as pdradon = pd.read_csv('./radon.csv', index_col=0)anoka_radon = radon.query('county=="ANOKA"').log_radon
4. Build a simple Bayesian model
from pymc3 import Model, Normal, Uniformwith Model() as radon_model:    μ = Normal(’μ’, mu=0, sd=10)    σ = Uniform(’σ’, 0, 10)
5. Compile and train the model
with radon_model:    dist = Normal('dist', mu=μ, sd=σ, observed=anoka_radon)
6. Draw some random samples from the trained Bayes model
from pymc3 import samplewith radon_model:    samples = sample(1000, tune=1000, cores=2, random_seed=12)
Now that you have caught up with these steps, let’s move on to the main topics of this post: Bayesian inference and evaluation
Given our trained Bayesian model, we can attempt to make a prediction or an inference. Suppose we are interested to find out the probability of any household in the US being exposed to log-radon concentration above 1.1 (ie. P(log-radon > 1.1)).
We can perform a posterior probability prediction. Remember this guy from our previous post?
But instead of estimating a parameter, θ (read: theta), from our observable data, y, we infer the log-radon concentration (z) as follows.
And similar to the Bayesian formula, we can find the probability of z given all observations, y, by the following formula.
The posterior prediction formula will sum up the probability of z given θ parameters multiplied by the probability of all θ given all observations y, for all values of possible θ
Using PyMC3, we first sample the posterior distribution 1000 times to get 1000 values of z (log-radon).
from pymc3 import sample_posterior_predictivewith radon_model:    pp_samples = sample_posterior_predictive(samples, 1000)
Then, we can find the mean value to retrieve the probability.
radon_thres = 1.1(pp_samples['dist'] > radon_thres).mean()>>> 0.39105769230769233
The probability of a US household in ANOKA having a log-concentration of Radon above 1.1 is approximately 39.11%.
But how do we know if our Bayesian model is any good?
One of the basic evaluation methods would be to compare our prediction against the observable data. You can choose from a plethora of statistical tests, such as the t-test or F-test, to compare the similarity between the two “populations”.
For our present use case, let’s use a graphic visualization to compare how our prediction fares with the empirical value.
First, let’s find a collection of samples that exceeds our log-radon concentration threshold of 1.1:
import seaborn as snsimport matplotlib.pyplot as pltp_thresh = [(sample>1.1).mean() for sample in pp_samples['dist']]
Then we plot the distribution of these samples with the empirical value.
ax = sns.distplot(p_thresh)plt.vlines((anoka_radon>1.1).mean(), *ax.set_ylim(), color='g')
The green vertical line illustrates the mean empirical probability that a household in ANOKA has log-radon exposure concentration that is above 1.1, while the histogram represents our sampling distribution from the trained Bayesian model.
The plot suggests that our initial assumption is not bad but there are rooms for improvements to make the distribution converge to the true empirical value. Further fine-tuning can be done by:
Changing our assumptions about the distributions of your parameters,Experimenting with different initial values for our Bayes model
Changing our assumptions about the distributions of your parameters,
Experimenting with different initial values for our Bayes model
It is also important to check for the sensitivity of our priors chosen earlier when building the model.
This can be done by building the same model but varying the prior distributions as follow.
from pymc3 import Flat, HalfCauchy, plot_posteriorwith Model() as sensitivity_check:        μ = Flat('μ')    σ = HalfCauchy('σ', 5)        dist = Normal('dist', mu=μ, sd=σ, observed=anoka_radon)        sensitivity_samples = sample(1000, cores=2)
And then plotting the distribution.
# Sensitivity evaluation distributionplot_posterior(sensitivity_samples, var_names=['μ'], ref_val=1.1)
And comparing it with the original.
# Original distributionplot_posterior(samples, var_names=['μ'], ref_val=1.1)
Comparing both plots, our priors seem to be insensitive to variation (ie. pretty robust), highlighting that we are one step to achieving our goal! The only thing left is to make further adjustments to our model by constantly experimenting with different priors or initial values.
We have discussed how to perform inference and evaluate our Bayesian models. In the next post, I will cover the underlying processes that govern Bayesian statistical programming, such as the Monte Carlo Markov Chain (MCMC) algorithm that helps us perform sampling. Stay tuned!
Do subscribe to my Email newsletter: https://tinyurl.com/2npw2fnz where I regularly summarize AI research papers in plain English and beautiful visualization.