Bayesian Thinking for Linear Regression @ Kaggle Days Meetup | by Souradip Chakraborty | Towards Data Science

One of the major motivations of this research is the fact that there has been an increasing focus on Deep model interpretability with the advent of more and more complex models. More is the complexity of the model, difficult it gets to have interpretability with respect to the outputs and a lot of research is going in the field of Bayesian thinking and learning.
But before understanding and being able to appreciate Bayesian in deep neural models, we should be well versed and adept with Bayesian thinking in linear models for example- Bayesian Linear regression. But there are very few good materials available online in a combined fashion which can give a clear motivation and understanding of the Bayesian Linear regression.
This was one of the major motivations for this blog and here I will try to give an understanding of how to approach the Linear regression from a Bayesian analysis standpoint.
Just before starting let's understand that we are very cleary with Maximum Likelihood estimates of Linear regression models. I have discussed this in one of my previous talks, please refer to the link.
Just to introduce very briefly the concept of the Maximum Likelihood Estimate in a Linear regression scenario.
The most important point to understand from this is that MLE gives you a point estimate of the parameter by maximizing the Likelihood P(D|θ).
Even, MAP which is Maximum a posteriori estimation maximizes the posterior probability P(θ|D), which also gives point estimation.
So, these methods don’t give you enough information about the distribution of the coefficients, whereas in Bayesian we estimate the posterior distribution of the parameters. Hence, in this case, the output is not a single value but a probability density/mass function.
Just to understand the intuition behind it.
Just one more step to go !!!
Before delving deep into Bayesian Regression, we need to understand one more thing which is Markov Chain Monte Carlo Simulations and why it is needed?
MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space. But why approximating the distribution and not calculating the exact distribution might be one question that you must be intrigued by.
So, it's almost impossible to compute and intractable due to the denominator and hence we go for approximating the posterior distribution using MCMC.
Monte Carlo methods help us in generating random variables following a given distribution. For example: -θ ~ N(mean, sigma**2), will help in generating multiple θ from the Normal distribution and there are numerous methods to do so.
Markov chain is a sequence of numbers where each number is dependent on the previous number in the sequence. Markov Chain Monte Carlo helps in generating random variables from a distribution where each value of θ is drawn from a known distribution with a mean equal to the previous value.
As you can understand, this is basically a random walk and there can be a lot of values generated which might not be necessary and relevant. Hence, we need to do an acceptance/rejection sampling of the values to generate the distribution of interest.
In this case the Metropolis-Hastings algorithm is used to refine the values by rejection and acceptance sampling.
We can also use Gibb’s sampling, where the goal is to find the posterior distribution P(θ1,θ2|y,x) which is done by obtaining the posterior conditional distributions P(θ1|θ2, y,x) and P(θ2|θ1, y,x). So, we generate
θ1 ~P(θ1|θ2, y,x), replace the value of generated θ1 in the second equation and generate θ2 ~ P(θ2|θ1, y,x) and we continue the process for several iterations to get the posterior.
Now, let’s illustrate the same with an example.
In the below example, I will be illustrating the Bayesian Linear Regression methodology firstly with Gibbs sampling. Here, I have assumed certain distributions for the parameters.
In this section, I will show you an illustration of using Gibbs sampling for Bayesian Linear regression. This section has been taken from the brilliantly illustrated Kieran R Campbell’s blog on Bayesian Linear regression.
So, to begin with, let's understand how the data is for our case. Let D be the dataset for the following experiment and D is defined as,
D = ((x1,y1) ,(x2,y2),(x3,y3) ...... (xn,yn)) is the data, where x1,x2.... belongs to the univariate feature space.
Y ~ N( b*x + c, 1/t), where Y is a random variable normally distributed with mean b*x + c and variance of 1/t, where t represents the precision.
In this case, we will be considering a univariate feature space (X) with two parameters, slope(c) and intercept (b). So, in this experiment, we will learn the posterior distributions of the above parameters c & b and the precision t.
Let's write down the assumed prior distributions of the above parameters
So, lets first write down the density function of a normal distribution and the log_pdf to have a generalizable form, which will be of use multiple times later.
Now it will be very easy for us to pen down the likelihood for our case which also follows Normal with mean bx+c and precision t.
Taking the logarithm of the same gives the below expression
Now comes the little complex part which is to derive the conditional posterior distribution for all the three parameters b,c,t.
Conditional Posterior Distribution for the intercept_c :
So, as we can see that this is the process to find out the conditional posterior distribution for the intercept c.
The code snippet of the above equation
def get_intercept_c(y, x, b, t, c0, tc):    n = len(y)    assert len(x) == n    precision = c0 + t * n    mean = tc * c0 + t * np.sum(y - b * x)    mean = mean/precision    return np.random.normal(mean, 1 / np.sqrt(precision)
Similarly, we can find the same for the slope b.
Conditional Posterior Distribution for the slope_b :
The code-snippet of the update for the slope is
def get_slope_b(y, x, c, t, b0, tb):    n = len(y)    assert len(x) == n    precision = tb + t * np.sum(x * x)    mean = tb * b0 + t * np.sum( (y - c) * x)    mean = mean/precision    return np.random.normal(mean, 1 / np.sqrt(precision))
Conditional Posterior Distribution for the precision_t :
The conditional posterior distribution gives b,c will not be like the above two as the prior follows Gamma distribution.
def get_precision_t(y, x, b, c, alpha, beta):    n = len(y)    alpha_new = alpha + n / 2    resid = y - c - b * x    beta_new = beta + np.sum(resid * resid) / 2    return np.random.gamma(alpha_new, 1 / beta_new)
Now, since we could get the closed distributional form for the parameters, now we can generate from the posterior distribution using MCMC simulations.
So, first, to run the experiment, I generated a sample data with known slope and intercept coefficients which we can validate from our Bayesian Linear regression. We have assumed for the experiment a=6, b = 2
# observed datan = 1000_a = 6_b = 2x = np.linspace(0, 1, n)y = _a*x + _b + np.random.randn(n)synth_plot = plt.plot(x, y, "o")plt.xlabel("x")plt.ylabel("y")
Now, after updating the parameters based on the equations and snippets shown above, we run the MCMC simulations for getting the true posterior distribution of the parameters.
def gibbs(y, x, iters, init, hypers):    assert len(y) == len(x)    c = init["c"]    b = init["b"]    t = init["t"]        trace = np.zeros((iters, 3)) ## trace to store values of b, c, t        for it in tqdm(range(iters)):        c = get_intercept_c(y, x, b, t, hypers["c0"], hypers["tc"])        b = get_slope_b(y, x, c, t, hypers["b0"], hypers["tb"])        t = get_precision_t(y, x, b, c, hypers["alpha"], hypers["beta"])        trace[it,:] = np.array((c, b, t))            trace = pd.DataFrame(trace)    trace.columns = ['intercept', 'slope', 'precision']            return traceiters = 15000trace = gibbs(y, x, iters, init, hypers)
On running the experiment for 15000 iterations, we see the trace plot to validate our hypothesis. There is a concept of a Burn-in period in MCMC simulations where we ignore the first few iterations as it doesn’t replicate samples from the true posterior rather random samples.
So as you can see, the sample posterior distribution replicated our assumptions and we have much more information about the coefficients and the parameters.
But it is not always possible to have a closed distributional form of the conditional posterior and hence we have to opt for a proposal distribution with acceptance & rejection sampling using the Metropolis-Hastings algorithm discussed above briefly.
Note: There are a lot of advantages of Bayesian thinking and reasoning and I will be coming up with subsequent materials on Variational Inference and Bayesian thinking in my upcoming blogs and materials. I will be delving into Bayesian Deep Learning and its advantages. So, keep reading and sharing knowledge.
P.S — This talk was part of the Kaggle Days Meetup Delhi-NCR session, please follow them for such materials and sessions. You will also get my video explaining the topic briefly.