The basics of Linear Regression | Towards Data Science

This post is a part of a series of posts that I will be making. You can read a more detailed version of this post on my personal blog by clicking here. Underneath you can see an overview of the series.
(a) What is machine learning?
(b) Model selection in machine learning
(c) The curse of dimensionality
(d) What is Bayesian inference?
(a) How linear regression actually works
(b) How to improve your linear regression with basis functions and regularization
(a) Overview of Classifiers
(b) Quadratic Discriminant Analysis (QDA)
(c) Linear Discriminant Analysis (LDA)
(d) (Gaussian) Naive Bayes
In this post, we will learn about linear regression, which is essential to a lot of machine learning, and will therefore be hugely beneficial to understand.
Given a training dataset of N input variables x with corresponding target variables t, the objective of linear regression is to construct a function h(x) that yields prediction values of t for new values of x.
The simplest linear model for regression is just known as linear regression, where the predictions are generated by
The first term is commonly called the intercept or bias parameter, and allows h to adapt to a fixed offset in the data. If we introduce a 1 as the first element of each input variable x, we can rewrite (1) with vector notation
In the first post of this series we went over polynomial regression with a 1-dimensional input variable. Now, we’re just doing linear regression (not polynomial), but we’re allowing our input variable to be D-dimensional, hence it becomes a vector instead of a scalar. For the sake of visualization, however, let’s stick to the same example dataset:
import numpy as npx = np.array([-1, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8, 1])t = np.array([-4.9, -3.5, -2.8, 0.8, 0.3, -1.6, -1.3, 0.5, 2.1, 2.9, 5.6])N = len(x)
Since our input variables are 1-dimensional, our parameter vector w is 2-dimensional, as it has a weight for the bias and a weight for the input variable.
To make sure we also find the bias parameter, we have to introduce a column of 1s in x. We can do this with the following code snippet
X = np.column_stack([np.ones(N), x])
So, how do we train the model? We’ll look at 2 different approaches of deriving the method of training this model.
Recall that training (or learning) refers to the process of estimating the parameters of our model, so when we ask how to train the model, it’s the same as asking how to estimate the values of w.
Like we did in the first post, we defined an objective function that calculated a measure of the performance of our model in terms of an error, and then we minimized this error with respect to our parameters.
This means we would find the parameters that would result in the least error. We’ll use the same objective function, the sum of squared errors (SSE), defined as
and we want to find values for w that minimizes E
Note that (2) is a quadratic function of the parameters w. Its partial derivatives with respect to w will therefore be linear in w, which means there is a unique minimum. This is a property of convex functions.
If we evaluate the objective function for values of our parameters w​ in a grid, then we can illustrate that our objective function has a unique minimum with a contour plot. This is shown below.
But how do we find the minimum of E?
We find the minimum of a function by taking the derivative, setting the derivative equal to 0, and solving for the function variable. To ease notation, let all our input variables be denoted by X with N rows (one for each input variable) and D+1 columns (one for each feature plus one for the bias), i.e.,
and let t denote a column vector of our target variables. We can now rewrite (2) as
If we now take the derivative with respect to w, we get
and setting this equal to 0 lets us solve for w
which are our estimated values for the parameters
Choosing the SSE as the objective function might seem a bit arbitrary though — for example why not just go with the sum of the errors? Why do we have to square them?
To show why this is a good choice, and why the solution makes sense, we are going to derive the same solution from a probabilistic perspective using maximum likelihood estimation (MLE).
To do this, we assume that the target variable t is given by our function h(x,w) with a bit of noise added
where we assume ε is normally distributed with mean 0 and standard deviation σ.
This lets us say that given an input variable x, the corresponding target value t is normally distributed with mean h(x,w) and standard deviation σ, i.e.,
Let’s take a moment to understand what is really going on.
The image below illustrates what (3) tells us. We are estimating some parameters w so our target variables t follow a normal distribution around the output values of h.
We can now use the entire dataset, X and t, to write up the likelihood function by making the assumption that our data points are drawn independently from (3). The likelihood function then becomes the product of (3) for all our input and target variable pairs, and is a function of w and σ
Now we want to maximize the likelihood, which means we want to determine the values of our parameters w and σ that maximizes (4). This might seem dauntingly difficult, but we can make it simpler with a handy trick. Since the logarithm is a monotonically increasing function, maximizing the log-likelihood is equivalent to maximizing the likelihood.
Taking the log of the likelihood gives us
Now it becomes evident why the SSE objective function is a good choice — the last term of (5) is the only part dependent on w and is the same as SSE.
Since the first term does not depend on w, we can omit it, and since the maximum of the likelihood function with respect to w does not change by scaling with the positive constant, then we see maximizing the likelihood is equivalent to minimizing the SSE objective function.
The maximum likelihood estimate for w is therefore the same as in our previous derivation
We can use the result of the maximum likelihood solution for w to find the value of the noise parameter σ. If we insert the maximum likelihood estimate for w in the log-likelihood, take the derivative, and set it equal to 0, then we can solve for σ
Since we have a probabilistic model now, we make predictions that are probability distributions over t instead of just point estimates. This is done by substituting the maximum likelihood solutions for w and σ into (3)
We can find w and σ with the following code snippet
w = np.linalg.inv(X.T @ X) @ X.T @ tsigma_sq = sum((t - X @ w)**2) / len(t)
Underneath is a chart of the line we’ve estimated from w along with the uncertainty parameter σ.
We can find the parameters for a linear regression through ordinary least squares or maximum likelihood estimation.
Usually in linear regression we have a scalar parameter that is not multiplied by the input called the intercept or bias​.
The process of estimating the values of the parameters is called the training or learning process.
Since the logarithm is a monotonically increasing function, maximizing the likelihood function is the same as maximizing the log-likelihood function.