Linear regression explained. Learn how this supervised machine... | by Zolzaya Luvsandorj | Towards Data Science

Linear regression is probably the most well-known machine learning algorithm out there. It is often the first algorithm to encounter when studying or practicing data science because of its simplicity, speed, and interpretability. If you want to sharpen your understanding of linear regression, this post shows the math behind the algorithm with an emphasis on the normal equation and how to interpret the parameters of a linear regression.
We will see a few code examples written in Python, however, the reader can absorb the conceptual knowledge without knowing Python.
Linear regression is a machine learning algorithm to predict a continuous target.
The linear regression equation is represented by the following:
Linear regression could be further split into two categories depending on the number of features used. If a single feature is used to predict the target, it is called simple linear regression. If multiple (i.e. two or more) features were used, it is known as multivariate or multiple regression.
Now, let’s learn how the model learns the optimal model parameters (i.e. intercept and coefficients). When training linear regression, we want to find the best parameter combination such that they result in the lowest error across all our training examples compared to any other combinations. For a given set of parameters, we can measure the total error using the mean squared error (aka MSE):
Errors are squared such that positive and negative errors don’t cancel each other. There are two ways how we can find the best set of parameters:◼ Using the normal equation◼ Using an optimisation algorithm
We will need to know the basics of matrix transpose, matrix multiplication, and matrix inverse to follow the upcoming normal equation example. Before we dive-in, let’s start with a quick refresher on these topics.
Matrix transpose: Here’s the formula for transposing a 2x2 matrix:
You can imagine that you are flipping the values along the diagonal line stretching from top left to bottom right. Here’s an example:
Matrix multiplication: Here’s the formula for multiplying two 2x2 matrices:
Here’s an example:
Inverse matrix: In order for a matrix to have an inverse matrix, a matrix must be a square matrix (i.e. number of rows = number of columns). Here’s the formula to find the inverse matrix of a 2x2 matrix:
A matrix will not have an inverse if its determinant (i.e. ad-bc) is zero. Such matrix is called a singular matrix.
Here’s an example finding an inverse matrix:
Now, we are ready to get familiar with the normal equation:
Let’s rewrite our linear equation such that the intercept is a special coefficient where the corresponding feature x0 takes a constant value of 1:
Let’s find the parameters of a simple linear regression using the normal equation on a toy dataset with one explanatory feature x1 and a target y so that it is manageable to calculate ourselves.
From below, we see X, a 5x2 matrix for the features. The first column shows x0 and second column shows x1 for the 5 training examples. First, we will multiply X^T by X:
Now, let’s find its inverse:
Next, we will multiply it by X_T:
Lastly, we will multiply it by the actual target values to find the best parameters:
That was it! We find that intercept is 0.3588 and coefficient is 0.2647. We can also do the same calculation in Python to calculate the parameters:
import numpy as npimport pandas as pdpd.options.display.precision = 4from sklearn.linear_model import LinearRegression# Create sample datatrain = pd.DataFrame({'x1': [8, 15, 6, 16, 20],                       'y': [4, 3, 1, 5, 6]})train
# Add x0=[1, 1, .., 1] for the intercept such that X is 5x2 matrixx0 = np.ones(train.shape[0])[:, np.newaxis]X = np.concatenate((x0, train.drop(columns='y')), axis=1)# Reshape y from 1D to 2D such that Y is 5x1 matrixY = train['y'].values.reshape(-1,1)# Find best parameters for the modeltheta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Y)print(f"Intercept: {theta[0][0]:.4f}")print(f"Coefficient for x1: {theta[1][0]:.4f}")# Predicttrain['y_hat_manual'] = X.dot(theta)train['residual'] = train['y'] - train['y_hat_manual']train
Let’s quickly check it against Scikit-learn’s output:
# Check parameters against sklearnmodel = LinearRegression()model.fit(train[['x1']], train['y'])print(f"Intercept: {model.intercept_:.4f}")print(f"Coefficient for x1: {model.coef_[0]:.4f}")# Check predictions against sklearntrain['y_hat_sklearn'] = model.predict(train[['x1']])train
Perfect, the results match!
Of note, there are times where normal equation won’t work if X^T ⋅ X is singular (i.e. the determinant is zero).
Another approach to train the model is to use an optimisation algorithm such as gradient descent to find the best parameters that minimises the total error. An optimisation algorithm solves for the optimal combination of the parameters iteratively. When training a linear regression with an optimisation algorithm, having features on the same scale can help converge faster to the global minimum. In practice, a choice of optimisation algorithm depends on the implementation.
Intercept tells us the expected target value when all features are 0.
In our example, target is expected to be 0.2647 when feature x1 is zero:
Coefficient tells us the expected change in target value when the feature of interest increases by one unit while holding all other features constant. The sign of the coefficient tells us the relationship between the feature and the target. For instance, positive coefficient indicates a positive relationship.
When feature x1 increases by one unit, the target is expected to increase by 0.3588:
That was it for this post! Hope you enjoyed deepening your knowledge on linear regression. We have also refreshed basic linear algebra (e.g. matrix multiplication and matrix inverse) along the way. Lastly, being able to explain how to interpret linear regression results is helpful when explaining the drivers of the model to others. If you are keen to learn more, check out this resource on linear regression assumptions.
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my article. If you are interested, here are links to some of my other posts:
◼️️ K-Nearest Neighbours explained◼️️ Comparing Random Forest and Gradient Boosting◼️️ How are decision trees built?◼️️ Pipeline, ColumnTransformer and FeatureUnion explained◼️️ FeatureUnion, ColumnTransformer & Pipeline for preprocessing text data
Bye for now 🏃 💨
Aurelien Geron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2017 - Chapter 4