Normal Equation in Python: The Closed-Form Solution for Linear Regression | by Suraj Verma | Towards Data Science

In this article, we will implement the Normal Equation which is the closed-form solution for the Linear Regression algorithm where we can find the optimal value of theta in just one step without using the Gradient Descent algorithm.
We will first recap with Gradient Descent Algorithm, then talk about calculating theta using a formula called Normal Equation and finally, see the Normal Equation in Action and plot predictions for our randomly generated data.
Machine Learning from scratch series —
Part 1: Linear Regression from scratch in Python
Part 2: Locally Weighted Linear Regression in Python
Part 3: Normal Equation Using Python: The Closed-Form Solution for Linear Regression
Part 4: Polynomial Regression From Scratch in Python
medium.com
towardsdatascience.com
We have,
X →Input data (Training Data)
y →Target variable
theta →The parameter
y_hat →Prediction/hypothesis (dot product of theta and X).
Loss Function →MSE loss or mean squared error loss (y_hat-y)2
m →the number of training examples.
n →number of features
First, we initialize the parameter theta randomly or with all zeros. Then,
Calculate the prediction/hypothesis y_hat using the equation 1 above.Then use the prediction/hypothesis y_hat to calculate MSE loss like this — (y_hat-y)2.Then take the partial derivative(gradient) of the MSE loss with respect to the parameter theta .Finally use this partial derivative(gradient) to update the parameter theta like this — theta := theta -lr*gradient , where lr is the learning rate.Repeat steps 1 to 4 until we reach an optimal value for the parameter theta .
Calculate the prediction/hypothesis y_hat using the equation 1 above.
Then use the prediction/hypothesis y_hat to calculate MSE loss like this — (y_hat-y)2.
Then take the partial derivative(gradient) of the MSE loss with respect to the parameter theta .
Finally use this partial derivative(gradient) to update the parameter theta like this — theta := theta -lr*gradient , where lr is the learning rate.
Repeat steps 1 to 4 until we reach an optimal value for the parameter theta .
Gradient Descent is an iterative algorithm meaning that you need to take multiple steps to get to the Global optimum (to find the optimal parameters) but it turns out that for the special case of Linear Regression, there is a way to solve for the optimal values of the parameter theta to just jump in one step to the Global optimum without needing to use an iterative algorithm and this algorithm is called the Normal Equation. It works only for Linear Regression and not any other algorithm.
Normal Equation is the Closed-form solution for the Linear Regression algorithm which means that we can obtain the optimal parameters by just using a formula that includes a few matrix multiplications and inversions.
To calculate theta , we take the partial derivative of the MSE loss function (equation 2) with respect to theta and set it equal to zero. Then, do a little bit of linear algebra to get the value of theta.
This is the Normal Equation —
If you know about the matrix derivatives along with a few properties of matrices, you should be able to derive the Normal Equation for yourself.
For reference — Derivation of the Normal Equation for linear regression
You might think what if X is a non-invertible matrix, which usually happens if you have redundant features i.e your features are linearly dependent, probably because you have the same features repeated twice. One thing you can do is go and find out which features are repeated and fix them or you can use the np.pinv function in NumPy which will also give you the right answer.
Calculate theta using the Normal Equation.Use the theta to make predictions.
Calculate theta using the Normal Equation.
Use the theta to make predictions.
Check the shapes of X and y so that the equation matches up.
Let’s take the following randomly generated data as a motivating example to understand the Normal Equation.
import numpy as npnp.random.seed(42)X = np.random.randn(500,1)y = 2*X + 1 + 1.2*np.random.randn(500,1)X.shape, y.shape>>((500, 1), (500,))
Here, n =1 which means the matrix X has only 1 column and m =500 means X has 500 rows. X is a (500x1) matrix and y is a vector of length 500.
Let’s write the code to calculate theta using the Normal Equation.
See comments (#).
def find_theta(X, y):        m = X.shape[0] # Number of training examples.     # Appending a cloumn of ones in X to add the bias term.    X = np.append(X, np.ones((m,1)), axis=1)        # reshaping y to (m,1)    y = y.reshape(m,1)        # The Normal Equation    theta = np.dot(np.linalg.inv(np.dot(X.T, X)), np.dot(X.T, y))        return theta
See comments (#).
def predict(X):        # Appending a cloumn of ones in X to add the bias term.    X = np.append(X, np.ones((X.shape[0],1)), axis=1)        # preds is y_hat which is the dot product of X and theta.    preds = np.dot(X, theta)        return preds
See comments (#).
# Getting the Value of theta using the find_theta function.theta = find_theta(X, y)theta>>array([[1.90949642],        [1.0388102 ]]# Getting the predictions on X using the predict function.preds = predict(X)# Plotting the predictions.fig = plt.figure(figsize=(8,6))plt.plot(X, y, 'b.')plt.plot(X, preds, 'c-')plt.xlabel('X - Input')plt.ylabel('y - target / true')
The cyan line shows the predictions for all the values of X .
We found the optimal values of theta in just one step, and the theta we found is the Global Minimum of the MSE loss function for the given data.
If the algorithm you want to use is Linear Regression and exactly Linear Regression and,
If n (number of features) is small.
If m (number of training examples) is small i.e. around 20,000.
Normal Equation is a good algorithm to consider to build your machine learning model.
For questions, comments, concerns, talk to me in the response section. More articles on ML from scratch are coming soon.
Machine Learning from scratch series —
Part 1: Linear Regression from scratch in Python
Part 2: Locally Weighted Linear Regression in Python
Part 3: Normal Equation Using Python: The Closed-Form Solution for Linear Regression
Part 4: Polynomial Regression From Scratch in Python