2 ways to train a Linear Regression Model-Part 1
By: Jeffy Merin Jacob
Tags: Machine Learning, Data Science, Linear Regression

One of the simplest Supervised Machine Learning models out there is Linear Regression. In this post, we will discuss two different ways we can train a Linear Regression model.

Let me list them out really quickly before I move into explaining each one in detail:

1. Train using closed-form equation

2. Train using Gradient Descent

The first way directly computes the model parameters that best fit the model to the training set and the second computes it iteratively. I will talk about closed-form equation in this post and will create a Part-II talking about Gradient Descent and variants of it.

But first, how does a linear model make predictions?

Linear regression consists of finding the best-fitting straight line through the data. This best-fitting line is what we call the regression line. A linear model makes prediction by computing the weighted sum of input features, plus a constant called the bias term¹

The equation for a Linear Regression model prediction can be given by:

How do we train this model?

You see the different θ’s in the equation? Those are called model parameters. To train a model, we need to find the value of θ (model parameters) that minimizes the cost function. There are 2 ways to do this! But first, let’s define our cost function.

The most common measure of performance of a regression model is RMSE. So, we need to find values of θ that minimize RMSE. But in practice, it is much easier to minimize MSE. Let’s say the cost function we are trying to minimize is Mean Squared Error. The MSE is calculated using the below equation

Now that we have our cost function defined, let’s go ahead and jump into the first way of finding the best model parameter values (θ).

“Closed-form” Equation

There is a mathematical equation that computes the model parameters directly. This equation is called the Normal Equation. You can read more about Normal Equations here. The Normal equation is given by:

But inserting all the values that we have into this equation; we will be able to compute the best model parameters that minimize our cost function. Easy right? Let’s test the equation on a linearly generated data.

X = 2*np.random.rand(100,1)

#add X0 = 1 to each instance(bias term)

X_b = np.c_[np.ones((100,1)), X]

y = 2+4*X +np.random.randn(100,1) fig = plt.figure()

plt.plot(X,y,"b.")

plt.axis([0,2,0,10])

plt.title("Randomly generated Linear data")

plt.show()

fig.savefig('result1.png')

# Normal equation

theta_cap = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y) print(theta_cap)

The function we used to generate the data was 2+4x+noise. The normal equation has computed values of θo and θ1 that are close enough! Let’s write some code to do prediction

X_new = np.array([[0],[4]])

X_new_b = np.c_[np.ones((2,1)), X_new] y_predict = X_new_b.dot(theta_cap)

print(y_predict) plt.plot(X_new, y_predict,"r-")

plt.plot(X,y,"b.")

plt.title("Linear Regression model prediction")

plt.axis([0,2,0,15])

plt.show()

While the normal equation computes the inverse of X^T X, the computational complexity is almost O(n^3), which if you haven’t figured out is bad! There is another approach that solves this in O(n^2) time. This approach computes the pseudoinverse of X using a matrix factorization technique called Singular Value Decomposition. In simple terms, SVD divides the matrix X into the matrix multiplication of 3 matrices. More about SVD can be found here. A simple implementation of this can be done using pinv() function from Numpy’s linear algebra model.

np.linalg.pinv(X_b).dot(y)

Why SVD is more efficient than Normal Equation?

a. Normal Equation works only if X^T X is invertible or singular. Pseudoinverse on the other hand is always defined.

b. As mentioned above, SVD has less computational complexity than Normal equation.

But both these approaches are very slow when the number of features and instances are large. Which is why the next way of training the model might be better for such cases. I will discuss that in my Part-II post.

Let’s end the post by doing a simple Scikit-Learn implementation of Linear Regression on the generated dataset and see if we get model parameters similar to the one computed by Normal Equation.

from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()

lin_reg.fit(X,y)

print(lin_reg.intercept_)

print(lin_reg.coef_)

And lo and behold, it does! Notice how sklearn gives us the same result as computing pinv(). That is because sklearn computes the pseudoinverse of X.

Conclusion

In this blog we learnt:

How linear models make predictions — the general equation for Linear Regression How to train a model and minimize the cost function — the equation of Mean Squared Error The 1st method of training a Linear Regression model using Closed Form equation — the Normal Equation A simple implementation of the Normal Equation Computing pseudoinverse using Singular Value Decomposition Scikit-learn implementation of Linear Regression

If you enjoyed this blog please leave a clap or comment. It lets me know that my blog was helpful :) Happy learning, folks!

References

1. Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. Book by Aurelien Geron.