The Math behind Gradient Descent and the Normal Equation for Linear Regression
Tags: Data Science, Gradient Descent, Machine Learning, Artificial Intelligence, Towards Data Science

The Math behind Gradient Descent and the Normal Equation for Linear Regression

Gradient Descent and the Normal Equation for Linear Regression with a Practical Implementation in Python Moeedlodhi Follow Aug 4, 2020 · 7 min read

When I started my machine learning journey, math was something that always intrigued me and still does.

I for one believe that libraries such as scikit learn have indeed done wonders for us when it comes to implementing the algorithms but without an understanding of the maths that goes into making the algorithm, we are bound to make mistakes on complicated problems.

In this article, I will be going over the math behind Gradient Descent and the derivation behind the Normal linear Equation and then implementing them both on a dataset to get my coefficients.

A Linear Regression model. Red dots are the true y values and the blue line is the model line

The Normal Equation

Looks pretty straight forward, no? When I was getting started with Linear Regression and trying to get an understanding of the different ways to calculate the coefficients, The Normal Equation was by far my most favorite method to find coefficients but where does this equation come from? Well, let us take a look.

The first thing we have to understand is that the Mean Squared Error or MSE for short is a metric that measures how well our model performs. The lower the MSE, the closer our predictions are to the actual values of y.

As can be seen in the equation, the lesser the difference between the actual and predicted values of y, the lower the MSE. And that is the goal, to have as low of an MSE as possible.

But can we use the MSE to find our coefficients? Yes, we can.

Enter the Cost Function

Linear Regression Cost function

As you might have noticed, the MSE equation and the cost function are the same but instead of the predicted values ŷ, we have written it in terms of Theta.

Here, ŷ is nothing but the dot product of the transpose of Theta and values of X.

Plugging the value of h(Θ) back into out cost function and simplifying the equation gives us:

Note, I have replaced the summation symbol with matrix multiplication which yields the same result.

Now using matrix transpose identities, we further simplify it to get:

And now, taking the derivate of J(Θ) concerning Theta and setting it zero,

we find the minimum point of the cost function(Calculus 101):

and on further simplification, we get:

Gradient Descent:

Gradient descent as the name suggests is going down the gradient till we hit the lowest point possible

The cost function is nothing but a quadratic equation, and all quadratic equations have a “Global minimum” where the value of the function is lowest and the gradient is zero.

So how does gradient descent work?

To put it simply, we take initial guesses of our Theta values to find the gradient and then subtract that gradient from out initial Theta values to get new values of Theta and repeat the process till we converge to a global minimum point.

Mathematically, the equation looks somewhat like:

here, the α represents the learning rate which basically prevents our value of Theta from “overshooting” if the gradient is too large.

A high value of the learning rate causes the function to miss its target whereas a small learning rate helps to converge easily to the Global minimum.

Note: The learning rate shouldn’t be too small as that can unnecessarily increase the computational time. A good estimate would be from 0.01 to 0.001 but it depends on the case at hand.

Practical Implementation in Python

In this part, I will be implementing both the Gradient Descent and the Normal Equation on a Regression dataset from Andrew ng’s “Introduction to Machine Learning Course”.

The dataset is a list of prices for houses based on the total square footage and the number of bedrooms they have.

Here the price is our dependent variable y and X is our set of independent variables.

Note: I have added an extra column “intercept_term” as that will be used to find our intercept as the name suggests.

Before separating our variables and performing the calculation, we have to realize that our values have varying scales.

What I mean is that the bedroom column is on a range of one to five whereas the sq_ft columns have varying values in the hundreds.

Just because a variable has a higher value doesn’t mean it’s more important but this high value can somewhat influence our model negatively.

So in Linear Regression, it’s always a good practice to Standardize our variables so that they are on the same scale. We would like our variables to have a mean of zero and a standard deviation of one.

Standardization formula

Using Scikit learns built-in StandardScaler module, we will standardize our entire variables to bring it on the same scale.

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

data_new = sc.fit_transform(data)

S = pd.DataFrame(data_new)

S.drop(3,axis=1,inplace=True)

S.columns = ['sq_ft','bedrooms','price']

S['ones'] = S['bedrooms']/S['bedrooms']

S

After Standardization, we get

Note: our previous “intercept_term” standardized to zero, so we added it again.

We will separate sq_ft, bedrooms, and ones columns as our X dataframe and the price column will be denoted as y.

X_array = S.drop('price',axis=1)

X_array

y = S['price']

X_array = np.array(X_array).reshape(46,3)

y_array = np.array(y).reshape(46,1)

Note: We will be converting our columns into numpy arrays as so we can perform numpy matrix operations on our variables.

First up: Normal Equation

I will divide the Normal Equation into two parts, the first will be the inverse of the dot product of X_transpose with X and the second will be the dot product of X_transpose with y

And then, I will take the dot product of both my results.

first_term = np.linalg.inv((np.dot(x_transpose,X_array)))

second_term = np.dot(x_transpose,y_array)

Theta = np.dot(first_term,second_term)

Theta



The result I get is:

I have a 3x1 matrix, Two being the coefficients and the other being the intercept term.

Gradient Descent

m = 96 # number of samples

alpha=0.01

x_transpose = X_array.transpose()

for iter in range(0, 100000):

hypothesis = np.dot(x, theta)

loss = hypothesis - y_array





gradient = np.dot(x_transpose, loss) / m

theta = theta - alpha * gradient # update

print(theta)

I defined my number of sample variables as m, my learning rate as alpha, and set my number of iterations to 100000.

as the for loop is run, our values of theta are updated for every iteration till we reach the point where Theta does not converge further and the result is.

So our result matches, what more?

Well, let's give the scikit learns the Linear Regression module a try and see what it gives us.

Let's visualize our results. Even though this is a multivariate regression problem, Let's visualize our price variable against the sq_ft variable.

Note: The U variable is nothing but a linear equation with one of the coefficients as the gradient. (0.883)

The line captures the data pretty well.

Hopefully, this introduction to Gradient Descent and the Normal equation will prove fruitful and beneficial.