Gradient Descent, Normal Equation, and the Math Story.
By: Evangelos Patsourakos
Tags: Machine Learning, Data Science, Gradient Descent, Normal Equation, Mathematics

In a world where data is becoming more valuable than gold, machine learning is trying to use these data for marketing, customer satisfaction, problem-solving and many many other reasons. However, the question arises, how we extract the maximum possible value from a given dataset? This is a question that I will attempt to tackle in this article.

# Import the required libraries

#

# import pandas

import pandas as pd

# import numpy

import numpy as np

np.seterr(all='warn')



# import matplotlib for visualization

import matplotlib.pyplot as plt



# Read your collected data

data = pd.read_csv("car_pricing.csv")



# Project the first 5 rows of your data

data.head()

Let’s make a hypothesis, shall we?

Suppose we have a magic empty box. The way that magic box works is the following. We put a paper with a written value inside, and once we open it again, a new paper appears with a prediction. And that’s the idea behind it.

The hypothesis is a formal way to describe our prediction function. It’s a Greek word “υπόθεση”, and it means “idea or explanation for something that is based on known facts but has not yet been proven”. Therefore, our goal is, given a training set, to learn a function h: X → Y so that h(x) is a “good” predictor for the corresponding value of y.

When the variable that we want to estimate is continuous, such as car pricing, which will be our example for this article, we tend to call this learning problem a regression problem.

Enter the cost function

In the previous section, we said that “… h(x) is a “good” predictor…”. However, we haven’t defined yet a way to calculate the accuracy of our hypothesis function. This measurement is calculated from the cost function. How we do that?

To make our prediction, we have a given set of data; it’s called the training set. This dataset contains price samples from the automotive industry. The idea is to try to predict these values again using our hypothesis function and to calculate how far the prediction is from the real one.

There are plenty of metrics that measures how well our model performs. It will be the next article so stay tuned!

The metric that we will use is the MSE or Mean Squared Error. It’s the most popular among the others for regression algorithms, and it looks like the following equation:

where:

i — index of sample,

ŷ — predicted value,

y — expected value,

n — number of samples in the dataset.

As shown in the equation, the smaller the difference between the actual and predicted values of y, the lower the MSE. And that is the goal, to have the lowest possible MSE because of the fact that the cost function uses the MSE as a metric to estimate an average difference of all the results of the hypothesis with inputs from x’s and the actual output y’s.

Finally, the last piece to assemble the cost function is the regression line. We try to make a straight line (defined by hθ (x)) that passes through the data points. Since we are trying to draw a straight line, our formula will be:

We are now ready to assemble the cost function by replacing y_hat with our hypothesis.

Normal Equation

At this point in the article, I want to introduce the vectorization of the functions. The reason that we do such a thing is about the computational efficiency from programming libraries and make our formulas shorter and easier to read. Let’s say that we have four cars whose prices we want to predict.

Our target variable(dependent variable) will be price, and the independent variable will be the horsepower of the car.

Convert our features and target variables into matrixes, vectors:

Multiply our features by our parameters:

Eh voila! This method will allow us to make a massive prediction of our data set and make our cost function:

to be more like:

To simplify more the equation, we will use one of the matrix transpose identities (AB)^T = B^T A^T and then we will use associative multiplication.

Due to the second term and the third term are the same, we can add them:

Then the question arises, how do we minimize this equation? The answer lies in graph analysis and Fermat’s Theorem. The Extreme Value Theorem tells us that the minimum and maximum of a function must be where the derivative is either zero or non-existent. So, we need to find the derivative of the new J(θ) to be equal to 0.

Find the derivative of A:

Therefore for each parameter n:

And if we convert all of these derivates into vectors then:

Find the derivative of B:

Therefore for each parameter n:

And if we convert all of these derivates into vectors then:

The third term does not contain any θ parameter and therefore the derivate of,

will be zero:

All the pieces are ready. Let’s assemble the puzzle from equation (1),(2) and (3):

This is the theory of the normal equation. However, we must test the method to prove that it works properly. It’s time to apply it to our car pricing problem. First, we will initialize some variables and data.

Tip: Using this formula does not require any feature scaling, and you will get an exact solution in one calculation: there is no “loop until convergence”.

# Initialize target variable

y = data['price'].to_numpy()



# initialize number of training examples

m = y.size



# Get your feautures

X = data

X = X[["horsepower"]].to_numpy()



# Add a column of intercept point

X = np.hstack((np.ones((m,1)), X))

Next, we will move on to the implementation of the normal equation:

def normalEquation(X, y):

"""

Computes the solution to linear regression using the normal equation formula.



Parameters

----------

X : array_like

The dataset of shape (m x n+1).



y : array_like

The value at each data point. A vector of shape (m, ).



theta : array_like

A vector of shape (n+1, ) which represents the parameters.

"""



# Initialize theta

theta = np.zeros(X.shape[1])



# Apply the formula

theta = np.dot((np.linalg.inv(np.dot(X.T,X))),np.dot(X.T,y))



# Return the theta parameters

return theta # Apply the normal equation

theta = normalEquation(X,y)



# print theta to screen

print('Theta found by normal equation: {0}, {1}'.format(theta[0], theta[1])) Theta found by normal equation: -3721.7614943227672, 163.263060969648 plt.scatter(data["horsepower"], data["price"])

plt.title("Our hypothesis fit")

plt.xlabel("horsepower")

plt.ylabel("price")



plt.plot(data["horsepower"], X.dot(theta), '-', label='Linear regression', color='red')

plt.legend(loc='upper right', shadow=True, fontsize='small', numpoints=1)



plt.show()

Finally, we will make prediction price for the Yaris which cost around 10,000 euros.

predict = np.dot([1, 72], theta)

print('For horsepower = 72, we predict that Yaris ACTIVE 2020 should cost: {:.2f} euros/dollars

'.format(predict)) For horsepower = 72, we predict that Yaris ACTIVE 2020 should cost: 8033.18 euros/dollars

Gradient descent

Αnother way to find out the parameters that make our hypothesis perform well with our data is Gradient descent. I will use the example of my previous post to give you the idea of the method. Imagine standing on top of a valley with a ball in your hand. To find out how far this ball can go, we let it roll to the deepest part of the valley. Since at the beginning of the movement of the ball, the valley is steeper, the ball descent faster. However, as soon as it is about to reach the bottom, the surface will become more vertical, and the ball will slow down. We know that we have succeeded when our cost function is at the very bottom of the pits in our graph, i.e. when its value is the minimum.

Mathematically the method depends on the tangent created by the ball and the cost function. When the slope is negative, the value of θᵢ increases and when it is positive, the value of θᵢ decreases. So let’s create the function based on this theory.

From the calculus, we know that its derivative calculates the slope of a function. Therefore our function will be as follows:

where,

The scientists have gone one step further and add it one more variable to the derivate of the cost function J to accelerate the formula. This scalar makes the gradient more or less steep to coverage faster to the minimum point, and it’s called learning rate.

The learning rate makes our equation powerful, but it comes with a bit of a risk. The value of α can make gradient descent to convergence or not convergence at all.

Andrew Ng — Coursera

In the first graph, α is too small, that makes our method slow. On the other hand, in case that α is too large, the steps of each iteration will be that big that it will overshoot the minimum over and over. Therefore, it is suggested to try different values of α between -1 and 1 to find the sweet spot. Usually, the machine learning engineers use α=0.01 or α=0.1 and increase or decrease the learning rate from there.

As a result, our gradient descent algorithm for two parameters will be:

That was the theory of gradient descent. It’s about time to translate it into code!

Tip: Normalize your data: z= (xᵢ — μ)/σ

# Initialize target variable

y = data['price'].to_numpy()



# initialize number of training examples

m = y.size



# Reshape y or transpose it

y = y.reshape(m, 1)



# Get your feautures

X = data

X = X[["horsepower"]].to_numpy()



# Normalize your data

X_norm = X.copy()

mu = np.zeros(X.shape[1])

sigma = np.zeros(X.shape[1])



mu = np.mean(X, axis = 0)

sigma = np.std(X, axis = 0)

X_norm = (X - mu) / sigma

X = X_norm



# Add a column of intercept

X = np.hstack((np.ones((m,1)), X))



# Initialize fitting parameters

theta = np.zeros((2,1))



# Initialize gradient descent settings

iterations = 2000;

alpha = 0.001; def computeCost(X, y, theta):

"""

Compute cost for linear regression.



Parameters

----------

X : array_like

The values that represents the input features + the intercept point

of shape (m x n+1), where m is the number of examples, and n is the

number of features.

y : array_like

The target variable. It is a vector of shape (m, ).



theta : array_like

A vector of shape (n+1, ) which represents the parameters.



Returns

-------

J : float - The value of the cost function.



"""



# number of training examples

m = np.size(y, axis=0)



# Initialize cost variable

J = 0



# initialize hypothesis

h_theta = np.dot(X, theta)



# Calculate the cost

J = (1/(2 * m)) * np.sum(np.square(np.dot(X, theta) - y))



# Return Cost

return J



def gradientDescent(X, y, theta, alpha, num_iters):

"""

Performs gradient descent to learn theta



Parameters

----------

X : array_like

The values that represents the input features + the intercept point

of shape (m x n+1), where m is the number of examples, and n is the

number of features.

y : array_like

The target variable. It is a vector of shape (m, ).



theta : array_like

A vector of shape (n+1, ) which represents the parameters.



Returns

-------

theta: array_like - The optimal parameters

J_history : array_like - The history of the cost.



"""

# number of training examples

m = np.size(y, axis=0)



# Initialize some useful values

J_history = []



# are passed by reference to functions

theta = theta.copy()



for iter in range(num_iters):

# initialize hypothesis

h_theta = np.dot(X, theta)



# calculate the thetas

# (2,1) = (2,1) - ( 2, 1454 ).dot(1454x1 )

theta = theta - (alpha / m) * (X.T).dot(h_theta - y)



# Save the cost J in every iteration

J_history.append(computeCost(X, y, theta))



return theta, J_history

Print the theta cost and theta and compare the results with the normal equation.

# run gradient descent

theta, J_history = gradientDescent(X, y, theta, alpha, iterations)



# print theta to screen

print('Theta found by gradient descent: {0}, {1}'.format(theta[0], theta[1])) Theta found by gradient descent: [11481.70029204], [5569.60294698]

Always plot a Cost over iteration graph to make sure that the cost decreasing.

plt.title("Cost over iterations")

plt.plot(J_history, label='Cost function', color='red') [<matplotlib.lines.Line2D at 0x11b80238>]

plt.scatter(data["horsepower"], data["price"])

plt.title("Our hypothesis fit")

plt.xlabel("horsepower")

plt.ylabel("price")



plt.plot(data["horsepower"], X.dot(theta), '-', label='Linear regression', color='red')

plt.legend(loc='upper right', shadow=True, fontsize='small', numpoints=1)



plt.show()

X_yaris = (72 - mu) / sigma

X_yaris = np.array([1, X_yaris[0]])



predict = X_yaris.dot(theta)

print('For horsepower = 72, we predict that Yaris ACTIVE 2020 should cost: {:.2f} euros/dollars

'.format(predict[0])) For horsepower = 72, we predict that Yaris ACTIVE 2020 should cost: 6947.09 euros/dollars

Conclusion

It’s cheaper to get a Yaris ACTIVE by using gradient descent, so consider to suggest it to your car dealer! Besides that, the normal equation is an accurate method that finds the global minimum but the inversion term make the calculation hard. Therefore if your dataset it’s too large you will need to much computer power to make it happen. On the other hand, gradient descent, it’s a simple concept that can be applied regardless of the size of your dataset.

Hope that you enjoyed it as much as I did! You can also find the Jyputer notebook on Github, thank you!