Introduction to Gradient Descent with linear regression example using PyTorch
By: C Kuan
Tags: Data Science, Deep Learning, Gradient Descent, Machine Learning, Programming

Introduction

In machine learning, usually, there is a loss function (or cost function) that we need to find the minimal value. Gradient Descent is one of the optimization methods that is widely applied to do the job.

In this post, I will discuss the gradient descent method with some examples including linear regression using PyTorch.

Calculus

One method to find a function’s max or min, it to find the point(s) where the slope equals zero. The max or min of the function will be the solution of the derivative of a function equals zero.

Take this function as an example:

f(x)=x**2–5x+5, image by author

the derivative of the function is:

so let f’(x) = 0, we can find x=2 as the solution. This means the slope of the function equals zero at x=2, and the minimal value of the function is f(2)=1.

This method finds the analytical solution, however, it is sometimes difficult to apply this method to complex functions in real applications. Therefore, we need numerical methods to find the approximate solution, gradient descent is one of the methods.

Gradient Descent

The mathematical explanation of the gradient can be found in this link.

For one dimension function f(x), the gradient is represented as f’(x). For the multi-dimension function, it is represented as below.

From wiki: If the gradient of a function is non-zero at a point p, the direction of the gradient is the direction in which the function increases most quickly from p, and the magnitude of the gradient is the rate of increase in that direction.

The gradient descent tries to approach the min value of the function by descending to the opposite direction of the gradient.

It updates the parameters (here, x) iteratively to find the solution. So first, we need an initial guess (x0) of the solution, then calculate the gradient based on the initial guess, then based on the calculated gradient to update the solution (x). It can be explained in this formula:

t is the iteration and r is the learning rate.

So, first, we have an initial guess x0, by the equation above, we can find the x1:

then by x1, we can find x2:

by repeating this process several times (epoch), we should be able to find the parameters (x) where the function at its min value.

Example 1:

Let’s take the previous function as an example:

and apply it to the gradient descent equation

rearrange:

Let’s set the learning rate = 0.1 and initial guess x0=7. We can then easily update and calculate the x1, x2, x3… and f(x1), f(x2), f(x3)…

In this example:

You can see the parameter (x) approaches to 2 and the function approaches its min value 1.

Let’s put it on the graph:

Gradient Descent (learning rate = 0.1) (image by author)

What if we use a smaller learning rate (0.01)?

Gradient Descent (learning rate = 0.01) (image by author)

As expected, it took more iterations to reach the minimum.

If we increase the learning to 0.3, it reached the minimum faster than 0.1.

Gradient Descent (learning rate = 0.3) (image by author)

If we increase it further to 0.7, it started to overshoot.

Gradient Descent (learning rate = 0.7) (image by author)

If we increase it to 1, it cannot reach the minimum value at all.

Gradient Descent (learning rate = 1) (image by author)

So the learning rate is very important to gradient descent. If it is too small, a lot of iterations are needed, if it is too big, it might not be able to reach the minimum.

Now, we have an idea of how the gradient descent work, let’s try to apply that to a simple machine learning model — linear regression.

Example 2 — linear regression:

Linear regression example

Linear regression is an approach to find a linear relationship between two variables. It finds a line,

that describes the data points shown in the figure. m is the slope of the line and c is the intercept. The task is to find the line (m and c) that best fits the data points.

Loss function

We use Mean Squared Error (MSE) to measure the error of the line and data points.

It averages the square of the sum of the differences between points and the line.

We can rearrange the equation:

We want to find the m and c, which gives the minimum value of MSE.

Gradient Descent

Just like the previous example, we need to find the gradient.

Then we can use this equation to update m and c:

Again, we need an initial guess for m and c, let’s start with m=0 and c=0, with learning rate = 0.0001

#Gradient Descent for i in range(epochs):

Y_pred = m*X + c

d_m = (-2/n) * sum(X * (Y - Y_pred))

d_c = (-2/n) * sum(Y - Y_pred)

# Update m

m = m - r * d_m

# Update c

c = c - r * d_c

mse=(1/n) * sum((Y - m*X + c)**2)

The m and c can be updated based on the gradient. Let’s put it on the graph.

Gradient Descent- linear regression example, learning rate = 0.0001. (image by author)

Again, a carefully chosen learning rate is important, if the learning rate is increased to 0.01, the calculation will not converge.

Gradient Descent — cannot find optimal m and c, learning rate = 0.01. (image by author)

In this simple example, we are calculating the gradient by taking the derivative of the function ourselves, it might become difficult for a more complicated problem. Luckily, PyTorch provides a tool to automatically compute the derivative of nearly any function.

Pytorch approach

Let’s define the line:

def f(x, params):

m, c= params

return m*x + c

and the loss function — Mean Squared Error:

def mse(preds, targets): return ((preds-targets)**2).mean()

Again, we start by m=0, c=0, the requires_grad_(), is used here to calculate the gradient.

params = torch.zeros(2).requires_grad_()

Then we can predict the y values based on our first parameter, and plot it.

preds = f(X_t, params)

Gradient Descent by Pytorch — initial guess. (image by author)

Then we can calculate the loss:

loss = mse(preds, Y_t)

and the gradient by this PyTorch function:

loss.backward()

after this we can check the gradient:

params.grad

it returns a tensor, which is the gradient: tensor([433.6485, 18.2594])

Then we update the parameter using the gradient and learning rate:

lr = 1e-4

params.data -= lr * params.grad.data

params.grad = None

and predict the y using these new parameters:

Gradient Descent by Pytorch — epoch 2(image by author)

We need to repeat this process several times, let’s make a function:

def apply_step(params):

preds = f(X_t, params)

loss = mse(preds, Y_t)

loss.backward()

params.data -= lr * params.grad.data

params.grad = None

return pred

Then we can run for several epochs. Looks like the loss is decreasing through epochs.

Put it on a graph

Gradient Descent by Pytorch (image by author)

This is it! By using PyTorch, we can easily calculate the gradient and perform the gradient descent for machine and deep learning models.

Thanks for reading.