Introduction to Gradient Descent with linear regression example using PyTorch | by C K | Towards Data Science

In machine learning, usually, there is a loss function (or cost function) that we need to find the minimal value. Gradient Descent is one of the optimization methods that is widely applied to do the job.
In this post, I will discuss the gradient descent method with some examples including linear regression using PyTorch.
One method to find a function’s max or min, it to find the point(s) where the slope equals zero. The max or min of the function will be the solution of the derivative of a function equals zero.
Take this function as an example:
the derivative of the function is:
so let f’(x) = 0, we can find x=2 as the solution. This means the slope of the function equals zero at x=2, and the minimal value of the function is f(2)=1.
This method finds the analytical solution, however, it is sometimes difficult to apply this method to complex functions in real applications. Therefore, we need numerical methods to find the approximate solution, gradient descent is one of the methods.
The mathematical explanation of the gradient can be found in this link.
For one dimension function f(x), the gradient is represented as f’(x). For the multi-dimension function, it is represented as below.
From wiki: If the gradient of a function is non-zero at a point p, the direction of the gradient is the direction in which the function increases most quickly from p, and the magnitude of the gradient is the rate of increase in that direction.
The gradient descent tries to approach the min value of the function by descending to the opposite direction of the gradient.
It updates the parameters (here, x) iteratively to find the solution. So first, we need an initial guess (x0) of the solution, then calculate the gradient based on the initial guess, then based on the calculated gradient to update the solution (x). It can be explained in this formula:
t is the iteration and r is the learning rate.
So, first, we have an initial guess x0, by the equation above, we can find the x1:
then by x1, we can find x2:
by repeating this process several times (epoch), we should be able to find the parameters (x) where the function at its min value.
Let’s take the previous function as an example:
and apply it to the gradient descent equation
rearrange:
Let’s set the learning rate = 0.1 and initial guess x0=7. We can then easily update and calculate the x1, x2, x3... and f(x1), f(x2), f(x3)...
In this example:
You can see the parameter (x) approaches to 2 and the function approaches its min value 1.
Let’s put it on the graph:
What if we use a smaller learning rate (0.01)?
As expected, it took more iterations to reach the minimum.
If we increase the learning to 0.3, it reached the minimum faster than 0.1.
If we increase it further to 0.7, it started to overshoot.
If we increase it to 1, it cannot reach the minimum value at all.
So the learning rate is very important to gradient descent. If it is too small, a lot of iterations are needed, if it is too big, it might not be able to reach the minimum.
Now, we have an idea of how the gradient descent work, let’s try to apply that to a simple machine learning model — linear regression.
Linear regression is an approach to find a linear relationship between two variables. It finds a line,
that describes the data points shown in the figure. m is the slope of the line and c is the intercept. The task is to find the line (m and c) that best fits the data points.
Loss function
We use Mean Squared Error (MSE) to measure the error of the line and data points.
It averages the square of the sum of the differences between points and the line.
We can rearrange the equation:
We want to find the m and c, which gives the minimum value of MSE.
Gradient Descent
Just like the previous example, we need to find the gradient.
Then we can use this equation to update m and c:
Again, we need an initial guess for m and c, let’s start with m=0 and c=0, with learning rate = 0.0001
#Gradient Descentfor i in range(epochs):   Y_pred = m*X + c   d_m = (-2/n) * sum(X * (Y - Y_pred))   d_c = (-2/n) * sum(Y - Y_pred)   # Update m   m = m - r * d_m   # Update c   c = c - r * d_c   mse=(1/n) * sum((Y - m*X + c)**2)
The m and c can be updated based on the gradient. Let’s put it on the graph.
Again, a carefully chosen learning rate is important, if the learning rate is increased to 0.01, the calculation will not converge.
In this simple example, we are calculating the gradient by taking the derivative of the function ourselves, it might become difficult for a more complicated problem. Luckily, PyTorch provides a tool to automatically compute the derivative of nearly any function.
Pytorch approach
Let’s define the line:
def f(x, params):    m, c= params    return m*x + c
and the loss function — Mean Squared Error:
def mse(preds, targets): return ((preds-targets)**2).mean()
Again, we start by m=0, c=0, the requires_grad_(), is used here to calculate the gradient.
params = torch.zeros(2).requires_grad_()
Then we can predict the y values based on our first parameter, and plot it.
preds = f(X_t, params)
Then we can calculate the loss:
loss = mse(preds, Y_t)
and the gradient by this PyTorch function:
loss.backward()
after this we can check the gradient:
params.grad
it returns a tensor, which is the gradient: tensor([433.6485, 18.2594])
Then we update the parameter using the gradient and learning rate:
lr = 1e-4params.data -= lr * params.grad.dataparams.grad = None
and predict the y using these new parameters:
We need to repeat this process several times, let’s make a function:
def apply_step(params):  preds = f(X_t, params)  loss = mse(preds, Y_t)  loss.backward()  params.data -= lr * params.grad.data  params.grad = None  return pred
Then we can run for several epochs. Looks like the loss is decreasing through epochs.
Put it on a graph
This is it! By using PyTorch, we can easily calculate the gradient and perform the gradient descent for machine and deep learning models.
Thanks for reading.