Gradient Descent Machine Learning Optimization Algorithm from Scratch in Python
By: Dhiraj K
Tags: Python, Gradient Descent, Numpy, Machine Learning, Optimization

Let's discover the fundamentals of Gradient Descent

Introduction:

Gradient descent Machine Learning method is an optimization algorithm that is used to find the local minima of a differentiable function. It can be used in Linear Regression as well as Neural Network.

In the realm of Machine Learning, It is used to find the values of parameters of a differentiable function such that the loss is minimized.

Let us understand the gradient descent algorithm with a simple practical example. Imagine a blind hiker is trying to get down a hill to its lowest point as shown in the image above. There is no way for the hiker to see which direction to go as he is blind. However, there is one thing hiker understands clearly If he is going down, its the right progress and if he is going up, it is wrong progress.

Gradient Descent from Scratch in Python

Therefore, if he keeps taking small steps, that takes him downwards, he will be able to get down the lowest point on the hill.

Here, taking small steps can be considered as a learning rate, and the height above the lowest point can be considered as the loss.

Also reaching the lowest point of the hill can be considered as a convergence

which indicates no further possibility of going down, and the loss is minimum.

Note that this is one of the posts in the series Machine Learning from Scratch. You may like to read other similar posts like Linear Regression from Scratch, Logistic Regression from Scratch, Decision Tree from Scratch, Neural Network from Scratch

You may like to watch a more detailed video version of this article as below:

Implementation:

Let's implement the gradient descent from scratch using Python.

We will start by importing the required libraries.

# Import the required Libraries

import pandas as pd

import numpy as np

Then let's define the function we want to optimize. The function we are considering is y = (x-5)*(x-5)

# Gradient of y with respect to x

gradient_of_y = lambda x: 2(x-5)

Initialize the value of x

#Initial value of x close to zero

current_x_value = 0.1

Define the maximum number of iteration that can be done.

# maximum number of iterations that can be done

maximum_iterations = 500

Define Learning rate, current iteration, and previous step size.

# Learning Rate

learning_rate = 0.01

current_iteration = 0

previous_step_size = 0.5

Finally, you need to run a while loop to optimize the loss

Gradient Descent from Scratch in Python

Note that the minimum value of x found is 4.99, which is very close to 5, as expected.

Gradient Descent from Scratch in Python

End Notes:

In this article, we implemented the Gradient Descent Algorithm from scratch in Python. Note that when the control is coming out of the while loop, we are able to print the value of x, which is the minima of x found by gradient descent algorithm.

As you can see, the minimum value is 4.99 quite close to 5 as expected.

Please note that the total iteration taken is around 342.

Hence, the Gradient Descent optimization Algorithm converges and finds the expected minima.

The associated code can be found here.

Happy Learning !!