Implementation of Gradient Descent in Python
Every machine learning engineer is always looking to improve their model’s performance. This is where optimization, one of the most…
Category: Machine Learning
Author: Deepak Battini

Implementation of Gradient Descent in Python
Every machine learning engineer is always looking to improve their model’s performance. This is where optimization, one of the most important fields in machine learning, comes in. Optimization allows us to select the best parameters, associated with the machine learning algorithm or method we are using, for our problem case. There are several types of optimization algorithms. Perhaps the most popular one is the Gradient Descent optimization algorithm. The first encounter of Gradient Descent for many machine learning engineers is in their introduction to neural networks. In this tutorial, we will teach you how to implement Gradient Descent from scratch in python. But first, what exactly is Gradient Descent?
What is Gradient Descent?
Gradient Descent is an optimization algorithm that helps machine learning models converge at a minimum value through repeated steps. Essentially, gradient descent is used to minimize a function by finding the value that gives the lowest output of that function. Often times, this function is usually a loss function. Loss functions measure how bad our model performs compared to actual occurrences. Hence, it only makes sense that we should reduce this loss. One way to do this is via Gradient Descent.
A simple gradient Descent Algorithm is as follows:
Obtain a function to minimize F(x)
Initialize a value x from which to start the descent or optimization from
Specify a learning rate that will determine how much of a step to descend by or how quickly you converge to the minimum value
Obtain the derivative of that value x (the descent)
Proceed to descend by the derivative of that value multiplied by the learning rate
Update the value of x with the new value descended to
Check your stop condition to see whether to stop
If condition satisfied, stop. If not, proceed to step 4 with the new x value and keep repeating algorithm
Implementing Gradient Descent in Python
Here, we will implement a simple representation of gradient descent using python. We will create an arbitrary loss function and attempt to find a local minimum value for that function.
Our function will be this — f(x) = x³ — 5x² + 7
We will first visualize this function with a set of values ranging from -1 and 3 (arbitrarily chosen to ensure steep curve)
Here is the result:

We will then proceed to make two functions for the gradient descent implementation:
The first is a derivative function: This function takes in a value of x and returns its derivative based on the initial function we specified. It is shown below:
The second is a Step function: This is the function where the actual gradient descent takes place. This function takes in an initial or previous value for x, updates it based on steps taken via the learning rate and outputs the most minimum value of x that reaches the stop condition. For our stop condition, we are going to use a precision stop. This means that when the absolute difference between our old and updated x is greater than a value, the algorithm should stop. The function will also print out the minimum value of x as well as the number of steps or descents it took to reach that value.
This function is shown below:
Next, we proceed to plot the gradient descent path as shown below:


The importance of Gradient Descent in Machine Learning is one that will be encountered all through your machine learning journey. This is why it is imperative that you understand the inner workings of this algorithm. This tutorial has introduced you to the simplest form of the gradient descent algorithm as well as its implementation in python. Now, you have an intuitive understanding of this algorithm and you are ready to apply it to real world problems.
Please check the complete iPython notebook code here.
Original Post: https://tech-quantum.com/implementation-of-gradient-descent-in-python/