Implementation of Gradient Descent in Python
By: Deepak Battini
Tags: Machine Learning, Python, Gradient Descent, Programming

Every machine learning engineer is always looking to improve their model’s performance. This is where optimization, one of the most important fields in machine learning, comes in. Optimization allows us to select the best parameters, associated with the machine learning algorithm or method we are using, for our problem case. There are several types of optimization algorithms. Perhaps the most popular one is the Gradient Descent optimization algorithm. The first encounter of Gradient Descent for many machine learning engineers is in their introduction to neural networks. In this tutorial, we will teach you how to implement Gradient Descent from scratch in python. But first, what exactly is Gradient Descent?

What is Gradient Descent?

Gradient Descent is an optimization algorithm that helps machine learning models converge at a minimum value through repeated steps. Essentially, gradient descent is used to minimize a function by finding the value that gives the lowest output of that function. Often times, this function is usually a loss function. Loss functions measure how bad our model performs compared to actual occurrences. Hence, it only makes sense that we should reduce this loss. One way to do this is via Gradient Descent.

A simple gradient Descent Algorithm is as follows:

Obtain a function to minimize F(x) Initialize a value x from which to start the descent or optimization from Specify a learning rate that will determine how much of a step to descend by or how quickly you converge to the minimum value Obtain the derivative of that value x (the descent) Proceed to descend by the derivative of that value multiplied by the learning rate Update the value of x with the new value descended to Check your stop condition to see whether to stop If condition satisfied, stop. If not, proceed to step 4 with the new x value and keep repeating algorithm

Implementing Gradient Descent in Python

Here, we will implement a simple representation of gradient descent using python. We will create an arbitrary loss function and attempt to find a local minimum value for that function.

Our function will be this — f(x) = x³ — 3x² + 7

We will first visualize this function with a set of values ranging from -1 and 3 (arbitrarily chosen to ensure steep curve)

import numpy as np

import matplotlib.pyplot as plt

%matplotlib inline creating the function and plotting it



function = lambda x: (x ** 3)-(3 *(x ** 2))+7



#Get 1000 evenly spaced numbers between -1 and 3 (arbitratil chosen to ensure steep curve)

x = np.linspace(-1,3,500)



#Plot the curve

plt.plot(x, function(x))

plt.show()

Here is the result:

We will then proceed to make two functions for the gradient descent implementation:

The first is a derivative function: This function takes in a value of x and returns its derivative based on the initial function we specified. It is shown below:

def deriv(x):



'''

Description: This function takes in a value of x and returns its derivative based on the

initial function we specified.



Arguments:



x - a numerical value of x



Returns:



x_deriv - a numerical value of the derivative of x



'''



x_deriv = 3* (x**2) - (6 * (x))

return x_deriv

The second is a Step function: This is the function where the actual gradient descent takes place. This function takes in an initial or previous value for x, updates it based on steps taken via the learning rate and outputs the most minimum value of x that reaches the stop condition. For our stop condition, we are going to use a precision stop. This means that when the absolute difference between our old and updated x is greater than a value, the algorithm should stop. The function will also print out the minimum value of x as well as the number of steps or descents it took to reach that value.

This function is shown below:

def step(x_new, x_prev, precision, l_r):



'''

Description: This function takes in an initial or previous value for x, updates it based on

steps taken via the learning rate and outputs the most minimum value of x that reaches the precision satisfaction.



Arguments:



x_new - a starting value of x that will get updated based on the learning rate



x_prev - the previous value of x that is getting updated to the new one



precision - a precision that determines the stop of the stepwise descent



l_r - the learning rate (size of each descent step)



Output:



1. Prints out the latest new value of x which equates to the minimum we are looking for

2. Prints out the the number of x values which equates to the number of gradient descent steps

3. Plots a first graph of the function with the gradient descent path

4. Plots a second graph of the function with a zoomed in gradient descent path in the important area



'''



# create empty lists where the updated values of x and y wil be appended during each iteration



x_list, y_list = [x_new], [function(x_new)]

# keep looping until your desired precision

while abs(x_new - x_prev) > precision:



# change the value of x

x_prev = x_new



# get the derivation of the old value of x

d_x = - deriv(x_prev)



# get your new value of x by adding the previous, the multiplication of the derivative and the learning rate

x_new = x_prev + (l_r * d_x)



# append the new value of x to a list of all x-s for later visualization of path

x_list.append(x_new)



# append the new value of y to a list of all y-s for later visualization of path

y_list.append(function(x_new))



print ("Local minimum occurs at: "+ str(x_new))

print ("Number of steps: " + str(len(x_list)))





plt.subplot(1,2,2)

plt.scatter(x_list,y_list,c="g")

plt.plot(x_list,y_list,c="g")

plt.plot(x,function(x), c="r")

plt.title("Gradient descent")

plt.show()



plt.subplot(1,2,1)

plt.scatter(x_list,y_list,c="g")

plt.plot(x_list,y_list,c="g")

plt.plot(x,function(x), c="r")

plt.xlim([1.0,2.1])

plt.title("Zoomed in Gradient descent to Key Area")

plt.show()

Next, we proceed to plot the gradient descent path as shown below:

#Implement gradient descent (all the arguments are arbitrarily chosen)



step(0.5, 0, 0.001, 0.05)

The importance of Gradient Descent in Machine Learning is one that will be encountered all through your machine learning journey. This is why it is imperative that you understand the inner workings of this algorithm. This tutorial has introduced you to the simplest form of the gradient descent algorithm as well as its implementation in python. Now, you have an intuitive understanding of this algorithm and you are ready to apply it to real world problems.

Please check the complete iPython notebook code here.

Original Post: https://tech-quantum.com/implementation-of-gradient-descent-in-python/

Also, Read