Gradient Descent, the key for ML algorithms?
By: Sharath T
Tags: Machine Learning, Cost Function, Artificial Intelligence, Optimization, Gradient Descent

Views expressed by the author are his own and done in his personal capacity. These are no way representative and/or reflective of the organization he is associated with.

Background

As part of the machine learning journey, we typically start with linear regression and gradually get onto the concept of the cost function, and the challenge starts from here to get the cost function optimized. Though we can do a trial and error for a small set of data elements, it becomes unviable as we start increasing the data.

To address this optimization problem, we have a technique and theory associated called “Gradient Descent” which will come in handy in getting the cost function minimized. Gradient Descent is not limited to linear regression but can be extended to logistic regression and further to neural networks also.

You may be wondering on why we need to learn all these stuff when we can find built-in function to deal with it directly? Good question, and it is always good to have some insights into the internal workings of it as it may help in visualizing the problems better and thus can help in our modeling for all the complex problems or scenarios tomorrow. Also, this gives an opportunity to peep into the libraries used such that the same can be leveraged for other problems at hand. So having a solid foundation on how things work is always a recommended approach in dealing with complex topics. Another point to remember is the built-in functions are limited to some specific languages but it is not consistently available in all. With the ground set now on the background, let’s jump onto the details.

Gradient Descent — Overview

As the problem is to optimize the cost function, let’s start from there

For a given hypothesis

The cost function can be written as

i=1 to m where m is the no. of sample data points. This is also called as Mean square Error.

Let’s initialize some values to theta0 and theta1 and start computing J(ɵ0, ɵ1), we can arrive at the convergence (with J, the cost function getting to minimum) after some iterations of trial and error, and it can turn out to be an inefficient approach as we start increasing the no. of data points, also this may lead to higher costs overall.

What are the alternatives to get to the minimum value of our cost function with limited steps? The Gradient Descent algorithm comes to the rescue here and helps us in arriving at the optimum value of cost function with minimal steps, thus it helps in concluding on the best fit for a given equation from the many possible opportunities.

To get a better intuition on this, let’s plot a graph with all the data points, assuming that the graph plotted has many troughs, there can be multiple local minimums possible but there can be always only one global minimum. Can we ever reach the global minimum or do we need to settle down with local minimum is something we can review in the subsequent paragraphs, okay?

If the data when plotted transforms into a convex shape, like a bowl shape, with small ridges sideways and a big trough, then we can make incremental improvements progressing step by step. Every step takes us closer to the bottom of the trough, and when we come to a stage where there is no further improvement possible, then we can conclude that the global minimum has arrived. Though we have an interim local minimum, they being very small and not have any impact on the overall direction of the curve, we are bound to reach the global minimum.

To keep it simple, the graph is plotted with only one value

If the data is not getting us a pure convex shape, then it can have a lot of ups and downs which are more or close to equal in their depth leaving the options open for arriving at the global minimum. How is that we can conclude that we arrived at the global minimum? We can either take a brute force approach trying all such possibilities to arrive at the final one, or we can restrict the trials to a few random ones prior to finalizing on the global minimum. Keeping in view of the costs associated, in majority cases, it will be concluded at the first encounter of the global minimum.

In the diagram shown below, we may need to compromise on the global minimum which is easily accessible or has to go through the tedious process of getting to the actual one.

To keep it simple, the graph is plotted with only one theta value

Though we can easily explain this visually picking a graph, the real challenge will be to understand the logic behind the scenes. Let’s get to that by going through the flow step by step.

Implementation

First thing we need to understand on this journey is about finding Slope, this can be easy for a line picking two points and putting it in the formula (y2-y1)/(x2-x1) but how can we find the slope of a curve where we may need to work with a single point instead of two. On a side note, the slope will help in identifying the rate at which the value of y changes with respect to each change in x.

There are multiple ways of handling this like assuming a couple of points closer to the point in the subject and going with the regular formula in calculating the slope, keep in mind that this may not be accurate but can come close to the actual. The other way or the preferred way to approach this is to find the derivative of the function and substitute the point in context. We may need to adopt partial derivatives if working with multiple features.

In the function mentioned above, there are two variables ɵ0, ɵ1 hence we need to use partial derivatives approach keeping one of them as constant while dealing with the other.

α is the learning rate, it helps in deciding the size of the steps to be taken in transitioning towards the convergence.

Fixed size steps may overshoot the global minimum hence the steps have to be gradually brought down as we start approaching the minimum, this can be easily achieved by the slope which can be computed at every step, thus the steps can be adjusted based on the slope hence it gets minimized as we reach close to the convergence.

There may be some questions on where to start with respect to the learning rate and the number of iterations.

We typically can start small say 10 or 50 iterations with a learning rate of 0.01.

Play around with the learning rate first by increasing, decreasing it. We can stop playing around with this as soon as we see a constant decreasing trend which is leading to a minimum cost.

From here, we can fix the learning rate and play around with no. of iterations to arrive at the minimum cost possible.

The other way to arrive at the minimum cost is to get to the point where the slope becomes zero or comes close to zero.

Programming Logic

Coming to the programming language implementation, there is no standard built-in function in Python but Tensorflow has one which abstract the complete implementation details from the user. To keep it neutral, let’s put some pseudo code which can be translated into any programming language based on the user choice.

Review the cost value adjusting the values of learning rate and no. of iterations until the convergence has arrived.

Challenges

This algorithm has a couple of issues to be careful while working on it.

The algorithm may never converge, this is quite possible if the incremental steps taken are big such that it crosses the global minimum without getting onto the convergence path

If the steps are slow, it may get converged but will take more steps so this is either not advisable

What we need is a balanced approach in picking the learning rate such that it will neither take more time nor gets us into a non-converged state.

Variants of Gradient Descent

The Gradient Descent will be arrived considering all the training data available, however, this is going to be time-consuming as the computation in each step has to include all the data set resulting in higher costs. There are some variants in the form of Stochastic and Mini Batch versions which takes a subset of the data than working on the full training set. The difference between the two is that the Stochastic picks only a single data point whereas Mini batch version takes a subset of data points. In both cases, the global minimum can be achieved but the path it takes may look little rough and ad-hoc.

Final Comments

As learned from the above details, Gradient Descent is one of the key concepts in Machine learning and is applicable to various algorithms.

As ML is spreading its wings into all the important aspects of our day to day to lives, the precision is going to be the key, and if we don’t get the things right, the impact will be high on some of these sensitive applications — be it self-driving cars or medical diagnosis, etc. Overall Gradient Descent will not only help in optimizing the costs but helps in ensuring better accuracy which is the need of the hour with machines taking over the control. Agree?