Gradient Descent — A drop in the ocean
By: Sagar Panda
Tags: Explained, Gradient, Gradient Descent, Machine Learning, Loss Function

Its often said, “An ocean is nothing but the accumulation of a zillion water droplets”. The same can be compared with the field of data science, while the field seems to be as vast as a human mind can comprehend, the drops are rather limited and in this write up, we would be focusing on an underlying algorithm for most of the techniques that’s being used across the field.

Before “what” ask “why” ???

If you have a conscience, you must be familiar with the simple term “saving” that can be associated with almost everything like time, memory, cost etc. The same is true with every data science technique that you implement. Its not just about arriving at a solution, its all about how optimally do you arrive. Cutting to the chase, every machine learning algorithm has some or the other kind of loss function and gradient descent is used to find the parameters that minimize this loss function for the model.

Key terms before a deep dive:

Loss function: Describes how well the model will perform given the current set of parameters (weights and biases) and is calculated at every iteration. Cost function: Its the average of all loss functions and is calculated once for the entire model. Slope: Its the actual tangent or the derivative to the curve of the function that connects the 2 variables. Its a measure of the rate of change of a function f(x) with respect to the x .

4. Gradient: The gradient is like the slope, except here, the function whose rate of change we wish to study depends on more than one variable, i.e. f = f(x,y) . In other words f is multidimensional. It would be convenient if we had one operator which would operate on f and give a single output.

This is the gradient operator

Notice that the gradient acts on a scalar function but returns a vector value. This is crucial since each component of the gradient indicates the rate of change with respect to that particular dimension. Thus the slope is the special case of the gradient operator acting on a single dimensional function.

5. Global minima: The smallest overall value of a function over its entire range.

6. Learning Rate: The Gradient descent algorithm multiplies the gradient by a number (Learning rate or Step size) to determine the next point.

For example: having a gradient with a magnitude of 4.2 and a learning rate of 0.01, then the gradient descent algorithm will pick the next point 0.042 away from the previous point.

Source: Towards Data Science

Breaking the Algorithm:

In order to descend into a valley, there are 2 factors that matter the most: Direction and Speed. Similarly the G.D needs a direction and a speed with which it can arrive at the global minima.

The direction is provided by the gradient where as the speed or the steps by which to descend is provided by the learning rate.

Now let’s see how the G.D actually function.

Repeat until hit convergence*:

Given the gradient, calculate the change in the parameters with the learning rate. Re-calculate the new gradient with the new value of the parameter. Repeat step 1. Here is the formula of gradient descent algorithm:

Source: Samuel Trendler

*Convergence is a name given to the situation where the loss function does not improve significantly, and we are stuck in a point near to the minima.

Gradient Descent variants

There are three variants of gradient descent based on the amount of data used to calculate the gradient:

Batch gradient descent

Stochastic gradient descent

Mini-batch gradient descent

Batch Gradient Descent:

Batch Gradient Descent, aka Vanilla gradient descent, calculates the error for each observation in the dataset but performs an update only after all observations have been evaluated.

It’s not often used, because of huge consumption of computational resources, as the entire dataset needs to remain in memory.

Stochastic Gradient Descent:

Stochastic gradient descent (SGD) performs a parameter update for each observation. So instead of looping over each observation, it just needs one to perform the parameter update.

SGD is usually faster than batch gradient descent, but its frequent updates cause a higher variance in the error rate, that can sometimes jump around instead of decreasing.

Mini-Batch Gradient Descent:

It is a combination of both bath gradient descent and stochastic gradient descent. Mini-batch gradient descent performs an update for a batch of observations. It is the algorithm of choice for neural networks, and the batch sizes are usually from 50 to 256.

Comparison of the 3 types of GD. (Source: Hackernoon)

That’s it!

This algorithm will move on to base to optimize many of the other machine learning algorithms including the most advanced neural networks. After all it’s just a drop in the Ocean.

Follow me for detailed explanations in the field of Data Science.