Gradient Descent- A Race to find the Global Minimum
By: Varun Bhatia
Tags: Normalization, Regression, Deep Learning, Gradient Descent, Machine Learning

A basic neural network basically comprises of the input layer, the hidden layer(s) and finally the output layer with some network parameters (weights and biases).

A simple Neural Network

We can see the presence of a loss function i.e the difference between the actual value(y) and the predicted value(ŷ) from the neural network.

In order to reduce this loss we use an optimizer.There are various types of optimizers one of them being Gradient Descent.

Gradient Descent is a very generic optimization algorithm capable of finding optimal solutions to a wide range of problems.

The general idea of Gradient Descent is to tweak parameters iteratively in order to minimize the loss function.

In short what the Gradient descent does is to find out the best weights(or parameters) in order to reduce the loss function by modifying all the weights(or parameters) with the help of backpropagation through the network.

Eq1: Modifying old weights to obtain new weights

Here λ is known as the learning rate.

How does Gradient Descent help us?

Suppose you are lost in the mountains in a dense fog, you can only feel the slope of the ground below your feet. A good strategy to get to the bottom of the valley quickly is to go downhill in the direction of the steepest slope. This is exactly what Gradient Descent does: it measures the local gradient of the error function with regards to the parameter θ, and it goes in the direction of descending gradient.Once the gradient is zero, you have reached a minimum.

We start by filling θ with random values (this is called random initialization), and then we improve it gradually, taking one baby step at a time, each step attempting to decrease the cost(loss) function (e.g., the MSE), until the algorithm converges to a minimum.

Effect of slope on weight calculation

As per our equation to calculate the new or optimized weights(Eq1), slope of the function plays a very important part.

Suppose we draw a tangent to the random point in order to find the slope, if the right hand side of that tangent is pointing downwards(Fig.1), then it determines that the slope is negative and hence the new weight will be more than the old weight and therefore will move towards the right of the curve.

Fig.1

And if the right hand side of that tangent is pointing upwards(Fig.2), then it determines that the slope is positive and hence the new weight will be less hence the old weight and therefore will move towards the left of the curve.

Fig. 2

Effect of Learning Rate Parameter(λ)

An important parameter in Gradient Descent is the size of the steps, determined by the learning rate parameter. If the learning rate is too small, then the algorithm will have to go through many iterations to converge, and hence will take a long time.

Too small Learning Rate

On the other hand, if the learning rate is too high, we might jump across the valley and end up on the other side, possibly even higher up than we were before. This might make the algorithm diverge, with larger and larger values, failing to find a good solution.

High Learning Rate

Summary

At last simplifying the above explained process of Gradient Descent: