All about Gradient Descent in Machine Learning and Deep Learning!
By: Harshit Dawar
Tags: Loss Function, Deep Learning, Data Science, Gradient Descent, Machine Learning

Ever wondered how the machine learning algorithms give us the optimal result, whether it is prediction, classification or any other? How those algorithms work? What is the maths behind those algorithms which leads to the result, which can be used as a tool to assess real world problems? Gradient Descent algorithm is the one which makes the magic possible.

Gradient Descent is the backbone of every machine learning algorithm and it also acts as a base for many deep learning optimizers. It works on a simple mechanism, and it is to find the optimal weights for the loss function by iterating over the error curve of values shown above, by which the error in the model predictions and the actual values is minimum.

Consider, a case in which we manually used let’s say 15 values and calculated the loss for each of those values according to any loss function, in this case we can do because the values are 15 only, it is very small number, but it will not lead to the optimal result, and in real world we have to use many values in order to conclude a value which minimises the loss i.e. the error in model prediction and the real world values. Here comes Gradient Descent into action.

Actually what this algorithm does is, it just calculates the derivative(gradient means derivative in this equation) of the function which has to be minimized and constantly works in the direction to minimize the loss. Now, there will be some people who may ask that calculating the derivative of a function isn’t a big deal, but my question to those people is that can they calculate the derivative of an equation which is of degree 100? I know that it is not possible for humans. That is why we need to take help of gradient descent algorithm.

The approach of this algorithm is to minimize the derivative and make the point converge at the global minimum of the loss function curve.

Loss function curve

Now, consider a point is at the left part on the boundary of the above curve, then if we draw a line vertical line through that point as shown in the curve, then definitely the slope of that line is negative, & from here if we keep on increasing the value of the loss function or in other words if we keep on adding the gradient(which is negative), it will converge at the global minimum. If the point is present initially at the right part on the boundary of the curve shown above, from that point if we draw a line, it’s slope will be positive, and now we have to subtract the gradient because it is positive in order to reach to the global minimum.

So, according to the above explanation, gradient descent decides the loss function in such a way which satisfies both the conditions of gradient that is positive and negative.

Till here, according to my explanation gradient descent work can be represented as:

weights = weights — derivative(loss function) → Equation (1)

There can be some problem in the above equation in the case suppose the derivative is too large or too small, then there will be a problem to reach the converging point i.e. minimum point in the loss function graph at which optimal weights are located.

In order to deal with the above problem we use a parameter called Learning rate which is used to control the above problem. Learning rate is multiplied by derivative(loss function) which helps to control the growth of derivative(loss function).

Problems associated with inefficient learning Rate

In the above shown graph, problems of inefficient learning rate is clearly depicted, and we can clearly see that if learning rate is large, then it will make derivative(loss function) shoot in a particular direction, and if it is small then derivative(loss function) will reach to the global minimum but it will take a very long period of time.

So, we have to choose learning rate accordingly, & the best value of it can be found by experimentation.

Now, the final equation becomes: