Gradient Descent
By: Hemangi Kinger
Tags: Machine Learning, Ml Gradient Descent, Gradient Descent

In our day-to-day lives, we are optimizing variables based on our personal decisions and we don’t even recognize the process consciously. We are constantly using optimization techniques all day long, for example, while going to work, choosing a shorter route in order to minimize traffic woes, figuring out and managing a quick walk around the campus during a snack break, or scheduling a cab in advance to reach the airport on time.

Optimization is the ultimate goal, whether you are dealing with actual events in real-life or creating a technology-based product. Optimization is at the heart of most of the statistical and machine learning techniques which are widely used in data science.

Optimization for Machine Learning

Accuracy is the word with which we are most concerned, while we are dealing with problems related to machine learning and artificial intelligence. Any rate of errors cannot be tolerated while dealing with real-world problems and neither should they be compromised.

Let us consider a case of self-driving cars. The model fitted in the car detects any obstacles that come in the way and takes appropriate actions, which can be slowing down the speed or pulling on the brakes and so on. Now we need to keep this in mind that there is no human in the car to operate or withdraw the actions taken by the self-driving car. In such a scenario, suppose the model is not accurate. It will not be able to detect other cars or any pedestrians and end up crashing leading to several lives at risk.

This is where we need optimization algorithms to evaluate our model and judge whether the model is performing according to our needs or not.

Optimization may be defined as the process by which an optimum is achieved. Optimization techniques are performed on the training data and then the validation data set is used to check its performance. Here we will talk about a optimization technique gradient descent

Gradient Descent

The whole concept of Gradient descent is to minimize the cost function of a ML model

It uses the below formula to minimize the cost of a function

I know I know this is not going to make sense if I just throw some formulas and call it a day

So let’s take step by step

First let me explain you how a ML model works and what a cost function is

Machine Learning Model

Consider a bunch of data points in a 2 D space. Assume that the data is related to the area and price of a group of houses. We are trying to predict some kind of relationship between these quantities so that we could predict the price of some new houses afterwards given the area of the houses. This is essentially a simple example of a supervised Machine Learning technique.

Let us now draw an arbitrary line in space that passes through some of these data points. The equation of this straight line would be Y = mX + b where m is the slope and b is its intercept on the Y-axis.

Predictions

With a known set of inputs and their corresponding outputs, a machine learning model attempts to make predictions according to the new set of inputs.

Error

The Error would be the difference between the two predictions.

This relates to the idea of a Cost function or Loss function.

Cost Function

A Cost Function/Loss Function evaluates the performance of our Machine Learning Algorithm.

A Cost Function/Loss Function tells us “how good” our model is at making predictions for a given set of parameters. The cost function has a curve and a gradient, the slope of this curve helps us to update our parameters and make an accurate model.

Let’s say, there are a total of ’N’ points in the dataset and for all those ’N’ data points we want to minimize the error. So the Cost function would be the total squared error i.e

Now a question may arise,

Why do we take the squared differences and simply not the absolute differences?

Because the squared differences make it easier to derive a regression line. Indeed, to find that line we need to compute the first derivative of the Cost function, and it is much harder to compute the derivative of absolute values than squared values. Also, the squared differences increase the error distance, thus, making the bad predictions more pronounced than the good ones.

Gradient Descent in Action

So as you may have got the idea till now We need to minimize the cost function which will also result in a lower error between the predicted values and the actual values which also denotes that the algorithm has performed well in learning.

How do we minimize this?????

If we look carefully, our Cost function is of the form Y = X² . In a Cartesian coordinate system, this is an equation for a parabola and can be graphically represented as :

As mentioned above the cost function is of the form

Now in order to minimize the function mentioned above, firstly we need to find the value of X which will produce the lowest value of Y (in this case it is the red dot). With lower dimensions (like 2D in this case) it becomes easier to locate the minima but it is not the same while dealing with higher dimensions. For such cases, we need to use the Gradient Descent algorithm to locate the minima.

Now how would you reach the minima the redo dot

Consider that you are walking along the graph below, and you are currently at the ‘green’ dot. Your aim is to reach the minimum i.e the ‘red’ dot, but from your position, you are unable to view it.

Think of gradient descent as hiking down to the bottom of a valley.

The equation below describes what gradient descent does: b is the next position while a represents his current position.

Δf(a) is nothing but the cost of going from point a to point b

Where as gamma in the middle is the Learning Rate ie in our analogy of moving down the mountain is how big of a step you should take

IMPORTANCE OF THE LEARNING RATE

How big the steps are gradient descent takes into the direction of the local minimum are determined by the learning rate, which figures out how fast or slow we will move towards the optimal weights.

For gradient descent to reach the local minimum we must set the learning rate to an appropriate value, which is neither too low nor too high. This is important because if the steps it takes are too big, it may not reach the local minimum because it bounces back and forth between the convex function of gradient descent (see left most image below). If we set the learning rate to a very small value, gradient descent will eventually reach the local minimum but that may take a while (see the right most image).

HOW TO MAKE SURE IT WORKS PROPERLY

A good way to make sure gradient descent runs properly is by plotting the cost function as the optimization runs. Put the number of iterations on the x-axis and the value of the cost-function on the y-axis. This helps you see the value of your cost function after each iteration of gradient descent

If gradient descent is working properly, the cost function should decrease after every iteration.

When gradient descent can’t decrease the cost-function anymore and remains more or less on the same level, it has converged.

Conclusion

The point of this article was to demonstrate the concept of gradient descent. We used gradient descent as our optimization strategy