The concept and intuition behind Gradient Descent Algorithm.
By: Ravindra Sah
Tags: Ml Gradient Descent, Gradient Descent, Gradient Descent Python

Note: This is the second part of the linear regression tutorial and is followed by Introduction to linear regression and cost function. Check out the contents given below to follow along with the tutorial.

Contents: Introduction to linear regression and cost function, How the Gradient Descent Algorithm works, Mathematics of Gradient Descent.

Part 2: Gradient Descent

Why do we need a Gradient Descent?

In short to minimize the cost function, But How? Let’s see

The cost function only works when it knows the parameters’ values, previously in part one we manually choose the parameters’ value each time but during the algorithmic calculation, once the parameters’ values are randomly initialized it’s the gradient descent who have to decide what params value to choose in the next iteration in order to minimize the error, it’s the gradient descent who decide by how much to increase or decrease the params values.

Analogy: How Gradient Descent works?

Gradient Descent

Gradient Descent

Gradient Descent

What did you learn from the game? In the beginning, you try with learning rate (alpha)=1 but you fail to reach the minimum, because of the larger steps it overshoots the minimum. In the next game, you try with alpha=0.1, and this time you managed to reach the bottom very safely. what if you had tried with alpha=0.01, well, in that case, you will be gradually coming down but won’t make it to the bottom, 20 jumps are not enough to reach the bottom with alpha=0.01, 100 jumps might be sufficient. while solving a real-world problem, normally alpha between 0.01–0.1 should work fine but it varies with the number of iterations that the algorithm takes, some problems might take 100 or some might even take 1000 iterations. Based on these factors you can try with different values of alpha. Although tuning alpha value is one of the important tasks in understanding the algorithm I would suggest you look at other parts of the algorithm also like derivative parts, minus sign, update parameters and understand what their individual roles are.

learning rate

Gradient Descent in python

Gradient descent python code

Until now we are just using a single parameter to calculate cost function and algorithms. What the cost function looks like and how does the algorithm works when we have two or more parameters? See the figure below for intuitive understanding. Imagine yourself somewhere at the top of the mountain and struggling to get down the bottom of the mountain blindfolded.

Source: Coursera

The algorithm working principle is the same for any number of parameters, it’s just that the more the parameters more the direction of the slope. In the previous example of the bowl-shaped curve, we just need to look at the slope of theta1, But now the algorithm needs to look for both directions in order to minimize the cost function. Do you remember our predicted housing price problem? Now it’s time to solve the problem and predict housing prices for an area of 3000ft square or for any size(area). To solve this problem we are going to use two parameter values i.e bias term(theta0) and weight(theta1) and we are going to code gradient descent again for two params values. let’s do it.