Implementing Gradient Descent for multilinear regression from scratch.
By: Gunand Mayanglambam
Tags: Python, Data Science, Deep Learning, Gradient Descent, Machine Learning

Gradient Descent is one of the most popular optimization algorithms that every Data science enthusiast should have a deep understanding of this topic.

Here, in this blog, my target is to make even the newbie to the Machine Learning field should be able to understand easily. The only prerequisite is just basic python. In this blog, I will be using the Boston house price dataset, which is a toy dataset provided by sklearn library.

About the Dataset: It has 506 records,13 features, and 1 target variable.

Gradient Descent is an iterative algorithm use in loss function to find the global minima. The loss can be any differential loss function. The different types of loss functions are linear loss, logistic loss, hinge loss, etc. For our dataset, we will be using linear loss because the target is a continuous variable.

In the above figure, considering the loss function as a convex function, on the x-axis, we have the parameter weight and on the y-axis, we have a loss value, the blue curve line is the loss function. At first, we are randomly initializing the weight “w” and using the Gradient descent algorithm the final weight “w” is obtained which gives the minimum loss.

When we say we are building a Linear Regression model, It is nothing but we are trying to find a straight line(one feature) or a hyperplane(multiple features) that best fits the data. And the equation of a line is represented by “mx+b=0” where “m” is the slope and “b” is bias. We can also refer to “m” as weight and “b” as intercept.

For example, if we have 5 features then the equation of hyperplane is represented by :

Now Let’s see how to find the best fit line using Gradient Descent.

In the above line equation, “m” and “b” are the parameter we need to update using Gradient descent to find the best fit line (when I say the best fit line, it is nothing but finding minima in loss function) and “x” is the given input data.

The equation to update weight and bias:

where “dl/dw” is derivative of loss w.r.t weight, “dl/db” is derivative of loss w.r.t bias, and “n” is the total number of records. Here, the weight is a vector with size=13(we have 13 features).

Steps for Gradient Descent:

Standardize the data:

2. Initialize the parameter and hyper_parameter

3. Find derivatives of loss w.r.t weight and bias.

Loss Equation

Derivatives of loss w.r.t “weight”

Derivatives of loss w.r.t “bias” or “intercept”

4. Update the weight and bias till we get the global minima.

The above figure is the plot between loss and the number of the epoch.

After each epoch, the loss is reduced

Initially, the loss is decreased drastically till the 1000 epoch

After the 1000 epoch, there is a minimal decrease in the loss.

This shows that we have reached a global minimum.

5. Using the final weight vector and bias, we can predict the output

This is the final weight and bias after the 2000 epoch

The above table is the comparison between the actual target and the model predicted target.

Lastly, if you want to see the entire code, you can click on this link Github.

Here, is the link for implementation of Stochastic Gradient Descent for multilinear regression on the same dataset: link

If You Enjoyed this article: