Linear Regression Using Gradient Descent Algorithm .
By: Prince Sharma
Tags: Machinelerning, Linear Regression, Deep Learning, Gradient Descent, Machine Learning

The values of m and c are updated at each iteration to get the optimal solution

Linear Regression

In statistics ,Linear Regression is the modelling technique which investigate the relationship between a independent variable and the dependent variable.

If we assume x be a independent variable and y be a dependent variable then we can define the linear relationship between these two as follows:

This is the equation of line that we studied in the mathematics of high schools , where the m represent the slope of line and the c represent the intercept of the line. Today we are going to use this to train our model by given data set and we will also predict the Y for a given value of X. Our challenge today is to determine the value of m and c, such that the line corresponding to those values is the best fitting line or gives the minimum error.

Loss Function

Basically the Loss Function is the error in our predicted value of m and c .Our goal is to minimize the error by getting the most accurate value of m and c. we are using here mean squared error function to calculate the loss . There are three step in this function

Difference between the actual Y and the predicted value of Y.

where y = mx + c

2. Square the difference.

3. Finding the mean of the squared equation.

Mean Squared Error Equation

now we have the mean squared error equation in this equation. where n is the no. of value of X , y is the actual value , and yₚ represents the predicted value .

Gradient Descent Algorithm

Gradient descent algorithm is an iterative optimization algorithm to find the minimum of a function. Here that function is our loss function.

This show how Gradient Algorithm Works.

In this diagram, there is a valley and a person who only want to reach at the goal ,Now the person goes down and take large steps as the slope is steep at that moment and when is come close to goal the slope become less steep and that’s why he take small steps .He puts his step based on how much he is closed to his goal.

Let’s try applying gradient descent to m and c and approach it step by step:

Initially let m = 0 and c = 0. Let L be our learning rate. This controls how much the value of m changes with each step. L could be a small value like 0.0001 for good accuracy. Calculate the partial derivative of the loss function with respect to m, and plug in the current values of x, y, m and c in it to obtain the derivative value D.

Dₘ is the value of the partial derivative with respect to m. Similarly lets find the partial derivative with respect to c, Dc :

3. Now we update the current value of m and c using the following equation:

4. We repeat this process until our loss function is a very small value or ideally 0 (which means 0 error or 100% accuracy). The value of m and c that we are left with now will be the optimum values.

Now going back to our analogy, m can be considered the current position of the person. D is equivalent to the steepness of the slope and L can be the speed with which he moves. Now the new value of m that we calculate using the above equation will be his next position, and L×D will be the size of the steps he will take. When the slope is more steep (D is more) he takes longer steps and when it is less steep (D is less), he takes smaller steps. Finally he arrives at the bottom of the valley which corresponds to our loss = 0.

Now with the optimum value of m and c our model is ready to make predictions !

Implementing the Model

Now let’s convert everything above into code and see our model in action !