An Introduction to Linear Regression & Gradient Descent
By: Caroline Pacheco Do E.Silva
Tags: Linear Regression, Gradient Descent, Inteligencia Artificial, Mathematics, Machine Learning

Linear Regression is one of the most popular supervised machine learning. It predicts values within a continuous range, (e.g. sale prices, life expectancy, temperature, etc) instead of trying to classify them into categories (e.g. car, bus, bike and others). The main goal of the linear regression is to find the best fit line which explains the relationship between the data.

Fig.1: Linear regression simulation.

The best-fitting line (or regression line) is represented by:

where m is the slope of the line known as angular gradient and b is the point at which the line crosses the y-axis also known as linear gradient.

The main challenge to find the regression line is to determine the value of m and b, such that the line corresponding to those values is the best fitting line, therefore, the line that provides the minimum error.

How to find the best m e b values?

A possible alternative would be to use methods like Ordinary Least Squares (OLS) — which is an analytical and non-iterative solution. OLS is a type of linear least squares method to estimate the unknown parameters. The OLS is defined by:

where x is the independent variables, x ̄ is the average of independent variables, y is the dependent variables y ̄ is the average of dependent variables.

The OLS can be a good option to solve the problem of linear regression, because it has coefficients and linear equations. Nevertheless, apply OLS to complex and non- linear machine learning algorithms, such as Neural networks, Support Vector Machines, etc. will not be feasible. This is due to the fact that the OLS solution isn’t scalable.

Instead of OLS we will find the numerical approximation by an iterative method. The Gradient descent is one of the best optimisation algorithms that approximate a solution by an iterative procedure able to efficiently explore the parameter space, instead of obtaining an exact analytical solution.

Understanding Gradient Descent Concept

Suppose you’re at the top of a mountain, and you have to reach a lake which is at the lowest point of the mountain. A twist is that you’re blind. Therefore, what will you do to reach the lake?

This is exactly the main idea behind the Gradient Descent does. Its goal is to reach the lowest point of the mountain. The mountain represents the data plotted in a space, the size of the step you move is the learning rate, feeling the incline around you and decide which is higher — is calculating the gradient of a set of parameter values. The chosen direction is where the cost function reduces. The lowest point in the mountain is the value -or weights- where the cost function reached its minimum (the parameters where our model presents more accuracy).

Illustration of how the Gradient descent algorithm works.

Let’s understand Linear Regression algorithm step by step

Suppose the regressor algorithm was given as input a set of points. After replacing the values of all points in Eq. (1), the algorithms returns the optimal values for m and b. You might ask yourself how did the regressor find this solution? We will describe below the steps normally employed by a regressor algorithm to find the best fitting line using Gradient Descent.

Step 1: In the beginning, m and b receive random values to begin with. These values are then replaced in Eq.(1).

Step 2: It’s very likely that the line found in [Step 1] is wrong. To determine how much this line is incorrect, we will calculates the total error of this line using the cost function. We can calculate the cost function using different methods such as Mean Square Error, Quadratic loss, L2 Loss Mean Square Error (MSE), etc. The cost function can be defined by:

the cost function above measures the squared difference between an observation’s actual reg and predicted value y (corresponds the points of the data).

Step 3: Now that we know how wrong our function is, we need to adjust the function to minimise this error. To minimise cost function we use will Gradient Descent . By replacing the y value (Eq. (1)) in Eq.(3), we obtain:

There are two parameters (coefficients) in our cost function we need to adjust: m and b values. Therefore, we will use the Partial Derivatives to determine the impact each one has on the final prediction. To adjust the m and b values we will use the following equations:

Updating the m value

We will start by calculating the derivative of the cost function in relation to m in Eq.(4). Remembering the cost function presented in Eq. (3):

Thus:

Applying the chain rule:

We obtain:

Replacing (reg = mx + b) in the above equation, we get: