Understanding & Comparing Linear Regression Using OLS and Gradient Descent
By: Ganesh Patil
Tags: Ordinary Least Square, Optimization Algorithms, Linear Regression, Gradient Descent, Machine Learning

Photo by Clarisse Croset on Unsplash

There are many articles available online explaining the process of gradient descent as well as the process of Ordinary Least Squares (OLS). However, it is very difficult to find a single compilation with an explanation of why do we need two algorithms, what are the basic differences between them, and which algorithm should be used in which situation? In this article, I will try to address these questions in detail. The rest of the article is organized as follows:

1. Linear Regression warm-up

2. Ordinary Least Square method

3. Gradient Descent method

4. Conclusion

Linear Regression warm-up

Regression is an inferential statistical methodology where we use sample dataset and derive an equation to estimate the properties of larger population. In regression, a single continuous variable Y, also called as dependent/target variable, is expressed as a function of multiple independent variables. Linear regression models the target variable with a line or a hyperplane, depending on the number of dimensions of input.

Assuming that in the problem at hand the dependent and independent variables are linearly related, and impact of different variables are additive, the equation of a typical linear regression can be written as below:

Where,

Y -> Dependent variable

X1, X2 -> Independent variables

B0, B1 and B2 -> The model coefficients that represents the mean change in the dependent variable for each 1-unit change in an independent variable when you hold all of the other independent variables constant

E -> Unpredictable error component which implies the error that cannot be explained by any of the covariates

In order to ensure that the model estimates the dependent variable correctly for all the members of the population, in an ideal scenario the error term is expected to be zero. However, it is difficult to achieve this feat in real world. Thus, we try to minimize this error as much as possible and bring it close to zero.

If we rearrange the equation of linear regression, the error term can be represented as:

The error term will be minimum if the difference between actual value and observed value is minimum. If we look at it closely, the only levers we have to ensure that the error term E is as low as possible are values of model coefficients B0, B1 and B2 as all other values are actual values.

There are different ways to find the optimum values of B0, B1 and B2 to minimize the error term E. Two of the most popular methods that we are going to compare here are:

1. Ordinary Least Square (OLS)

2. Gradient Descent

Let’s have a quick look at how these approaches work:

Ordinary Least Square (OLS)

OLS method tries to minimize the sum of the squares of deviations of observations from the estimated values i.e. it tries to minimize the sum of square of equation (2) above for all the observations.

In mathematical form, the aim of linear regression using OLS is to select B0, B1 and B2 so as to minimize:

This equation is also called as loss function or cost function as it provides the cost of or loss from interpreting the data using a linear regression.

As per the rules of calculus we can obtain this minimum value by taking the partial derivative of cost function as below:

Please note that using OLS we get the exact solutions; the coefficients are obtained in single iteration by solving the closed form partial derivative equations. Moreover, solving the equation makes the whole process highly transparent and easy to comprehend.

Linear regression using OLS is vulnerable to the problem of multicollinearity. As mentioned in the explanation of equation (1) above, linear regression coefficients represent the mean change in the dependent variable for each 1-unit change in an independent variable when you hold all of the other independent variables constant. Presence of multicollinearity compromises this definition as change in one variable effectively changes the other and hence impacts the dependent variable. This reduces the precision of coefficient estimates and in turn the effectiveness of the model.

Another way to visualize the linear regression equation is using matrices of weights, inputs and outputs as:

Please note that solving OLS, involves lot of heavy matrix computations. Assuming that there are N observations and V number of independent variables where N >>> V, the computation complexity of OLS involves:

A. Multiplication of XTX: O(V2N)

B. Multiplication of XTY: O(VN)

C. Inverse of XTX: O(V3)

While these computations are fast when we have small handful of independent variables, as the dimensionality increases the complexity reaches to extremely high level for large values of N (which is the case in most of the ML problems).

Moreover, albeit unlikely there are instances when the matrix XTX is not invertible. A matrix can be non-invertible because of 2 reasons:

i. It is not a square matrix: Which will never be the case with XTX

ii. Determinant of the matrix is zero: One of the reasons this could be possible is if the columns of X (i.e. independent variables) are linearly dependent on each other i.e. highly collinear

Different statistical software packages use different workaround for this problem bringing in approximations in the coefficient computations. The ways to compute XTX when its not invertible or otherwise is not in the scope of this article. The point I am trying to drive home here is that for large values of N the closed form solution becomes unworkable and computationally expensive.

Gradient Descent (GD)

Just to recap, gradient descent is one of the ways (like OLS) to find optimum values of regression coefficients (B0, B1 and B2) to bring the error term E in linear regression close to zero. Unlike OLS, gradient descent does not try to solve any closed for solution. It achieves the same objective by more crude way of starting with random values of regression coefficients (B0, B1 and B2) and iteratively changing them to achieve the lowest value of error term E.

For the purpose of simplicity and to visualize the GD in two-dimensional space, let’s assume that this is a univariate regression equation and hence have only two moving parameters B0 and B1. Thus, the error term in equation (2) will change to:

The objective of gradient descent is to minimize the following mean squared function:

Where, n is a total number of data points in the data.

If you are wondering why squared error, then please understand that the objective is to bring the error term as close to zero as possible. Any negative value is equally not acceptable to achieve our objective. Moreover, because we are summing over all the data points, not taking square would result in misleading error term because positive and negative errors would cancel each other. Squaring solves this issue as the negative errors would be considered equally bad as positive errors.

Division by (2 * n) is to reduce the scale and simplify the computations. Note that irrespective of division by (2 * n) the result will be the same.

The gradient descent is an iterative optimization algorithm. To understand this process effectively, it is very important to visualize it.

If we plot any non-linear mathematical equation in two-dimensional space, it is expected to have minimum values as shown in the sample example for one such functions below:

Where, global minimum is the absolute minimum value of the function and local minimum is the minimum value within a smaller range of the function.

In our case, the function is the error function of linear regression described in equation (4) and the objective is to find the global minimum value of this error function that is dependent on B0 and B1. If we compute the squared error function in equation (4) for multiple values of B0 and B1 and plot it on two-dimensional space, we get a parabolic curve as shown below:

To start with, gradient descent assumes random values of parameters B0 and B1 and computes the error term in equation (4). Let’s assume that the random values of B0 and B1 gives use the cost shown by blue circle in the graph below:

It is important to understand here that we don’t know the actual cost function graph in the beginning. We are just randomly computing cost value in a hope to minimize it using properties of calculus irrespective of (as long as it is differentiable and convex) underlying function.

Once the initial random position on cost function is secured, the gradient descent starts an iterative process wherein it tries to move closer to optimal solution by reducing the error term in each iteration. To achieve this movement towards optimum solution, gradient descent utilizes the constantly changing slope of the cost function. The idea is that at the optimal solution, the slope of the cost function is zero whereas for all other points there is some slope which can be utilized as the direction of movement of the cost function.

The slope of the cost function in equation (4) can be easily computed by taking partial derivative of error function with respect to B0 and B1 as shown below:

Based on the values of these slopes the gradient descent decides the direction with lower descent and moves in that direction. It repeats this process until it reaches to the optimal solution.

Mathematically, this process can be explained using following equations:

The α in equations above, is called a learning rate which decides the step size to take in the direction of lower descent after each iteration. The value of α plays an important role in how fast the gradient descent converges to the optimum solution. If α is too small, the process will take small steps in each iteration and hence will take large number of iterations to reach the minima. On the other hand, it α is too large the process will keep toggling between the different points without ever reaching the minima.

The best way to find the correct value of α is to run the algorithm for multiple values of α, review the final cost value obtained and adjust the value to obtain the lowest value of cost function.

Using gradient descent, we try to achieve the “good enough” optimum solution instead of an exact solution. In most of the cases we don’t reach the exact global minima but obtain the pretty close approximation which is enough for large scale problems which are difficult to solve using closed form solutions.

The process works in exact same manner for multivariate cost functions. The only difference would be that the movement along the cost function could be in any direction of the largest descent instead of two directional movement shown above for univariate cost function.

The method of gradient descent is a generic methodology containing series of iterations that can be applied to any function. The illustration above is just an instance of its application in optimizing the cost function of linear regression. The GD works best in case of convex cost functions as shown above. It can also be applied to quasi-convex and non-convex surfaces too. However, in such cases it runs the risk of slowing down or finding local minima or saddle point instead of global minima.

Conclusion

To summarize, the key difference between OLS and GD are as below: