Linear Regression — Basics that every ML enthusiast should know
By: Aditi Mittal
Tags: Machine Learning, Linear Regression, Mathematics, Statistics, AI

Even after the development of more complex and powerful ML algorithms in recent times, linear regression is still hard to beat because of its versatility, robustness, and simplicity.

Linear regression was originally developed in the field of statistics and is studied as a model for understanding the relationship between input and output numerical variables, but has been borrowed by ML. Now, it is classified both as a statistical algorithm and a machine learning algorithm.

When there is a single input variable, the model is called simple linear regression and when there are multiple input variables, the method is known as multiple linear regression.

Linear Regression Graph ( Source: http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/R/R5_Correlation-Regression/R5_Correlation-Regression_print.html)

Representation

Linear regression is represented by a linear equation that combines a set of input values (x) to the predicted output for that set of input values (y) where both x and y are numeric. (OBVIOUSLY!) This linear equation assigns a factor to each input value, known as coefficient (B). One coefficient is also added to give the line an additional degree of freedom (e.g. moving up and down on a two-dimensional plot) and is known as the bias coefficient.

In a simple regression model, the representation would be:

y = B0 + B1*x

It represents a simple line. But in higher dimensions when we have more than one input values, this line is called a plane represented by:

y=B0 + B1*x1+B2*x2+B3*x3

y is also called dependent variable whereas x’s are known as independent variables.

Ordinary Least Squares (OLS)

In linear regression, you need to find the equation of the line that best fits your data. But now the question is “HOW?”

The best-fit line can be attained by minimizing the sum of the squared difference between the data points and the regression line. This is called Ordinary Least Squares (OLS).

That means, we need to

find the values for B0 and B1 that minimizes the sum of squared errors where squared error for observation i can be calculated by using the given formula:

(yi - (B0 + B1*xi))^2

And the sum of squared error is just the sum of

(yi - (B0 + B1*xi))^2 for all observations.

This approach treats the data as a matrix and uses linear algebra operations to estimate the optimal values for the coefficients. It means that all of the data must be available and you must have enough memory to fit the data and perform matrix operations.

Gradient Descent

When there are one or more inputs, we can use a process of optimizing the values of the coefficients by iteratively minimizing the error on the training data. This operation is called Gradient Descent.

In this, we start with random values for each coefficient and then the sum of the squared errors is calculated for each pair of input and output values. A learning rate is used as a scale factor and the coefficients are updated in the direction towards minimizing the error. The process is repeated until a minimum sum squared error is achieved or no further improvement is possible. When using this method, a learning rate (alpha) parameter is selected that determines the size of the improvement step to take on each iteration.

Regularization

Regularization minimizes the sum of the squared error of the model on the training data and also reduces the complexity of the model (like the number or absolute size of the sum of all coefficients in the model).

Two popular examples of regularization procedures for linear regression are:

Lasso Regression: where Ordinary Least Squares is modified to also minimize the absolute sum of the coefficients (called L1 regularization).

Ridge Regression: where Ordinary Least Squares is modified to also minimize the squared absolute sum of the coefficients (called L2 regularization).

These methods are effective to use when there is collinearity in your input values and ordinary least squares would overfit the training data.

Outliers and Influential Observations

After a regression line has been computed for a group of data, a point which lies far from the line is known as an outlier. Such points may represent erroneous data, or may indicate a poorly fitting regression line.

If a point lies far from the other data in the horizontal direction, it is known as an influential observation. The reason for this distinction is that these points have may have a significant impact on the slope of the regression line.

Metrics for model evaluation

R-Squared value

This value ranges from 0 to 1. Value ‘1’ indicates predictor perfectly accounts for all the variation in y. Value ‘0’ indicates that predictor ‘x’ accounts for no variation in ‘y’.

R^2 = 1 - (residual sum of squares / total sum of squares)

1. Regression sum of squares (SSR)

This gives information about how far estimated regression line is from the horizontal ‘no relationship’ line (average of actual output).

2. Sum of Squared error (SSE)

This tells us about how much the target value varies around the regression line .

3. Total sum of squares (SSTO)

This tells how much the data point move around the mean.

Conclusion

Linear Regression is the easy to understand and easiest to pick up algorithm. We have Ordinary Least Mean Square, Gradient descent and regularization techniques to find the line that best fits our data. We also discussed various metrics that are used in linear regression.