Regression and its variants: Simple, Multiple, LASSO, Ridge and Stepwise Regression
By: Mahbubul Alam
Tags: Regression, Ridge Regression, Lasso Regression, Data Science, Machine Learning

Let’s say as a data scientist you want to analyze a student dataset that has records of their height and weight. You observe that there is a missing value in the weight column. Can you make a prediction of this missing value?

A hypothetical student dataset

Regression helps solve these kinds of problems. In real-world machine learning applications, regression models are frequently used to make predictions of unknown values based on known features (e.g. determining the price of a used car based on make, model, mileage etc.) and it is done by building a statistical relationship between an unknown variable (a.k.a dependent variable) and known features (a.k.a. independent variables). This relationship can be expressed as:

dependent_variable = f(independent_variable_1, independent_variable_2 ….)

This statistical relationship can take many different forms depending on the type of predictors, outcomes and the function used to build the relationship. Below are five types of regression models that are frequently encountered. I’m describing them in a way to explicitly link one type to the next so that it is easy to understand the similarities and differences.

1) Simple linear regression

As the name suggests, simple regression is literally the simplest form of all regressions. It has only one dependent variable and is explained by one independent variable, both represented in a two-dimensional scatter diagram. The example above — the relationship between height and weight — is a classic example of simple linear regression. The mathematical formulation of a simple linear regression model takes the following form:

Weight = b0+ b1*Height + e

where, b0 = intercept, b1 = coefficient and e = error term

2) Multiple regression

Multiple regression is similar to simple linear regression, but in this case, instead of one, there will be multiple independent variables. If we follow the above example again and suppose weight is predicted not just by height but an additional variable — let’s say age — then it’s multiple regression. In the mathematical formulation we just add that additional variable:

Weight = b0+ b1*Height + b2*Age + e

3) LASSO regression

So, multiple regression is better than simple linear regression, right? Since there are more predictors the prediction must be more accurate! Not really, not always. Sometimes simpler models perform better than a complex one. The multiple regression above has 2 independent variables, which is still quite simple, but what if there are 20 or 200 variables in the dataset? That’s where data scientists need to think hard about how many and which features to keep and which ones to let go.

In machine learning, feature selection is an important step to eliminate overfitting, which is also the case in regression. So in LASSO, if there are too many features, some of them are completely eliminated. And it is done by setting the coefficients to zero. This process is known as “L1 Regularization” in machine learning vocab.

4) Ridge regression

In LASSO regression, we simplify regression equations that have too many features and we do so by completely eliminating some of them. Ridge regression serves the same purpose, i.e. simplifies models, but instead of eliminating features entirely it minimizes their effect.

So in ridge regression, the feature coefficients are shrunk to close to zero but not exactly zero (a process known as L2 Regularization). In machine learning ridge regression models, a hyperparameter called lambda is used to control the weight of the penalty associated with the loss function.

Since no feature is completely eliminated, ridge regression is not used for feature selection.

5) Stepwise regression

So far we’ve been looking into various ways of manipulating features to enhance model performance. There is an old school, yet equally effective approach for finding better models, known as “Stepwise regression”.

As the name suggests, in stepwise regression you start with the simplest model (e.g. 1 dependent and 1 independent variable), then evaluate its performance. Then you add another variable and evaluate the performance again and compare two models to find the better one. This process is repeated until the best performing model is found.

An estimator called Akaike Information Criterion (AIC) is used to compare model performance in stepwise regression.

Summary

Regression quantifies the relationship between a dependent and one or more independent variables and there are many different ways to establish and quantify this relationship. In this article, I focused on 5 types of regression which, as a data scientist, you will most frequently encounter. One thing to remember, although there are differences between them, they are not entirely different classes. There are more similarities than differences. So when learning regression (or any modeling family for that matter), it is better to learn them altogether rather than one at a time.