Brief Introduction to Regularization: Ridge, Lasso, and Elastic Net
By: Sai Nikhilesh Kasturi
Tags: AI, Statistics, Data Science, Supervised Learning, Machine Learning

Supervised Machine Learning is a currently used buzz word by many organizations to identify and solve business problems. There are two types of algorithms commonly used — Classification and Regression. In this article, we will focus on Regression. Regression analysis are the models which predict a continuous outcome. A few examples include predicting the housing market prices, sales of a retail store, weather prediction and many more!

What Is Regression?

Regression searches for relationships among variables.

For example, you can observe several houses of a locality and try to understand how their prices depend on the features, such as area of the house, type of utilities available, type of roof, role, city in which it is located, number of bedrooms and so on.

This is a regression problem where data related to each house represent one observation. The presumption is that the area of the house, type of utilities available, type of roof, role, city in which it is located are the independent features, while the price depends on them.

Generally, in multivariate analysis, you would like to seek out a function that maps some features or variables to others sufficiently well.

The dependent features are called the dependent variables, outputs, or responses. The independent features are called the independent variables, inputs, or predictors.

Regression problems usually have one continuous and unbounded variable . The inputs, however, are often continuous, discrete, or maybe categorical data like gender, nationality, brand, and so on.

It is a standard practice to denote the outputs with 𝑦 and inputs with 𝑥. If there are two or more independent variables, they will be represented because the vector 𝐱 = (𝑥₁, …, 𝑥ᵣ), where 𝑟 is that the number of inputs.

When Do You Need Regression?

Typically, you would like regression to answer whether and the way some phenomenon influences the other or how several variables are related. for instance , you’ll use it to work out if and to what extent the experience or gender impact salaries.

Regression is additionally useful once you want to forecast a response employing a new set of predictors. For instance, you’ll attempt to predict electricity consumption of a household for subsequent hour given the outdoor temperature, time of day, and number of residents therein household. Regression is employed in various fields: Supply chain planning, marketing then on.

First, we need to understand the basics of regression and what parameters of the equation are changed before getting playing around with various models. Simple linear regression, also known as ordinary least squares (OLS) attempts to minimize the residuals. The error, in this case, is the difference between the actual value and its predicted value.

The equation for this model is referred to as the cost function.

One common problem is the features of the data showing multi-collinearity, which is explained as predictor variables are correlated to each other and to the response variable. To picture this, let’s say we’re doing a study that looks at a response variable — patient weight, and our predictor variables would be height, sex, and diet. The problem here is that height and sex are also correlated and can inflate the standard error of their coefficients which may make them seem statistically insignificant.

The general solution to this is: reduce variance at the cost of introducing some bias. This approach is called regularization and is almost always beneficial for the predictive performance of the model. To make it sink in, let’s take a look at the following plot.

As the model complexity, which in the case of linear regression can be thought of as the number of predictors, increases, estimates’ variance also increases, but the bias decreases. The unbiased OLS would place us on the right-hand side of the picture, which is far from optimal. That’s why we regularize: to lower the variance at the cost of some bias, thus moving left on the plot, towards the optimum.

This graphic illustrates what bias and variance are. Imagine the bull’s-eye is the true population parameter that we are estimating, β, and the shots at it are the values of our estimates resulting from four different estimators — low bias and variance, high bias and variance, and the combinations thereof.

Both the bias and the variance are desired to be low, as large values result in poor predictions from the model. In fact, the model’s error can be decomposed into three parts: error resulting from a large variance, error resulting from significant bias, and the remainder — the unexplained part.

To produce a more accurate model of complex data we add a penalty term to the OLS equation. A penalty adds a bias towards certain values. These are known as L1 regularization(Lasso regression) and L2 regularization(ridge regression).

Ridge Regression

Ridge regression uses L2 regularization which adds the following penalty term to the OLS equation.

The L2 term is equal to the square of the magnitude of the coefficients. In this case, if lambda(λ) is zero then the equation is the basic OLS, but if it is greater than zero then we add a constraint to the coefficients. This constraint leads to minimized coefficients (aka shrinkage) that trend towards zerom the larger the worth of lambda. Shrinking the coefficients results in a lower variance and successively a lower error value. Therefore Ridge regression decreases the complexity of a model but doesn’t reduce the amount of variables, it rather just shrinks their effect.

Bias-Variance Trade-Off in Ridge Regression

Incorporating the regularization coefficient in the formulas for bias and variance gives us

From there you can see that as λ becomes larger, the variance decreases, and the bias increases. This poses the question: how much bias are we willing to accept in order to decrease the variance? Or: what is the optimal value for λ? There are two ways we could tackle this issue. A more traditional approach would be to settle on λ such some information criterion, e.g., AIC or BIC, is that the smallest.

Lasso Regression

Lasso regression uses the L1 penalty term and stands for Least Absolute Shrinkage and Selection Operator. The penalty applied for L2 is equal to absolutely the value of the magnitude of the coefficients:

Similar to ridge regression, a lambda value of zero spits out the essential OLS equation, but given an appropriate lambda value lasso regression can drive some coefficients to zero. The larger the worth of lambda the more features are shrunk to zero. This will eliminate some features entirely and provides us a subset of predictors that helps mitigate multi-collinearity and model complexity. Predictors not shrunk towards zero signify that they’re important and thus L1 regularization allows for feature selection (sparse selection). As a result, for top values of λ, many coefficients are exactly zeroed under lasso, which isn’t the case in ridge regression.

Elastic Net

A third commonly used model of regression is that the Elastic Net which includes penalties from both L1 and L2 regularization:

In addition to setting and selecting a lambda value elastic net also allows us to tune the alpha parameter where 𝞪 = 0 corresponds to ridge and 𝞪 = 1 to lasso. Simply put, if you connect 0 for alpha, the penalty function reduces to the L1 (ridge) term and if we set alpha to 1 we get the L2 (lasso) term. Therefore we will choose an alpha value between 0 and 1 to optimize the elastic net. Effectively this may shrink some coefficients and set some to 0 for sparse selection.

Implementation of Linear, Ridge, and Lasso on a dataset

So now let us play around Lasso and Ridge by applying both the algorithms to a housing dataset and then comparing their performance.

Loading the data.

Once we load the data, we do exploratory data analysis, clean the data and apply different pre-processing techniques.

checking for the missing values

After the running Linear Regression and Linear Regression with hyper parameter tuning on the above dataset. The code and results are :

The above output shows that the RMSE, one of the two evaluation metrics. Running basic linear regression and running regression after getting the best parameters from grid search cv and evaluation metrics.

After running both Lasso and Lasso with hyper parameter tuning on the above data set. The code and results are :

Code for lasso

Lasso Regression can also be used for feature selection because the coeﬃcients of less important features are reduced to zero.

The above output shows that the RMSE values for the Lasso Regression model for different values of alpha on the training and test data respectively.

After the running both Ridge and Ridge with hyper parameter tuning on the above data set. The code and results are :

Ridge

The above output shows that the RMSE values for the Ridge Regression model for different values of alpha on the training and test data respectively.

Comparison of all the models

As you can see from the comparison chart above, we can see that with adding a penalty term to the OLS equation. The accuracy of the models on the complex data are better.

Elastic Net

Elastic Net first emerged as a result of critique on lasso, whose variable selection can be too dependent on data and thus unstable. The solution is to combine the penalties of ridge regression and lasso to get the best of both worlds. Elastic Net aims at minimizing the following loss function:

where α is that the mixing parameter between ridge (α = 0) and lasso (α = 1).

End Notes

To summarize, here are some salient differences between Lasso, Ridge and Elastic-net:

Lasso does a sparse selection, while Ridge doesn’t .When you have highly-correlated variables, Ridge regression shrinks the 2 coefficients towards each other.

Lasso is somewhat indifferent and usually picks one over the opposite . Elastic-net may be a compromise between the 2 that attempts to shrink and do a sparse selection simultaneously.

Ridge penalizes the larger errors quite it penalizes the smaller ones (as they’re squared within the penalty term).

Lasso penalizes them more uniformly. this might or might not be important. during a forecasting problem with a strong predictor, the predictor’s effectiveness is shrunk by the Ridge as compared to the Lasso.

You can find the whole code in my GitHub repository.

Do you have any questions about related to this post? Leave a comment and ask your question and I will do my best to answer it.

Thanks for reading! ❤