Intro to Regularization With Ridge And Lasso Regression with Sklearn | by Bex T. | Towards Data Science

Linear Regression a.k.a. Ordinary Least Squares is one of the easiest and most widely used ML algorithms. But it suffers from a fatal flaw — it is super easy for the algorithm to overfit the training data.
For the simplest case — 2D data, everything just clicks visually: Line of best fit is the line that minimizes the sum of squared residuals (SSR):
The formula is not that hard too:
But as the number of predictor variables (or dimensions) increases, the coefficients β_i also tend to get very large. With large coefficients, it is easy to predict nearly everything — you just take the relevant combination of individual slopes (βs) and you get the answer. That’s why it is common for linear regression models to overfit the training data.
Another problem with LR is that it does not care about the weights of the features. As long as there is a linear relationship between them, you get a model with OLS minimized. But in reality, you do want your model to care about the weight of each feature. For example, if you are trying to predict the number of new-born babies in a town, an obvious factor would be the number of fertile women.
However, the number of storks in that particular town would be completely irrelevant. But if these figures coincidentally represent a linear relationship and you use both of them for linear regression, you get a crappy model but still with OLS minimized.
And in practice, this is a real and common problem: today’s datasets contain many, many features both useful and useless to the target you are trying to predict.
You can run the notebook for this article here on Kaggle.
To understand how Ridge Regression solves the above two problems, we have to understand two other important topics: bias and variance.
Variance is always associated with a model failing to perform well on a new (test) set. High variance leads to overfitting which makes sense because overfit models generate vastly different, inconsistent results for different datasets.
In contrast, bias is the inability of the model to generalize on the training data. A model with too much bias fails to generalize both on training and test sets.
Ideally, the perfect model would have low bias and low variance but that is easier said than done. Because, bias and variance are in a trade-off relationship over model complexity.
Model complexity is determined by the number of dimensions input to the model, i. e. the number of features.
As linear regression is unbiased (does not care about predictors), it fits training data well. But the fit is too specific resulting in a model with very high variance putting Linear Regression at the far right of the Model Complexity Plot:
Both problems related to bias and overfitting can be elegantly solved using Ridge and Lasso Regression. The equation for the line of best fit is still the same for the new two regressions:
What is changed is the cost function. Both Ridge and Lasso Regression introduce a new hyperparameter to the OLS — λ. Let’s first look at the cost function of Ridge:
Apart from OLS (the first part), ridge regression squares every individual slope of the feature variables and scales them by some number λ. This is called the Ridge Regression penalty. What this penalty essentially does is shrink all coefficients (slopes). This shrinkage has a double effect:
We avoid overfitting with lower coefficients. As lambda is just some constant, it has the same scaling effect on all coefficients. For example, by choosing a low value for lambda like 0.1, we get to scale all coefficients both large and small.
This scaling also introduces some bias to the completely unbiased LR. Think about it this way — if you square a large number, you get an even larger number. By choosing a low value like 0.01 you get to shrink the number a lot. But if you square small numbers such as numbers lower than 1, instead of getting a larger number, you get an even smaller number. And multiplying it by 0.01, you manage to make it even smaller. This way, ridge regression gets to make important features more pronounced and shrink unimportant ones close to 0 which leads to a more simplified model.
You might be saying that the added sum of scaled, squared slopes will be bigger which does not fit the training data as well as the plain-old OLS. It is true but starting with a slightly worse fit, Ridge and Lasso provide better and more consistent predictions in the long run. By introducing a small amount of bias, we get a significant drop in variance.
Let’s see Ridge in action using Scikit-learn. Ridge follows the same API as any other model offered by sklearn. We will work on the Ames Housing Dataset from Kaggle. Using a subset of features, we will predict house prices using Ridge in this section and Lasso in the next one. To make things short, I am loading an already-processed dataset (feature selected, scaled, and imputed):
First, we will fit a LinearRegression model and compare its performance to Ridge using MAE (Mean Absolute Error). Before we do this, there are a few preprocessing required like feature scaling the variables and filling in missing values. So, I will create a simple Pipeline instance to deal with them:
If you want to know more about pipelines using sklearn check out this article or notebook on Kaggle.
Now, let’s first fit a Linear Regressor:
As you can see, the testing score is significantly lower than training’s which suggests overfitting. Now, let’s try Ridge:
We get almost identical results for Ridge because we chose too small a value for alpha. If you go back to the cost function of Ridge, you can see that if we set lambda to 0, we get plain-old OLS back:
Note that in sklearn API, the hyperparater lambda is given as alpha. Don't get confused.
Now, instead of blindly trying out a bunch of values for alpha, we can use cross-validation using RidgeCV. Instead of taking a single value for alpha, RidgeCV takes a list of possible alphas and tries them out using cross-validation, just like GridSearch:
We are passing alphas in the range of 1 and 100 with a step of 5 under a 10-fold cross-validation. After the fit is done, we can get the best alpha using .alpha_ attribute:
>>> ridge.alpha_86
Let’s finally evaluate a Ridge with that hyperparameter and compare it with Linear Regression:
Unfortunately, even with the best value for alpha, we get almost identical results to Linear Regression. (I realized this was a bad dataset for showing Ridge and Lasso much later while writing the article😶).
Lasso regression is also very similar to Ridge. There is only a minor difference in the cost function:
Instead of taking the square of each coefficient, Lasso regression takes the absolute values of them. The rest are practically the same. Let’s see the regressor in action on a new dataset. I will load in the built-in diamonds dataset from Seaborn:
diamonds = sns.load_dataset('diamonds')diamonds.head()
Using all of the features, we will predict the price using Lasso:
Lasso and LassoCV are imported in the same way. We will find the best value of alpha using CV:
It looks like choosing a very low alpha is good enough. Let’s now fit with that value and evaluate the performance:
Lasso regression did a fairly good job. One really cool thing about Lasso regression is that it can be used for feature selection. Because of how it works under the hood, Lasso regression has the ability to shrink unimportant parameters all the way down to zero. We can check this by plotting the above fitted lasso's coefficients using its coef_ attribute:
As you can see, except for carat which seems to be the deciding factor for the diamond price, all other coefficients are shrunk to close to 0:
That is the main difference between Ridge and Lasso. Lasso can shrink coefficients all the way to 0 while Ridge never can.
In conclusion, Ridge and Lasso are two of the most important regularization techniques out there. Both of them solve the problems of plain-old LR by introducing some bias leading to lower variance and thus avoiding overfitting. The hyperparameter lambda controls how severe the regularization is for features with different magnitudes.
Even though we covered a lot, we didn’t take a deep look into the hairy math that enables these two elegant algorithms. So, I am leaving a few references to sources that might help out:
Can’t recommend this enough: StatQuests on Ridge and Lasso
Math behind Linear, Ridge and Lasso Regression
The Math Behind Ridge Regularization and Lasso Regularization
Lasso Regression causes sparsity while Ridge Regression doesn’t! — Unfolding the math