Don’t Over Complicate using Regularization
By: Richard Mei
Tags: Lasso Regression, Ridge Regression, Python, Regression

Have you ever done regression where you have low training error but and extremely high test error? Well the solution there is to use regularization!

When you have a over-fit regression model, you may end up having a lot of coefficients for each variable meaning a complex model. This will do very well on your training data of course, but once its introduced to new data (like in the test) then the performance would go down dramatically. This results in a high variance in your results and thereby not as a great as a model. Regularization will help reduce the complexity of the model with one of two ways. The first is LASSO (L1) and the second is RIDGE (L2). The only difference between these two methods are how exactly they deal with penalizing a coefficient inside the cost function.

( LASSO = Least Absolute Shrinkage and Selection Operator)

Looking at the chart above, can you spot the difference between the ridge and lasso cost functions? On the right there is a w-squared term and and absolute-w term. So, LASSO’s cost function has a term that penalizes by the absolute value of the coefficients while Ridge has the square of the coefficients. The entire right side of both equations is the penalty term. It’s known as the lambda. So depending on the lambda term you decide to use, the type of regularization will matter! Moreover, this means that lambda is a hyperparameter.

So to summarize the breakdown all in one line, there are two regularization methods dependent on which penalty your occur in your regularization, LASSO or Ridge. LASSO has a absolute value penalization term while Ridge has a square penalization term.

More in-depth

The the penalty terms results in LASSO to have the ability to drive coefficients to actually zero, so that means LASSO would be more effective for higher dimensional data. This leads to simultaneously doing variable selection as well! For the penalty term in ridge, the coefficients only tend to go towards zero. So if you have a few features/lower dimensions then this may potentially work better.

When using regularization, it’s important to have your data scaled. For example, if you are comparing the number of rooms vs price of a house, then the unscaled coefficients would be either small or very large as per magnitude of the original variable. This means the large coefficient would be regularized more harshly than the other solely because it was a bigger number. More over here’s some potential effects of regularization:

Reduce Model Complexity

Prevent overfitting

Variable Selection

Counter multi-collinearity

Conclusion

Overall, that was a quick glimpse of regularization without too much math. It’s an important concept to know that isn’t too difficult to understand, but it is much harder to implement. For example, when you use regularization, how do you know which regularization to use? How do you pick the right lambda value? This boils down to picking the better model and there is no exact right answer since it really depends on the problem at hand. At the end, regularization is a strategy that you can use. If you want to give it a shot in Python you can use the SKLearn linear model library.