The game of Regularization. An intuitive explanation of the…
By: Arpan Das
Tags: Artificial Intelligence, Machine Learning, Data Science, AI, Tutorial

The game of Regularization

An intuitive explanation of the different regularization techniques used in machine learning

In machine learning regularization is a method to solve over-fitting problem by adding a penalty term with the cost function. Let’s first understand,

what is over-fitting and why it is a problem.

While solving a machine learning problem, we first train our model with the training data and finally generalize the solution (tests how well the concepts learned by a machine learning model apply to specific data points not seen by the model when it was learning). If our model is too complex it results in

higher test error though the training error is much lower. But a natural question is why? What do you mean by a complex model? Let’s take an example.

Suppose we are solving a simple binary classification problem with a logistic regression model. Let’s red points are class A and blue points are class B. Now

source : wikepedia

if we try to fit our decision boundary perfectly, there is also a chance that we will also fit the noisy points(potential outliers).As a result the model will not learn proper pattern about the data(and for smaller datasets it will just memorize all the points), which may leads to higher generalization(read test) error.

To avoid over-fitting in models we add a regularization term, that will help to reduce the generalization error.Let’s list down the various options we have:

L1 regularization: If we look at the cost function, l1 regularizer is the sum of absolute values of coefficients. Simply it is added to maintain the value

cost function with L1 regularization

of cost function within a range. The main advantage of using l1 regularization is it creates sparsity in the solution(most of the coefficients of the solution are zero), which means the less important features or noise terms will be zero. It makes l1 regularization robust to outliers.

2. L2 regularization: In case of l2 regularization we add squared values of

cost function with L2 regularization

coefficients instead of absolute values. In both of the methods lambda is a hyperparameter, which is used to tune the regularization. The main difference between l1 and l2 in term of result is, for less important features the coefficients with l2 term will be very low but never be exactly zero.

3. Elastic Net: When both l1 and l2 regularization terms are used together as penalty, it is called elastic net regularization.

That’s cool. But How to choose the best regularization for me? That mostly depends on the problem at hand. We can always take the help of cross validation to select better performing one for the particular problem. For large dimensional data l1 provides better feature interpretability by giving zero coefficients for less important features.

End Notes:

In this article we discuss what is over-fitting and how we can use regularization to minimize the generalization error. Across the broad spectrum of machine learning, you will find it is nothing but a game of bias variance trade off, the well fitted model we have the better result we get. Hope this article will help you to get the basics right. Happy Learning!

References:

In case you want to dive deep,please follow these links below