Regularization in Machine Learning: Connect the dots
By: Vamsi Chekka
Tags: Artificial Intelligence, Bias Variance Tradeoff, Linear Regression, Regularization, Machine Lear

Breakdown

Following are the various steps we will walk together and try gaining an understanding.

Context

2. Prerequisites

3. Problem of Overfitting

4. Objective

5. What is Regularization?

6. L2 Norm or Ridge Regularization

7. L1 Norm or Lasso Regularization

8. Lasso (L1) vs Ridge (L2) Regularizations

9. Conclusion

1. Context

In this post, we will consider Linear Regression as the algorithm where the target variable ‘y’ will be explained by 2 features ‘x1’ and ‘x2’ whose coefficients are β1 and β2.

2. Prerequisites

First up, lets get some minor prerequisites out of the way in order to understand their use down the line.

Linear Regression

Optional: Refer Chapter 3 in the link below to gain understanding about Linear Regression.

Cost Function of Ordinary Least Squares (OLS) Regression

N — number of samples

p — number of independent variables or features

x — feature

y — actual target or dependent variable

f(x) — estimated target

β — coefficient or weight corresponding to each feature or independent var.

Gradient Descent represented as Contour plot

In Fig 1(a) below, Gradient Descent is represented in 3-dim. J (β) represents the error for corresponding β1 and β2. Red color represents the zones where error is high and blue color represents the zones where error is minimum. Using Gradient Descent, the coefficients β1 and β2 would be identified where the error is minimum.

Fig 1: Gradient Descent Projection as Contour

In Fig 1(b) above, the corresponding error from Cost Function of Gradient Descent is projected on to 2-dim with corresponding colors.

Fig 2: Gradient Descent on axes of β1 and β2

In Fig 2 on the left, Gradient Descent Contour for the Simple Linear Regression is represented in 2-dimensional format on the co-ordinate system of coefficients β1 and β2.

Lets keep this understanding aside for a while and move on to address few others basics.

3. Problem of Overfitting

Assuming we have trained a model using Linear Regression on Training Set and increased the performance to satisfactory levels, our next step is to validate against the Test Set / unseen data. In most cases, there will be a drop in the accuracy levels. What has happened?

In our endeavor to increase model accuracy during training, our model would try to fit and learn as many data points as possible. In other words as goodness of fit increases, model moves from linear to quadratic to polynomial (i.e. Model Complexity is growing). This is due to Overfitting. Following image from Andrew Ng’s Machine Learning course helps understand this visually.

Higher degree model and large coefficients increase the variance significantly leading to Overfitting.

4. Objective :

Our model needs to be robust to predict well in real world i.e. out of sample data or the data it has not seen during training. The overfitting during training is what prevents it from being robust. In order to avoid this overfitting scenario and increase the model robustness, we try to

1. Shrink the coefficients or weights of features in the model

2. Getting rid of high degree polynomial features from the model

How do we achieve this? In comes Regularization. Let’s explore it now.

5. What is Regularization?

Suppose say, we have trained our Linear Regression model and identified the coefficients β1 and β2 at the global minimum. In order to meet our objective; if we try shrink the coefficients of features that were learnt earlier, the model loses accuracy. This loss in accuracy needs to be explained by something else to maintain the accuracy levels. This responsibility will be taken up by the Bias portion of model equation (Bias: part of model equation that is not dependent on the feature data).

The Bias portion will be tweaked in the following ways:

1. Variance needs to be reduced and Bias needs to be increased: Since Variance is a function of coefficients β1, β2; Bias will also be tagged to them and changed as a function of coefficients β1, β2. The reason why L1 and L2 Norms of coefficients β1, β2 are used.

2. Generalization: This function of coefficients β1, β2 needs to be generalized and work in the entire co-ordinate space (positive and negative values on axes). That is the reason we find modulus operator in L1 and L2 Norm equations.

3. Regularization Parameter ‘λ’: Since both Variance and Bias are functions of coefficients β1, β2, they will be directly proportional. This will not work. Hence we need an additional parameter that can regulate the size of Bias term. This regulator is the Regularization parameter ‘λ’

4. ‘λ’ is a Hyper-parameter: If ‘λ’ was a parameter, Gradient Descent would nicely set it to 0 and travel to the global minimum. Hence, the control on ‘λ’ cannot be given to Gradient Descent and needs to be kept out. It will not be a parameter but a Hyper-parameter.

After understanding the changes to Bias term, lets move in to some functions of coefficients β1, β2 that will be used for Regularization.

6. L2 Norm or Ridge Regression

L2 Norm is Euclidean distance norm of the form |β1|² + |β2|².

Modified Cost function with L2 Regularization is as follows:

Let’s assume λ = 1 and is out of picture for a while

β1 and β2 can vary in the process of finding the global minimum error. With the regularization term (highlighted in blue in above equation); a particular value of (β1,β2) will generate a bias output β1² + β2². There can be multiple values that can generate the same bias like (1,0),(-1,0), (0,-1), (0.7,0.7) and (0,1). In case of L2 Norm, the various combinations of β1 and β2 that generate a particular same Bias, form a circle. For our example, let’s consider a bias term = 1. Fig 6(a) represents this circle.

Fig 6: L2 Norm

Fig 6(b) indicates the Gradient Descent Contour plot of Linear Regression problem. Now, there are 2 forces at work here.

Force 1: Bias term pulling β1 and β2 to lie somewhere on the black circle only.

Force 2: Gradient Descent trying to travel to the global minimum indicated by green dot.

Both the forces pull and finally settle near the point of intersection indicated by ‘Red cross’.

Remember our assumptions for Fig 6(b) until now were Bias term = 1 and λ = 1. Now get rid of one assumption related the Bias term.

Imagine the ‘Black circle’ changing in size and Gradient Descent settling down at a different point where cost is minimum. The assumption for λ = 1 is still in place. This minimum cost observed is for λ = 1

Next get rid of the other assumption λ = 1. Provide a different value say λ = 0.5.

Repeat the Gradient Descent process for λ = 0.5 and check the cost. This process can be repeated for various values of λ in order to identify the optimum value of λ whose cost function gives the least error.

The overall objective is always to have low cost after the Gradient Descent. Hence the values of λ, β1,β2 are identified keeping that as objective.

By the end of this process, the Variance would have been reduced i.e. coefficients are reduced in magnitude. Let’s move over to another type of function of β1,β2.

7. L1 Norm or Lasso Regression

L1 Norm is of the form |β1| + |β2|.

Modified Cost function for L1 Regularization is as follows:

Fig 7(a) indicates the shape of L1 Norm.

Fig 7: L1 Norm

Fig 7(b) indicates the L1 Norm with the Gradient Descent Contour plot. From Variance reduction standpoint, the same logic discussed in previous section is valid here as well.

Let’s move over to another important aspect of Lasso Regularization that we will discuss in next section.

8. Lasso (L1) vs Ridge (L2) Regularization

Fig 8(a) shows the area of L1 and L2 Norms together. For same amount of Bias term generated, the area occupied by L1 Norm is small. But L1 Norm doesn’t concede any space close to the axes. This is what causes the point of intersection between the L1 Norm and Gradient Descent Contour to converge near the axes leading to feature selection.

Fig 8: L1 vs L2 Norms

Fig 8(b) indicates the L1 and L2 Norms along with Gradient Descent Contours of different Linear Regression problems. Except for one case, L1 Norm converges on or very close to the axes and hence removing feature from the model.

Each quadrant in here has a Gradient Descent contour for a different Linear Regression problem. Green, Blue, Brown colors indicate they are related to different Linear Regression problems. The RED circle in each contour intersects the Ridge or L2 Norm. The BLACK circle in each contour intersects the Lasso or L1 Norm.

This indicates that L1 Norm or Lasso Regularization acts as Feature Selector along with reducing Variance.

9. Conclusion

Variance Reduction is synonymous with Bias Variance Trade-off.

Credits: