Regularization — A Technique Used to Prevent Over-fitting
By: Kiprono Elijah Koech
Tags: Python, Data Science, Regression, Machine Learning

Lets consider another approach for explaining regularizers through visualizations.

So, suppose that we have a regression problem with only two features then the regularized cost function will be

Equation 8: Ridge and Lasso Regression for two-features problem.

Without loss of generality, we can impose a constraint on the regularization term, such that, β₁²+β₂²<=s for Ridge Regression and |β₁|+|β₂|<=s for some non-negative value s.

L₁ and L₂ regularizers are the common choices for regularization but in the real sense there are many different possible choices of regularization with different effect on the fitted coefficients.

The regularizers are constructed from the family of Lₚ vector norms, that is,

Equation 8: The Lₚ norm. For p=1, we have rectilinear/Manhattan distance and for p=2, we have the famous Euclidean norm.

Figure 3: Different regularization functions (L_p norms)

In Figure 3 and Equation 8: for p=1, we have L₁ regularization (Lasso Regression) and for p = 2 we have L₂ regulation (Ridge Regression). We can easily visualize these norms by sketching the isosurfaces (Figure 3) for different values of p and we can “conveniently” choose s=1 and and sketch the isosurfaces as follows.

Figure 4: Inspired by An Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani (https://doi.org/10.1007/978-1-4614-7138-7)

For p = 1 the constraint will be |β₁|+|β₂|<=s and the isosurface is a rotated square. We can interpret this to mean that increase in value of one parameter β then there must be exact offset by decrease in the other.

For p=2, we will impose the constraint |β₁²+β₂²|<=s and the result is a unit circle (of course, letting s=1).

If p is large (for example, at p=4 as shown in the figure), the isosurface tends to approach a square and for small p (say, p =0.5 as shown in the figure) the shape is very peaked where we are allowed to have large values of one parameter if and only if the value of the other parameter is tending to zero.

The most common regularizers: L₁ and L₂ norms acts on the cost function (unregularized) and therefore in essence, we can see this kind of regularization as a sort of balance between two different losses (unregularized cost and the norm, the regularization term).

The unregularized term determines how well the model fits the data before regularization and it is minimized at the β_hat point. The regularization term, on the other hand, is minimized at the origin where all the parameters are equals to zero.

It increases, in the case of the l₂ norm quadratically away so the combination of the data plus the regularization will be minimized at some point that touches both surfaces (the circle shape and the contors) since in order to reduce the regularization we would have to leave this isosurface and increase the data loss and vice-versa.

L₁, on the other hand, encourages sparse parameter vectors because the contours can touch the rotated square exactly at the axis meaning that one of the coefficient will be zero and consequently the corresponding feature will be eliminated. In this way, L₁ works as a form of feature selection since less important features can be forced to be zero.