Preventing Overfitting with Lasso, Ridge and Elastic-net Regularization in Machine Learning | by Susan Maina | Towards Data Science

Tackling the limitations of linear regression using L1 and L2 regularization methodologies to explore bias-variance tradeoff in Machine learning
Linear Regression models are very popular because they are easy to understand and interpret. However, in practice, linear models cannot effectively express non-linear relationships and are very prone to overfitting when we have many features, especially if the features are not useful predictors for the target feature.
Let us first investigate a simple linear regression model, which is a model where there is only one input feature x and target feature y. The complete workflow is here on Github.
We will use a simulated dataset similar to the one I used in this article on model complexity. The simulation process of the dataset is available here on my Github page. The dataset contains 2 columns, x and y, each with 100 values, which when plotted reveal a cosine curve pattern.
Below is the true cosine curve.
After importing the numpy, pandas, matplotlib and seaborn libraries, we read in our dataframe.
df = pd.read_csv(‘cosine_df.csv’)df.head()
Next, we will train a linear regression model and predict the outcomes of our cosine dataset. Remember that the underlying pattern is a curve, and a simple linear regression model fits a straight line through the data.
To automate the model-training process, I created a function that takes in the data to be fitted and an instance of a model, separates the dataset into target and features, fits the model, makes predictions on the training data and plots the predictions. It then returns the trained model.
def model_fitter(data, model):    features = data.drop('y', axis = 1)    target = data['y']    model.fit(features, target)    preds = model.predict(features)    plt.scatter(data.x, data.y)    plt.plot(data.x, preds, 'k--')    plt.show()    return model
Now, let us use our model_fitter function to train a linear model on our data, but first, we import the library from sci-kit learn.
from sklearn.linear_model import LinearRegressionlm = model_fitter(df, LinearRegression())
The result is the expected straight linear regression line, which translates to an underfit model that does not represent the curve of our dataset.
Let us display the estimated y_intercept (β0) and the coefficient of input feature x (β1)
print(lm.intercept_)print(lm.coef_)### Results-0.09248137706996012[0.02712842]
A way to increase the complexity of a linear regression model is by increasing the input features and this will lead to overfitting as shown below.
Here, we will use a modified version of our simulated dataset created in the same notebook here on Github. In addition to our 2 columns (x and y), there are 99 other columns (s1 to s99) that contain random values with no predictive influence and in no way related to our 2 important features, x and y.
Using panda’s read_csv function, we will call our dataframe suspect_df and display the first 5 rows.
suspect_df = pd.read_csv('cosine_df_extra_columns.csv')suspect_df.head()
This time we will fit a multiple linear regression because there are multiple input features. More about that here. We follow the same steps for the linear regression model created above and save the model as lm_s.
lm_s = model_fitter(suspect_df, LinearRegression())
Wow! Our model is completely overfit. With the 100 input features and 100 observations(rows), there are enough coefficient combinations for the model to memorize every individual observation creating a perfectly overfit model. This model would perform poorly on new unseen data.
Note: It is common in real-world datasets for analysts to collect and retain a lot of features just to be on the safe side, or to develop many features during feature engineering. These features may lead to overfitting if they are useless as predictors.
Below are the coefficients for the 100 features (including x).
print(lm_s.coef_)
The task of a linear regression model is to reduce the cost function and come up with the best estimates of β0 and β1 coefficients that represent the true patterns in the data.
Regularization, in the context of linear regression, is the technique of penalizing these model coefficients, consequently reducing overfitting. This is by adding a penalty factor to the cost function (cost function + penalty on coefficients) minimizing both the cost function and the penalty. The lambda value, or λ, controls how much we minimize the penalty factor and controls the degree of fit of the model.
There are 2 types of regularization; L1 and L2. Each is penalized differently leading to different behaviors.
L1 regularization. This approach penalizes the absolute size of the coefficients.
L2 regularization. This approach penalizes the squared size of the coefficients.
From the formulas above, we can see how the penalties change the cost function.
Regularization results in the coefficients being shrunk or in some cases reduced to zero removing that feature entirely and its influence the model, taking care of multicollinearity issues. Features with higher coefficients have a greater influence on the target variable. The penalty function can be tuned by adjusting the λ constant.
The regularization algorithms we will describe in this post differ in the amount of l1 or l2 penalties applied to the cost function.
LASSO regression
LASSO regression
LASSO stands for Least Absolute Shrinkage and Selection Operator. This completely relies on the L1 penalty, which can reduce the coefficients’ sizes so small that they can get to 0, leading to automatic feature selection (features with a 0 coefficient do not influence a model).
Since λ (the “strength” of the penalty) can and should be tuned, a stronger(larger) penalty leads to more coefficients pushed to zero.
We will use sci-kit learn’s Lasso class. We will run our model_fitter function to instantiate a Lasso model, fit it, plot the predictions and return the model. There is some randomization involved therefore we pass a random_state value to the model instance.
from sklearn.linear_model import Lassolasso = model_fitter(suspect_df, Lasso(random_state=2021))
The resulting model is an underfit horizontal regression line at the mean of the target feature, or the mean model. Let us display the model coefficients.
print(lasso.coef_)
We see that all the coefficients were forced down to zero, including the predictive x feature.
A solution to this problem is to tune the penalty strength λ and in sci-kit learn, this parameter is called alpha. The default value is alpha=1.0. We should therefore reduce it because as explained earlier, higher λ values lead to more coefficients forced to 0, and we want fewer coefficients as zero.
Let us use alpha=0.1 and observe the results.
lasso_point1 = model_fitter(suspect_df, Lasso(random_state=1234, alpha=0.1))
Now the model is tending towards overfitting. It seems that a good alpha value is between 0.1 and 1.0. But first display the coefficients for a lower alpha (0.1).
print(lasso_01.coef_)
We now see that about two-thirds of the coefficients are 0. The model has performed feature selection. In Lasso, different coefficients reach zero at different rates and according to the model, the last features to be zeroed out are the most important ones.
We can run a for-loop to investigate the effect of different alpha values between 0.1 and 1.0 and determine the best model.
values = [0.1, 0.3, 0.5, 1.0]for n in values:    plt.title('Alpha value: {}'.format(n))    model_fitter(suspect_df, Lasso(random_state=2021, alpha = n))
We see that the model’s complexity is reducing as the alpha values are increasing. Note that the intercept is not affected by the lambda (alpha) hyper-parameter.
When the alpha value approaches 0, the model is a linear regression model (refer to our multiple linear regression model that was overfit above). As alpha increases, the variance decreases while the bias increases, and the model becomes the global mean.
2. Ridge Regression
Ridge (not an acronym) completely relies on the L2 penalty which leads to coefficients pushed closer to zero, but not completely zero leading to feature shrinkage.
A bigger λ, or penalty strength, leads to more coefficients approaching zero.
We first import the Ridge class from sklearn.linear_model. Then we call our function model_fitter passing in the suspect_df and an instance of the Ridge regression model. The default alpha value is alpha=1.0.
from sklearn.linear_model import Ridgeridge = model_fitter(suspect_df, Ridge(random_state=2021))
We immediately note that the model is overfit. Let us look at the model coefficients.
print(ridge.coef_)
The coefficients seem rather high when using Ridge’s default. Therefore, we need to shrink the coefficients by increasing the alpha value, reducing the effect of the noisy columns on the model.
Let us use a for-loop to investigate a number of alpha values on different Ridge regression models. The alpha values are 1, 1000, 3000, 5000 and 50000.
values = [1, 1000, 3000, 5000, 50000]for n in values:    plt.title('Alpha value: {}'.format(n))    model_fitter(suspect_df, Ridge(random_state=2021, alpha = n))
We note that the model bias increases as the alpha value increases. At the highest alpha value of 50000, the regression line is the mean (average) of the target feature, or the mean model.
print(df.y.mean())###Results-0.007254917297325034
Let us check the coefficients at alpha=30000.
ridge_30000 = model_fitter(suspect_df, Ridge(random_state=2021, alpha = 30000))print(ridge_30000.coef_)
Take note of the high negative powers in the values. As the alpha value approaches infinity, more coefficients are approaching zero. This explains the feature shrinkage aspect discussed earlier.
3. Elastic-Net regression
The Elastic-net regression model is a compromise between Lasso and ridge. It combines both Lasso and ridge penalties by applying a ratio. In sci-kit learn, this ratio is called l1_ratio and it defines what proportion of L1 penalty is to be applied to the model. The l1_ratio together with the alpha (or strength of the penalty) should be tuned.
Ridge and Lasso regression are special cases of Elastic-Nets if you set the ratio to completely favor one of the penalty types over the other.
Note: Elastic-net regression should be used if you wish to combine the Lasso and Ridge functionalities, which usually results to better performance when tuned properly. In sci-kit learn, you can set the ratio (l1_ratio) to either 1 (Lasso) or 0 (Ridge), but sci-kit’s implementation of elastic-net, in this case, is not the best. In the situation where the dataset favors either Ridge or Lasso regression, you should use the separate Ridge or Lasso classes for optimal results. Also, they are simpler to tune.
To implement an Elastic-net regression model, we need to first import the ElasticNet class from sklearn.linear_model. We will then call our model_fitter function and pass in the suspect_df dataframe and an instance of ElastcNet model. The default values are alpha=1.0 and l1_ratio=0.5
from sklearn.linear_model import ElasticNetenet = model_fitter(suspect_df, ElasticNet(random_state=2021))
The model appears to be the underfit mean model, similar to the Lasso(alpha=1.0). Let us display the coefficients.
print(enet.coef_)
Again, just like the Lasso, the coefficients have been forced to zero. Similar results are observed for any l1_ratio greater than 0.5.
I then trained a model where l1_ratio=0.1. This ratio is close to zero meaning the L2 penalty of Ridge regression is favored. Below is the model.
enet_point1 = model_fitter(suspect_df, ElasticNet(random_state=2021, l1_ratio = 0.1))
print(enet_point1.coef_)
We observe something similar to the Lasso regression with a low alpha value with the model applying some feature engineering. This is because an l1_ratio of 0.1 means that the model is still applying some L1 penalty which causes some coefficients to 0. However, further decreasing the l1_ratio towards 0 (eg 0.01,0.001, etc) applies more of the L2 penalty resulting in non-zero coefficients.
Differences between L1 and L2 penalties
The L1 norm is more robust to outliers but is unstable for different lambda values as the model changes significantly with different lambda values. L2 is better if the outliers are important because the squaring of the coefficients leads to an exponential increase in cost for these features. Also, L1 has a built-in feature selection by shrinking the coefficients to exactly 0, whereas L2 shrinks them to near 0 resulting in lower sparsity.
This has been a basic introduction of Regularization techniques for reducing overfitting in Linear Regression for machine learning. We used Lasso, Ridge, and Elastic-net models which apply either the L1 or L2 penalties to the cost functions to reduce the magnitude of the coefficients, reducing the model variance.
When tuning the hyperparameters such as the alpha and the l1_ratio to get the best model, it is practical to have a separate training set and a test set to train the model on and test the efficiency of the trained model respectively. You can employ cross-validation mechanisms for hyperparameter optimization and select the best-performing model, keeping in mind the bias-variance trade-off.