Hyperparameters in Lasso and Ridge | Towards Data Science

In this post, we are first going to have a look at some common mistakes when it comes to Lasso and Ridge regressions, and then I’ll describe the steps I usually take to tune the hyperparameters. The code is in Python, and we are mostly relying on scikit-learn. The guide is mostly going to focus on Lasso examples, but the underlying theory is very similar for Ridge.
At first, I did not really see that there would be a need for another guide on this topic — it’s a pretty basic concept after all. However, when I recently wanted to confirm something, I realised that a good chunk of the guides out there are either very academic, too simple, or just plain wrong. One very common source of confusion is that there is always a dozen different ways to calculate the same thing in sklearn.
So without further ado here’s my 2 cents on the topic.
Lasso and Ridge are both regularising methods, they aim to regularise complex models by introducing penalty factors. They are great at reducing overfitting, handling multicollinearity, or just automatic feature engineering. This might seem counterintuitive, but by making the model work harder to explain the training data, we get a better understanding of the underlying structure, and thus better generalisation and better fits on the test data.
Following sklearn’s formulas, this is the expression that is minimised in the LinearRegression model, the so-called Ordinary Least Squares:
Where X matrix is the independent variables, w is the weight a.k.a. coefficients, and y is the dependent variable.
The Ridge regression takes this expression, and adds a penalty factor at the end for the squared coefficients:
Here, α is the regularisation parameter, this is what we are going to optimise. The model penalises large coefficients and tries to more evenly distribute the weights. In layman’s terms, this is what the Ridge model does:
X1, we see that you are doing a great job, and if not for the penalising factor, we would be putting a lot of weight on you. However, X2 is just slightly worse than you, and if we split the weights between the two of you, we will get a much lower penalty, hence a much better overall score.
Lasso does a similar thing but uses the sum of absolute values (the l1 norm) of the weights for the penalty.
Note: there is also a n_samples in the sklearn formula, which is the number of observations, which should not change for the same X and y. I have found no explanation as to why that is there, perhaps to compare different models, please let me know if you have a better idea.
Lasso will start decreasing the coefficients of variables that are not so important, and it is also possible to decrease coefficients down to 0. In layman’s terms:
X1, your minimal contribution to the overall score is noted. However, in light of the latest penalty scores, we will have to let you go from the regression. It’s just not worth keeping you around.
It’s worth noting that you can also combine the two penalties in the same model with an Elastic Net. You need to optimise two hyperparameters there. In this guide, we are not going to discuss this option.
If you want to follow the code, here is a list of all the libraries you will need:
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.metrics import \    r2_score, get_scorerfrom sklearn.linear_model import \    Lasso, Ridge, LassoCV,LinearRegressionfrom sklearn.preprocessing import \    StandardScaler, PolynomialFeaturesfrom sklearn.model_selection import \    KFold, RepeatedKFold, GridSearchCV, \    cross_validate, train_test_split
In this section, we are going to discuss a few general tips and common mistakes to avoid when it comes to regularised regressions. The examples use the Boston housing data, you can download it from Kaggle.
Processing the data:
column_names = \    ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',\     'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']data = pd.read_csv("../datasets/housing.csv", \    header=None, delimiter=r"\s+", names=column_names)y = data['MEDV']X = data.drop(['MEDV'], axis = 1)
As the title suggests: it’s absolutely a requirement to scale your variables for a regularised regression. (As we know, a linear transformation like scaling has no impact on the predictions of a vanilla linear regression.) It becomes pretty clear why you have to scale for a regularised regression if you have a closer look at the formulas: if one of your variables happens to be on a very small scale, its coefficient will be large, and as such, it will be much more punished due to the penalisation. Or vice versa, a variable that is on a huge scale will get small coefficients and be less impacted by the penalty. This is true for Lasso and Ridge.
Let’s say you do something like the following.
(Once again, the example is without scaling, will not yield correct results, don’t do this. Also, please note that there are other issues apart from the scaling, we will return to them shortly.)
# don't copy!!!cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)lasso_alphas = np.linspace(0, 0.2, 21)lasso = Lasso()grid = dict()grid['alpha'] = lasso_alphasgscv = GridSearchCV( \    lasso, grid, scoring='neg_mean_absolute_error', \    cv=cv, n_jobs=-1)results = gscv.fit(X, y)print('MAE: %.5f' % results.best_score_)print('Config: %s' % results.best_params_)
This will be the result:
MAE: -3.37896Config: {'alpha': 0.01}
However, if you scale your X variable beforehand, you’ll usually get a better score. To scale, we can use StandardScaler from sklearn. This method centres variables around 0 and makes the standard deviation equal to 1.
sc = StandardScaler()X_scaled = sc.fit_transform(X)X_scaled = pd.DataFrame(data = X_scaled, columns = X.columns)
If we replace X with X_scaled in the code block above, we get:
MAE: -3.35080Config: {'alpha': 0.08}
Yes, not much improvement, but that is due to a number of factors we will see later. Most importantly, the Boston housing data is a quite nice, tailored toy example for linear regression, so we can’t improve the predictions that much.
Summary: Use StandardScaler to scale independent variables before regularisation. No need to adjust the dependent variable.
If you pick 0 for the alpha parameter in either Lasso and Ridge, you are basically fitting a linear regression, because there is no penalty applied on the OLS part of the formula.
The sklearn documentation actually discourages running these models with an alpha = 0 argument due to computational complications. I have not met a case when it caused any computational problems, it has always given the same results as a LinearRegression model.
Summary: There is no point in picking alpha = 0, that is simply the linear regression.
In the examples above, we walked through an array of alphas, tried them all, and picked the one with the highest score. However, like always when you use a GridSearchCV, it is advisable to do multiple sweeps. Find the area where the highest alpha is, and then do a more granular check.
In my experience, especially with Lasso, it’s a common mistake to pick the lowest non-zero parameter, when in reality the optimal parameter is a much smaller number. See the example in the second half.
Note: Of course, we will never find the actual optimal number with a grid search method, but we can get close enough.
You can also visualise the results. This is how the unscaled version looks like:
For each alpha, GridSearchCV fit a model, and we picked the alpha where the validation data score (as in, the average score of the test folds in the RepeatedKFold) was the highest. In this example, you can see that there is probably not a crazy spike between 0 and 0.01. It’s still incorrect of course because we did not scale.
This is the plot of the scaled version:
Once again, it looks OK, probably nothing weird going on between 0.07 and 0.09.
Summary: Plots are your friend, observe the alpha curve. Make sure that the alpha you picked is in a nice “curvy” area.
You might be tempted to calculate in a different way to check your results. As mentioned earlier, sklearn usually has a bunch of different ways to calculate the same thing. For one, there is a LassoCV method that combines Lasso and GridSearchCV in one.
You might try something like this to get the best alpha (not going to use the not scaled version anymore in examples):
lasso = LassoCV(alphas=lasso_alphas, cv=cv, n_jobs=-1)lasso.fit(X_scaled, y)print('alpha: %.2f' % lasso.alpha_)
This will return:
alpha: 0.03
Wait, wasn’t this alpha for the same data 0.08 above? Yes. And the reason for the difference? LassoCV uses an R2 score, and you cannot change it, while earlier, we specified MAE (well, minus MAE, but that’s just so we can maximise and remain consistent) in our GridSearchCV object. This is what we had in the code I warned you not to copy:
scoring='neg_mean_absolute_error'
See, the problem is, sklearn has dozens of scoring methods, have a look at the list. You can certainly pick let’s say the max_error to measure your model’s performance as well. However, the model optimises for the squared differences. I think using anything derived from the squared differences is much more consistent. After all, LassoCV uses R2, so maybe that’s a sign?
This whole “optimising on one basis but then comparing performance on another” is actually quite visible on the plots above. Notice how the green line increases for a while. That is the training score. Under normal circumstances, it should not perform better after we impose a penalty factor.
Normally, this is the shape of the curves you would see. Training data score decreases immediately, validation data score increases for a while and then decreases:
Summary: Use R2 or another squared difference-based model as the primary scoring for a regression.
In this section, I’m going to walk us through the method I use to prepare the data and fit regularisation regressions.
I’m not going to detail the data up until getting X and y. I’m using a version from this U.S. County National Health Rankings data collection to generate the results below, but it really doesn’t matter for this example.
So, assuming you have a nice and clean X and y, the next step is to set aside a test dataset, using the handy train_test_split. Pick any number for my_random_state, if you want to make the results reproducible.
X_train , X_test, y_train, y_test = train_test_split(    X, y, test_size=1000, random_state=my_random_state)
The next step is to include the PolynomialFeatures. We save the results in the poly object, which is important, we’ll use it later.
poly = PolynomialFeatures(    degree = 2, include_bias = False, interaction_only = False)
This will produce all the 2nd-degree polynomial combinations of the variables. It’s important to note that we set include_bias to False. That is because we do not need an intercept column, the regression model itself will include one.
This is how we can transform and rename X. It assumes that you kept X in a pandas DataFrame, and it takes some tweaking to keep the column names usable. If you don’t want the names, you only need the first line.
X_train_poly = poly.fit_transform(X_train)polynomial_column_names = \    poly.get_feature_names(input_features = X_train.columns)X_train_poly = \    pd.DataFrame(data = X_train_poly,         columns = polynomial_column_names )X_train_poly.columns = X_train_poly.columns.str.replace(' ', '_')X_train_poly.columns = X_train_poly.columns.str.replace('^', '_')
After this is done, the next step is to scale. This is even more important now that we introduced the polynomials, and the magnitude will be all over the place without scaling.
sc = StandardScaler()X_train_poly_scaled = sc.fit_transform(X_train_poly)X_train_poly_scaled = pd.DataFrame( \        data = X_train_poly_scaled, columns = X_train_poly.columns)
Here comes the tricky part. If we want to use the test dataset, we need to apply the same steps.
However, we do not need to fit the objects again. Well, it doesn’t matter with poly, but for sc, we want to keep the method we used to fit X_train_poly. And yes, that means that the test data will not be perfectly standardised, and that is fine. So instead of fit_transform, we use transform.
X_test_poly = poly.transform(X_test)X_test_poly_scaled = sc.transform(X_test_poly)
You might have wondered how to generate the plots we used above. I’m using two functions, building on the libraries listed way above. The first function draws a plot:
And the second one is essentially a grid search, with some additional stuff: it also runs test scores and saves the plot of course.
I would not really like to go into details here, I think it’s pretty self-explanatory, and we will see examples on how to call it later.
There’s one thing, which I think is pretty cool: sklearn has a get_scorer function that returns a scorer object based on its sklearn string code. E.g.:
scorer = get_scorer('r2')scorer(model, X_test, y_test)
Now we have one additional way to calculate the same thing.
Once the process is set up like that, all we need to do is run the functions for different alpha arrays.
An interesting point of the process is that we are also plotting the test scores:
take the training datasets and an alpha;do cross-validation, save the training and validation scores;pretend that that is the alpha we choose and fit a model without cross-validation on the whole training data;calculate the score this model would achieve on the test data, and save the test score.
take the training datasets and an alpha;
do cross-validation, save the training and validation scores;
pretend that that is the alpha we choose and fit a model without cross-validation on the whole training data;
calculate the score this model would achieve on the test data, and save the test score.
This is not something you would do in “real life” (unless you do a Kaggle competition) because now there is the possibility of optimising for your test dataset. We only include it here to illustrate the model performance. The red line is going to be the test score on different alphas.
We will also need a cross-validation object, there is no one good answer here, this is an option:
cv = KFold(n_splits=5, shuffle=True, random_state=my_random_state)
To illustrate my point on the importance of multiple-step parameter search, let’s say we want to check these alphas:
lasso_alphas = np.linspace(0, 0.02, 11)
After we run the function:
chosen_alpha, max_validation_score, test_score_at_chosen_alpha = \    regmodel_param_test(        lasso_alphas, X_train_poly_scaled, y_train,         cv, scoring = 'r2', model_name = 'LASSO',         X_test = X_test_poly_scaled, y_test = y_test,         draw_plot = True, filename = 'lasso_wide_search')print("Chosen alpha: %.5f" % \    chosen_alpha)print("Validation score: %.5f" % \    max_validation_score)print("Test score at chosen alpha: %.5f" % \    test_score_at_chosen_alpha)
The results:
Chosen alpha: 0.00200Validation score: 0.82310Test score at chosen alpha: 0.80673
Does that mean we found the optimal alpha? You might look at the plot, see the nice spiky spike, decide that that is high enough. Well, not quite. If we run it on a more granular level:
lasso_alphas = np.linspace(0, 0.002, 11)
This is the result, please note that 0.02, the rightmost point is where we had the spike in the previous chart:
Chosen alpha: 0.00060Validation score: 0.83483Test score at chosen alpha: 0.82326
Had we stopped earlier, we would have picked an alpha that results in a 2% lower overall test R2, and I think that is quite material.
Right, the title of the post includes Ridge, and we haven’t discussed it yet, apart from the theory introduction.
The reason is quite simple: it works exactly like Lasso, you just probably want to pick different alpha parameters, and you pass on ‘Ridge’ in the model_name argument. The same problem with Ridge (I’m not including the part where we search for the alphas range to examine):
You will notice that the point we pick based on the blue line no longer seems to be the best for the red line. Yes, that is true, but I think it was a coincidence that it worked out so well for the Lasso model.
So there you have it, that’s how I do hyperparameter tuning for Lasso and Ridge. I hope you found it helpful, the main points again:
remember to scale your variables;
alpha = 0 is just the linear regression;
do multiple steps when searching for the best parameter;
use a squared difference based score to measure performance.