LASSO Regression Tutorial. Bias-variance evaluation using LASSO... | by Benjamin Obi Tayo Ph.D. | Towards Data Science

LASSO regression is an example of regularized regression. Regularization is one approach to tackle the problem of overfitting by adding additional information, and thereby shrinking the parameter values of the model to induce a penalty against complexity. The 3 most popular approaches to regularized linear regression are the so-called Ridge Regression, Least Absolute Shrinkage and Selection Operator (LASSO) and Elastic Net method.
In this tutorial, I’ll focus on LASSO, but an extension to Ridge and Elastic Net is straightforward.
Suppose we would like to build a regularized regression model on a dataset with n observations and m features.
LASSO regression is an L1 penalized model where we simply add the L1 norm of the weights to our least-squares cost function:
where
By increasing the value of the hyperparameter alpha, we increase the regularization strength and shrink the weights of our model. Please note that we don’t regularize the intercept term w0. Note also that alpha = 0 corresponds to standard regression analysis.
Depending on the regularization strength, certain weights can become zero, which makes the LASSO method a very powerful technique for dimensionality reduction.
1) For given alpha, simply minimize the cost function to find the  weights or model parameters w.2) Then compute the norm of w (excluding w0) using the equation below:
We shall illustrate the LASSO technique using the cruise ship dataset cruise_ship_info.csv.
import numpy as npimport pandas as pdimport matplotlib.pyplot as plt
df = pd.read_csv("cruise_ship_info.csv")df.head()
In another article (Feature Selection and Dimensionality Reduction Using Covariance Matrix Plot), we saw that a covariance matrix plot can be used for feature selection and dimensionality reduction. Using the cruise ship dataset cruise_ship_info.csv, we found that out of the 6 predictor features [‘age’, ‘tonnage’, ‘passengers’, ‘length’, ‘cabins’, ‘passenger_density’], if we assume important features have a correlation coefficient of 0.6 or greater with the target variable, then the target variable “crew” correlates strongly with 4 predictor variables: “tonnage”, “passengers”, “length, and “cabins”. We, therefore, were able to reduce the dimension of our feature space from 6 to 4.
cols_selected = ['Tonnage', 'passengers', 'length', 'cabins','crew']df[cols_selected].head()
X = df[cols_selected].iloc[:,0:4].values    # features matrix y = df[cols_selected]['crew'].values        # target variable
a. Split dataset into training and testing sets
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split( X, y,                                       test_size=0.4, random_state=0)
b. Standardize features
from sklearn.preprocessing import StandardScalersc_y = StandardScaler()sc_x = StandardScaler()y_std = sc_y.fit_transform(y_train[:, np.newaxis]).flatten()X_train_std = sc_x.fit_transform(X_train)X_test_std = sc_x.transform(X_test)y_train_std = sc_y.fit_transform(y_train[:, np.newaxis]).flatten()
c. Implement LASSO regression
from sklearn.linear_model import Lassofrom sklearn.metrics import r2_scorealpha = np.linspace(0.01,0.4,10)r2_train =[]r2_test =[]norm = []alpha = np.linspace(0.01,0.4,10)for i in range(10):    lasso = Lasso(alpha = alpha[i])    lasso.fit(X_train_std,y_train_std)    y_train_std = lasso.predict(X_train_std)    y_test_std = lasso.predict(X_test_std)    r2_train = np.append(r2_train,              r2_score(y_train,sc_y.inverse_transform(y_train_std)))    r2_test = np.append(r2_test,              r2_score(y_test,sc_y.inverse_transform(y_test_std)))    norm = np.append(norm,np.linalg.norm(lasso.coef_))
d. Visualization of output
plt.figure(figsize=(8,6))plt.scatter(alpha,r2_train,label='r2_train')plt.plot(alpha,r2_train)plt.scatter(alpha,r2_test,label='r2_test')plt.plot(alpha,r2_test)plt.scatter(alpha,norm,label = 'norm')plt.plot(alpha,norm)plt.ylim(-0.1,1)plt.xlim(0,.43)plt.xlabel('alpha', size = 14)plt.ylabel('R2_score',size = 14)plt.legend()plt.show()
We observe that, as the regularization parameter alpha increases, the norm of the regression coefficients become smaller and smaller. This means more regression coefficients are forced to zero, which tends to increases bias error (over simplification). The best value to balance bias-variance tradeoff is when alpha is kept low, say alpha = 0.1 or less. This method should be compared with principal component analysis (PCA) before deciding which to use for dimensionality reduction.
Ridge and Elastic net regressions could be implemented in the same way. For more information, see the following book: Raschka, Sebastian, and Vahid Mirjalili. Python Machine Learning, 2nd Ed. Packt Publishing, 2017.
Training a Machine Learning Model on a Dataset with Highly-Correlated Features.Feature Selection and Dimensionality Reduction Using Covariance Matrix Plot.Raschka, Sebastian, and Vahid Mirjalili. Python Machine Learning, 2nd Ed. Packt Publishing, 2017.Benjamin O. Tayo, Machine Learning Model for Predicting a Ships Crew Size, https://github.com/bot13956/ML_Model_for_Predicting_Ships_Crew_Size.
Training a Machine Learning Model on a Dataset with Highly-Correlated Features.
Feature Selection and Dimensionality Reduction Using Covariance Matrix Plot.
Raschka, Sebastian, and Vahid Mirjalili. Python Machine Learning, 2nd Ed. Packt Publishing, 2017.
Benjamin O. Tayo, Machine Learning Model for Predicting a Ships Crew Size, https://github.com/bot13956/ML_Model_for_Predicting_Ships_Crew_Size.