Feature Selection Using Regularisation | by Akash Dubey | Towards Data Science

Regularisation consists in adding a penalty to the different parameters of the machine learning model to reduce the freedom of the model and in other words to avoid overfitting. In linear model regularisation, the penalty is applied over the coefficients that multiply each of the predictors. From the different types of regularisation, Lasso or L1 has the property that is able to shrink some of the coefficients to zero. Therefore, that feature can be removed from the model.
In this post I will demonstrate how to select features using the Lasso regularisation classification problem. For classification I will use the Paribas claims dataset from Kaggle.
Importing important libraries
Importing important libraries
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinefrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import Lasso, LogisticRegressionfrom sklearn.feature_selection import SelectFromModelfrom sklearn.preprocessing import StandardScaler
2. Loading the dataset
data = pd.read_csv(‘paribas.csv’, nrows=50000)data.shape
data.head()
3. Selecting Numerical Columns
In practice, feature selection should be done after data pre-processing, so ideally, all the categorical variables are encoded into numbers, and then we can assess how deterministic they are of the target, here for simplicity I will use only numerical variables to select numerical columns:
numerics = ['int16','int32','int64','float16','float32','float64']numerical_vars = list(data.select_dtypes(include=numerics).columns)data = data[numerical_vars]data.shape
4. Separating the data into training and tests set
X_train, X_test, y_train, y_test = train_test_split(    data.drop(labels=['target', 'ID'], axis=1),    data['target'],    test_size=0.3,    random_state=0)X_train.shape, X_test.shape
5. Scaling the data, as linear models benefits from feature scaling
scaler = StandardScaler()scaler.fit(X_train.fillna(0))
6. Selecting features using Lasso regularisation using SelectFromModel
Here I will do the model fitting and feature selection, altogether in one line of code. First I specify the Logistic Regression model, and I make sure I select the Lasso (L1) penalty.Then I use the selectFromModel object from sklearn, which will select in theory the features which coefficients are non-zero.
sel_ = SelectFromModel(LogisticRegression(C=1, penalty='l1'))sel_.fit(scaler.transform(X_train.fillna(0)), y_train)
7. Visualising features that were kept by the lasso regularisation
sel_.get_support()
In the above output, the output labels are index wise. So Trueis for the features that lasso thought is important (non-zero features) while False is for the features whose weights were shrinked to zero and are not important according to Lasso.
8. Make a list of with the selected features.
selected_feat = X_train.columns[(sel_.get_support())]print('total features: {}'.format((X_train.shape[1])))print('selected features: {}'.format(len(selected_feat)))print('features with coefficients shrank to zero: {}'.format(      np.sum(sel_.estimator_.coef_ == 0)))
Number of features which coefficient was shrank to zero :
np.sum(sel_.estimator_.coef_ == 0)
9. Identifying the removed features
removed_feats = X_train.columns[(sel_.estimator_.coef_ == 0).ravel().tolist()]removed_feats
10. Removing the features from training an test set
X_train_selected = sel_.transform(X_train.fillna(0))X_test_selected = sel_.transform(X_test.fillna(0))X_train_selected.shape, X_test_selected.shape
L2 regularisation does not shrink coefficients to zero
# Separating the data into train and test set X_train, X_test, y_train, y_test = train_test_split(    data.drop(labels=['target', 'ID'], axis=1),    data['target'],    test_size=0.3,    random_state=0)X_train.shape, X_test.shape
For comparison, I will fit a logistic regression with a Ridge regularisation, and evaluate the coefficients :
l1_logit = LogisticRegression(C=1, penalty='l2')l1_logit.fit(scaler.transform(X_train.fillna(0)), y_train)
Now, Lets count the number of coefficients with zero values :
np.sum(l1_logit.coef_ == 0)
So, Now number of coefficients with zero values is zero. So, now it is clear that Ridge regularisation (L2 Regularisation) does not shrink the coefficients to zero.
As we can see, the logistic regression we used for the Lasso regularisation to remove non-important features from the dataset. Keep in mind that increasing the penalisation c will increase the number of features removed. Therefore, we will need to keep an eye and monitor that we don’t set a penalty too high so that to remove even important features, or too low and then not remove non-important features.
For feature selection using Random forest :