Feature Selection using Logistic Regression Model | by Satyam Kumar | Towards Data Science

Feature Engineering is an important component of a data science model development pipeline. ‘More data leads to a better machine learning model’, holds true for the number of instances but not for the number of features.
A data scientist spends most of the work time preparing relevant features to train a robust machine learning model. A raw dataset contains a lot of redundant features that may impact the performance of the model. Feature Selection is a feature engineering component that involves the removal of irrelevant features and picks the best set of features to train a robust machine learning model.
Feature Selection methods reduce the dimensionality of the data and avoid the problem of the curse of dimensionality. I have broadly discussed 7 feature selection techniques in one of my previous articles:
towardsdatascience.com
In this article, we will discuss how to remove redundant features from the data using a logistic regression model with L1 regularization.
Regularization is a technique used to tune the model by adding a penalty to the error function. Regularization can be used to train models that generalize better on the test or unseen data and prevents the algorithm from overfitting the training dataset.
L2 regularization refers to the penalty which is equivalent to the square of the magnitude of coefficients, whereas L1 regularization introduces the penalty (shrinkage quantity) equivalent to the sum of the absolute value of coefficients.
L1 regularization introduces sparsity in the dataset, and it can use to perform feature selection by eliminating the features that are not important. Lasso or L1 regularization shrinks the coefficients of redundant features to 0, therefore those features can be removed from the training sample.
The credit card fraud detection dataset downloaded from Kaggle is used to demonstrate the feature selection implementation using Lasso Regression model
Read the dataset and perform feature engineering (standardize) to make it fit to train a logistic regression model.
Train a best-fit Logistic Regression model on the standardized training sample.
Compute the coefficients of the Logistic Regression model using model.coef_ function, that returns with the weight vector of the logistic regression dividing plane.
The dimensionality of the coefficient vector is the same as the number of features in the training dataset.
The coefficient values equating to 0 are the redundant features and can be removed from the training sample. Observing from the above snapshot of the coefficient vector, we have 7 features that have coefficient values of 0.
coef = model.coef_[0]imp_features = pd.Series(X_std.columns)[list(coef!=0)]X_train = X_train[imp_features]X_test = X_test[imp_features]
Prior to feature selection implementation, the training sample had 29 features, which were reduced to 22 features after the removal of 7 redundant features.
The parameter ‘C’ of the Logistic Regression model affects the coefficients term. When regularization gets progressively looser or the value of ‘C’ decreases, we get more coefficient values as 0. One must keep in mind to keep the right value of ‘C’ to get the desired number of redundant features.
A higher value of ‘C’ may consider important features as redundant, whereas lower values of ‘C’ may not exclude the redundant features.
Lasso Regression (Logistic Regression with L1-regularization) can be used to remove redundant features from the dataset. L1-regularization introduces sparsity in the dataset and shrinks the values of the coefficients of redundant features to 0.
It is a very useful technique or hacks to reduce the dimensionality of the dataset by removing the irrelevant features.
There are various other techniques for feature selection. I have discussed 7 such feature selection techniques in one of my previous articles:
towardsdatascience.com
[1] Scikit-learn documentation: https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_path.html
Thank You for Reading