[Machine Learning] How to do feature selection | by Jay Hui | Towards Data Science

When I had an interview for a data science-related job, the interviewer asked me the following question. Afterwards, I also asked the same question to the candidate when I was an interviewer: Given a large dataset (more than 1,000 columns with 100,000 rows (records)), how will you select the useful features to build a (supervised) model?
This is a good question to distinguish your data science knowledge between a student and a professional level. When you are a student, you learn algorithms from a beautiful and cleansed dataset. However, in the business world, data scientists put a lot of effort into data cleansing to build a machine learning model.
Back to our question:
Given a large dataset (more than 1000 columns with 10000 rows (records)), how do you select the useful features to build a (supervised) model?
In fact, there is no absolute solution to this question but is to test your logical thinking and explanation skill. In this post, I would share a few methodologies (feature selection) to handle it.
The Kaggle notebook is uploaded here, where the dataset I use is the prudential life insurance underwriting data:
www.kaggle.com
This task aims to predict the risk level (underwriting) of the life insurance applicant. For more implementation of feature selection, you may check the Scikit-learn article as well.
First, we read the data:
Here I am going to do some simple data cleansing. Most machine learning model implementations do not accept string inputs, and hence we have to transform them into numeric values. As our target is is an ordinal target (i.e., supervised learning), transforming the categorical variable by the basic OneHotEncoder suffers from the “curse of dimensionality” (usually it takes more time in model training as well).
Instead, what we do for the categorical column is as follows: Based on its categorical value, we calculate the corresponding mean values of the Y target (by “GroupBy”) and rank it from 0 to the number of unique categorical values that column minus 1. Such mapping is available for testing data or production later.
Then, we fill in the missing value as a standard procedure:
Suppose you have 200 football players, and you would like to select 11 of them to form the best football team. How would you select them?
In the analogy of the football player selection, the unsupervised approach evaluates each player's basic information, such as height, BMI, age, and other health markers. These are not football-specific (unsupervised), but most excellent football players should have a good physical foundation.
Drop columns with a high missing rateThe first and easiest approach is to drop columns in the unsupervised approach. We can drop columns that have too many missing values.
(59381, 127)(59381, 126)
Drop columns with a small varianceAnother way is to drop columns with too small a variance since these columns generally give little information.
number of columns after dropping by variance threshold: 16
PCAPCA is a more advanced way to perform feature selection. The main advantage is that the transformed features are now independent; however, the transformed features are hard to interpret. You can check this article for the PCA implementation of Scikit-learn.
A more sophisticated approach is to do it by supervised learning. Back to our analogy of football team selection, we do it in two rounds:
In the first round, we evaluate the football skills (supervised), such as penalty kick, shooting, short-pass ability, for each player and rank them. Suppose we can select the top 50 players out of the 200 players now.
In the second round, since we want to find the best combination of 11 players out of the 50 players, we need to evaluate how these 50 players will cooperate. We will finally find the best 11 players. (Why do we not do the second round directly? Running every iteration takes a lot of time, so we need the preliminary test in the first round.)
Technically speaking, the first round is the “feature selection by model,” and the second round is “Recursive Feature Elimination” (RFE). Let’s go back to machine learning and coding now.
Feature selection by modelSome ML models are designed for the feature selection, such as L1-based linear regression and Extremely Randomized Trees (Extra-trees model). Comparing to L2 regularization, L1 regularization tends to force the parameters of the unimportant features to zero. (Do you know why?) The extremely randomized trees split the leaf randomly (not through information gain or entropy). The important features should still be more important than the unimportant features (measured by the impurity-based feature importance). We evaluate it with three models:
CPU times: user 15.9 s, sys: 271 ms, total: 16.2 sWall time: 14.3 s========== LogisticRegression ==========Accuracy in training: 0.4138598854833277Accuracy in valid: 0.41020945163666983Show top 10 important features:
========== ExtraTreesClassifier ==========Accuracy in training: 0.3467497473896935Accuracy in valid: 0.3467213977476055Show top 10 important features:
========== RandomForestClassifier ==========Accuracy in training: 0.3473391714381947Accuracy in valid: 0.34581622987053995Show top 10 important features:
We also plot the model importance rankings for each model:
Since L1-based logistic regression has the highest accuracy, so we will only select the top 60 features (from the graph) by logistic regression:
selected_model = ‘LogisticRegression’number_of_features = 60selected_features_by_model = importance_fatures_sorted_all[importance_fatures_sorted_all[‘model’] == selected_model].index[:number_of_features].tolist()
Recursive Feature Elimination (RFE)The second part is to select the best combination of features. We do it by “Recursive Feature Elimination” (RFE). Instead of building one model, we build n models (where n = the number of features). In the first iteration, we train the model by all 60 features and calculate the cross-validation accuracy and all columns' feature importance. Then we drop the least important feature, and so we have 59 features now. Based on these 59 features, we repeat the above process, and we end at the last single feature. This method takes time but will give you a reliable feature importance ranking. If time is not allowed for the large dataset, one can consider do it by sampling or dropping more features at each iteration.
CPU times: user 7min 2s, sys: 334 ms, total: 7min 2sWall time: 26min 32s
As you can see, the validation accuracy will finally saturate (around 0.475) as trained with more features. We can check our feature importance ranking now:
Again, there is no Golden Rule to perform the feature selection. In the business world with production, we have to balance the hardware capability, required time, stability of the model, and the model performance. After finding the best combination of columns, we can now select the best set of hyper-parameter.