Superior Feature Selection By Combining Multiple Models | Towards Data Science

There are many feature selection methods in Machine Learning. Each one may give different results depending on how you use them, so it is hard to trust a single method entirely. Wouldn’t it be cool to have multiple methods cast their own vote on whether we should keep a feature or not? It would be just like the Random Forests algorithm, where it combines the predictions of multiple weak learners to form a strong one. It turns out, Sklearn has already given us the tools to make such a feature selector on our own.
Together, using those tools, we will build a feature selector that accepts an arbitrary number of Sklearn models. All these models will give votes on which features we should keep, and we make decisions by gathering all the votes across models (democracy).
Before we move on to building the selector, let’s brush up on some of the topics required. Firstly, almost all Sklearn estimators that yield predictions have either .coef_ and .feature_importances_ attributes after being fitted to the training data.
.coef_ attribute mostly occurs in models given under sklearn.linear_model submodule:
As the name suggests, the above are coefficients calculated by fitting the line of best fit for Linear Regression. Other models follow a similar pattern and yield the coefficients of their internal equation:
Models in sklearn.tree and sklearn.ensemble work differently, and they compute the importance or weight of each feature under .feature_importances_ attribute:
Unlike the coefficients of linear models, the weights add up to 1:
np.sum(dt.feature_importances_)1.0
Regardless of the model, the feature contributes less and less to the overall prediction as its weight or coefficient decreases. This means that we can drop features with close to 0 coefficients or weights.
Recursive Feature Elimination (RFE) is a popular feature selection algorithm. It automatically finds the best number of features to keep to achieve the best performance for a given model. Below is a simple example:
The above code is used to find the smallest number of features to achieve the best performance for the Lasso Regression model.
After fitting to the training data, RFECV has .support_ attribute which gives a boolean mask, with True values for the features that should be kept:
We can then use this mask to subset the original data:
X.loc[:, rfecv.support_]
The core of the custom feature selector will be this RFECV class. I didn't go into detail about how it works, but my previous article focused on it. I recommend reading it before continuing:
towardsdatascience.com
We will be using the Ansur Male dataset mainly because it contains many features (98 numeric) about body measurements of 6000 US Army Personnel:
We will be trying to predict weight in pounds, and to do that, we need to reduce model complexity — i. e., create a model with as much predictive power as possible using as few features. Currently, there are 98, and we will be trying to decrease that number. Also, we will be dropping the column which records weight in kilograms.
ansur.drop("weightkg", axis=1, inplace=True)
Our first model will be Lasso Regressor, and we will plug it into RFECV:
We are storing the boolean mask generated from the Lasso in lasso_mask, and you are going to see why in a bit.
Next, we will do the same for two more models: Linear Regression and GradientBoostingRegressor:
Now, we have the votes as boolean masks in three arrays: lasso_mask, gb_mask and lr_mask. Since True/False values represent 1 and 0s, we can add the three arrays:
The result will be an array with counts of how many times all models chose each feature. Now, we can set a threshold of votes to decide whether we will keep the feature or not. This threshold depends on how conservative we want to be. We can set a strict threshold where we want the feature to have been chosen by all 3, or we can choose 1 as a threshold to be safe:
Now, the final_mask is a boolean array with True values if all of the 3 estimators chose a feature. We can use it to subset the original data:
>>> X.loc[:, final_mask].shape(4082, 39)
As you can see, the final mask chose 39 columns to keep out of 98. You can use this subset of the dataset to create a less complex model. For example, we will choose a Linear Regression model because we can expect body measurements to be linearly correlated:
Even though it takes some work to get to the final results, it will be worth it because the combined power of multiple models can surpass any other feature selection method.
We only chose 3 models in the examples, but you can include as many models as you wish to make the results more robust and trustworthy.
At this point, a logical step is to wrap all this code in a function or even a custom Sklearn transformer, but custom transformers are a topic for another article. To reinforce the ideas above and give you an outline, let’s review the steps we have taken:
Choose an arbitrary number of Sklearn estimators that have either .coef_ or .feature_importnances_ attributes. The more estimators, the more robust the results will be. However, multiple models come at a cost - as RFECV uses cross-validation under the hood, the training times will be computationally expensive for ensemble models and large datasets. Also, make sure to choose estimators depending on the type of the problem - remember to pass either classification or regression-only estimators for RFECV to work.Plug all chosen models into RFECV class and make sure to save each round's boolean mask (accessed via .support_). To speed things up, you can tweak the step parameter so that an arbitrary number of features are dropped in each elimination round.Sum up the masks from all estimators.Set a threshold for the vote count. This threshold depends on how conservative you want to be. Convert the votes array into a boolean mask using this threshold.Subset the original data using the final mask for final model evaluation.
Choose an arbitrary number of Sklearn estimators that have either .coef_ or .feature_importnances_ attributes. The more estimators, the more robust the results will be. However, multiple models come at a cost - as RFECV uses cross-validation under the hood, the training times will be computationally expensive for ensemble models and large datasets. Also, make sure to choose estimators depending on the type of the problem - remember to pass either classification or regression-only estimators for RFECV to work.
Plug all chosen models into RFECV class and make sure to save each round's boolean mask (accessed via .support_). To speed things up, you can tweak the step parameter so that an arbitrary number of features are dropped in each elimination round.
Sum up the masks from all estimators.
Set a threshold for the vote count. This threshold depends on how conservative you want to be. Convert the votes array into a boolean mask using this threshold.
Subset the original data using the final mask for final model evaluation.
How to Use Variance Thresholding For Robust Feature Selection
How to Use Pairwise Correlation For Robust Feature Selection
Powerful Feature Selection With Recursive Feature Elimination
RFECV Sklearn documentation
Sklearn Official Feature Selection User Guide
Intro to Object-Oriented-Programming For Data Scientists
My 6-Part Powerful EDA Template That Speaks of Ultimate Skill
How to Use Sklearn Pipelines For Ridiculously Neat Code