Feature Selection: How to Throw Away 95% of Your Features and Get 95% Accuracy | by Samuele Mazzanti | Towards Data Science

Can you recognize these handwritten digits?
You probably had no trouble spotting respectively a 0, a 3 and an 8.
If so, you were able to classify them correctly even if only 25% of the original image was shown, while the remaining 75% was covered with red pixels. This was an easy task because the “relevant” pixels were visible, while only the “irrelevant” and “redundant” ones had been hidden.
This was a simple example of “feature selection”.
Feature selection is a process that is carried out in machine learning when, before feeding the data to a predictive model, some features are removed. If your data is in the form of a table, this simply means dropping many columns from your table.
Why bothering with feature selection? Can’t I just throw all the data inside my predictive model and let him doing the dirty work?
Actually, there are several reasons for which you may want to do feature selection:
Memory. Big data take big space. Dropping features means that you need less memory to handle your data. Sometimes there are also external constraints (for instance, Google’s AutoML allows you to use no more than 1,000 columns. Thus, if you have more than 1,000 columns, you will be forced to keep only some of them).
Time. Training a model on less data can save you much time.
Accuracy. Less is more: this also goes for machine learning. Including redundant or irrelevant features means including unnecessary noise. Frequently, it happens that a model trained on less data performs better.
Interpretability. A smaller model means also a more interpretable model. Just picture if you had to explain a model based on thousands of different factors: it would be unfeasible.
Debugging. A smaller model is easier to mantain and troubleshoot.
In this article, we will compare some methods for feature selection. Our playground dataset will be the world-famous “MNIST”.
MNIST is a dataset consisting of 70,000 black-and-white images of handwritten digits. Each image is 28 x 28 (= 784) pixels. Each pixel is encoded as an integer from 1 (white) to 255 (black): the higher this value the darker the color. As a convention, 60,000 images are used as train set and 10,000 as test set.
The data can be imported into Python through a Keras command (we’ll also reshape the dataset to make it 2-dimensional, as a table):
from keras.datasets import mnist(X_train, y_train), (X_test, y_test) = mnist.load_data()X_train = X_train.reshape(60000, 28 * 28)X_test = X_test.reshape(10000, 28 * 28)
For instance, let’s print out the values of the 8th row:
print(X_train[7, :])
This is the outcome:
Through Matplotlib, we can also display the corresponding image:
import matplotlib.pyplot as pltplt.imshow(X_train[7, :].reshape(28, 28), cmap = 'binary', vmin = 0, vmax = 255)
Our task is to select a small number of columns (i.e. pixels) that are sufficient to achieve a good level of accuracy when fed to a predictive model.
There are many possible strategies and algorithms to perform feature selection. In this article, we will put to the test 6 of them:
3.1 F-Statistic
F-Statistic is the outcome of the ANOVA F-test. This test is computed as the ratio: between-group variability / within-group variability, where the group is the target class.
The idea is that a pixel is relevant when the variability between the groups (images in class 0, images in class 1, ..., images in class 9) is high and the variability within the same groups is low.
from sklearn.feature_selection import f_classiff = f_classif(X_train, y_train)[0]
3.2 Mutual Information
Mutual information is a measure of the mutual dependence between two variables.
Since the formula of MI requires knowing the probability distribution of each variable (and normally we don’t know distributions), the scikit-learn implementation employs a nonparametric approximation based on k-nearest neighbors distances.
from sklearn.feature_selection import mutual_info_classifmi = mutual_info_classif(X_train, y_train)
3.3 Logistic Regression
If the target variable is categorical (as in our case), a logistic regression may be fitted to the data. Then, the relative importance of the features can be used to rank them from the most to the least relevant.
from sklearn.linear_model import LogisticRegressionlogreg = LogisticRegression().fit(X_train, y_train)
3.4 LightGBM
The same can be done with any predictive model. For instance, LightGBM.
from lightgbm import LGBMClassifierlgbm = LGBMClassifier(    objective = 'multiclass',    metric = 'multi_logloss',    importance_type = 'gain').fit(X_train, y_train)
3.5 Boruta
Boruta is an elegant algorithm designed in 2010 as a package for R. The aim of Boruta is to tell whether or not each feature has some kind of relationship with the target variable. So, the output of Boruta is more a yes/no for each feature, rather than a ranking of features.
(If you are curious to know more about the functioning of Boruta, I wrote a post about it: Boruta Explained Exactly How You Wished Someone Explained to You).
from boruta import BorutaPyfrom sklearn.ensemble import RandomForestClassifierboruta = BorutaPy(    estimator = RandomForestClassifier(max_depth = 5),     n_estimators = 'auto',     max_iter = 100).fit(X_train, y_train)
3.6 MRMR
MRMR (which stands for “Maximum Relevance Minimum Redundancy”) is an algorithm designed in 2005 for feature selection. The idea behind MRMR is to identify a subset of features having a high relevance with respect to the target variable and a small redundancy with each other.
(If you are curious to know more about the functioning of MRMR, I wrote a post about it: MRMR Explained Exactly How You Wished Someone Explained to You).
from mrmr import mrmr_classifmrmr = mrmr_classif(X_train, y_train)
All these algorithms provide a “ranking” of the features (except for Boruta, which has a yes/no outcome). Let’s see how the rankings change according to the different algorithms.
At this point, a natural question is:
What method should I choose for feature selection?
As always in data science, the best choice is to test different approaches, and see which one gives better results on your data. Therefore, let’s try them on MNIST.
We will take the feature rankings provided by the 5 methods (since Boruta does not provide a ranking) and see what accuracy can be achieved when training a predictive model on the first K features (for K up to 40).
import pandas as pdfrom catboost import CatBoostClassifierfrom sklearn.metrics import accuracy_scorealgos = ['f', 'mi', 'logreg', 'lightgbm', 'mrmr']ks = [1, 2, 5, 10, 15, 20, 30, 40]accuracy = pd.DataFrame(index = ks, columns = algos)for algo in algos:    for nfeats in ks:    feats = ranking[algo][:n_feats]        clf = CatBoostClassifier().fit(      X_train[:, feats], y_train,      eval_set = (X_test[:, feats], y_test),      early_stopping_rounds = 20    )                    accuracy.loc[k, algo] = accuracy_score(      y_true = y_test, y_pred = clf.predict(X_test[:, cols])))
These are the outcomes:
In this case, MRMR outperformed the other algorithms.
As the plot shows, a predictor trained on the most relevant 40 pixels identified by MRMR reached 95.54% accuracy on test images!
This is pretty impressive, especially if we consider that:
40 pixels are only 5% of the whole image (which consists of 28 x 28 = 784 pixels);
we have used a predictive model (CatBoost) with no further tuning, thus this performance can probably be improved further.
Thus, in the case of MNIST, we could throw away 95% of our data and still get more than 95% accuracy (which corresponds to an area under ROC of 99.85%!).
Even if MNIST is a “simple” dataset, the main takeaways are valid for most real-world datasets. Often, a high level of accuracy can be achieved with just a small portion of features. An effective feature selection allows you to build data pipelines that are more efficient in terms of memory, time, accuracy, interpretability and ease of debugging.
Thank you for reading! I hope you found this post useful.
The results shown in this article are fully reproducible through this code: https://github.com/smazzanti/mrmr/blob/main/notebooks/mnist.ipynb.
I appreciate feedback and constructive criticism. If you want to talk about this article or other related topics, you can text me at my Linkedin contact.