Feature selection using Python for classification problems | by Richard Liang | Towards Data Science

Including more features in the model makes the model more complex, and the model may be overfitting the data. Some features can be the noise and potentially damage the model. By removing those unimportant features, the model may generalize better.
The Sklearn website listed different feature selection methods. This article is mainly based on the topics from that website. However, I have collected different resources about the theory behind those methods and added them to this article. In addition, I applied different feature selection methods on the same data set to compare their performances.
After reading this article and the resources I provided in the article, you should be able to understand the theory behind feature selection as well as how to do it using python.
We use the iris data set. For more information about this data:
https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html
iris = load_iris()X = iris.datay = iris.targetprint(X[: 5, :])print(X.shape)output: [[5.1 3.5 1.4 0.2] [4.9 3.  1.4 0.2] [4.7 3.2 1.3 0.2] [4.6 3.1 1.5 0.2] [5.  3.6 1.4 0.2]](150, 4)
The data have four features. To test the effectiveness of different feature selection methods, we add some noise features to the data set.
np.random.seed(100)E = np.random.uniform(0, 1, size=(len(X), 10))X = np.hstack((X, E))print(X.shape)output:(150, 14)
The data have 14 features now.
Before applying feature selection method, we need to split the data first. The reason is that we only select features based on the information from the training set, not on the whole data set. We should hold out part of the whole data set as a test set to evaluate the performance of the feature selection and the model. Thus the information from the test set cannot be seen while we conduct feature selection and train the model.
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=100, test_size=0.3)print(X_train.shape)output:(105, 14)
We will apply the feature selection based on X_train and y_train.
VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features. https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html#sklearn.feature_selection.VarianceThreshold
Obviously, our data doesn’t have any zero-variance feature. But for demonstration purpose, I still apply the method here.
sel_variance_threshold = VarianceThreshold() X_train_remove_variance = sel_variance_threshold.fit_transform(X_train)print(X_train_remove_variance.shape)output:(105, 14)
The data still has 14 features, none of the features was removed.
Univariate feature selection works by selecting the best features based on univariate statistical tests. We compare each feature to the target variable, to see whether there is any statistically significant relationship between them. It is also called analysis of variance (ANOVA). When we analyze the relationship between one feature and the target variable, we ignore the other features. That is why it is called ‘univariate’. Each feature has its test score. Finally, all the test scores are compared, and the features with top scores will be selected.
(1) use chi-square test.
For more information about the chi-square test, read:
statistics: http://vassarstats.net/textbook/, chapter 8
Sklearn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2
Read the source code if you want to know more about how Sklearn apply chi-square test: https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/feature_selection/univariate_selection.py#L172
sel_chi2 = SelectKBest(chi2, k=4)    # select 4 featuresX_train_chi2 = sel_chi2.fit_transform(X_train, y_train)print(sel_chi2.get_support())output:[ True  True  True  True False False False False False False False False False False]
The first four elements in the array are true, which means the first four features were selected by this method. Since these features are the original features in the data, the chi-square test performs well.
(2) use f test
statistics
one-way ANOVA :
http://vassarstats.net/textbook/, chapter 14
ANOVA for regression:
http://facweb.cs.depaul.edu/sjost/csc423/documents/f-test-reg.htm
Sklearn
f test for classification:
https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif
f test for regression:
https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression
sel_f = SelectKBest(f_classif, k=4)X_train_f = sel_f.fit_transform(X_train, y_train)print(sel_f.get_support())output:[ True  True  True  True False False False False False False False False False False]
The f test can also correctly select the original features.
(3) use mutual_info_classif test
for classification: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif
for regression: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html#sklearn.feature_selection.mutual_info_regression
sel_mutual = SelectKBest(mutual_info_classif, k=4)X_train_mutual = sel_mutual.fit_transform(X_train, y_train)print(sel_mutual.get_support())output:[ True  True  True  True False False False False False False False False False False]
In sum, three univariate feature selection methods produce the same result.
We use the iris data as a classification problem. For regression problems, similarly, we can use f_regression, mutual_info_regression to conduct the feature selection.
Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from the current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.
About RFE in Sklearn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html#sklearn.feature_selection.RFE
(1) use the logistic regression as the model
model_logistic = LogisticRegression(solver='lbfgs', multi_class='multinomial', max_iter=1000)sel_rfe_logistic = RFE(estimator=model_logistic, n_features_to_select=4, step=1)X_train_rfe_logistic = sel_rfe_logistic.fit_transform(X_train, y_train)print(sel_rfe_logistic.get_support())output:[False  True  True  True False False False False False False False False False  True]print(sel_rfe_logistic.ranking_)output:array([ 3,  1,  1,  1,  9,  8,  7,  6,  5,  4,  2, 11, 10,  1])
The features which were selected are rank 1. The result shows that the recursive feature elimination selects only part of the original features and one noise feature. This is not an ideal result we want. Let’s try another model:
(2) use the random forest as the model
model_tree = RandomForestClassifier(random_state=100, n_estimators=50)sel_rfe_tree = RFE(estimator=model_tree, n_features_to_select=4, step=1)X_train_rfe_tree = sel_rfe_tree.fit_transform(X_train, y_train)print(sel_rfe_tree.get_support())output:[ True  True  True  True False False False False False False False False False False]
Using random forest as the model in the recursive feature selection can select the right features in this case.
SelectFromModel is a meta-transformer that can be used along with any estimator that has a coef_ or feature_importances_ attribute after fitting. The features are considered unimportant and removed if the corresponding coef_ or feature_importances_ values are below the provided threshold parameter.
Compared to univariate feature selection, model-based feature selection consider all feature at once, thus can capture interactions. The model used for the feature selection doesn’t need to be the same model for the training later.
Sklearn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel
(1) L1-based feature selection
linear model with L1 penalty can eliminate some of the features, thus can act as a feature selection method before using another model to fit the data.
model_logistic = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=10000, penalty='l1')sel_model_logistic = SelectFromModel(estimator=model_logistic)X_train_sfm_l1 = sel_model_logistic.fit_transform(X_train, y_train)print(sel_model_logistic.get_support())output:[ True False  True  True False False False False False  True False False False False]
It selects the wrong features! The second feature in the original dataset was ignored while one feature from the noise was selected.
(2) Tree-based feature selection
model_tree = RandomForestClassifier(random_state=100, n_estimators=50)model_tree.fit(X_train, y_train)print(model_tree.feature_importances_)sel_model_tree = SelectFromModel(estimator=model_tree, prefit=True, threshold='mean')        # since we already fit the data, we specify prefit option here      # Features whose importance is greater or equal to the threshold are kept while the others are discarded.X_train_sfm_tree = sel_model_tree.transform(X_train)print(sel_model_tree.get_support())output:[0.11844633 0.09434048 0.21340848 0.33708242 0.02019553 0.03081254 0.02317242 0.01962394 0.02407251 0.02193159 0.03289007 0.01836624 0.01811144 0.027546  ][ True  True  True  True False False False False False False False False False False]
It selects the correct original features.
In summary of all the above features selection methods: for this particular data set, using the logistic model as recursive feature elimination or model selection select the features incorrectly. On the other hand, all other feature selection methods select the first four features correctly.
(1) Before feature selection
model_logistic = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=10000)model_logistic.fit(X_train, y_train)predict = model_logistic.predict(X_test)print(confusion_matrix(y_test, predict))print(classification_report(y_test, predict))output:[[16  0  0] [ 0 11  0] [ 0  2 16]]              precision    recall  f1-score   support           0       1.00      1.00      1.00        16           1       0.85      1.00      0.92        11           2       1.00      0.89      0.94        18    accuracy                           0.96        45   macro avg       0.95      0.96      0.95        45weighted avg       0.96      0.96      0.96        45
(2) After feature selection
we use the result from the feature selection based on the chi-square test.X_train_chi2 is the data after the feature selection to feed into the model.
model_logistic = LogisticRegression(solver=’saga’, multi_class=’multinomial’, max_iter=10000)model_logistic.fit(X_train_chi2, y_train)
We also need to transform the test data because the number of features was changed.
X_test_chi2 = sel_chi2.transform(X_test)print(X_test.shape)print(X_test_chi2.shape)output:(45, 14)(45, 4)
Only use the features in the test set that are corresponding to the remaining features in the training set. Four features in this case.
predict = model_logistic.predict(X_test_chi2)print(confusion_matrix(y_test, predict))print(classification_report(y_test, predict))output:[[16  0  0] [ 0 11  0] [ 0  1 17]]              precision    recall  f1-score   support           0       1.00      1.00      1.00        16           1       0.92      1.00      0.96        11           2       1.00      0.94      0.97        18    accuracy                           0.98        45   macro avg       0.97      0.98      0.98        45weighted avg       0.98      0.98      0.98        45
In sum, feature selection removes the noise and generalize the model better, thus improve the model performance.
We can experiment on adding different noise. for example:
E = np.random.uniform(0, 10, size=(len(X), 20))
If the number of noise features increases to 20 and the magnitude of the noise is bigger, then the feature selection might select the wrong features. However, performance is still improved!
Before feature selection[[16  0  0] [ 0 10  1] [ 0  4 14]]              precision    recall  f1-score   support0                  1.00      1.00      1.00        16           1       0.71      0.91      0.80        11           2       0.93      0.78      0.85        18accuracy                               0.89        45   macro avg       0.88      0.90      0.88        45weighted avg       0.90      0.89      0.89        45After feature selection:[[16  0  0] [ 0 10  1] [ 0  1 17]]              precision    recall  f1-score   support0                  1.00      1.00      1.00        16           1       0.91      0.91      0.91        11           2       0.94      0.94      0.94        18accuracy                               0.96        45   macro avg       0.95      0.95      0.95        45weighted avg       0.96      0.96      0.96        45
In this article, I used different feature selection methods to the same data. Then I evaluate their performances. I first added noise to the iris dataset to form a new dataset. Then different feature selection methods were applied to this new dataset. If the noise is small compared to the original data and the number of noise feature is relatively small, then most of the methods can discover the original feature correctly.
Comparing using all the features (original features + noise features) to train the model, the model performs better if we only use the remaining features after the feature selection.
Note: The full Jupiter notebook can be downloaded here:
https://github.com/musicpiano/data-science-project/blob/master/compare-different-feature-selection-methods.ipynb
scikit-learn.org
[2] Andreas C. Müller, Sarah Guido (2016). Introduction to Machine Learning with Python: A Guide for Data Scientists. O’Reilly