Feature Selection in Python
By: George Pipis
Tags: Python, Machine Learning, Feature Selection, Scikit Learn, Feature Importance

Feature Selection in Python

We will provide a walk-through example of how you can choose the most important features. For this example, we will work with a classification problem but can be extended to regression cases too by adjusting the parameters of the function.

We will work with the breast-cancer dataset. Let’s start:

import pandas as pd

import numpy as np

from scipy import stats

from sklearn.ensemble import RandomForestClassifier

from sklearn.feature_selection import SelectFpr, chi2, SelectKBest, SelectFwe, f_classif, SelectFdr import matplotlib.pyplot as plt

%matplotlib inline https://www.kaggle.com/uciml/breast-cancer-wisconsin-data?select=data.csv

df = pd.read_csv("data.csv") df = pd.read_csv("data.csv") # replace M with 1 and B with 0

my_map = {

'M':1,

'B' :0

} df['diagnosis'] = df['diagnosis'].map(my_map) # remove the id column

df.drop(['id'], axis=1, inplace=True) df

Statistical Significant features with t-test

Since our target is binary, we can compare the values of the independent variables for each group (0,1) by applying t-test .

my_important = [] for c in df.columns[1:]:

pvalue = stats.ttest_ind(df.loc[df.diagnosis==1][c], df.loc[df.diagnosis==0][c])[1]

if pvalue<0.05:

my_important.append(c)

print(f'The variable {c} is statistically significant with a pvalue = {pvalue:.2}')

else:

print(f'The variable {c} is NOT statistically significant')

And we get:

The variable radius_mean is statistically significant with a pvalue = 8.5e-96

The variable texture_mean is statistically significant with a pvalue = 4.1e-25

The variable perimeter_mean is statistically significant with a pvalue = 8.4e-101

The variable area_mean is statistically significant with a pvalue = 4.7e-88

The variable smoothness_mean is statistically significant with a pvalue = 1.1e-18

The variable compactness_mean is statistically significant with a pvalue = 3.9e-56

The variable concavity_mean is statistically significant with a pvalue = 1e-83

The variable concave points_mean is statistically significant with a pvalue = 7.1e-116

The variable symmetry_mean is statistically significant with a pvalue = 5.7e-16

The variable fractal_dimension_mean is NOT statistically significant

The variable radius_se is statistically significant with a pvalue = 9.7e-50

The variable texture_se is NOT statistically significant

The variable perimeter_se is statistically significant with a pvalue = 1.7e-47

The variable area_se is statistically significant with a pvalue = 5.9e-46

The variable smoothness_se is NOT statistically significant

The variable compactness_se is statistically significant with a pvalue = 1e-12

The variable concavity_se is statistically significant with a pvalue = 8.3e-10

The variable concave points_se is statistically significant with a pvalue = 3.1e-24

The variable symmetry_se is NOT statistically significant

The variable fractal_dimension_se is NOT statistically significant

The variable radius_worst is statistically significant with a pvalue = 8.5e-116

The variable texture_worst is statistically significant with a pvalue = 1.1e-30

The variable perimeter_worst is statistically significant with a pvalue = 5.8e-119

The variable area_worst is statistically significant with a pvalue = 2.8e-97

The variable smoothness_worst is statistically significant with a pvalue = 6.6e-26

The variable compactness_worst is statistically significant with a pvalue = 7.1e-55

The variable concavity_worst is statistically significant with a pvalue = 2.5e-72

The variable concave points_worst is statistically significant with a pvalue = 2e-124

The variable symmetry_worst is statistically significant with a pvalue = 3e-25

The variable fractal_dimension_worst is statistically significant with a pvalue = 2.3e-15

So, the statistically significant variables are the:

my_important ['radius_mean',

'texture_mean',

'perimeter_mean',

'area_mean',

'smoothness_mean',

'compactness_mean',

'concavity_mean',

'concave points_mean',

'symmetry_mean',

'radius_se',

'perimeter_se',

'area_se',

'compactness_se',

'concavity_se',

'concave points_se',

'radius_worst',

'texture_worst',

'perimeter_worst',

'area_worst',

'smoothness_worst',

'compactness_worst',

'concavity_worst',

'concave points_worst',

'symmetry_worst',

'fractal_dimension_worst']

Random Forest Feature Importance

We can also run a model, like Random Forest and see which are the most important features.

clf = RandomForestClassifier( n_estimators=50) X = df.drop(['diagnosis'], axis=1)

y = df.diagnosis model = clf.fit(X,y)

feat_importances = pd.DataFrame(model.feature_importances_, index=X.columns, columns=["Importance"])

feat_importances.sort_values(by='Importance', ascending=False, inplace=True) feat_importances.plot(kind='bar')

Let’s see the 10 most important features according to Random Forest:

feat_importances.index[0:10]

We get:

Index(['concave points_worst', 'perimeter_worst', 'radius_worst', 'concave points_mean', 'area_worst', 'perimeter_mean', 'area_mean', 'concavity_worst', 'area_se', 'perimeter_se'], dtype='object')

Feature Selection with Scikit-Learn

We can wor with the scikit-learn . You can find more details at the documentation. We will provide some examples:

k-best

It selects the k most important features. In our case, we will work with the chi-square test. Keep in mind that the new_data are the final data after we removed the non-significant variables.

selector = SelectKBest(score_func=chi2, k=5)

new_data = selector.fit_transform(X, y) mask = selector.get_support()

new_features = X.columns[mask]

new_features

We get:

Index(['perimeter_mean', 'area_mean', 'area_se', 'perimeter_worst', 'area_worst'], dtype='object')

Fwe

This is similar to had we did at the beginning with the t-test. It can be done with a chi-square test or with ANOVA (wherein binary case is the same as the t-test)

Chi-Square

# chi-ssquare

selector = SelectFwe(score_func=chi2, alpha=0.05)

new_data = selector.fit_transform(X, y) mask = selector.get_support()

new_features = X.columns[mask]

new_features

We get:

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',

'concavity_mean', 'concave points_mean', 'radius_se', 'perimeter_se',

'area_se', 'radius_worst', 'texture_worst', 'perimeter_worst',

'area_worst', 'compactness_worst', 'concavity_worst',

'concave points_worst'],

dtype='object')

ANOVA

selector = SelectFwe(score_func=f_classif, alpha=0.05)

new_data = selector.fit_transform(X, y) mask = selector.get_support()

new_features = X.columns[mask]

new_features

We get:

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',

'smoothness_mean', 'compactness_mean', 'concavity_mean',

'concave points_mean', 'symmetry_mean', 'radius_se', 'perimeter_se',

'area_se', 'compactness_se', 'concavity_se', 'concave points_se',

'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst',

'smoothness_worst', 'compactness_worst', 'concavity_worst',

'concave points_worst', 'symmetry_worst', 'fractal_dimension_worst'],

dtype='object')

FDR

The false discovery rate (FDR) takes into account the multiple comparisons.

selector = SelectFdr(chi2, alpha=0.05)

new_data = selector.fit_transform(X, y) mask = selector.get_support()

new_features = X.columns[mask]

new_features

We get:

Index(['radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean',

'compactness_mean', 'concavity_mean', 'concave points_mean',

'radius_se', 'perimeter_se', 'area_se', 'radius_worst', 'texture_worst',

'perimeter_worst', 'area_worst', 'compactness_worst', 'concavity_worst',

'concave points_worst'],

dtype='object')

From the Model

We can keep the most important features derived from the model. Let’s consider again the random forest:

from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(estimator=RandomForestClassifier(n_estimators=50)).fit(X, y)

mask = selector.get_support()

new_features = X.columns[mask]

new_features

We get:

ndex(['perimeter_mean', 'area_mean', 'concavity_mean', 'concave points_mean',

'radius_worst', 'perimeter_worst', 'area_worst',

'concave points_worst'],

dtype='object')

Discussion

This was an example of how you can get a good idea of which variables are the most important for your model. Keep in mind that you should always run exploratory data analysis. For binary cases, a box plot is always an appropriate plot. Let’s see the box plot of the concave points_worst for both being and malignant tumors.

Finally, you can find a detailed explanation of the Chi-Square test.