Feature selection in Python using the Filter method | by Renu Khandelwal | Towards Data Science

Prerequisites: Feature selection understanding
The filter method ranks each feature based on some uni-variate metric and then selects the highest-ranking features. Some of the uni-variate metrics are
variance: removing constant and quasi constant features
chi-square: used for classification. It is a statistical test of independence to determine the dependency of two variables.
correlation coefficients: removes duplicate features
Information gain or mutual information: assess the dependency of the independent variable in predicting the target variable. In other words, it determines the ability of the independent feature to predict the target variable
Filter methods are model agnostic
Rely entirely on features in the data set
Computationally very fast
Based on different statistical methods
The filter method looks at individual features for identifying it’s relative importance. A feature may not be useful on its own but maybe an important influencer when combined with other features. Filter methods may miss such features.
Select independent features with
High correlation with the target variable
Low correlation with another independent variable
Higher information gain or mutual information of the independent variable
Data set available here
Import the required libraries
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinefrom sklearn.model_selection import train_test_splitfrom sklearn.feature_selection import mutual_info_regression, mutual_info_classif
Reading the data in the data set
dataset= pd.read_csv(‘c:\\data\\auto-mpg.csv’)dataset.head(2)
dataset.info()
We see horsepower is not float but the data above shows that horsepower is numeric.
Checking the presence of categorical features
dataset.describe(include=’O’)
Updating the horsepower feature to int and filling all nulls with 0
dataset[‘horsepower’] = pd.to_numeric(dataset[‘horsepower’], errors=’coerce’).fillna(0).astype(int)
We now check again for the presence of categorical variables
dataset.describe(include=’O’)
We see that horsepower is no more a categorical variable and Car name is the only categorical variable.
Creating a labelEncoder for Car name to encode Car names with a value between 0 and n_classes-1. In our case n_classes for Car name is 305
from sklearn.preprocessing import LabelEncoderlabelencoder = LabelEncoder()X_en= dataset.iloc[:, 8].valuesX_en = labelencoder.fit_transform(X_en)
Creating the input features X and target variable y
X= dataset.iloc[:,1:7]X[‘Car’] = X_eny= dataset.iloc[:,0].values
Create a data set with all the input features after converting them to numeric including target variable
full_data= X.copy()full_data[‘mpg’]= yfull_data.head(2)
Identify input features having high correlation with target variable.
Here we print the correlation of each of the input feature with the target variable
importances = full_data.drop(“mpg”, axis=1).apply(lambda x: x.corr(full_data.mpg))indices = np.argsort(importances)print(importances[indices])
Plotting this data for visualization
names=['cylinders','displacement','horsepower','weight','acceleration','model year', 'car']plt.title('Miles Per Gallon')plt.barh(range(len(indices)), importances[indices], color='g', align='center')plt.yticks(range(len(indices)), [names[i] for i in indices])plt.xlabel('Relative Importance')plt.show()
We want to keep features with only a high correlation with the target variable. This implies that the input feature has a high influence in predicting the target variable.
We set the threshold to the absolute value of 0.4. We keep input features only if the correlation of the input feature with the target variable is greater than 0.4
for i in range(0, len(indices)):    if np.abs(importances[i])>0.4:        print(names[i])
X= dataset[ [‘cylinders’, ‘displacement’, ‘horsepower’, ‘weight’, ‘acceleration’, ‘model year’]]
We now have reduced the input features from 7 to 6. Car name was dropped as it was not having a high correlation with mpg(miles per gallon)
Identify input features that have a low correlation with other independent variables.
Iterating through all the filtered input features based on step 1 and checking each input feature correlation with all other input features.
We will keep input features that are not highly correlated with other input features
for i in range(0,len(X.columns)):    for j in  range(0,len(X.columns)):        if i!=j:            corr_1=np.abs(X[X.columns[i]].corr(X[X.columns[j]]))            if corr_1 <0.3:                print( X.columns[i] , " is not correlated  with ", X.columns[j])            elif corr_1>0.75:                print( X.columns[i] , " is highly  correlated  with ", X.columns[j])
displacement, horsepower, cylinder, and weight are highly correlated. We will keep only keep one of them.
Based on the above result we keep cylinders, acceleration and model year and remove horsepower, displacement, and weight
X= dataset[ [‘cylinders’, ‘acceleration’, ‘model year’]]
Find the information gain or mutual information of the independent variable with respect to a target variable
mi = mutual_info_regression(X, y)
Plotting the mutual information
mi = pd.Series(mi)mi.index = X.columnsmi.sort_values(ascending=False)mi.sort_values(ascending=False).plot.bar(figsize=(10, 4))
We now have our feature importance to predict the miles per gallon. Miles per gallon can be predicted based on the number of cylinders in the car, the year car was manufactured ad the acceleration.
Filter method for feature selection is thus model agnostic, simple and easy to interpret
code available here