Different Feature Selection Techniques
By: Abhay Parashar
Tags: Artificial Intelligence, Machine Learning, Learning, Education, Data

In this blog, I am going to discuss four techniques using which we can select features.

1) Univariate Selection

Univariate Selection examines each feature individually to determine the strength of the relationship of the features with the output feature. In this, some Statistical tests are performed that can be used to select those features that have the strongest relationship with the output variable.

Scikit-learn library provides SelectKBest class that can be used with a score function to find the importance of features with respect to the output variable.

### Importing Libraries

from sklearn.feature_selection import SelectKBest

from sklearn.feature_selection import chi2 ##Statistical test ### Apply SelectKBest Algorithm

ordered_rank_features=SelectKBest(score_func=chi2,k=20)

ordered_feature=ordered_rank_features.fit(X,y) ### Creating a DataFrame with Features and their importance score

dfscores=pd.DataFrame(ordered_feature.scores_,columns=["Score"])

dfcolumns=pd.DataFrame(X.columns)

features_ranks=pd.concat([dfcolumns,dfscores],axis=1) ### Displaying the Feature rank dataframe with top 10 features

features_ranks.columns=['Features','Score']

features_ranks.nlargest(10,'Score')

“Top Features With Their Importance Score”

2) Ensemble Methods

In this method, we make use of ExtraTreesClassifier to find the feature importance in the dataset. This technique gives us a score for each feature of data, the higher the score more relevant it is to predict the output feature.

### Importing Libraries

from sklearn.ensemble import ExtraTreesClassifier

import matplotlib.pyplot as plt ### Creating an object

model=ExtraTreesClassifier() ### Fitting model on the data

model.fit(X,y) ### Checking feature importance for all features

print(model.feature_importances_) ### Visulizing top 10 important Features

ranked_features=pd.Series(model.feature_importances_,index=X.columns)

ranked_features.nlargest(10).plot(kind='barh')

plt.show()

“Top 10 Features”

3) Correlation

A correlation matrix is a table that is used to show the relationship between all the features of a data frame. Each cell om the table used to show the correlation between the two variables. The higher the value of correlation the more they are related to each other.

Using the correlation matrix we can find highly correlated features and then we can simply remove them by defining a threshold value.

### Visulizing Correlation Matrix

import seaborn as sns

corr=df.iloc[:,:-1].corr()

top_features=corr.index

plt.figure(figsize=(20,20))

sns.heatmap(df[top_features].corr(),annot=True)

### Defining a Threshold value

threshold=0.51 ##Values that are related to each other more than 50% ### Returning all the values that are above threshold

def correlation(dataset, threshold):

col_corr = set()

corr_matrix = dataset.corr()

for i in range(len(corr_matrix.columns)):

for j in range(i):

if abs(corr_matrix.iloc[i, j]) > threshold:

colname = corr_matrix.columns[i]

col_corr.add(colname)

return col_corr ### Calling the function(DATAFRAME,THRESHOLD_VALUE)

correlation(df.iloc[:,:-1],threshold)

>0.51

4) Mutual Information

Mutual information is calculated between two variables and measures the reduction in uncertainty for one variable given a known value of the other variable. Mutual information between two random variables X and Y can be stated as follows:

I(X ;Y) = H(X) — H(X | Y)

I(X ;Y) is the mutual information for X and Y

H(X) is the entropy for X

H(X | Y) is the conditional entropy for X given Y

In scikit-learn, we have a class name mutual_info_classif that can be used to find mutual information for each feature with respect to the dependent variable.

### Importing Libraries

from sklearn.feature_selection import mutual_info_classif ### Finding the mutual information

mutual_info=mutual_info_classif(X,y) ### Converting the information into an dataframe

mutual_data=pd.Series(mutual_info,index=X.columns) ### Viewing only top 10

mutual_data.sort_values(ascending=False)[:10]