Choosing the best ones: Feature Selection
By: Apurva Misra
Tags: Data, Data Analytics, Data Science, Feature Selection, Data Visualization

“Feature Importance”, “Feature Selection” are different phrases meaning the same thing- finding the features which contribute the most information towards the learning task. In this post, we will learn how to utilize Scikit learn to retrieve the best features.

Why do we need it?

There are multiple ways of extracting features that create new features in a different vector space altogether such as Principle Component Analysis (PCA), but there are times when we want to know the importance of the existing features before generating any secondary features. This increases interpretability and allows us to reduce the feature set to form simpler models as well as avoid overfitting.

For instance, a company wants to classify flowers based on their height and colour. There are two kinds of flowers: “Sunflower” and “Marigold”, which are to be classified based on the features- height and shade of yellow as given below. Since the two flowers have almost similar variations in the colour the height acts as an important differentiator. Marigolds on average are shorter than Sunflowers. Height is an important feature as it has the highest spread and has more differentiation between the two flowers when projected on the “height” axis. Hence, the company need not put in the resources to collect data on the colour of the flowers and can instead form a good classifier based on the feature “height”.

Marigold (purple color circles) and Sunflowers (yellow color circles)

There are three ways of doing Feature selection as given:

Filter Approach: Use statistical properties to filter out poorly informative/redundant features. This approach is applied before using any classifier/regressor. Selected feature subset should have less correlation to each other (avoid redundancy) and a large correlation with class labels (increase relevance). Wrapper Approach: Involves feature subset generation and its evaluation on the classifier/regressor. There are multiple algorithms for subset generation such as Exhaustive search (generating all possible subsets), Bidirectional Search, Genetic algorithm, etc. Embedded approach: Feature selection is done during the training of a classifier/regressor such as using a regularisation term which gives weighting to the features accordingly.

AIM: Getting rid of irrelevant (no contribution towards the learning task) or redundant (no additional information added) features.

Wrappers generally achieve better accuracy than filters since they are tuned to the classifier. Most of the feature selection solutions in the Wrapper approach of Scikit learn rely on recursive feature selection/elimination. The drawback of this methodology arises when features have multicollinearity- two or more features are highly linearly related. Hence, if two features x1 and x2 are highly correlated, dropping x1 would not drop the accuracy of the model as x2 would be able to provide the necessary information and vice versa. Eventually, both of them would be given a low weightage even though the information gained through choosing at least one of them might be important.

In this post, we try clustering the features based on the correlation and then choosing one of the features from each of the clusters for forming a reduced feature set. Wisconsin Breast Cancer Database is used here from UCI, the code and dataset can be found here , having 10 attributes with 2 classes - benign and malignant. There are 458 benign cases and 241 malignant. It has the following attributes: ‘sample_code_number’, ‘clump_thickness’, ‘uniformity_of_cell_size’, ‘uniformity_of_cell_shape’, ‘marginal_adhesion’, ‘single_epithilial_cell_size’, ‘bare_nuclei’, ‘bland_chromatin’, ‘normal_nucleoli’, ‘mitoses’.

Code for forming the clusters and selecting features

Clusters based on correlation on the left and correlation heat map on the right

We use Spearman’s correlation coefficient to find the correlation between the features and then make hierarchical linkage plot where two features connected by a link which is shorter in height has high correlation, for instance, “uniformity of cell size” and “uniformity of cell shape” have high correlation as they are connected together have a height of around 0.18.

Hence, we can choose any threshold to define our clusters, in the above diagram we have used a threshold of 0.6 (red horizontal line). The three clusters would be:

cluster1: mitoses cluster2: clump_thickness cluster3: uniformity_of_cell_size, uniformity_of_cell_shape, marginal_adhesion, single_epithilial_cell_size, bare_nuclei, bland_chromatin, normal_nucleoli.

Random Forest is used as a classifier for wrapper approach and one feature is chosen from each cluster to avoid any redundancy as shown in the code. The increase in accuracy with the addition of each feature can be observed below:

Reduced feature set and it's corresponding accuracy

As we can see, the addition of the feature “mitoses” from the first cluster leads to a drop in accuracy. Here, we have just added the first feature from each cluster, but it will be even more useful to iterate over all the features from each cluster to choose the best one.

Feature selection is one of the most necessary processes needed to interpret and simplify the model and there are numerous ways such as Permutation feature importance which reduces the number of times the model has to be retrained. The above methodology is one of the ways I looked into, to drop features as part of my research.

Additional tips: Use the Point-biserial correlation coefficient to find correlation between a feature and binary labels.