Feature Selection and EDA in Machine Learning
By: Destin Gong
Tags: Feature Selection, Data Science, Python, Machine Learning, Exploratory Data Analysis

Box plot is used as the visual representation of ANOVA analysis

box plot code

Box plot displays the distributions of groups of numerical data through their quantiles. Each box shows how spread out the data is within group and putting boxes side by side indicates the difference among groups. It is aligned with ANOVA test which also analyze the degree of variance between-group compared to within-group. If the relative variability is large, for instance “Total_Revolving_Bal” and “Total_Cnt_Chng_Q4_Q1” shown below, then it may be an indication of that these features could contribute to predicting the labels. Let’s find out whether this can be quantified by the ANOVA test in filter methods.

box plot results (image by author)

box plot results (image by author)

If you would like to have a more thorough understanding of EDA, feel free to read my article on “Semi-Automated Exploratory Data Analysis (EDA) in Python”.

Feature Selection

This article introduces two types of feature selection methods: filter method and wrapper method. The fundamental difference is that filter method evaluates the feature importance based on statistical tests such as Chi-Square, ANOVA etc, whereas wrapper method iteratively assessed the performance of subsets of features based the performance of models generated by these features.

Filter Methods

filter method illustration (image by author)

Filter methods give a score to each feature by evaluating its relationship with the dependent variable. For classification problems with categorical response variables, I am using these three major scoring functions: Chi-Square (score_func = chi2), ANOVA (score_func = f_classif), and Mutual Information (score_func = mutual_info_classif). To create a feature selection model, we need the SelectKBest() function, then specify which scoring functions to utilize and the how many variables to select.

selection_model = SelectKBest(score_func=score_function, k=variable_counts)

I would like to know how these two parameters, scoring function and the number of variables, would affect the accuracy of the model trained on the selected features.

Firstly, to create the carry out the feature selection and examine the performance of the model built upon it, I define a feature_selection function with following steps:

import required libraries

create a feature selection model based on two parameters: score_function (e.g. chi square) and variable counts (e.g. ranging from 1 to all features)

train a logistic regression model based on selected features only

calculates the accuracy score

import libraries

define feature_selection function

Secondly, to test how score functions and variable counts would affect the model performance, I iteratively passing different combinations of two parameters, “variable_counts” and “score_function”, using the following code.

filter method accuracy chart code

Filter Methods with Data Visualization

The result was generated in a data frame format and then use line chart to demonstrate how the accuracy progress as the number of selected features grows. As shown, except mutual information method, the accuracy score stabilizes around 0.88 after reaching 8 features.

filter method accuracy (image by author)

Afterwards, let’s investigate what is the score of each feature based on various approaches. This time we will use bar charts to visualize the scores has been allocated to features according to chi-square, anova or mutual information.

feature score code

feature score by different statistics (image by author)

As you can see, different approach scores the same feature differently, but some features always appear higher up on the list. For example, “Total_Revolving_Bal” is always among the top 3, which is aligned with the findings from the box plot in the bivariate EDA. And “Card_Category” does have high feature importance compared to other categorical variables, which can be explained by the grouped bar chart.

Wrapper Methods

wrapper method illustration (image by author)

Wrapper Methods find the optimal subsets of features by evaluating the performance of machine learning models trained upon these features. Since it incorporates model into the feature selection process, it requires more computational power. This article covers two main wrapper methods, forward selection and backward elimination. To perform forward selection and backward elimination, we need SequentialFeatureSelector() function which primarily requires four parameters:

model: for classification problem, we can use Logistic Regression, KNN etc, and for regression problem, we can use linear regression etc k_features: the number of features to be selected forward: determine whether it is forward selection or backward elimination scoring: evaluation metrics to determine the model performance, e.g. classification problem — accuracy, precision, recall etc; regression problem — p-value, R-squared etc

Forward Selection

Forward Selection starts with no features in the model and incrementally adds one feature to the feature subset at a time. During each iteration, the new feature is chosen based on the evaluation of the model trained by the feature subset.

Since the machine learning model is wrapped within the feature selection algorithm, we need to specify a model as one of the input parameters. I choose Logistic Regression for this classification problem and accuracy as the evaluation metrics. There is a slight difference in calculating the accuracy in the wrapper method compared to the filter method. Since we only fit the training set to the wrapper model, the accuracy score returned by the wrapper method itself is purely based on the training dataset. Therefore, it is necessary to train an additional model on the selected features and further evaluated based on the test set.

To achieve this, I used the code below to import the required libraries, as well as create and evaluate the logistic regression model built upon the wrapper method.

import libraries

forward selection code

Backward Elimination

Simply put, it is just the opposite of the forward selection, starting with including all features to train the model. Then, features are iteratively removed from the feature subset based on whether they contribute to the model performance. Similarly, logistic regression and accuracy are used as the model and evaluation metrics correspondingly.

backward selection code

Wrapper Methods and Data Visualization

Similar to the filter method, I enveloped both forward selection and backward elimination into a “for loop”, in order to examine whether the variable counts would matter to the accuracy score.

wrapper method accuracy code

As shown in the line chart, the accuracy grows rapidly when the feature counts is less than 4 and then remains stable around 0.88 afterwards.

wrapper method accuracy chart (image by author)

In this dataset, since there are only around 20 features in total, it may be hard for feature selection to yield any significant impacts on model performance. However, it is undeniable that data visualization can help us to decide which features and how many features are suitable for the dataset or the objectives. This principle can definitely be extended to other dataset with more variables.

Take Home Message

This article covers two fundamental techniques of feature selection:

Filter Methods: baed on chi-square, ANVOA and mutual information

baed on chi-square, ANVOA and mutual information Wrapper Methods: based on forward selection and backward elimination

We also look at how to use data visualization to better understand the feature properties and additionally to select an appropriate number of features.

More Articles Like This