Using the Chi-Squared test for feature selection with implementation
By: Dr. Saptarsi Goswami
Tags: Data Science, Machine Learning, Feature Selection, Scikit Learn, Chi Square Test

Using the Chi-Squared test for feature selection with implementation

Let’s approach this problem of feature selection using Chi-Square a question and answer style. If you are a video guy, you may check out our youtube lecture on the same.

Question 1: What is a feature?

For any ML or DL problem, the data is arranged in rows and columns. Let’s take the example of a titanic shipwreck problem.

The rows consist of data of the passengers, actually, each row consists of data about a single passenger. This is typically called an instance, entity, or observation.

The columns describe the passengers like their Sex, Age, etc. These are called features or attributes.

Titanic Data, Observations, Features (Imag Source Author)

Question 2: What are the different types of features?

Without getting into too many details, let’s follow the above example:

Age, Fare, Pid are numeric attributes. However, Pid does not divulge any pattern, as it is routinely assigned either serially or randomly.

Name, Sex, etc. are categorical attributes ( They can be further divided into nominal like Sex and ordinal like ‘Size’ having values like small, medium, and large)

A special class is a categorical variable with two possible values or binary variables like Sex or the variable survived

In the context of a problem, the value of a variable we want to predict is a dependent variable, and the variable or features on the basis of which we want to determine the same are called independent variables.

Question 3: What is feature selection?

In the titanic shipwreck dataset, the problem to solve is given the characteristics of passengers, it is to be determined whether the passenger survived or not ( i.e. whether he got access to a boat or not). So, this is a classification problem.

If the original dataset we have 8 features about the passenger and a classification model brings about 90% classification accuracy, the objective of feature selection is to select maybe 3 or 4 out of the 8 and still achieve similar accuracy.

Question 4: Why feature selection?

With a smaller number of features:

The models are more interpretable

Model training is faster and so is space required by the model

Question 5: What is the connection of feature selection with chi-square?

Actually, it’s not about chi -Squared rather it’s about hypothesis testing. A typical setup for the same is:

H0: The attribute sex has no role to play in the survival of a passenger ( The feature is not important)

H1: The attribute sex has a role to play in survival (The feature is important)

It may be noted, these techniques are called univariate techniques as we inspect each feature separately.

Chi-Square is to be used when the feature is categorical, the target variable is any way can be thought as categorical. It measures the degree of association between two categorical variables. If both are numeric, we can use Pearson’s product-moment correlation, and if the attribute is numerical and there are two classes we can use a t-test if more than two classes we can use ANOVA

We can start with a simple visualization when we approach the effect of Sex on survival status.

Titanic Survival Status versus Sex (Image Source: Author)

The above visualization clearly tells, that ratio of males is to females is not the same across the two classes. Hypothesis testing is a more formal way of doing the same. The Chi-Squared statistics are calculated using the following formula where “O” stands for observed or actual and “E” stands for expected value if these two categories are independent. If they are independent these O and E values will be close and if they have some association then the Chi-squared value will be high.

Chi-Squared Formula (Source: Author)

Sex of Passengers against Survival (Source: Author)

The above table shows Sex vs survival status of the passengers. This is also called the contingency table.

If they are independent then the male-female ratio across survived and not survived should be the same.

A ratio of 449/1313 survived

If there is no dependence or association 463 * 449/1313 of the females should survive i.e 158.13

This is the expected count under independence

The calculation for all four cases are given below

Chi-Squared Calculation Observed vs Expected (Image: Author)

These Chi-Square statistics are adjusted by the degree of freedom which varies with the number of levels the variable has got and the number of levels the class variable has got.

It may be noted Chi-Square can be used for the numerical variable as well after it is suitably discretized.

Question 6: How to implement the same?

Importing the libraries

from sklearn import datasets

from sklearn.feature_selection import chi2

from sklearn.feature_selection import SelectKBest

We are going to do feature selection on the wine dataset

Let’s look at the chi-square scores

# Loading wine data

X,y=datasets.load_wine(return_X_y=True)

# k = 4 tells four top features to be selected

# Score function Chi2 tells the feature to be selected using Chi Square

test = SelectKBest(score_func=chi2, k=4)

fit = test.fit(X, y)

fit.scores_

It will print the values as follows

array([5.44549882e+00, 2.80686046e+01, 7.43380598e-01, 2.93836955e+01,4.50263809e+01, 1.56230759e+01, 6.33343081e+01, 1.81548480e+00,9.36828307e+00, 1.09016647e+02, 5.18253981e+00, 2.33898834e+01,1.65400671e+04])

Now, if we want to select the top four features, we can do simply the following

X_new=test.fit_transform(X, y)

Endnote:

Chi-Square is a very simple tool for univariate feature selection for classification. It does not take into consideration the feature interactions. This is best suited for categorical variables, hence as wide application in textual data.

References:

[1] https://youtu.be/hkDfi5j-6lM

[2] https://machinelearningmastery.com/feature-selection-machine-learning-python/

[3] https://www.kaggle.com/saptarsi/feature-selection-using-chi-square/