Using the Chi-Squared test for feature selection with implementation | by Dr. Saptarsi Goswami | Towards Data Science

Let’s approach this problem of feature selection using Chi-Square a question and answer style. If you are a video guy, you may check out our youtube lecture on the same.
Question 1: What is a feature?
For any ML or DL problem, the data is arranged in rows and columns. Let’s take the example of a titanic shipwreck problem.
The rows consist of data of the passengers, actually, each row consists of data about a single passenger. This is typically called an instance, entity, or observation.
The columns describe the passengers like their Sex, Age, etc. These are called features or attributes.
Question 2: What are the different types of features?
Without getting into too many details, let’s follow the above example:
Age, Fare, Pid are numeric attributes. However, Pid does not divulge any pattern, as it is routinely assigned either serially or randomly.
Name, Sex, etc. are categorical attributes ( They can be further divided into nominal like Sex and ordinal like ‘Size’ having values like small, medium, and large)
A special class is a categorical variable with two possible values or binary variables like Sex or the variable survived
In the context of a problem, the value of a variable we want to predict is a dependent variable, and the variable or features on the basis of which we want to determine the same are called independent variables.
Question 3: What is feature selection?
In the titanic shipwreck dataset, the problem to solve is given the characteristics of passengers, it is to be determined whether the passenger survived or not ( i.e. whether he got access to a boat or not). So, this is a classification problem.
If the original dataset we have 8 features about the passenger and a classification model brings about 90% classification accuracy, the objective of feature selection is to select maybe 3 or 4 out of the 8 and still achieve similar accuracy.
Question 4: Why feature selection?
With a smaller number of features:
The models are more interpretable
Model training is faster and so is space required by the model
Question 5: What is the connection of feature selection with chi-square?
Actually, it’s not about chi -Squared rather it’s about hypothesis testing. A typical setup for the same is:
H0: The attribute sex has no role to play in the survival of a passenger ( The feature is not important)
H1: The attribute sex has a role to play in survival (The feature is important)
It may be noted, these techniques are called univariate techniques as we inspect each feature separately.
Chi-Square is to be used when the feature is categorical, the target variable is any way can be thought as categorical. It measures the degree of association between two categorical variables. If both are numeric, we can use Pearson’s product-moment correlation, and if the attribute is numerical and there are two classes we can use a t-test if more than two classes we can use ANOVA
We can start with a simple visualization when we approach the effect of Sex on survival status.
The above visualization clearly tells, that ratio of males is to females is not the same across the two classes. Hypothesis testing is a more formal way of doing the same. The Chi-Squared statistics are calculated using the following formula where “O” stands for observed or actual and “E” stands for expected value if these two categories are independent. If they are independent these O and E values will be close and if they have some association then the Chi-squared value will be high.
The above table shows Sex vs survival status of the passengers. This is also called the contingency table.
If they are independent then the male-female ratio across survived and not survived should be the same.
A ratio of 449/1313 survived
If there is no dependence or association 463 * 449/1313 of the females should survive i.e 158.13
This is the expected count under independence
The calculation for all four cases are given below
These Chi-Square statistics are adjusted by the degree of freedom which varies with the number of levels the variable has got and the number of levels the class variable has got.
It may be noted Chi-Square can be used for the numerical variable as well after it is suitably discretized.
Question 6: How to implement the same?
Importing the libraries
from sklearn import datasetsfrom sklearn.feature_selection import chi2from sklearn.feature_selection import SelectKBest
We are going to do feature selection on the wine dataset
Let’s look at the chi-square scores
# Loading wine dataX,y=datasets.load_wine(return_X_y=True)# k = 4 tells four top features to be selected# Score function Chi2 tells the feature to be selected using Chi Squaretest = SelectKBest(score_func=chi2, k=4)fit = test.fit(X, y)fit.scores_
It will print the values as follows
array([5.44549882e+00, 2.80686046e+01, 7.43380598e-01, 2.93836955e+01,4.50263809e+01, 1.56230759e+01, 6.33343081e+01, 1.81548480e+00,9.36828307e+00, 1.09016647e+02, 5.18253981e+00, 2.33898834e+01,1.65400671e+04])
Now, if we want to select the top four features, we can do simply the following
X_new=test.fit_transform(X, y)
Endnote:
Chi-Square is a very simple tool for univariate feature selection for classification. It does not take into consideration the feature interactions. This is best suited for categorical variables, hence as wide application in textual data.