Chi-Squared Test for Feature Selection with implementation in Python | by Yaser Sakkaf | Towards Data Science

There are various techniques in machine learning to determining whether our input features are relevant to the outcome to be predicted or not. We can use these techniques to find out and remove unimportant features that do not contribute to the outcome of our model. In a way we select only those features that actually matter, hence these techniques are called as Feature Selection methods. Chi-Squared test can be used for Feature Selection. Let’s dive into the details.
There are many methods to determine feature importance which are mainly divided into two groups:
1) Filter feature selection methods2) Wrapper feature selection methods
We will only see the first one since our Chi-Squared test falls in this category.
Briefly, Filter feature selection methods are those that use some statistical techniques (like Chi-Squared test) considering the data type of the input and target variable to evaluate the relationship between them.
There are many other statistical measures that can be used for filter-based feature selection. As such, the choice of statistical measures is highly dependent upon the feature and target variable data types.
The data types can either be numerical or categorical.
When the data type of our feature to be tested and the target variable are both categorical (i.e. we have a classification problem) we can use Chi-Squared test.
The Pearson’s Chi-Squared test, or just Chi-Squared test, is named after the mathematician Karl Pearson. It is also called a “goodness of fit” statistic.
Let’s see how to perform this test along with an example in 8 simple steps.
For example’s sake let’s consider a very simple dataset with only two columns. We will find out if Gender is related/dependent/associated to the interest in sports or not.
╔════════╦══════════╗║ Gender ║ Interest ║╠════════╬══════════╣║ Male   ║ Cricket  ║║ Female ║ Tennis   ║║ Male   ║ Cricket  ║║ Male   ║ Football ║║ ....   ║ ....     ║╚════════╩══════════╝
A Hypothesis Test is a statistical method which evaluates two statements(hypothesis) and determines which statement is true. In hypothesis testing the initial statement is called a Null Hypothesis denoted as H0 and the second one that is usually complementary to the first one is called the Alternate Hypothesis denoted as H1.
For our example, the hypothesis are:H0: The features Gender and Interest are independent (which means they are not associated).H1: Gender and Interest are not independent (which means they are associated).
Basically, the significance level denoted by α (alpha) is a measure of how certain we want to be about our results.
Or in simple words significance level is used to determine whether the null hypothesis should be rejected or not. For the null hypothesis to be rejected the p-value(explained later) should be less than the significance level.
Lower α values are generally preferred which may be in the range of 0.01 to 0.10.
We choose α = 0.05
A table showing the frequency distribution of one variable in rows and another in columns, used to study the correlation between the two variables is called a contingency table (also known as a cross tabulation or crosstab). You will be more clear after seeing the table below:
╔══════════════╦═════════╦══════════╦════════╦═══════════╗║              ║ Cricket ║ Football ║ Tennis ║ Row Total ║╠══════════════╬═════════╬══════════╬════════╬═══════════╣║ Male         ║      20 ║       30 ║     15 ║        65 ║║ Female       ║      20 ║       15 ║     30 ║        65 ║║ Column Total ║      40 ║       45 ║     45 ║       130 ║╚══════════════╩═════════╩══════════╩════════╩═══════════╝
We can see that out of 130 candidates, in the dataset, there are 20, 30 and 15 males interested in Cricket, Football and Tennis respectively. Similarly 20, 15 and 30 Females are interested in Cricket, Football and Tennis respectively.
The values in the table are called as observed values.
We calculate the expected frequecy count for each cell. The formula to count expected frequncy is:
E = (row total * column total)/grand total
The Expected frequency for first cell (i.e. Male-Cricket) will be:
E1 = (65 * 40)/130 = 20
We calculate Expected Frequency for rest of the cells and get the table below where values in brackets ‘[]’ denote expected frequencies:
╔══════════════╦═════════╦════════════╦═══════════╦═══════════╗║              ║ Cricket ║  Football  ║  Tennis   ║ Row Total ║╠══════════════╬═════════╬════════════╬═══════════╬═══════════╣║ Male         ║ 20 [20] ║  30 [22.5] ║ 15 [22.5] ║        65 ║║ Female       ║ 20 [20] ║  15 [22.5] ║ 30 [22.5] ║        65 ║║ Column Total ║ 40      ║ 45         ║ 45        ║       130 ║╚══════════════╩═════════╩════════════╩═══════════╩═══════════╝
The formula to calculate Chi-square value or Χ 2 is:
where Χ is not the english aphabet we know but the 22nd Greek alphabet Chi.As shown, Χ 2 is the summation of the squared difference between Observed and Expected frequencies divided by the Expected frequency for all the cells. The calculations are shown below:
Χ2 = ((20-20)^2/20) + ((30-22.5)^2/22.5) + ((15-22.5)^2/22.5) + ((20-20)^2/20) + ((15-22.5)^2/22.5) + ((30-22.5)^2/22.5)Χ2 = 0 + 2.5 + 2.5 + 0 + 2.5 + 2.5Χ2 = 10
The degrees of freedom can be calculated as:
df = (total_rows - 1) * (total_cols - 1)
There are 2 rows and 3 columns in the contingency table, hence our degrees of freedom is 2.
We can see the Chi Square distribution tables like this one to find the p-value using the Chi Square and degrees of freedom values.
If you opened the link to the table, choose the degrees of freedom value (2) on the left, follow along its row to the closest number to the Chi-Square value (10), and then check the corresponding number in the first row to get the p-value which is 0.005
There are also some websites that calculate the p-value for you like this one.Here we get the p-value as .0067, we can use any one of them, let’s go ahead with the one obtained from the table.
In simple words if our p value is less than the significance value we reject the Null Hypothesis and if our p value is greater than the significance value we do not reject it.
Since 0.005 is less than our significance value of 0.05 we reject the Null Hypothesis which means that there is as association between Gender and the sport of interest.
# required importsfrom scipy.stats import chi2_contingency# we create contingency table same as abovetable = [[20, 30, 15], [20,  15,  30]]# Get chi-square value , p-value, degrees of freedom, expected frequencies using the function chi2_contingencystat, p, dof, expected = chi2_contingency(table)# select significance valuealpha = 0.05# Determine whether to reject or keep your null hypothesisprint('significance=%.3f, p=%.3f' % (alpha, p))if p <= alpha:    print('Variables are associated (reject H0)')else:    print('Variables are not associated(fail to reject H0)')
For simplicity I created a contingency table sing a list of lists. To create a contingency table out of your dataframe columns you can use this crosstab function available in pandas.
We have seen how to perform a Chi-Squared Test to check if there some relation between two categorical variables or not.
In very few lines we can implement this test using Python’s scipy library.
Hope you liked it. Suggestions are always welcome.