How to Test for Statistically Significant Relationships Between Categorical Variables with Chi Square | by Lisa Chen | Towards Data Science

A while back, I was analyzing market research data for a survey my company released to gauge customer sentiments towards a new product idea. Many of the variables were categorical (i.e. male, female, which of these features is most / least appealing) and after some initial research, I decided a chi-squared test for independence would be the best way to discern if there were certain segments of the population more or less likely to like a feature.
However, I noticed many of the tutorials for implementing post hoc testing were in SPSS or R, with almost no native packages in Python. This inspired me to put together a Python tutorial for chi square test for independence and post hoc testing to understand which categories are significant and relationship directionality (i.e. more or less likely to want a feature).
To demonstrate, I will be using the results of a survey where 8,415 Americans rated how comfortable they would be with getting on a domestic flight within the next month.
They survey results allowed me to slice by age and political affiliation, so my 2 research questions were:
Does age group impact attitudes towards flying (ATF)? If so, how?Does political affiliation impact attitudes towards flying (ATF)? If so, how?
Does age group impact attitudes towards flying (ATF)? If so, how?
Does political affiliation impact attitudes towards flying (ATF)? If so, how?
A short explanation of how the chi-squared test works is you first assume the null hypothesis, which is no relationship between variable A (i.e. age) and variable B (i.e. ATF). You calculate the expected value for each cell based on the distribution of your variables, and compare with the observed value. The assumption here is that any variance in observations is from distributions in the data, not any underlying relationship.
As an example, let’s calculate the expected value of respondents between 18–24 selecting “Very Comfortable.” 19.76% of all respondents were 18–24, and 12.77% of all respondents said they were very comfortable with flying soon.
Expected Value = 19.76% x 12.77% x 501 (all observations) = 12.64. So in this case, the observed value was almost spot on for the expected value.
The difference between the observed value and the expected value is used to calculate the chi-squared statistic, which — in conjunction with degrees of freedom, is used to determine if the difference between observed and expected is significant or due to sampling error. (More in depth detail on chi-square testing here).
I decided to tackle the relationship between age and ATF first. I pivoted my dataset to create a contingency table, which is the format accepted by the chi2_contingency SciPy method.
import pandas as pdimport numpy as npfrom scipy.stats import chi2_contingencyfrom scipy.stats import normimport math# Reading in my csv with age and attitudes towards flyingraw_df = pd.read_csv(“age_flying.csv”)# Pivoting the dataframe and assigning Age as the index, Status as the column headings, and percentage of respondents as the valuespivot_df = raw_df.pivot(index=’Age’, columns=’Status’, values=’Percentage’)
At this point, my dataframe looked like the following:
Nothing stood out to me as being particularly higher or lower than the expected value for the cell, but I decided to run the test just in case.
chi2, p, dof, ex = chi2_contingency(pivot_df, correction=False)print(chi2, '{:.10f}'.format(p))
My output was 14.885706463917337 0.5330249996, which means my p-value was 0.53 and I fail to reject the null hypothesis (H0: there is no relationship between age and attitude towards flying in the next month).
I moved on to my next research question to investigate if there is a relationship between political affiliation and ATF.
I ran the same process listed above, but the output of my chi-squared test this time was 79.59995957590428 0.0000000002. My p-value was significantly under the commonly accepted 0.05 alpha value, which means we’re in business!
At this point, all I know is that there is some sort of statistically significant relationship between political affiliation and ATF. Post hoc testing will enable me to understand which categories are contributing to this significance, and how.
My process is:
For each combination of political affiliation and ATF, calculate the adjusted standardized residuals between the observed values and the expected values. Standardized residuals is another name for z-scores. Adjustment means I adjust for different frequency counts between columns and rows.Calculate the p-values of the z-scores, which will tell me how probable the difference between the observed and expected value isApply a p-value correction to account for running multiple tests on the same dataset, which typically leads to Type 1 errors.
For each combination of political affiliation and ATF, calculate the adjusted standardized residuals between the observed values and the expected values. Standardized residuals is another name for z-scores. Adjustment means I adjust for different frequency counts between columns and rows.
Calculate the p-values of the z-scores, which will tell me how probable the difference between the observed and expected value is
Apply a p-value correction to account for running multiple tests on the same dataset, which typically leads to Type 1 errors.
The formula to calculate adjusted standardized residuals is as follows:
First, I turned the pivot table back into a regular dataframe. I did this because pivot tables have less functionality and I can’t access columns or rows by name.
observed_vals = pivot_df.reset_index(drop=True)observed_vals = observed_vals.rename_axis(None, axis=1)observed_vals.set_index([["Dem", "Ind", "Not_sure", "Other", "Rep"]], inplace=True)observed_vals = observed_vals.reset_index()observed_vals.rename(columns={"index": "Party"}, inplace=True)
Next, I created a new dataframe with the expected values. Luckily, the chi2_contingency method outputs an array of expected values.
expected_vals = pd.DataFrame(ex)expected_vals.set_index([["Dem", "Ind", "Not_sure", "Other", "Rep"]], inplace=True)expected_vals = expected_vals.reset_index()expected_vals.rename(columns={"index": "Party", 0: "SW_Comfortable", 1:"SW_Uncomfortable", 2:"V_Comfortable", 3:"V_Uncomfortable", 4:"dont_know"}, inplace=True)
Then, I created two tables for the row-wise totals and column-wise totals. These values are the n in the (1- RowMarginal/n) * (1 - ColumnMarginal/n) part of the equation.
# Creating a column-wise sum table, adding back the column with party namescol_sum = pd.DataFrame(observed_vals.sum(axis=1), columns=["count"])col_sum.insert(0, "Party", ["Dem", "Ind", "Not_sure", "Other", "Rep"], True)# Creating a row-wise sum tablerow_sum = pd.DataFrame(observed_vals.iloc[:,1:].sum(axis=0), columns=["count"]).reset_index()row_sum.rename(columns={"index":"Status"}, inplace=True)
At this point, I had all of the information I needed to start calculating adjusted standardized residuals (try saying that 10 times, fast). I wrote a script to iterate through every combination of political affiliation and ATF, and calculate the adjusted standardized residual and accompanying p-value.
I applied a Bonferroni Correction to the p-value, which lowered my p-value threshold to 0.002 to reject the null hypothesis (0.05 / 25 comparisons). P-value corrections are applied when multiple tests are done on the same dataset (25, in my case), which increases chances of false positives.
# Find all the unique political parties for pairwise comparisonparties = list(raw_df["Party"].unique())# Find all the unique attitudes towards flying for pairwise comparisonstatus = list(raw_df["Status"].unique())# Iterate through all combinations of parties and statusfor p in parties:    for s in status:        observed = float(observed_vals.loc[observed_vals.Party == p][s].values[0])        expected = float(expected_vals.loc[expected_vals.Party == p][s].values[0])        col_total = float(col_sum[col_sum["Party"] == p]["count"].values[0])        row_total = float(row_sum[row_sum["Status"] == s]["count"].values[0])        expected_row_prop = expected/row_total        expected_col_prop = expected/col_total        std_resid = (observed - expected) / (math.sqrt(expected * (1-expected_row_prop) * (1-expected_col_prop)))        p_val = norm.sf(abs(std_resid))        if p_val < 0.05/25:            print(p, s, std_resid, p_val)
All of you are probably cringing at the double for-loop, but whatever — it gets the job done, okay? I configured the script to only print relationships with a p-value less than the corrected p-value, and the following turned out to be statistically significant at p < 0.002:
In this article, I ran briefly through the process of conducting a chi-squared test for independence and post hoc test, focusing primarily on the technical implementation. However, I think understanding the theory behind the implementation is crucial for debugging and interpreting results, so I linked resources that I personally found helpful so you can review if necessary.
Let’s circle back to my two original research questions and see what conclusions we dug up:
Does age group impact attitudes towards flying (ATF)? If so, how? NoDoes political affiliation impact attitudes towards flying (ATF)? If so, how? Yes, and quite significantly so! Democrats are more likely to have a “Very Uncomfortable” ATF compared to the average, Republicans are more likely to have a “Very Comfortable” ATF compared to the average, people who are not sure of their political affiliation are less likely to have a “Very Uncomfortable” ATF, but more likely to put “Don’t know.”
Does age group impact attitudes towards flying (ATF)? If so, how? No
Does political affiliation impact attitudes towards flying (ATF)? If so, how? Yes, and quite significantly so! Democrats are more likely to have a “Very Uncomfortable” ATF compared to the average, Republicans are more likely to have a “Very Comfortable” ATF compared to the average, people who are not sure of their political affiliation are less likely to have a “Very Uncomfortable” ATF, but more likely to put “Don’t know.”
I found the last insight to be very interesting — people who do not have strong political affiliations are more likely to not have an opinion towards flying. One possible explanation could be respondents just breezed through questions and didn’t put much thought into their response, which led to selecting the most neutral answers (i.e. not sure, don’t know).
Another, more interesting explanation could be that people who are politically engaged are more likely to watch the news and regularly consume media about current events. Since many media outlets are focusing on COVID-19, people who stay up-to-date will be exposed to more debates between authority figures with strong opinions about pandemic best practices (i.e. public health and safety protocol, travel bans). Conversely, those without strong political affiliations may tune in less compared to their partisan peers, and therefore are exposed to less polarizing opinions on how best to handle travel.
At this point, however, that is purely conjecture on my part. Now it’s up to you guys to calculate the adjusted standardized residual for my hypothesis and let me know if I’m right 😉