Fisher’s exact test from scratch with Python | by Tobias Roeschl | Towards Data Science

In this article, I want to give a brief review of Fisher’s exact test and then continue to show its implementation from scratch with Python. After that, we want to validate our result by comparing it to the output we get from Python’s built-in function.
Sir Ronald Fisher (17 February 1890 - 29 July 1962) was arguably one of the most famous statisticians of the 20th century. One of his greatest contributions to modern statistics was “Fisher’s exact test”.
Fisher’s exact test is used to determine whether there is a significant association between two categorical variables in a contingency table. Fisher’s exact test is an alternative to Pearson’s chi-squared test for independence. While actually valid for all sample sizes, Fisher’s exact test is practically applied when sample sizes are small. A general recommendation is to use Fisher’s exact test- instead of the chi-squared test - whenever more than 20 % of cells in a contingency table have expected frequencies < 5.1
In contrast to other statistical tests (e.g. z-test, chi-squared test), where p-values are calculated using an approximation to the true distribution (e.g. normal distribution, χ2-distribution) Fisher’s exact test yields exact p-values, hence the name. That’s why some authors prefer Fisher’s exact test over Pearson’s chi-squared test whenever (computationally) possible — even with larger sample sizes.2
Fisher’s exact test is used to calculate the probability that proportions as or more extreme than the proportions we observed are caused by random chance— given these marginal totals of the contingency table. So we assume that the margins are fixed and use permutations and combinations to determine the probability of every possible contingency table using the following formula:
The formula above gives the exact hypergeometric probability p of observing this arrangement of data in a particular contingency table, assuming the given margins, on the null hypothesis of independence.
Fisher introduced his test by the “Lady tasting tea”- experiment3: At a tea party, a female colleague of Fisher’s claimed that she could distinguish whether tea or milk was added to the cup first. Fisher proposed to test the lady’s hypothesis with an experiment. He prepared eight cups of tea, four of each with milk or tea first. He then presented the eight cups to the Lady and asked her to determine which cups were filled with milk or tea first, respectively. She knew that there were four cups of each type but did not know the order. The null hypothesis (H0) was that the Lady’s guessing abilities were no better than chance, in other words, the row and the column variable were independent. The alternative hypothesis (Ha) was that she would guess systematically more right or wrong than expected by chance alone. The following result was obtained:
Since the margins are fixed by the experimental design — the Lady knew that there were 4 cups where milk was poured in first and 4 cups where tea was poured in first — we have one degree of freedom and can, therefore, complete an individual contingency table if we know the number of milk poured first cups she got correct. To calculate the probability to obtain, let’s say the contingency table where three milk poured first cups were identified correctly, we are using permutations and combinations. There are “8 choose 4” possible ways to pick four of the milk poured first cups. This will be our denominator. There are “4 choose 3” possible ways to guess three out of four milk first cups correctly. Then there are “4 choose 1” possible ways to incorrectly pick one of the four tea poured first cups as a milk first cup. And because these are independent, we get the following probability:
Now let’s calculate the p-value which is the probability of observing this or an even more extreme contingency table under the null hypothesis. Therefore, we will have to determine all possible contingency tables and their respective probabilities given our fixed margins. This is what we get:
Finally, we can calculate our p-value by adding up all those probabilities p(a) where p(a)≤p(3), because it was the corresponding contingency table to p(3) which was observed in the “Lady tasting tea” experiment. This will give us the following p-value:
Since the p-value comes out very high, we have no evidence to reject our null hypothesis. Although it is technically possible to report a one-sided p-value with Fisher’s exact test, it is recommended to always report the two-sided p-value.4
Now let’s consider the following example to implement Fisher’s exact test from scratch with Python. Let’s assume we are running a trial involving a new drug X to treat a particular disease and we want to determine if there are gender-related differences in responsiveness to this drug. Here is the contingency table that we observed after the trial:
From a first glance at the data, we might think that females and males respond differently to the drug X. But how do we know that this observation was not just solely due to chance? Let’s first state a null hypothesis H0: The gender of the patient and the responsiveness to the drug X are stochastically independent. Next, let’s calculate the chance of getting this or an even more extreme contingency table under the condition that the margins of the contingency table are fixed and the null hypothesis is true.
Let’s import some Python libraries:
import numpy as np
import pandas as pd
import scipy
import scipy.special

Next, we create the observed contingency table with pandas:
ar=np.array([[7.0, 2.0],[1.0, 12.0]])    
df=pd.DataFrame(ar, columns=["Responded", "No Response"])
df.index=["Males", "Females"] 
df 

... and add the marginal totals and the grand total:
df2=df.copy()
df2.loc['Column_Total']= df2.sum(numeric_only=True, axis=0)
df2.loc[:,'Row_Total'] = df2.sum(numeric_only=True, axis=1)
df2

Thereafter, we want to calculate the expected values for each cell in our contingency table. The expected values represent the numbers that should be found in each cell of the table if there’s no association between the two variables “gender” and “response to drug”. For each cell we calculate its expected value by multiplying its respective Row_Total with its respective Column_Total and finally divide by the grand total. For the cell in the upper left, the following would result:
In Python:
n=df2.at["Column_Total", "Row_Total"]  # grand total 

exp=df2.copy()               # create dataframe with expected frequencies
for x in exp.index[0:-1]:
    for y in exp.columns[0:-1]:
        # round expected values to nearest hundredths:
        v= (((df2.at[x, "Row_Total"]) * (df2.at["Column_Total", y])   )   /n ).round(2) 
        exp.at[x,y]=float(v)

exp

More than 20 % of cells in our contingency table have expected frequencies < 5, so we have to apply Fisher’s exact test. It is recommended to use floats when setting up the ‘original’ contingency table because Panda’sDataFrame.at -function somehow conserves the original datatype. Alternatively, we could use pandas.DataFrame.loc.
Now we want to define a function p(a) that takes the value of the upper left cell as an input and returns the probability of observing this particular contingency table given the fixed margins. We use Panda’s DataFrame.iloc-function to avoid label-location based indexing and Scipy’s special.binom-function to implement binomial coefficients.
After that we calculate the probability p(a) for every possible value of “a” given our fixed margins. We then take those probabilities where p(a)≤p(a=7) and add these values up to finally get our p-value.
Lastly, we want to validate our result by comparing it to the output we get from Python’s built-in function. Therefore, we import scipy.stats ...
... and are delighted to note that both p-values are identical.
The complete jupyter notebook can be found on my GitHub: https://github.com/BundleOfKent/Fisher-s-exact-test-from-scratch/releases/latest
References:
Kim H. Y. (2017). Statistical notes for clinical researchers: Chi-squared test and Fisher’s exact test. Restorative dentistry & endodontics, 42(2), 152–155. doi:10.5395/rde.2017.42.2.152Hess, A.S. and Hess, J.R. (2017), Understanding tests of the association of categorical variables: the Pearson chi‐square test and Fisher’s exact test. Transfusion, 57: 877–879. doi:10.1111/trf.14057Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.Meulepas, E. (1998), A Two‐Tailed P‐Value for Fisher’s Exact Test. Biom. J., 40: 3–10. doi:10.1002/(SICI)1521–4036(199804)40:1❤::AID-BIMJ3>3.0.CO;2-T
Kim H. Y. (2017). Statistical notes for clinical researchers: Chi-squared test and Fisher’s exact test. Restorative dentistry & endodontics, 42(2), 152–155. doi:10.5395/rde.2017.42.2.152
Hess, A.S. and Hess, J.R. (2017), Understanding tests of the association of categorical variables: the Pearson chi‐square test and Fisher’s exact test. Transfusion, 57: 877–879. doi:10.1111/trf.14057
Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.
Meulepas, E. (1998), A Two‐Tailed P‐Value for Fisher’s Exact Test. Biom. J., 40: 3–10. doi:10.1002/(SICI)1521–4036(199804)40:1❤::AID-BIMJ3>3.0.CO;2-T