McNemar’s exact test from scratch with Python: Evaluation of SARS-CoV-1 diagnostic test performance | by Tobias Roeschl | Towards Data Science

This article is about McNemar’s exact test and its implementation in Python. I want to begin with indications and test theory of McNemar’s exact test before we will address how to perform the test from scratch with Python. We will use real data from the SARS coronavirus epidemic in 2003 as an example. At the end, we want to validate our result by comparing it to the result we obtain with Python’s built-in function.
McNemar’s exact test is a statistical test used for correlated, binary data in 2x2 tables. McNemar’s exact test is the dichotomous equivalent to the paired t-test with the latter being applied to continuous variables. The test was published by American Professor of Psychology Quinn McNemar in 1947.1 Common applications for McNemar’s exact test in healthcare are datasets, which contain dependent (e.g. matched) pairs. These include pretest-posttest studies, case-control studies (e.g. twin studies) and statistical comparisons of diagnostic tests.
To introduce the math behind the test, I want to use a brief example: Let’s assume we want to examine whether there is an association between smoking and lung cancer. We therefore match 42 patients with lung cancer (=cases) to 42 control persons (= controls) based on age and family history of lung cancer. Next we obtain the smoking status of each of the 84 individuals. We get the following table:
Out of 42 matched pairs, there were 21 pairs where both, the healthy individual (=control) and the patient with lung cancer (=case), were smokers, 8 pairs where the lung cancer patient was a smoker but the control was not, 12 pairs where both were non-smokers and 1 pair where the cancer patient did not smoke but the control person did.
Note: This is not a ‘typical’ contingency table, since cells reflect pairs and not individuals! Consequently, we cannot use the usual analyses of contingency tables like Pearson‘s chi-squared test or Fisher‘s exact test, which both assume independent samples!
The 21 pairs, in which both cases and controls were exposed to the risk factor, provide no information about the association between smoking and lung cancer. Similarly, the 12 pairs, in which neither case nor control were exposed to the risk factor, provide no further information.
McNemar’s test solely uses data from the two discordant cells (“Case Smoker | Control Non-smoker” and “Case Non-smoker | Control Smoker”) in the table. If there was no association between smoking status and lung cancer, we would intuitively expect discordant pairs to be distributed roughly equally among these two cells. In other words, the discordant pairs could either fall in one discordant cell or in the other with equal chances (p=0.5) if our variables of interest are independent from each other (= null hypothesis).
The probability of observing a particular distribution of the 9 discordant pairs among both cells under the null hypothesis can be calculated with a binomial. In our example we get:
In hypothesis testing we are generally interested in the probability of observing a particular outcome — or an even more extreme outcome — under the null hypothesis. An even more extreme distribution of discordant pairs in this study would correspond to 9 pairs in “Case Smoker | Control Non-smoker” and zero pairs in “Case Non-smoker| Control Smoker”:
McNemar’s test is usually performed as a two-sided test because theoretically, all discordant pairs could also be found on the “Case Non-smoker | Control Smoker”-side of the table. That’s why we also have to add
and
to the two probabilities we calculated in the first place resulting in a two-sided p-value of 0.0176+0.00195+0.0176+0.00195= 0.039. Consequently, we would reject the null hypothesis of independence at the 0.05 significance level in this example.
We could have simplified the calculation of our p-value, as it can be mathematically shown, that
with b and c being the counts of the discordant cells in our table.
McNemar’s exact test is a non-parametric test, that has to be differentiated from McNemar’s chi-squared test, which uses a chi-square approximation to the binomial distribution. Like Fisher’s exact test, McNemar’s exact test is applicable with any sample size, but is obligatory with sparse data. The general rule of thumb is to use the exact test whenever the number of discordant pairs is less than 10.2
In late 2002 a new viral illness, which would later become known as severe acute respiratory syndrome (SARS), appeared in south-eastern China and rapidly spread to over 8,000 people worldwide causing 774 deaths. Due to global effort, the virus could be finally contained in 2004. The virus responsible for the outbreak in 2002 (SARS-CoV-1) is genetically closely related to the SARS-CoV-2 virus causing the ongoing pandemic of coronavirus disease 2019 (COVID-19).
One key measure to contain the spread of SARS in the early 2000s was rapid detection of human SARS cases. Like today, the quantitative RT-PCR assay was the most sensitive diagnostic tool for SARS CoV detection. However, a major disadvantage of PCR-based methods has been the necessity for a non-portable, laboratory setting.3 This can lead to RT-PCRs being impracticable in remote areas. A faster and logistically simpler alternative to PCR-based methods is the reverse transcription loop-mediated amplification test (RT-LAMP), which can be used in disease outbreaks in remote areas.
To compare the detection rates of both methods (quantitative RT-PCR vs. RT-LAMP), researchers conducted the following study4: Retrospective nasopharyngeal aspirates of 59 confirmed cases of SARS Cov were tested using both methods respectively. The following table was obtained:
At first sight, we can see that RT-LAMP correctly identified 42/59=71 % aspirates as cases compared to 46/59=78 % in RT-PCR. Looking at the data, we might be deluded into thinking that RT-LAMP might have inferior detection rates.
We now want to use Python to find out whether both diagnostic tests differ significantly regarding their detection rates. Since we are dealing with a binary outcome (“positive/negative”) and both diagnostic tests were repeated on the same nasopharyngeal aspirate respectively, we have to use McNemar’s test. The sum of discordant pairs is equal to 2+6=8. As our number of discordant observations is < 10, we have to use McNemar’s exact test.
In Python we start by importing some useful libraries:
import numpy as np
from statsmodels.stats import contingency_tables
import pandas as pd
from scipy.stats import binom

... and plug in our data:
ar=np.array([[40.0, 6.0],[2.0, 11.0]])    # Create DataFrame by passing a numpy array with our data. 
ct=pd.DataFrame(ar, columns=["RT-LAMP positive", "RT-LAMP negative"])
ct.index=["RT-PCR positive", "RT-PCR negative"] 
ct

Then we use a while loop to calculate the corresponding binomial probabilities, just like we did in the example before.
alist = []    # create list where binomial probabilites will be added to after each loop.

c= ct.iloc[1,0] # get cell counts of discordant pairs
b= ct.iloc[0,1]
N=ct.iloc[0,1]+ ct.iloc[1,0] # ... and their sum.


# the way we set up our while-loop makes it necessary to loop over the larger of the two counts b and c! 
if b>=c:   # ... otherwise we would not calculate tail probabilites in this setting!     
    while b<=N:
        P1= binom.pmf(b, N, 0.5, loc=0) # binomial probability
        alist.append(P1) # append values to list

        P2= binom.pmf(N-b, N, 0.5, loc=0) # calculate binomial probability for the other "tail" within the same loop.
        #( The calculation could be done more elegantly since P1=P2 by the way ...) 
        alist.append(P2)

        b=b+1 # increase b with every loop to calculate probabilites for the "even more extreme" outcomes...
        #... until it reaches the total number of discordant pairs N. 
    
else: # If c>b: 
    while c<=N: 
        P1= binom.pmf(c, N, 0.5, loc=0)
        alist.append(P1)

        P2= binom.pmf(N-c, N, 0.5, loc=0)
        alist.append(P2)

        c=c+1

alist

[0.10937500000000006,
 0.10937500000000006,
 0.031250000000000014,
 0.031250000000000014,
 0.003906250000000001,
 0.003906250000000001]
The sum of these binomial probabilities is our p-value:
Statsmodels is a Python module, that offers a built-in function to apply McNemar’s exact test directly:
As we can see, both statsmodels’ function and our own Python code give us a p-value of 0.29, which is essentially what was reported in the original paper.4 The authors concluded that the detection rates overall were not statistically significant from each other. Today LAMP-based methods are used in addition to PCR-based tests in the global effort to stop COVID-19.5
The complete jupyter notebook can be found on my GitHub!
McNemar, Quinn (June 18, 1947). “Note on the sampling error of the difference between correlated proportions or percentages”. PsychometrikaRufibach, K. Assessment of paired binary data. Skeletal Radiol 40, 1–4 (2011). https://doi.org/10.1007/s00256-010-1006-1Lee, Patricia. (2017). DNA amplification in the field: move over PCR, here comes LAMP. Molecular Ecology Resources. 17. 138–141. 10.1111/1755–0998.12548.Poon LL, Wong BW, Chan KH, et al. Evaluation of real-time reverse transcriptase PCR and real-time loop-mediated amplification assays for severe acute respiratory syndrome coronavirus detection. J Clin Microbiol. 2005;43(7):3457–3459. doi:10.1128/JCM.43.7.3457–3459.2005Minzhe Shen, Ying Zhou, Jiawei Ye, Abdu Ahmed Abdullah AL-maskri, Yu Kang, Su Zeng, Sheng Cai, Recent advances and perspectives of nucleic acid detection for coronavirus, Journal of Pharmaceutical Analysis,2020, ISSN 2095–1779, https://doi.org/10.1016/j.jpha.2020.02.010.
McNemar, Quinn (June 18, 1947). “Note on the sampling error of the difference between correlated proportions or percentages”. Psychometrika
Rufibach, K. Assessment of paired binary data. Skeletal Radiol 40, 1–4 (2011). https://doi.org/10.1007/s00256-010-1006-1
Lee, Patricia. (2017). DNA amplification in the field: move over PCR, here comes LAMP. Molecular Ecology Resources. 17. 138–141. 10.1111/1755–0998.12548.
Poon LL, Wong BW, Chan KH, et al. Evaluation of real-time reverse transcriptase PCR and real-time loop-mediated amplification assays for severe acute respiratory syndrome coronavirus detection. J Clin Microbiol. 2005;43(7):3457–3459. doi:10.1128/JCM.43.7.3457–3459.2005
Minzhe Shen, Ying Zhou, Jiawei Ye, Abdu Ahmed Abdullah AL-maskri, Yu Kang, Su Zeng, Sheng Cai, Recent advances and perspectives of nucleic acid detection for coronavirus, Journal of Pharmaceutical Analysis,2020, ISSN 2095–1779, https://doi.org/10.1016/j.jpha.2020.02.010.