Chi-squared tests to compare two machine learning models and determine whether they are random | by shashank kumar | Towards Data Science

A challenging classification problem has grappled you to your screen. You have been pouncing the keyboard for hours, analyzing data, debugging code, and executing algorithms. Your endeavors blossom into two machine learning models. But there’s a last hassle before you can smack the display down and gulp a cold one. You must evaluate the models. Often, evaluation goes beyond merely computing the test accuracy. You’d want to probe whether the performances of multiple models differ. If there are variations, are they statistically significant, or is it due to randomness? So let’s hunt these answers together. If you’re familiar with probability distributions (normal, binomial, and chi-squared), hypothesis testing, false positives, and false negatives, we’re good to go.
Imagine your classification problem was to predict whether a person was suffering from covid. The first model you constructed, M1, gave the following result.
Probabilistically speaking, there’s a 65% chance a data instance is positive and a 35% chance otherwise. One important question would be, is the model more effective than a random guesser? Random guessing entails that the predicted and actual classes are independent of each other. Essentially, such a model won’t learn anything valuable from data and merely make guesses. From the laws of independent probability, it then follows:
In chi-square tests, we pit the observations against the expected values. M1 labels 69 instances as positive. Now, if it is a random guesser, we can expect 35% of those instances to be false positive. Why? Because there’s a 35% chance that a test data instance is negative. This result can also be derived from the above equation with a slightly modified question. How many of the 100 instances would you expect to be false positives? In that case, we find P(Predicted=+ and Actual= — ). P(Predicted=+)= 69/100 = 0.69.
25.15% of the total data instances are likely to be classified as false positive if M1 is a random guesser. Hence, the number of false positives = 24.15% of 100=24.15. Similarly, we can find all the expected predictions of a random guesser.
Running a chi-squared independence test on the above two tables will verify whether M1 is a random guesser. It does so by computing a chi-squared statistic. Okay, what’s that?
I’ll prove the above equation is Chi-squared in the next section. The rest of the test is like any other hypothesis test.
H0(Null hypothesis): M1 is a random guesser
H1(Alternate hypothesis): M1 is not a random guesser
The M1 table lists the observed values, while the Random Guesser table lists the expected values. In other words, for our null hypothesis, we expect M1 to yield the results given in the RM table.
To find the value of a Chi-squared random variable, we also require DOF (degrees of freedom). It’s the number of values free to vary in a data table. While calculating the probabilities of actual and predicted class, we fixed four numbers — the number of actual positives and negatives, and the number of predicted positives and negatives.
If you now observe the above table, you’ll realize that if I give you any one value among the set-A, B, C, D-you can find the remaining variables. Ergo, two variables/values in the dataset are free to vary, so DOF=1.
From the chi-square statistic equation, we can find the test statistic is:
Before moving ahead, I would like to unravel the mathematics of the Chi-squared test. Feel free to skip if you aren’t a math buff.
If you’re still in this section, I believe you’re twitching to know why the above term is a Chi-squared variable. Let’s reformulate the problem. We’re looking for evidence of whether M1 is a random guesser. To do so, we hypothesize that it is a random guesser and tabulate the expected values. The values in the M1 table are what we observe, and we test the two sets of values using the Chi-squared test.
Fine, now observe the random guesser table. Think of each side, Predicted(+) and Predicted(-), like coins. For example, Predicted(+) is a coin with heads denoting positive and tails denoting negative. Ergo, if you toss the Predicted(+) coin and it lands heads, the data instance is classed as true positive. Else, if it lands tails, the data instance is classified as false positive. Does this remind you of a particular probability distribution?
Take T as the random variable describing the number of true positive instances (Heads by the coin analogy). Doesn’t T follow a binomial distribution with p=0.65 (probability of positive) and n=69 (total data instances)? Sure it does. I assume you know that binomial distribution can be approximated from a normal distribution, provided both np and n(1-p) exceed 10. Accordingly, we get a normal variable z~N(0,1)
We can manipulate the above term to get
That’s it we have dug the hidden Chi-squared terms. Remember that Z2 follows a Chi-squared distribution. Hence, the RHS is chi-square distributed. In the first term, T is the number of true positives, which we observe to be 58 and np=69 times 0.65=44.85. Similarly, in the second term, n-T is the number of false positives, which equals 11, and n(1-p) equals 24.15. Replace the variables with these values.
You may now consider the Predicted(-) as another coin and repeat the grunt work. Otherwise, believe me when I say that your troubles will yield the following:
There you go. The two Z2 terms compose the Chi-squared statistic.
You may run the below code in python to carry the remaining calculations. I’ll discuss the results.
import numpy as npimport pandas as pdfrom scipy.stats import chi2_contingencydata=[[58,7],[11,24]] #Model M1 table#Chi square statistic,pvalue,DOF,expected tablestat, p, dof, expected = chi2_contingency(data) print('Chi-square statistic=',stat)print('Pvalue=',p)alpha=0.05if p < alpha:    print('Not a random guesser')else:    print('Model is a random guesser')
Results:
Chi-square statistic= 32.884319981094166Pvalue= 9.780899116984747e-09
From the results, you can observe that the chi-square statistic is exceedingly high. Accordingly, compared to the alpha (0.05), the probability (p-value) of our null hypothesis standing true is insignificant. Thus, we can conclude that M1 is not a random guesser. Although, note that the final decision entirely depends on the alpha value. But as we generally take alpha as 0.05, we can be confident with our finding.
All right, we’re done evaluating the first model you made. Let’s run some tests on the second model.
You may carry a separate — ‘is it a random guesser’ — test for this model too. Although, it’s evident from the above values that M2 is more accurate than M1. It is impossible, hence, for M2 to be a random guesser. But does it genuinely outperform M1, or do the slight improvements stem from random chance. To find that, we’ll have to modify our approach.
The above table presents the distribution of correct and incorrect predictions by M1 and M2. (I have taken these values arbitrarily. For a real problem, you can find them easily by executing few lines of code.). Nothing is compelling about M1 and M2 making identical predictions (correct/incorrect). We are curious about their differences, so we have to heed the data fields where one model classifies correctly while the other does not (B and C).
If M1 and M2 were performing alike, values B and C should not differ much. In that scenario, both models will make nearly equal proportions of correct and incorrect predictions and consequently indicate homogeneity.
Now, by imagining B and C as two faces of a coin, we can carry a Chi-squared test to determine whether the values are different. We already established that the DOF for the above table is 1. The Chi-squared statistic is:
Again, do not itch to figure why the above is a chi-square variable. It can be proved by repeating what we did in the last section. You may try it if you desire. This particular method of differentiating two models is better known as Mcnemar’s test. Let’s carry with the test now and establish the hypotheses.
H0: M1 and M2 are the same
H1: M1 and M2 are different
import numpy as npimport pandas as pdfrom statsmodels.stats.contingency_tables import mcnemardata=[[77,5],[10,8]]res = mcnemar(data, exact=False)alpha=0.05print('Chi-square statistic:',res.statistic)print('Pvalue:',res.pvalue)if res.pvalue < alpha:    print('Models are different')else:    print('Models are same')Chi-square statistic: 1.0666666666666667Pvalue: 0.30169958247834494
Here, the P-value exceeds alpha, so we cannot discard H0. Hence, although M2 seemingly performs better than M1, the variations in performance are most likely due to random chance. We ought to deduce from the results to avoid arriving at any conclusion by solely analyzing classification accuracies.
In this article, I just discussed two types of evaluations. There are statistical procedures for evaluating umpteen models over multiple problems. ROC curves are extremely nifty when judging numerous machine learning algorithms. I have linked some resources for you to have a more comprehensive review of the above explanations.