The Most Important Statistical Test | by Wicaksono Wijono | Towards Data Science

The likelihood ratio test (LRT) “unifies” frequentist statistical tests. Brand-name tests like t-test, F-test, chi-squared-test, and so on are specific cases (or even approximations) of the LRT.
Thus, it is surprising that many people have never heard of LRT. When I help people with their statistical questions, most of the time they only need to perform LRT and they’re done. But then they ask what it is and how to perform it. If you must know a single statistical test by heart, it should be LRT.
This article will walk you through the theory and application of LRT. Basic understanding of mathematical statistics is assumed.
Though I lean Bayesian, any applied statistician must understand basic frequentist methods too. People often don’t have time to learn the theoretical overhead and superiors ask for p-values in reports.
As the name implies, we take the ratio of two likelihoods and transform it:
I apologize for the obtuse notation but it is consistent with the one in Wikipedia so please consult it for clarification. The numerator is the likelihood under the null hypothesis, while the denominator is the maximum likelihood under the union of the null and alternative hypotheses. I denote this quantity ΔD to mean the difference in deviance, -2 log likelihood + some constant (that cancels out from the subtraction), between the null model and the relaxed model. If this is unclear right now, don’t worry; as we work through some examples below, the LRT should become more clear.
Those who have seen AIC or BIC in passing might be puzzled by the seemingly arbitrary -2, but deviance is important because ΔD is asymptotically χ2-distributed with degrees of freedom (df) equal to the difference in df between the two models (the difference in number of estimated parameters). This amazing fact enables the majority of hypothesis testing.
The use of the LRT is justified by the Neyman-Pearson lemma, which states that the LRT is the most powerful test to compare two simple hypotheses, e.g. H0: θ = 0 and Ha: θ = 2.
Because the alternative hypothesis is usually not simple, like θ ≠ 0, we use maximum likelihood estimation (MLE) to reduce it to a simple one such as θ = 2. Commonly the MLE is done through (generalized) linear regression. Intuitively, if we cannot reject the null hypothesis at the MLE, then we cannot possibly reject at any other point, so the MLE is the only point that matters.
In human language, we are comparing two models: one with a constraint and one with a relaxation of the constraint. You often see it as
In the null model, we constrain the mean to be 0. In the alternative model, we allow the mean to be nonzero. We then need to specify a likelihood function. Here, to keep it familiar, we’ll posit that the observations come from a normal distribution with a known variance σ2 = 1. The MLE is the sample mean. Thus:
This looks gnarly, but the ΔD term looks familiar: regression sum of squares (SS)! (It should be scaled, but here the scale parameter is 1.) It can be re-expressed as total SS minus the residual SS, as in the second line. And the χ2 distribution is defined as the SS from a standard normal distribution, so this all makes sense. Because in the alternative model we estimate one extra parameter, ΔD follows the χ2 distribution with 1 df.
What if the variance is unknown? Then the null model estimates one parameter while the alternative model estimates two, so the difference in df is still 1. And in this case, we know this more familiarly as the ANOVA or F-test, which in this example is equivalent to the t-test on the intercept.
Remember my point about how all of these statistical tests with different names are actually the same thing (LRT)? Yeah. Here’s some R code to demonstrate the equivalence:
set.seed(123)x <- rnorm(50, 0.5, 1)model0 <- lm(x~0)model1 <- lm(x~1)anova(model0, model1)summary(model1)t.test(x)
You can think of deviance as a generalization of SS applied to distributions other than the normal. Though not completely accurate, the intuition is there. As an exercise, compare the binomial deviance to the log loss function that is sometimes optimized in classification models.
Other than testing sample means, we often want to test for independence in contingency tables. Statistics is riddled with many antiquated traditions and this is one of them.
Most would immediately jump to Pearson’s Chi-Squared test but this is merely an approximation (by Taylor expansion) of the G-test, a specific case of LRT. Before calculators, it was difficult to take logarithms so statisticians used the Chi-Squared test for computational ease. It is surprising that we now have calculators but still have not switched over to the G-test by default. (Another common alternative, Fisher’s Exact Test, provides exact p-values instead of relying on asymptotics but the factorials make it computationally impractical for large sample sizes.)
What is the likelihood? Under independence, we can think of the data as being generated from a multinomial distribution where the probability of falling in each cell is
With dependence, the probability of falling in each cell is
And we can compute the ΔD as:
which is equivalent to the test statistic in the Wikipedia page for G-test. This looks suspiciously like KL divergence, hm? Funny how many concepts are so interconnected with each other.
We can easily perform this test using Poisson regression: y~x1+x2 against y~x1+x2+x1*x2. (Equivalence to multinomial model left as an exercise. Oh no. I have become the thing I hated.)
When all the πN are large, Chi-Squared test and G-test yield similar p-values. The G-test should be preferable because the LRT is the most powerful test and the distribution of the test statistic more closely matches the χ2 distribution.
library(data.table)N <- 100 # play around with thisfactor1 <- sample(  c('A', 'B'),   N,   replace = TRUE,   prob = c(0.4, 0.6))factor2 <- ifelse(  factor1 == 'A',  sample(c('C', 'D'), N, replace = TRUE, prob = c(0.3, 0.7)),  sample(c('C', 'D'), N, replace = TRUE, prob = c(0.7, 0.3)))dat <- data.table(factor1, factor2)xtab <- table(dat)dat_count <- dat[,.N, by = .(factor1, factor2)]model_indep <- glm(  N~factor1+factor2,   data = dat_count,   family = 'poisson')model_dep <- glm(  N~factor1*factor2,   data = dat_count,   family = 'poisson')summary(model_indep)summary(model_dep) # the fitted match the observed so deviance = 0pchisq(model_indep$deviance, df = 1, lower.tail = FALSE) # G-testchisq.test(xtab) # Chi-Squared test
The above is the general procedure for LRT. We fit a simpler regression model and compare the deviance against a more complex model.
From the two previous examples, hopefully, it was clear:
Compute the maximum log likelihood under the constrained null modelCompute the maximum log likelihood when the constraint is relaxedCompare ΔD against the χ2 distribution with the appropriate df to get the p-value
Compute the maximum log likelihood under the constrained null model
Compute the maximum log likelihood when the constraint is relaxed
Compare ΔD against the χ2 distribution with the appropriate df to get the p-value
The model in 1) has to be nested under 2). After all, that’s what relaxing constraints means. Otherwise, ΔD doesn’t have the theoretical guarantees.
For instance, ΔD of y~x1 against y~x1+x2 is fair game because in the first model we constrain b2 = 0 while in the second model we relax this constraint.
However, ΔD of y~x1 against y~x2 is meaningless. In the first model we don’t have the constraint b1 = 0 while in the second model we do have the constraint b1 = 0. For b2, it’s vice versa.
Sometimes we have hypotheses like H0: θ ≤ 0 and Ha: θ > 0. To make sure the models are simple and nested, the null model should be the MLE when θ ≤ 0 while the alternative model should be MLE when θ is any real number.
Computing MLE by hand is cumbersome and most of the time the MLE does not have closed-form solutions. Luckily, we can harness the power of computers.
For linear models, statistical packages have built-in functions to compute MLE for GLMs, which cover most of the hypotheses you want to test. (Be careful! In Python, sklearn’s logistic regression function does NOT compute the MLE and will yield incorrect deviances. Use statsmodels instead.) I’d argue that GLM is the most important skill for applied statisticians because so many inferential tasks can be framed as GLM problems.
For more complicated likelihood functions (e.g. censoring and truncation), you need to write down the function and perform numerical optimization.
You can abandon dozens of different statistical tests and replace them with the LRT (mostly through GLM), which should be the most powerful test anyway. Learn how to frame questions as the correct regression formulas, choose the most reasonable distribution, and diagnose the models.
Writing the correct regression formula is the trickiest part. Adding too many variables can result in grossly incorrect inferences. Stepwise regression is a poster child of what not to do, while regularization methods are not MLE so we cannot compute ΔD. See a previous article to learn how to pick variables.