ANOVA and the Bonferroni Correction | by Michael Grogan | Towards Data Science

Performing a hypothesis test comes with the risk of obtaining either a Type 1 or Type 2 error.
Type 1 error: Rejecting a true null hypothesis
Type 2 error: Accepting a false null hypothesis
When analysing different groups, a one-way ANOVA can tell us if there is a statistically significant difference between those groups.
However, it cannot tell us which group is different from another. Moreover, when performing multiple hypothesis tests at once, the probability of obtaining a Type 1 error increases.
In statistics, this is known as the family-wise error rate, which measures the probability that a Type 1 error will be made across any particular hypothesis test.
It is calculated as follows:
1 — (1-α)^n
where:
α = the significance level for a given hypothesis test
n = total number of tests
For instance, if we are using a significance level of 0.05 and we conduct three hypothesis tests, the probability of making a Type 1 error increases to 14.26%, i.e. 1-(1–0.05)3 = 0.1426.
To guard against such a Type 1 error (and also to concurrently conduct pairwise t-tests between each group), a Bonferroni correction is used whereby the significance level is adjusted to reduce the probability of committing a Type 1 error. However, a downside of this test is that the probability of committing a Type 2 error also increases.
For this example, let us consider a hotel that has collected data on the average daily rate for each of its customers, i.e. the average price that the customer pays per day to stay at the hotel.
The hotel also has information on the distribution channel pertaining to each customer, i.e. Corporate, Direct, and TA/TO. The goal of the analysis is to determine the differences across means in ADR for each of these three groups. The original data was sourced from Antonio, Almeida and Nunes (2019) as referenced below, and 100 samples from each distribution channel were randomly selected.
Let’s start by conducting a one-way ANOVA in R.
> model <- aov(ADR ~ DistributionChannel, data = data)> summary(model)                     Df  Sum Sq Mean Sq F value   Pr(>F)    DistributionChannel   2  236636  118318   33.33 8.76e-14 ***Residuals           297 1054469    3550                     ---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
When analysing the results, we can see that the p-value is highly significant and virtually zero. This means we reject the null hypothesis that no significant differences exist between each group.
However, we can see that the ANOVA test merely indicates that a difference exists between the three distribution channels — it does not tell us anything about the nature of that difference.
Before performing the pairwise p-test, here is a boxplot illustrating the differences across the three groups:
From a visual glance, we can see that the mean ADR across the Direct and TA/TO distribution channels is higher than that of Corporate, and the dispersion across ADR is significantly greater.
However, we would like to analyse this in more detail using a pairwise t-test with a Bonferroni correction.
> pairwise.t.test(data$ADR, data$DistributionChannel, p.adjust.method="bonferroni")Pairwise comparisons using t tests with pooled SDdata:  data$ADR and data$DistributionChannel       Corporate DirectDirect 4.6e-11   -     TA/TO  2.4e-11   1P value adjustment method: bonferroni
When looking at the adjusted p-values, we can see that the differences between Corporate and Direct, and Corporate and TA/TO are highly significant as the p-values are near zero. Given that the Bonferroni correction has been used to guard against Type 1 errors, we can be more confident in rejecting the null hypothesis of no significant differences across groups.
That said, we can see that there exists a p-value of 1 between the Direct and TA/TO groups, implying that we cannot reject the null hypothesis of no significant differences between these two groups.
This has been a short introduction to pairwise t-tests and specifically, the use of the Bonferroni correction to guard against Type 1 errors. You have seen:
The limitations of using a one-way ANOVA
How to calculate the family-wise error rate
How to conduct a pairwise t-test using a Bonferroni correction and interpret the results
Many thanks for your time, and any questions or feedback are greatly appreciated.
Antonio, Almeida, Nunes (2019). Hotel Booking Demand Dataset
Statology: How to Perform a Bonferroni Correction in R
Statology: What is the Family-wise Error Rate?
Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as professional advice. The findings and interpretations in this article are those of the author and are not endorsed by or affiliated with any third-party mentioned in this article. The author has no relationship with any third parties mentioned in this article.