An Overview of Methods to Address the Multiple Comparison Problem | by Paulina Zheng | Towards Data Science

As data scientists, we are uniquely positioned to assess relationships between variables within a larger context and ultimately, tell stories with statistics.
In short, we use hypothesis testing to prove some hypothesis about our data.
This is a fairly straightforward process until we want to test a set of hypotheses simultaneously. In this post, I’ll review the problems with making such multiple comparisons and how to address them accordingly.
How do we first begin to investigate our belief about a situation? Well, we can conduct so-called hypothesis tests with the data available to us, in order to assess our claim.
Let’s consider drug A, which is currently in development. We are interested in whether drug A genuinely improves patients’ self-rated pain, compared to a placebo (a ‘sugar pill’ with no therapeutic effect).
How do we demonstrably measure and determine ‘effect’? It’s not enough to simply posit that drug A is better than placebo. We want to do so with specific measurements, based on the available data. For the purposes of this example, let’s assume that patients report pain using a numerical scale from 0–10. We can collect pain scores and then compare the mean of those scores across two groups: an experimental group (taking drug A) and a control group (taking placebo). It is important to note that each group is intended to be representative of a broader patient population.
For our hypothesis test, we must specify the following:
Null Hypothesis: What we are seeking to disprove, usually the ‘status quo’ (that there is no effect or difference). Here, our null hypothesis is that there is no difference between drug A and placebo in improving self-reported pain (the means are the same).
Alternative Hypothesis: Our effect of interest, or the antithesis of the null hypothesis. We would say that our alternative hypothesis is that there is a difference between drug A and placebo in improving self-reported pain (the means are not the same).
Significance level (α): Probability of incorrectly rejecting the null hypothesis, given that it’s true (false positive). This is always established at the beginning of every hypothesis test.
Of these, α is perhaps most relevant to the multiple comparison problem. It is important to first have a thorough understanding of how α is used in hypothesis testing.
If we find a difference in the mean pain scores across the two groups, does it actually mean anything? We can define boundaries that allow us to determine whether any observed differences are extreme enough to reflect an actual effect/relationship or are more likely to be due to chance alone.
Because we have the pain scores for every patient, we can describe the distribution of the scores for each group. For this example, we will say that for both groups, the data is roughly normally distributed.
Every hypothesis test is predicated on some type of distribution for the population in question. Any given distribution is centered around the mean (μ) and characterized by the spread (standard deviation, σ) of the data. Most hypothesis tests — and the assumptions thereof — are based on the normal, or Gaussian, distribution.
As evidenced above, for our patient population, 99.7% of our data should fall within 3 standard deviations of the mean. Using this distribution, we can determine how many standard deviations a given observation is from the mean. The ‘farther’ the observation is from the mean, the more unusual it is.
More technically, the number of standard deviations from the mean can be represented by a z-score. We use the z-score as a way of standardizing what we observe and is calculated as:
With the central limit theorem, we can assume that the data is distributed normally.
We can then calculate the test-statistic for our data. We need to compare this test-statistic to some threshold. If our test-statistic exceeds this threshold, then our finding is statistically significant and unlikely to be due to chance alone.
We use our pre-determined significance level (α) to define the threshold.
Typically, α is set to be 0.05, although this may vary depending on the field of study. What does this mean? It means that if you find an effect, you are willing to be 95% confident that your results are statistically significant and there is a 5% chance that it is due to chance alone.
We can look again at our standard normal distribution that we are using to inform all our calculations.
α defines the rejection region(s) for a Gaussian curve. Extreme results exist at the tail ends of the curve, farthest from the mean. The rejection region is chosen such that the probability is equal to α that it will contain the test statistic when the null hypothesis is true.
Because our hypothesis specified only that there was a difference between drug A and placebo, we test the relationship in both directions: drug A could be either better or worse than placebo. Therefore, it’s a two-tailed hypothesis test and we look at both ends of the Gaussian curve. We distribute α equally between the two tails so each tail will contain 0.025.
Here, the p-value guides our decision. The p-value is the probability of an effect at least as extreme as that observed in sample data, if the null hypothesis were true. If the test-statistic falls within the rejection region(s), then the p-value will be less than α. We can say that we are 95% confident that drug A significantly reduces patients’ self-reported pain, compared to placebo. There is a 5% chance that the observed effect is due to chance alone. Therefore, we reject the null hypothesis.
We want to establish α from the get-go because we want to control the probability of wrongfully rejecting the null hypothesis, or our type 1 error from the get-go. This is fairly straightforward for a single hypothesis test.
But what if we want to test for multiple hypotheses? You may encounter this in AB testing, when you want to compare multiple variations of a site against each other, or multiple symptoms treated in drug testing. You would need multiple null hypotheses then, one for each ‘variation’.
Each hypothesis will have an individual α*, which represents the type 1 error rate for that particular test. As the number of hypotheses increases, so too does the overall α for the set of tests because the individual α*’s accumulate. For a single hypothesis test at the α=0.05 level, the type 1 error rate is only 5%. There is only a 5% chance of erroneously rejecting the null hypothesis. For 2 hypothesis tests, however, the overall α becomes 0.10. The probability of erroneously rejecting at least 1 null hypothesis is 0.10. The overall α quickly grows with the addition of each new comparison:
Overall α ≤ 1-(1-α*)
Now, why not just restrict to a single test and avoid this messy complication? Efficiency and practical considerations may necessitate multiple comparisons. Let’s say we are interested in how drug A reduces pain, reduces incidences of headaches, etc. It makes sense to conduct a set of tests to understand the overall efficacy of drug A, based on multiple clinically meaningful measures.
So, we want to control our type 1 error. We can approach this in two different ways.
Control overall α (also known as family-wise error rate or FWER), which will affect the α* for each test. That is, we are controlling the overall probability of making at least one false discovery. Bonferroni and Sidak corrections all control FWER.Control false discovery rate (FDR). Where FWER controls for the probability for making a type 1 error at all, these procedures allow for type 1 errors (false positives) but control the proportion of these false positives in relation to true positives. This is done by adjusting the decision made for the p-value associated with each individual test to decide rejection or not. Because this will result in a higher type 1 error rate, it has higher power. This affords a higher probability of true discoveries. The step procedures control for FDR.
Control overall α (also known as family-wise error rate or FWER), which will affect the α* for each test. That is, we are controlling the overall probability of making at least one false discovery. Bonferroni and Sidak corrections all control FWER.
Control false discovery rate (FDR). Where FWER controls for the probability for making a type 1 error at all, these procedures allow for type 1 errors (false positives) but control the proportion of these false positives in relation to true positives. This is done by adjusting the decision made for the p-value associated with each individual test to decide rejection or not. Because this will result in a higher type 1 error rate, it has higher power. This affords a higher probability of true discoveries. The step procedures control for FDR.
Below, I’ll provide a brief overview of available correction procedures for multiple comparisons.
Bonferroni CorrectionThe most conservative of corrections, the Bonferroni correction is also perhaps the most straightforward in its approach. Simply divide α by the number of tests (m).
However, with many tests, α* will become very small. This reduces power, which means that we are very unlikely to make any true discoveries.
Sidak Correctionα* = 1-(1-α)^(1/m)
Holm’s Step-Down ProcedureAn update of the Bonferroni correction, this procedure is more powerful. Rather than controlling the FMER, Holm’s procedure controls for the false discovery rate (FDR) and performed after conducting all hypothesis tests and finding associated p-values at α within a set.
The step-down procedure is best illustrated with an example. Say we have three hypotheses, each with the associated p-value:
H1: 0.025H2: 0.003H3: 0.01
Step 1: Order p-values from smallest to greatest
H2: 0.003H3: 0.01H1: 0.025
Step 2: Use the Holm-Bonferroni formula for the first-ranked (smallest) p-value
α* = α/(n-rank+1)
α* = 0.05/(3–1+1) = 0.0167
Step 3: Compare the first-ranked p-value with the α* calculated from Step 2
0.003 < 0.0167
Because the p-value for H2 is less than the calculated α*, we can reject H2.
Move onto the next ranked p-value and repeat steps 2–3, calculating the α* for its respective rank and comparing it to that p-value. Continue until you reach the first non-rejected hypothesis. You would then fail to reject all following hypotheses.
Hochberg’s Step-Up ProcedureMore powerful than Holm’s step-down procedure, Hochberg’s step-up procedure also seeks to control the FDR and follows a similar process, only p-values are ranked from largest to smallest.
For each ranked p-value, it is compared to the α* calculated for its respective rank (same formula as Holm’s procedure). Testing continues until you reach the first non-rejected hypothesis. You would then fail to reject all following hypotheses.
For the purposes of this blog post, I will only briefly refer to two other well-known correction methods, which are a bit more involved than this blog post can accommodate.
Tukey’s ProcedureTukey’s procedure is a single-step multiple comparison test and is only applicable if you are interested in comparing all possible pairs of means; it does so simultaneously. It is essentially a t-test that corrects for FWER.
Dunnet’s CorrectionDunnet’s correction is similar to Tukey’s procedure except that it involves the comparison of every mean to a single control mean.
Both these procedures make use of the ANOVA test which allows you to test multiple groups, to see if there is a significant difference between any of the groups (null hypothesis: μ1 = μ2=...=μk). Tukey’s procedure or Dunnet’s correction can then allow you to identify which pairs are significantly different.
Certainly, making multiple comparisons is not simple. However, I hope this post has helped to elucidate the options that we, as data scientists, have in order to reduce false conclusions while at the same time, having enough flexibility to make some true conclusions.