When to combine hypotheses and adjust for multiple tests.

Recent issues of Health Services Research (HSR), the Journal of Health Economics, and Medical Care each contain articles that lack attention to the requirements of multiple hypotheses. The problems with multiple hypotheses are well known and often addressed in textbooks on research methods under the topics of joint tests (e.g., Greene 2003; Kennedy 2003) and significance level adjustment (e.g., Kleinbaum et al. 1998; Rothman and Greenland 1998; Portney and Watkins 2000; Myers and Well 2003; Stock and Watson 2003); yet, a look at applied journals in health services research quickly reveals that attention to the issue is not universal. This paper has two goals: to remind researchers of issues regarding multiple hypotheses and to provide a few helpful guidelines. I first discuss when to combine hypotheses into a composite for a joint test; I then discuss the adjustment of test criterion for sets of hypotheses. Although often treated in statistics as two solutions to the same problem (Johnson and Wichern 1992), here I treat them as separate tasks with distinct motivations. In this paper I focus on Neyman–Pearson testing using Fisher's p-value as the interpretational quantity. Classically, a test compares an observed value of a statistic with a specified region of the statistic's range; if the value falls in the region, the data are considered not likely to have been generated given the hypothesis is true, and the hypothesis is rejected. However, it is common practice to instead compare a p-value to a significance level, rejecting the hypothesis if the p-value is smaller than the significance level. Because most tests are based on tail areas of distributions, this is a distinction without a difference for the purpose of this paper, and so I will use the p-value and significance-level terms in this discussion. Of greater import is the requirement that hypotheses are stated a priori. A test is based on the prior assertion that if a given hypothesis is true, the data generating process will produce a value of the selected statistic that falls into the rejection region with probability equal to the corresponding significance level, which typically corresponds to a p-value smaller than the significance level. Setting hypotheses a priori is important in order to avoid a combinatorial explosion of error. For example, in a multiple regression model the a posteriori interpretation of regression coefficients in the absence of prior hypotheses does not account for the fact that the pattern of coefficients may be generated by chance. The important distinction is between the a priori hypothesis “the coefficient estimates for these particular variables in the data will be significant” and the a posteriori observation that “the coefficient estimates for these particular variables are significant.” In the first case, even if variables other than those identified in the hypothesis do not have statistically significant coefficients, the hypothesis is rejected nonetheless. In the second case, the observation applies to any set of variables that happen to have “statistically significant” coefficients. Hence, it is the probability that any set of variables have resultant “significant” statistics that drives the a posteriori case. As the investigator will interpret any number of significant coefficients that happen to result, the probability of significant results, given that no relationships actually exist, is the probability of getting any pattern of significance across the set of explanatory variables. This is different from a specific a priori case in which the pattern is preestablished by the explicit hypotheses. See the literatures on False Discovery Rate (e.g., Benjamini and Hochberg 1995; Benjamini and Liu 1999; Yekutieli and Benjamini 1999; Kwong, Holland, and Cheung 2002; Sarkar 2004; Ghosh, Chen, and Raghunathan 2005) and Empirical Bayes (Efron et al. 2001; Cox and Wong 2004) for methods appropriate for a posteriori investigation.