Betting odds and genetic associations.

“Multiple Comparisons? No Problem! ” read the title of an editorial in an issue ofEpidemiologyin 1991 (1). Attitudes toward multiple comparisons have swung like a pendulum, the view quoted above representing one end of a cycle that has been reversed in recent years. Few classical risk factor epidemiology studies test more than a few hundred associations between exposures and diseases, even in an exploratory mode. In molecular epidemiology, however, the growing feasibility of testing many thousands, or even millions, of single nucleotide polymorphisms (SNPs) associations in genome-wide studies have forced even the most liberal investigators to re-examine their positions on multiple hypothesis testing (2,3). These concerns have been reinforced by the observation that an alarming proportion of reported associations between genetic variants and diseases are not replicated(4–6). In the classical theory of multiple comparisons for analysis of variance (developed in the mid-twentieth century), it was recognized that ifg groups are compared, a total of g(g–1)/2 two-way comparisons are possible (7). This classical approach provides investigators with a means to report all possible comparisons without spuriously inflating their type I error rate. The probability of making at least one falsepositive statistical significance claim (i.e., the experimentwise type I error rate) increases with each additional test performed, even though the probability of any one particular hypothesis being rejected is unaffected by the number of further tests carried out. In analysis of variance, the multiple comparison issue is complicated by the fact that the comparisons are not mutually independent. In a simpler case where k independent hypotheses are tested at a statistical significance level , the experiment-wise type I error rate becomes 1–(1– ), leading to the familiar Bonferroni correction in which one must apply a level /k to each test to maintain a fixed experiment-wise statistical significance level of . In the mid-1970s, epidemiologists began to doubt that the evidence in support of a hypothesis should depend on the number of tests carried out in the same study (8–11). Indeed, investigators questioned the relevance of the experiment-wise type I error rate and argued that it is not the number of tests performed but rather the prior credibility of the hypotheses that is important for interpreting a set of observed associations. That is, when a hypothesis is unlikely to be true, a priori, we should require stronger evidence to be convinced of its truth. In the Bayesian theory of statistics, this argument leads to a formal numerical rule via Bayes theorem; in comparing two rival hypotheses, H1 andH0, the posterior odds for H1:H0 are obtained by multiplying the prior odds by the likelihood ratio, the ratio of the probabilities of observing the data if H1 or H0 were true. This likelihood ratio measures the weight of evidence for H1 overH0. In a valuable contribution to this issue of the Journal, Wacholder et al. (12)argued that the Bayesian paradigm is more relevant to molecular epidemiologic studies than are classical ideas of multiple testing. Few would argue that the simple theory of multiple testing applies naturally in the context of gene association studies, in part because it is often nearly impossible to define how to correct for the appropriate number of tests when using the Bonferroni method; e.g., is it the number of tests we have already carried out in a study, the number of tests we intend to carry out, or the number of tests that we and others might together intend to carry out? Likewise, it makes little sense to require two investigators faced with identical data to come to different conclusions simply because one has carried out more tests than the other. These difficulties in the Bonferroni approach are avoided by focusing attention on the prior credibility of hypotheses. Wacholder et al.(12) combine these Bayesian ideas with classical notions of statistical significance by reducing the data provided by an experiment to the single observation of whether a statistically significant association was found. If this binary observation is the only one available, then the likelihood ratio is the ratio of the statistical power of the test to the statistical significance level applied. Multiplication of the prior odds in favor of a true association by the likelihood ratio gives the posterior odds. For example, if the prior odds are 1000:1 against a true association that we have 90% statistical power to detect at the 5% statistical significance level, the posterior odds, after observing a statistically significant finding, are still 1000:18 ( 50:1) in favor of the null hypothesis. That is, the probability that such a positive finding will be false is . 982. Wacholder et al. term this probability the false-positive report probability (FPRP) and provide spreadsheet programs for carrying out these simple calculations. It is easy to see from the above example that, if the prior odds of a hypothesis are overwhelmingly in favor of no association, we must use very small levels in statistical significance testing, if most positive findings are not to be truly false. The approach of Wacholder et al. oversimplifies the analysis by assuming a simple binary choice between the null hypothesis of no effect and the hypothesis of an effect of known size. In a full Bayesian analysis, we would also need to specify a prior distribution for the size of effect, and the calculation of the posterior probability of association would then require the like-