Commentary on ‘Misunderstandings about Q and “Cochran's Q test” in meta analysis’

We congratulate David Hoaglin on this important and very timely critique of the manner in which the statistic Q is currently used in meta-analysis. While we are in agreement with Hoaglin’s [1] conclusions, we would like to elaborate on several points arising from the paper. The Q statistic is ubiquitous in any standard meta-analysis. It is often used to test for heterogeneity between the studies; it is used to calculate the popular I measure of inconsistency [2]; and it is used for moment estimation of the between-studies variance component τ, and therefore participates indirectly in numerous estimates of the combined effect. One important misunderstanding raised in the paper is the widely held belief that in any meta-analysis of K studies, the Q statistic has a chi-square distribution with K 1 degrees of freedom, and therefore, its expected value is K 1. This belief is not true. In fact, the distribution of the Q statistic very much depends on the effect measure used. As early as 1951 (3 years prior to the 1954 paper by Cochran that Hoaglin analyses so well in Section 3), Welch [3] and James [4] established that if the effect measure is a normally distributed sample mean, then the expected value of Q exceeds K 1. (Note that Q appears earlier in the 1937 paper of Cochran [5].) More recently, we have investigated the moments and the distribution of Q for several effect measures: for the difference of means [6], for standardised mean difference [7], for risk difference [8], and for log odds ratio [9]. In all cases, the mean of Q is not K 1; the mean can be either smaller or larger, depending on the effect measure in question (larger for the difference of means and smaller for the other measures). Further, the mean of Q will depend not only on the number of studies K but also on all of the other parameters involved (e.g. study sizes and weights). The distribution of Q itself can be approximated by various distributions: a scaled F for sample means, a chi-square with fractional degrees of freedom for standardised mean differences, or a gamma distribution for risk differences and log odds ratios. The use of the standard chi-square approximation to the distribution of Q in a homogeneity test may result in an overly conservative or an overly liberal test, once more depending on the effect measure used. These discrepancies are not trivial. As discussed in Section 6 of the paper by Hoaglin, moment-based estimation of the between-studies variance component τ is based on the first moment of Q (DerSimonian–Laird [10] and Mandel–Paule [11] methods), and the confidence interval for τ is often based on Q profiling [12], that is, on the use of the percentage points of the Q distribution. Both therefore should also be effect measure-dependent. In Section 5, Hoaglin discussed implications for I =[Q – (K 1)]/Q as a relative measure of heterogeneity. These implications include the wrong centering of I and a wrong distribution routinely used for calculation of its confidence intervals. Additionally, Hoaglin mentions the substantial probability of obtaining a large value of I even under homogeneity. Another, perhaps more important, drawback of I is its dependence on the sizes of the individual studies. Under heterogeneity, for a fixed K, the larger the study sizes, the larger is the value of I, converging to 100% in the limit. Hoaglin provided some critique of the paper by Rücker et al. [13] on this point. But the fact itself is easily