Statistically Speaking: methodological madness Heterogeneity in meta-analyses. Comparing apples and oranges?

Dear Ms. Method Matters, I saw a recent meta-analysis published in Anaesthesia on the analgesic effects of propofol total intravenous anaesthesia (TIVA) compared with inhalational anaesthetics [1]. I noticed that in some of the studies included in the metaanalysis, the primary outcome, pain, was assessed using a numerical rating scale, while other studies used a visual analogue score (from 0 to 100, or 0 to 10), instead [1, 2]. In addition, the meta-analysis [1] pooled data from studies which investigated the analgesic effects of propofol after different surgical procedures, some of which may have very different pain outcomes; for example, some studies were looking at laparoscopic surgery, whereas others were open surgical procedures. Can data from these studies be collated into a meta-analysis, can using fixed or random-effect models overcome this problem, or are the studies so different from each other that we would be comparing apples and oranges? Mystified with meta-analyses (Southampton) Dear Mystified, Meta-analysis is defined as the statistical analysis of a large collection of analysis results from individual studies for integrating the findings [3, 4]. As the number of clinical studies with similar protocols has grown, meta-analyses have become increasingly popular, since more reliable results can be obtained by systematically combining studies and the limits of size and scope of individual studies can be overcome. However, the goal of a metaanalysis is not simply to report the mean effect size of an intervention; it is also important to know how the effect sizes in the various studies are dispersed around the mean: in other words, is the impact of the intervention consistent, or does it vary across studies? The consistency of results across studies needs be determined to ascertain whether the findings can be applied in other populations. In an attempt to establish whether the results from studies are consistent, meta-analyses will often include a measure of heterogeneity, for example, Cochran’s Q, or I [5]. The test statistics Cochran’s Q or I determine whether there are true differences underlying the results of the collated studies (denoting heterogeneity), or whether the differences observed between studies are due to chance or sampling error (denoting homogeneity) [5]. Tests for heterogeneity are tests for the null hypothesis that all studies are investigating the same effect. Since meta-analyses pool data from studies which are diverse methodologically as well as clinically, some degree of heterogeneity is to be expected. The classical measure of heterogeneity is Cochran’s Q, which is calculated as the weighted sum of squared differences between individual study effects and the pooled effect across studies. Cochran’s Q has low power as a test of heterogeneity [6] when the number of studies is small, but too much power if the number of studies is large [5]. Since the reliability of Cochran’s Q as a measure of heterogeneity depends on the number of studies included in the meta-analysis, other measures of heterogeneity have been developed, the most popular of which is I, (I2 = 100% 9 (Qdegrees of freedom)/Q. I2 is presented as a percentage, and the interpretation is simple and intuitive. A rough guide for the interpretation of I is: 0–40%, may not represent