Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change

Key Summary Points The decision to calculate a summary estimate in a meta-analysis should be based on clinical judgment, the number of studies, and the degree of variation among studies. A random-effects model is a meta-analytic approach that incorporates study-to-study variability beyond what would be expected by chance. The DerSimonianLaird (DL) method, the earliest and most commonly used random-effects model, is the default method in many software packages. The DL method produces confidence bounds that are too narrow (and P values that are typically too small) when the number of studies is small or when there are substantive differences among study estimates. Alternative random-effects estimates based on small-sample adjustments, the profile likelihood, or hierarchical Bayesian models that perform better than the DL method are readily available in software packages. When it is appropriate to pool studies whose estimates vary widely, meta-analytic methods that provide a better accounting of uncertainty than the DL estimator should be used. The basic premise of meta-analysis is that the average of estimates provided by a group of studies is closer to the truth than the estimate provided by an individual study. This premise rests on the assumption that each study is a near-replication of a single experiment and that differences among study results are due only to chance. The technical jargon for this fundamental assumption is that each of the studies is estimating the same fixed effect, and the corresponding meta-analytic approach is dubbed the fixed-effects model. When studies are statistically heterogeneous and differences among their results cannot be explained by chance alone, the meta-analyst faces a conundrum. Qualitative heterogeneity among study designs, patient characteristics, and treatment and comparator regimens may be so great that it does not make sense to combine studies to derive a single summary estimate. However, when the qualitative and quantitative heterogeneity is not so great that a single number summarizing the evidence would be misleading, statistical models that incorporate the extra variability across studies not believed to be due to chance may be used to summarize the data. These models assume that the observed treatment effect for a study is a combination of a treatment effect common to all studies plus a component specific to that study alone. This extra, study-specific component is assumed to be random, hence the jargon that it is a random effect, with accompanying mathematical models dubbed random-effects models. The most widely used random-effects model is based on an estimator developed by DerSimonian and Laird in the mid-1980s and is known as the DerSimonianLaird (DL) estimator (1). Supplement. Alternative Random Effects Estimators An Example The Figure depicts a statistically heterogeneous set of studies followed by several methods of estimating their average effect. The example is from a 1985 meta-analysis by Collins and colleagues on the effect of administering a diuretic to women at risk for preeclampsia (11), and it is frequently used to illustrate different methods for estimating a common treatment effect when the body of evidence is heterogeneous (12, 13). The effect estimates from the individual studies range from a more than 4-fold statistically significant decrease in the odds of eclampsia with diuretics observed in the study by Fallis and colleagues (5) to an almost 3-fold nonsignificant increase in the study by Tervil and Vartiainen (9). A visual clue that these studies are statistically heterogeneous is that the confidence limits of several pairs of studies do not overlap. Figure. Heterogeneous evidence from Collins and colleagues meta-analysis of the effects of diuretics on preeclampsia (11). * The metafor package in R was used to compute the fixed-effects estimate and the DerSimonianLaird random-effects estimate. The metafor package in R was used to compute the KnappHartung small-sample adjustments, based on the DerSimonianLaird estimate. The small-sample (Skovgaard) estimate from the metaLik package in R was used to compute the profile likelihood estimate. The large-sample profile likelihood estimate produced a narrower CI that indicates a statistically significant effect (95% CI, 0.37 to 0.95). The hierarchical Bayesian estimate was computed using WinBugs and assumed a vague uniform (10, 10) prior distribution for . A sensitivity analysis assuming a vague (0.001, 0.001) on precision (1/2) produced a slightly smaller but statistically significant 95% CI (0.36 to 0.98). The Figure shows that different statistical approaches to combining data can produce results leading to different conclusions. The fixed-effects model, which is not appropriate for these data, shows a summary effect of 0.67, with 95% confidence limits (0.56 and 0.80) that are 19% less than and greater than that value. The DL random-effects estimate shows a slightly larger effect (odds ratio, 0.60), but the confidence limits are substantially wider33% less than (0.40) and greater than (0.89) the summary effect, albeit still highly statistically significant. Use of any of the other 3 random-effects estimators depicted in the Figure shows identical point estimates for the odds ratio of 0.60 but dramatically wider confidence limits that are 73% less than and greater than 0.60, with the upper limits all exceeding 1.00. The corresponding P values range from less than 0.001 for the fixed-effects model to 0.011 for the DL estimator and 0.070 or greater for the other random-effects models. Statistical Heterogeneity and Uncertainty The differences noted in the example are due to the ways that the models handle statistical heterogeneity. Statistical heterogeneity refers to variation in the true effects being estimated by each study. We characterize this variation by its SD, a statistic called . Assuming normality, we expect 95% of true effects to fall within2 of the central estimate. When odds ratios or relative risks are used, the normality is on a log scale, so that true study odds ratios or relative risks fall within a range of the estimate multiplied by e2. In the example, equals 0.48, so the true study effects are estimated to fall within 0.60e0. 96, or 2.6 times greater than or less than 0.60 (0.23 to 1.56). This range should be smaller than the actual smallest and largest study estimates, as is the case in this example, with the remainder of the variation assumed to be due to chance. The models vary in their assumption of how certain we are about ; this uncertainty is included in the meta-analytic CIs. The DL method assumes that our guess about is exactly correct, with no uncertainty; thus, confidence limits are too narrow and the P values are too small. In Collins and colleagues meta-analysis, which pooled a modest number of studies (n= 9) with statistically heterogeneous effects, the DL estimator provided the narrowest confidence limits among the random-effects options. In addition to , meta-analysts commonly use statistical tests, such as the Cochran Q test, or indices, such as the I 2 index, to help gauge heterogeneity of effects. Both the Cochran Q test and the I 2 index are dimensionless measures of statistical heterogeneity. Neither conveys information about actual variation in effect size, and both have low power to detect heterogeneity in situations involving 10 or fewer studies (14). The DL Estimator and Alternative Approaches The DL method appeared in the literature just as meta-analytic methods were being adopted to help reviewers quantitatively summarize evidence about medical interventions. It was relatively simple to compute and is still the standard estimator programmed into many meta-analysis software packages, including the RevMan software developed by the Cochrane Collaboration (15). As statisticians began in the 1990s to recognize the problems with the DL approach, theyincluding DerSimonian and Kacker (16)proposed a wide range of alternatives that better capture the uncertainty associated with statistical heterogeneity. These included random-effects estimators based on small-sample adjustments, such as the KnappHartung approach (17), likelihood-based methods (13, 18, 19), and hierarchical Bayesian models (20). The KnappHartung approach, one of the more recent methods, assumes that variances are estimated from small samples, makes small-sample adjustments to the variance estimates, and constructs confidence limits based on the t distribution with k 1 degrees of freedom. This estimator produces a wider confidence limit than the DL estimate. It may slightly overestimate the amount of uncertainty in some cases, particularly when dealing with 5 or fewer studies. It is available in some specialized meta-analysis programs and packages, such as the metareg program (21) in Stata (StataCorp, College Station, Texas) and the metafor package (22) in R (R Foundation for Statistical Computing, Vienna, Austria). Likelihood estimates, which are readily available in such commonly used statistical packages as SAS (SAS Institute, Cary, North Carolina), are computed using standard mixed-effects linear models (18, 19). The profile likelihood is a good method for computing confidence bounds. Unlike estimators based on maximum likelihood or restricted maximum likelihood methods, the profile likelihood allows for asymmetrical intervals and uncertainty in estimation of the between-study variance (2). Simulation studies show that it provides a substantially better accounting of uncertainty than the DL estimator (13, 23). The profile likelihood estimates are available in the metaan package (24) in Stata and the metaLik package (25) in R. The latter provides a more accurate but possibly conservative small-sample profile likelihood estimate of uncertainty (26). Bayesian random-effects models, which are based on an exact binomial distribution, perform well in many situations where others do poorly, particularl