Meta-analysis as evidence: building a better pyramid.

In following the practice of evidence-based medicine, when faced with a question about prevention or treatment the clinician should seek out the best evidence that addresses the question. If quality of evidence is considered a pyramid, what category should be placed at the peak? One dogma argues that it is the best-conducted randomized clinical trial (RCT) comprising patients similar to those seen by the clinician, reasoning that a well-done RCT mimics pure experimental conditions better than any other study design, hence minimizing the likelihood of confounding. A counterargument is that the best evidence is a systematic review with meta-analysis, because this approach can integrate all of the relevant evidence and provide a more reliable answer than a single study, however well conducted. The notion that a synthesis that includes mathematically combiningacompletebodyofevidenceprovidesthehighestlevel of evidence is attractive. However, as with most of evidencebased medicine, the principles are rational, consistent, and appealing, but in practice are fraught with practical challenges, ambiguities, and nuances. Moreover, a busy clinician faces tension betweensearchingforandassessingthebest-qualityprimaryevidence vs accepting the efficiency of using easily obtained but potentially inferior information as a shortcut to an answer. As a general principle, generating, summarizing, and understanding the best available evidence are essential for establishing the benefits and safety of interventions. Metaanalysis has become a valuable tool toward these ends. There has been a proliferation of guidelines by professional societies and others, aimed at ensuring that the best preventive interventions or treatment options are provided to the appropriate patients at the appropriate time; these guidelines often incorporate meta-analyses as a key evidence support for their recommendations. However, limitations of meta-analysis as a study design preclude consistently placing this evidence at the top of the pyramid, and a number of issues need to be resolved before that can happen. These are the problems that researchers, guideline developers, journal editors, and critical readers of the literature struggle with, and understanding the limitations of metaanalytic evidence is crucial for each of these stakeholders.1 One useful way to view these challenges is to divide them into 2 categories: heterogeneity and methodological dilemmas. Heterogeneity (variation in true effect sizes and in factors that might influence those effect sizes) is inherent in metaanalysis, not a problem to be solved. It includes clinical components (eg, diversity in patient populations or interventions) and statistical components (eg, random differences). There are statistical approaches to try to quantify some elements of heterogeneity, including the Q statistic (a measure of total within-study variance), the I2 statistic (the ratio of variability of results among studies to total observed variation), and τ2 (a measure of between-studies variance). Heterogeneity can be investigated and sometimes managed, but not eliminated as an issue. In some instances, helpful insights can be gained when the heterogeneity of findings of component studies can be related to characteristics of those studies (eg, disease severity, outcome definition, duration of follow-up, duration of treatment, or dose of a drug). For example, treatments might show different effects in patients with severe disease than in those with mild to moderate disease. Studies might also give different answers because of flaws in the design or conduct, such as excessive loss to follow-up.2 Modelingtounderstandheterogeneitycanbehelpful.However, relationships that would allow drawing conclusions about effects in different populations, or effects for an ideal study, do not always emerge from the analyses of heterogeneity. Study factors are often confounded with each other; for example, if high-dose studies were conducted in more severely ill populations and low-dose studies in less ill populations, it would be difficult if not impossible to separate the effects of dose from the effects of severity of illness. Most meta-analyses use the aggregate data as presented in the report of the primary study, but when individual patient–level data are not available important relationships can be missed and spurious relationships can be found.3,4 Moreover, inherent heterogeneity presents a challenge in practical clinical interpretation, even when the heterogeneity is assessed and its effects understood. For a single RCT, interpretation of the average effect size may exaggerate the benefit that most patients are likely to achieve5 and often does not provide the information needed for a clinician to understand how to apply the findings to a particular patient. With a meta-analysis, in which 5, 10, or 40 RCTs are combined, the interpretation of the average effect size for application to an individual patient is likely to be even more obscured. Because of the innate uncertainty generated by combining RCT data from disparate sources (which can include differences in patients, interventions, and assessed outcomes), the ability to make causal inferences can be limited; JAMA considers metaanalysis to represent an observational design, with measures that should be interpreted as associations rather than causal effects. Related article page 623 Opinion