Integrating Heterogeneous Pieces of Evidence in Systematic Reviews

Previous articles in this series described systematic reviews and how to find them [1, 2], discussed their role in practice and educational settings [3-6], and outlined important aspects of their conduct [7-9]. This article addresses a particularly challenging problem in conducting systematic reviews-integration of different types of evidence within a single review. We present strategies for integrating evidence from various primary studies that were conducted with different objectives, protocols, and designs. These strategies may be useful in a variety of situations in which heterogeneous evidence is used (for example, clinical decisions, decision analysis, economic analysis, practice guidelines, and health policy formulations). The strategies are intended primarily for reviewers who address broad questions (for example, reviewers interested in producing evidence-based guidelines). The strategies may also help reviewers with focused questions when the data available on a topic are particularly heterogeneous. We consider the following specific questions: 1) How can reviewers classify and structure heterogeneous research evidence? 2) What factors complicate integration of heterogeneous research evidence? 3) What strategies help integrate heterogeneous research evidence? Classifying and Structuring Research Evidence Just as any scientific inquiry moves from concrete observations to abstract concepts, reviewers must move from samples of data (individual pieces of evidence) to more general conclusions [10]. This process involves drawing together multiple pieces of evidence into a unified whole by categorizing and ordering data [11]. An important first step is to classify evidence as direct or indirect. Direct evidence directly relates an exposure, diagnostic strategy, or therapeutic intervention to the occurrence of a principal health outcome [12]. Principal health outcomes are those relevant to the patient, such as symptoms, loss of function, and death [13]. Whether a study provides direct evidence depends on methodologic design and the outcomes studied. For example, some randomized, controlled trials that compared diuretic and -blocker regimens with no therapy or placebo in hypertensive adults have directly shown that these therapies decrease cardiovascular morbidity and mortality [14]. Other trials comparing antihypertensive agents have shown decreases in blood pressure (a surrogate outcome) but have not directly demonstrated effects on cardiovascular morbidity and mortality (the principal outcomes). Evidence is indirect if two or more bodies of evidence are required to relate the exposure or intervention of interest to the principal health outcome [12]. Thus, one body of evidence may relate exercise (the intervention) to lower-extremity strength (an intermediate outcome), and another may relate lower-extremity strength to the probability of falls (the health outcome of principal interest); neither one alone directly relates exercise to falls. Other examples are intervention strategies with several substitutes, particularly when the various substitutes have been evaluated in different types of studies. For example, pravastatin, a 3-hydroxy-3-methylglutaryl-coenzyme A reductase inhibitor, has been shown to decrease low-density lipoprotein cholesterol levels and cardiovascular morbidity and mortality in men with moderate hypercholesterolemia and no history of myocardial infarction (primary prevention) [15]. Several of these inhibitors, including pravastatin, simvastatin, and fluvastatin, have been shown to decrease cardiovascular morbidity or mortality in persons with hypercholesterolemia and history of myocardial infarction (secondary prevention) ([16, 17], Oral presentation of the Lipoprotein and Coronary Atherosclerosis Study at American Heart Association Meeting, Anaheim, California, November 1996). Taken together, these data provide indirect rather than direct evidence that all 3-hydroxy-3-methylglutaryl-coenzyme A reductase inhibitors are effective in the primary prevention of cardiovascular disease. Much evidence about health care is indirect. The synthesis of indirect evidence or of pieces of direct evidence requires the creation of models that relate exposures or diagnostic or intervention strategies to principal health outcomes. Conceptually, this involves 1) identifying links that connect the exposures, diagnostic strategies, or interventions to principal health outcomes; 2) analyzing the evidence that pertains to each link; and 3) combining the links [12]. In essence, this approach breaks a complex problem into a series of smaller problems and formally theorizes the relations among those problems. Such models are often called evidence models. They provide reviewers with an analytic framework that clarifies the cause or natural history of a health problem, the sequence of intermediate effects that an exposure or diagnostic or intervention strategy must pass through to reach certain primary outcomes, and the range of potential adverse effects that need consideration [18, 19]. Factors Complicating Integration of Evidence Regardless of whether reviewers are synthesizing direct or indirect evidence, many factors can modify etiologic and prognostic associations, diagnostic accuracy, and therapeutic effectiveness. Study participants are often drawn from various settings and have a wide spectrum of baseline risk, disease severity, and sociodemographic and cultural characteristics. Exposures, diagnostic strategies, interventions, and comparison groups have varying formulations and intensities. Different outcome measures are used in different studies, and similar outcomes are measured or reported differently. Various study designs are used (Table 1), and heterogeneity of methodologic features occurs within a given design (Table 2). Table 1. Potential Sources of Research Evidence Table 2. Examples of Important Methodologic Features Associated with Different Types of Studies Heterogeneity of research evidence may concurrently exist at one or more levels. A review of several randomized, controlled trials that tested whether a particular drug class resulted in improved survival in similar groups of patients may be complicated only by judgments on the degree of homogeneity of the different drugs within the class. In practice, heterogeneity of only one factor in a given study or group of studies (for example, different drugs within a class) is relatively rare; several sources of complexity are usually present. For example, a recent systematic review of the effectiveness of stroke units included evidence from both randomized and nonrandomized, controlled clinical trials; these trials evaluated different models of stroke units, used different patient inclusion criteria, and had various outcome measures [20]. Systematic reviews that examined studies of methods for implementing clinical guidelines have included multifaceted management interventions directed toward different clinical conditions; systematic reviews of the efficacy of continuing medical education examined studies of a variety of educational activities among different groups of health care professionals working in different health care settings [21, 22]. A comprehensive review evaluating the association between cigarette smoking and lung cancer might integrate evidence from laboratory studies of genetic mutations with evidence from casecontrol and prospective studies of cancer in animals and humans. Heterogeneity is a double-edged sword. On the positive side, it may allow reviewers to examine consistency of findings across studies of various types and their applicability in a variety of patients and settings (that is, it may increase generalizability). It may also allow a more comprehensive picture of feasibility, acceptability, benefits, and harms associated with particular formulations of a diagnostic or therapeutic strategy. On the negative side, it may introduce ambiguity into the synthesis of evidence. Researchers conducting systematic reviews may be required to make judgments about the relevance of the heterogeneity, the legitimacy and relative uncertainty of particular pieces of evidence, the importance of missing evidence, the soundness of the model for linking the evidence, and the appropriateness of conducting a quantitative summary. Strategies for Integrating Heterogeneous Evidence Linking Multiple Pieces of Evidence Reviewers addressing broad questions that involve linkages among multiple bodies of both indirect and direct evidence need to use explicitly defined models. An example of a model that was used to guide a systematic review of screening for hearing impairment in elderly persons is given in Figure 1 [23]. The model was based on preset criteria for evaluating screening programs [24]. Frameworks for constructing models of causality, prognosis, effectiveness of diagnostic and intervention strategies, and specific relationships between surrogate and clinically meaningful outcomes are also available [12, 13, 18, 25, 26]. An example of a complex framework for assessing benefits and harms of a particular therapy is given in Figure 2. Figure 1. Model examining rationale for screening for hearing impairment. Figure 2. An evidence model for the pharmacologic treatment of obesity. Each link in a model represents a subquestion for which a systematic review could be conducted. In some instances, direct evidence that obviates the need to address certain intermediate linkages may be available. Reviewers select important linkages and perform a series of pertinent systematic reviews, each with a well-formulated question, specified inclusion criteria, explicit searching and selection techniques, and method of critical appraisal. Evidence tables for each subquestion can be developed (Table 3, Table 4). These can be accompanied by narrative summaries that identify the direction, magnitude, significance, and uncertainty of effects and highlight major issues