Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions

Over the 50 years since the first randomized, controlled trial (RCT) in clinical medicine, RCTs have become firmly established as the standard for determining which medical treatments are effective (1). The limitations of nonrandomized study designs are well known to researchers. The Canadian Task Force on the Periodic Health Examination was the first organization to incorporate an explicit hierarchy of different study designs to assist in the evaluation of evidence to support clinical recommendations (2). Similar classifications developed by other organizations all rank well-designed RCTs or meta-analyses of RCTs as the highest-quality evidence of clinical effectiveness. Some have argued that agencies should fund nonrandomized studies only when convinced that a randomized study is not feasible (3). Study designs other than RCTs remain critical for evaluating diagnostic (4) and prognostic strategies, as well as for assessing the harms of interventions (5). However, the debate continues on the role of nonrandomized studies in formulating recommendations about treatments (3, 6). Recent experiences with vitamin supplementation and hormone replacement therapy, in which large clinical trials failed to confirm benefits reported by multiple observational studies (7, 8), have focused renewed attention on the pitfalls of drawing conclusions from nonrandomized studies. Yet in some areas of health careamong them surgery, public health, and the organization of health care deliverymost evidence addressing the effectiveness of clinical or policy interventions rests on nonrandomized study designs. Systematic reviews therefore frequently need to include these studies in order to provide a more detailed picture of our current knowledge and its limitations for clinicians and policymakers. A distinguishing feature of the Evidence-based Practice Center (EPC) program of the Agency for Healthcare Research and Quality (AHRQ) is the wide range of topics addressed in its systematic reviews (9) and the resulting broad array of study designs included in these reviews. In this paper, we examine the use of study designs other than RCTs in EPC reports addressing questions of the effectiveness of treatment interventions. These reports, completed by 15 different EPCs, offer the opportunity to examine the approaches used and the challenges faced by EPC reviewers when they included nonrandomized studies and assessed their quality. We then offer recommendations for the use of these studies in systematic reviews of treatment interventions. Of the 107 EPC reports released between February 1999 and September 2004, 78 examined at least 1 question of efficacy or effectiveness of a clinical intervention (Figure 1). Twenty-seven of these reports restricted their review to RCTs (25 were of pharmacotherapy interventions, and 2 examined the effectiveness of medical devices). Forty-nine reports included evidence from study designs other than RCTs. We focus here on these reports, which examined pharmacotherapy, medical devices, surgery, complementary and alternative medicine, and behavioral interventions. Figure 1. Study designs used in Evidence-based Practice Center reports. Challenge: Study Design Terminology The terminology to describe different nonrandomized study designs is inconsistent in the clinical research literature (3), a pattern we also observed in the EPC reports. Herein we use the term nonrandomized study to mean a study with a design other than an RCT, including studies in which the investigator assigns treatment group on the basis of a nonrandom strategy (nonrandomized trial), observational studies in which the investigator does not assign treatments (such as casecontrol and cohort studies), and single-group studies (such as beforeafter studies and case series) without a comparison group. Challenge: When To Incorporate Nonrandomized Studies into EPC Reports No established guidelines address situations in which nonrandomized studies can or should be considered for inclusion in a systematic review or what study designs to consider. Study designs varied among the 49 EPC reports that included at least 1 nonrandomized study. Prospective cohort studies, single-group comparison studies, and nonrandomized trials were included in the largest number of reports (Figure 1). Most EPC reports did not state the rationale for including nonrandomized studies. Of the 19 reports that provided an explicit rationale for including these studies, all but 1 cited a lack of sufficient evidence from RCTs. In a review of tocolytics and antibiotics for preterm labor (10), the authors explicitly sought observational data despite availability of many RCTs. In most reports, the decision to include nonrandomized studies appears to have been made early in the review process, but several reviewers broadened or narrowed their inclusion criteria over the course of their review. A review of therapies for treatment-resistant epilepsy took an explicit best-evidence approach; the reviewers restricted the review to controlled studies when 5 or more such studies addressed a specific question and expanded the review to include case series of various surgical interventions when too few trials were identified (11). Other reviews expanded inclusion criteria on the basis of a more subjective assessment of the evidence from trials (12-16) or dropped plans to include nonrandomized studies after they identified numerous RCTs (17). The inclusion of different study designs varied with the nature of the clinical question that the review was addressing. Reviews of pharmacotherapy (n= 29), complementary and alternative medicine (n= 8), and behavioral interventions (n= 4) usually combined RCTs with evidence from nonrandomized trials, cohort studies, or beforeafter studies (Figure 2). On the other hand, reviews of surgical interventions (n= 16) were less likely to find relevant RCT evidence and more likely to include case series and beforeafter studies. Figure 2. Study designs by type of clinical question for Evidence-based Practice Center reports. n In addition to examining the number of reports that used nonrandomized studies, we examined contributions of these studies to the total body of evidence within each report. Many of the 49 EPC reports that included nonrandomized studies nonetheless derived most of their evidence from RCTs. In 18 reports, RCTs accounted for most (75%) of the studies used to address an effectiveness question. In 4 of these reviews, the reviewers did not explicitly use the nonrandomized studies to formulate conclusions (18-21). We identified many discrete situations in which nonrandomized studies were helpful in addressing questions on treatment effectiveness. The following examples illustrate 6 distinct reasons for including nonrandomized studies in a systematic review of treatment interventions. Difficulty Conducting Randomized Trials Randomized trials are increasingly becoming the standard with which to evaluate surgical and technologic interventions, but certain conditions can make randomization problematic. When interventions are complicated or potentially hazardous and have long-term implications for treatment, patients and clinicians are often reluctant to have treatments assigned by chance. For example, a review of pancreatic islet-cell transplantation for type 1 diabetes mellitus found only case series to assess effectiveness (22). Despite the limitations of case series, the finding that 76% of transplant recipients remained insulin independent for 1 year after transplantation provides fairly compelling evidence of benefit on that important outcome. In most of the 16 reports on surgical interventions, most available studies used nonrandomized designs. Such data may not always be sufficient for clinical recommendations, but excluding them a priori will deny patients, providers, and policymakers potentially useful information. Occasionally, results of nonrandomized studies may be so striking that it would be considered unethical to randomly assign patients. A careful examination of available nonrandomized studies may nonetheless provide a more accurate picture of the size of benefits, the factors that may modify effectiveness, and the harms of the intervention. A review of total knee replacement found no trials that compared arthroplasty to medical therapy or sham surgery and acknowledged that such studies would probably not be possible (13). Numerous studies, however, measured pain and function with validated instruments before and after surgery. When results were compared and pooled across different instruments using standardized mean differences (whereby the size of the treatment effect in each study is expressed as the difference in means between groups relative to the variability observed in the study), total knee replacement resulted in large and consistent improvements in function, corresponding to improvements of 2 to 4 standard deviations on these scales. The fact that such dramatic improvements are hardly ever seen spontaneously, were consistent across many studies, and persisted in follow-up beyond 5 years lends further credence to the findings despite limitations of the study design. Examining Long-Term Outcomes of Treatments Assessed in Trials Cohort studies and case series can more easily examine long-term outcomes that may be difficult to observe in RCTs. A review of treatment options for chronic hepatitis C (23) found numerous RCTs to address efficacy and safety of treatment with interferon and other antiviral agents using biochemical or pathologic end points, but data from trials on long-term outcomes such as clinical cirrhosis and hepatocellular carcinoma were rare. Cohort studies and case series with outcomes beyond 5 years for treated and untreated patients demonstrated effects of treatment that were generally consistent with intermediate outcomes in RCTs. This supported a conclusion that standard interferon-based therapy moderately decreases