When To Base Clinical Policies on Observational Versus Randomized Trial Data

Physicians and health care organizations face the substantial challenge of maintaining or improving health outcomes for patients while containing health care expenditures. Analysis of large databases is an efficient method for discovering new clinical policies that may help to achieve these goals [1]. As advances in information technologies continue to facilitate such analysis, physicians must decide whether to accept a new policy only on the basis of these analyses or wait for completion of at least one randomized trial. Research shows that physicians are more likely to accept a new policy when evidence stems from a randomized study [2]. However, physicians may choose not to conduct or have their patients participate in a randomized study for two primary reasons: They believe the patients would not directly benefit from a study, or resources to fund the study would be better spent on assuring provision of established therapies. In addition, other physicians or organizations might perform the needed studies. This article helps physicians and health care organizations decide whether to change a clinical policy on the basis of analysis of large databases compared with conducting or waiting for the results of a randomized trial. We first discuss the advantages and disadvantages of undertaking a randomized study, using digoxin for patients with heart failure and -carotene for patients with cancer as specific examples. We then describe a technique for estimating the potential costs and health consequences of having patients participate in a randomized trial. This technique has been used to help a large physician-managed organization decide whether to conduct a randomized study of folic acid supplementation to prevent cardiovascular disease among patients with end-stage renal disease. Basing Clinical Policies on Analysis of Large Databases or Waiting for Completion of Randomized Studies In the 1980s, analysis of several clinical databases showed that some patients who received digoxin for heart failure after myocardial infarction had a higher risk for death [3-5]. However, several questions needed to be resolved before discontinuation of digoxin therapy could be recommended [6]. Researchers asked what biological mechanisms could explain the link between digoxin and the increased risk for death. They also asked whether other factors, such as comorbid conditions, could explain the association between digoxin and the increased risk for death. Basic science research plays a pivotal role in answering the first question of identifying the mechanisms of a drug's effect on clinical outcomes. Powerful epidemiologic tools have been developed to address the second question by controlling for the possible influence of measurable comorbid conditions. However, such methods cannot control for the effects of unmeasured factors [7, 8]. The randomized trial has become an important tool for evaluating treatments because any subsequent difference in outcomes found between patient groups are more likely to be the direct consequence of treatment assignment [9, 10]. As a result, physicians might appropriately ask whether it would be prudent to wait for evidence from randomized studies before changing their policy. In the case of digoxin, several trials were completed and showed, contrary to concerns raised by observational studies, that digoxin was beneficial in selected patients [11-13]. The case of -carotene is also instructive. More than 20 basic science and epidemiologic studies provide evidence that -carotene may decrease the risk for cancer [14, 15]. Two of three randomized trials found, however, that patients receiving -carotene from nondietary sources had an increased risk for lung cancer and death [16-18]. These examples, in which findings of randomized studies differed significantly from predictions of epidemiologic and basic science studies, have led experts to caution against relying only on analyses of observational data to determine the direction of treatment effects [7, 8, 14, 19-23]. Despite the value of randomization, it is estimated that less than 20% of clinical policies are based on randomized studies [22]. Several factors may create a barrier to conducting a randomized study. First, the findings of observational studies may seem so compelling that randomization would be considered unethical [24]. Second, conducting a randomized study requires a commitment in time and money, the return on which may not be evident for many years. Third, physicians may believe that they and their patients will benefit more by waiting for the study to be done elsewhere. In the next section, we illustrate these issues by analyzing the potential costs and health consequences of a randomized study of folic acid supplementation to prevent cardiovascular disease among patients with end-stage renal disease. The analyses presented here were used by the medical directors of Satellite Dialysis Centers in Redwood City, California, in deciding whether to conduct a randomized study as part of a new quality-improvement initiative. Estimating the Cost-Benefit of Conducting a Randomized Study of Folic Acid Supplementation in Patients with End-Stage Renal Disease Basic science and epidemiologic studies have shown a correlation between increased levels of homocysteine in blood and an increased risk for cardiovascular disease [25-27]. Plasma homocysteine levels increase with the onset of chronic renal failure and remain elevated in more than 80% of patients receiving long-term dialysis [26]. Cardiovascular disease is the leading cause of death in patients with end-stage renal disease [28], and hyperhomocysteinemia is more prevalent in these patients than any other cardiovascular risk factor [27]. In patients without renal disease, homocysteine levels may return to normal with as little as 2.5 mg of oral folic acid per day [29]. However, homocysteine levels are difficult to normalize in patients with end-stage renal disease, even at supraphysiologic doses [30, 31]. Moreover, the costs and risks of administering high doses of folic acid to patients with end-stage renal disease are unknown. Physicians might want additional information about the consequences of folic acid supplementation before recommending a change from the current clinical policy. Analysis of Costs and Health Consequences How can the potential costs and benefits of a randomized study of a new treatment, such as prescribing higher doses of folic acid (for example, 5 mg/d) to decrease homocysteine levels in patients with end-stage renal disease, be estimated? First, a health economic model (cost-effectiveness or costbenefit) is needed to estimate the expected costs and health consequences of high doses of folic acid compared with the standard dose (1 mg/d) [32]. Such models have been used extensively to compare the consequences of medical treatments [32-34]. Model inputs include established concerns of modern technology assessments, such as treatment effects on life expectancy, quality of life, and costs of medical care [35]. Expert panels recommend that the combined effects of treatment on illness and death be summarized in these models in terms of quality-adjusted life expectancy [33]. We use a health economic model to determine whether a trial should be done and how many patients should be enrolled to prevent recommending the wrong treatment once the trial is completed [36]. Regardless of the type of health economic model used, physicians must decide how much is a reasonable investment in health programs, including the cost of research, to prolong quality-adjusted life expectancy by 1 year [33]. For example, choosing a low valuation (investing no more than $20 000 to prolong life by 1 year) [37] is likely to undervalue the research effort in the context of many currently accepted medical interventions, such as dialysis in patients with end-stage renal disease [38]. In contrast, if the valuation exceeds $200 000 per year of prolonged life, then resources may be spent on some types of research that could have been used for established interventions that would have yielded better health outcomes or had lower costs. Because no consensus exists on the optimal investment in treatments or research for improving quality-adjusted life expectancy [33], the effects of different valuations of health on trial designs are explored with sensitivity analyses. Using a costbenefit method, we constructed a health economic model to compare the effects of two different doses of oral folic acid among patients with end-stage renal disease undergoing dialysis. Such costbenefit analyses usually begin with a reference case, in which each input is assigned an initial average value. Details about the structure of the costbenefit model, inputs and assumptions, reference case results, and sensitivity analyses are available from the authors or on the World Wide Web (http://www-leland.stanford.edu/~ewrone). The model also includes consideration of the size of the target population (that is, the number of patients that physicians expect to benefit from the results of the trial) and the rate at which patients are expected to withdraw from the study. Sensitivity analyses are used to explore how conclusions from the reference case analysis change with reasonable variation from the initial input values. The top panel of Figure 1 shows a sensitivity analysis in which the cost benefit of using high-dose folic acid instead of standard-dose folic acid is plotted as a function of the relative risk (RR) for death between the two treatments. The RR equals the probability of death associated with high-dose folic acid divided by the probability of death associated with standard-dose folic acid. If the RR is low, then the costbenefit difference is negative and high-dose folic acid is preferred. If the RR is high, then the costbenefit difference is positive and standard-dose folic acid is preferred. Figure 1. Expected costbenefit differ