Observational Studies of Treatment Effectiveness: Some Cautions

The last decade has seen an extraordinary proliferation of evidence that statins decrease mortality in patients who have, or are at risk for developing, cardiovascular disease (1, 2). The benefits extend to high-risk patients with normal cholesterol levels (2) and have led some to suggest that all individuals older than 55 years of age should receive statins as part of a polypill (3). In this issue, Spencer and colleagues (4) add to the evidence in favor of statins. They conclude that statin users with an acute coronary syndrome are less likely to present with ST-segment elevation or myocardial infarction than those not receiving statins (odds ratio, 0.78 [95% CI, 0.70 to 0.86]). More surprisingly, they also conclude that starting statin therapy after admission markedly decreases in-hospital mortality (odds ratio, 0.38 [CI, 0.30 to 0.48]). Whereas randomized trials have provided much of the previous evidence supporting statins (1), Spencer and colleagues have conducted an observational study. This form of research design should always raise concern that bias in some form accounts for the findings. In observational studies, patients and their physicians select treatment on the basis of clinical need or preference, which can result in differences in clinical outcomes solely because of differences between those who do and do not receive a treatment. In contrast, random assignment makes it likely that the frequency of all patient characteristics, known and unknown, will be the same in the treatment groups. This powerful feature makes the results of randomized trials less susceptible to bias than observational studies. Biases associated with observational studies are common and have previously misinformed clinical practice. For example, the results of recent randomized trials showed harm from hormone replacement therapy (5, 6), while previous observational studies had suggested a benefit. Such experiences have convincingly illustrated the dangers of using observational studies to determine the effectiveness of medical therapies. If the pretreatment characteristics of the patients in the treated and untreated groups are almost identical or the treated patients are clearly sicker, the results of observational studies showing a beneficial effect of therapy are more believable. Exactly this pattern occurred with the first major finding of Spencer and colleagues: Statin users had better outcomes even though they appeared to have had more prehospital comorbid illnesses. Another reason for confidence is the 22% relative risk reduction, which is similar to findings in other studies of statins in heart disease. However, despite all of these confidence-inspiring findings, it is still possible that unmeasured confounders accounted for this result. The most striking finding of Spencer and colleagues' study is the magnitude of the reduction in mortality associated with starting statin therapy in the hospital among patients who had previously not received statins (that is, a 62% risk reduction relative to statin nonusers), and the rapidity with which the mortality reduction occurred (during the initial hospital admission). Frankly, this result seems too good to be true. Perhaps the best available evidence about early statin use (24 to 96 hours after admission) relative to placebo in acute coronary syndromes stems from a randomized trial of 3086 patients (7). This trial found no impact on death and only a 16% relative decrease in the composite primary end point of death, myocardial infarction, cardiac arrest, and recurrent ischemia at 16 weeks. Furthermore, the relative risk reduction observed in randomized studies of patients with a variety of indications for statin therapy has consistently been about 25% (1, 2). Selection bias is another cause for concern. One reason for suspecting it is the significant differences between treatment groups in medical history, clinical presentation, and use of preventive medications such as angiotensin-converting enzyme inhibitors and -blockers. Multivariate statistical analyses can adjust for these known discrepancies, but they cannot adjust for unmeasured differences between statin and nonstatin groups. Differences in noncardiovascular conditions, such as renal disease, and in unmeasured sociodemographic characteristics could affect the outcomes measured by Spencer and colleagues. In addition, although statin use was associated with reduced mortality, it was not associated with a lower risk for myocardial infarction or strokean unexplained and seemingly inconsistent finding. Survivor treatment selection bias (8) and competing medical issues (9) are two potential problems with observational studies of treatment efficacy. They may have affected Spencer and colleagues' finding that starting statin therapy during hospitalization resulted in a lower risk for hospital complications or death. Survivor treatment selection bias occurs because patients who live longer have more opportunities to begin receiving a treatment than those who die early; thus, even an ineffective treatment may appear beneficial. The presence of competing medical issues can divert attention from seemingly less urgent care to more pressing clinical issues. For example, those who died during the first day of hospitalization in Spencer and colleagues' study may not have had the opportunity to take statins because they died before receiving the drugs. Another possibility is that more pressing competing issues could delay the start of statin therapy; for example, the treating clinicians might have thought that arranging an urgent cardiac catheterization in a critically ill patient was more important than starting statin therapy. These biases could lead to an association between not using statins and poor outcomes. Unfortunately, Spencer and colleagues did not present the kinds of analyses needed to detect or account for these biases, such as ensuring that all patients in the study had survived for a certain time period, assessing exposure to statin therapy after that time period, and using time-dependent covariates to represent statin use in the analysis (8). Can we adequately account for bias in observational studies? Researchers who use observational data are increasingly turning to propensity scores, which can create groups of patients who have a similar likelihood of receiving a therapy (10). Unfortunately, despite the claims of some authors, propensity scores have the same problem as traditional multivariate analysesthey cannot adjust study results for treatment group differences in the frequency of unknown and therefore unmeasured confounders. Our concerns about Spencer and colleagues' study notwithstanding, observational studies of treatment can provide useful information. They are especially valuable for studying patterns of real-world clinical practice and medication adherence and comparing them with evidence-based guidelines or standards of practice. They are also useful for detecting rare side effects of drugs, drugdrug interactions (11), and the impact of changes in practice or policy on an entire population (12). What are the clinical implications of Spencer and colleagues' study? Clinical trial evidence has established the effectiveness of statins in individuals who are at high risk for acute coronary syndrome, so such patients should already be receiving them before an admission for acute coronary syndrome. Clinicians can justify starting statin therapy early after admission for an acute coronary syndrome because postadmission statins were effective in a previous randomized trial (7). Furthermore, in vitro studies provide a biological basis for rapid effects of statins: immediate inhibition of smooth-muscle cell proliferation and stimulation of re-endothelialization (13). However, given the potential for bias in selecting patients to start statin therapy after admission, we believe that Spencer and colleagues overestimated the magnitude of this benefit of statins. In making this conclusion, we take support from another large observational study, which found no relationship between early initiation of statin therapy and improved outcomes (14). The current evidence leaves us uncertain about whether starting statin therapy at the time of admission for an acute coronary syndrome is more effective than starting this therapy later. The matter is important to resolve, and additional large randomized trials are needed to clarify the issue. While we wait for the results, we can reflect on the many reasons for caution in drawing firm conclusions from observational studies of treatment effectiveness.