Resolving conflicting comparative effectiveness research in critical care

Comparative effectiveness research (CER) has received keen interest from healthcare professionals, funding agencies, and researchers as a way to improve healthcare quality, efficiency, and costs. Broadly defined, CER includes both observational studies and randomized controlled trials (RCTs). Of these, RCTs are widely considered near the top of the hierarchy of scientific evidence. In critical care, however, RCTs are clearly not feasible for every research question. They are expensive, time consuming, restricted to narrow patient populations, and usually conducted in an academic community setting. All of these issues limit their practicality and generalizability. As a result, observational research is an important and increasing complement to RCTs as we develop best practice in the intensive care unit (ICU).(1) In contrast to RCTs, observational research is often more efficient and generalizable.(2) Administrative databases and clinical registries offer opportunities to study large, diverse patient populations with relative ease, and emerging statistical techniques such as propensity scores and instrumental variables purport to overcome the indication biases that traditionally plague observational studies of treatment effects.(3) Yet even in these cases, the results of observational trials often conflict with RCTs, leading to erroneous conclusions and inappropriate treatment recommendations.(4) In these cases, it is left to the consumers of research—patients, clinicians and policy makers—to reconcile the results. In this issue of Critical Care Medicine, Rimmer and colleagues present us with that very challenge.(ref) They report a multi-national, multi-center observational study of recombinant human activated protein C (rhAPC, drotrecogin alfa) among 7392 subjects with septic shock enrolled from 1997 to 2007. They observed an overall low rate of rhAPC use among patients with septic shock (4.7%), and a significant association between rhAPC and 30-day mortality in a propensity matched analysis (hazard ratio = 0.72, 95%CI 0.52–1.00, p=0.05). Such findings in favor of rhAPC use are consistent an early randomized trial and multiple subsequent observational studies.(5, 6) And yet, Eli Lilly and Company conducted a confirmatory, randomized trial (PROWESS-SHOCK) to refine target population and risk-benefit ratios for rhAPC. This trial found no significant benefit for rhAPC on 28-day mortality, and the drug was withdrawn worldwide.(7) So when studies conflict, what is the “right” answer? The first inclination is to rely on RCTs, and indeed there are many reasons why their conclusions are likely to be most reliable and valid. In contrast to other study designs, through the act of randomization RCTs protect against a common source of confounding termed “indication bias.” This bias arises from the notion that patients who receive a treatment are fundamentally different than those who don’t.(8) For example, ICU patients with severe sepsis who require vasopressors are more severely ill than those who don’t. If we were to study the outcomes of patients who received vasopressors against controls in a large cohort study, our results may typically show that vasopressors are associated with harm. Yet these results would likely be due to the fact that treated patients were sicker in ways that cannot be measured through traditional approaches. To attempt to overcome indication bias in their observational study of rhAPC, Rimmer and colleagues applied propensity score methods. They used all the variables they could measure to predict the probability a patient would receive rhAPC (“the propensity score”). Then they matched subjects receiving rhAPC to control subjects who did not on the propensity score. The propensity score matching helps balance baseline characteristics – a sort of pseudo-randomization. Yet in the end, propensity score methods are only as good as the variables measured in the dataset. The cannot account for unmeasured covariates, especially the subtle “unmeasurables” that govern the decision to prescribe a drug like rhAPC. Thus they are far from a panacea as a method to remove bias.(3) Only randomization can fully overcome these biases, supporting the role of RCTs as the best available clinical evidence (5). Yet there are also valid reasons why observational studies can also be “right”, and can indeed provide better data that even RCTs. Unlike RCTs, which typically are conducted in the academic setting under ideal conditions and with innumerable exclusion criteria, observational studies provide invaluable knowledge of practice patterns, treatment effects among “real” patients, and study phenotypes or sub-phenotypes not necessarily enrolled in pre-approval randomized trials. They also may study patient-centered outcomes not mandated by drug approval agencies, and could include more patients with greater power to detect small differences in treatment effects. Negative RCTs may be negative not because the drug doesn’t work, but because the study was not conducted in the right setting, in the right patients, or with the power to detect clinically meaningful effect sizes in small subgroups. Ignoring this reality downplays the important role of observational studies in CER, so long as that research is rigorously conducted. Recently, the Patient-Centered Outcomes Research Institute (PCORI) offered methodological standards for observational research that compliment standards for reporting.(9, 10) These provide a roadmap for the future of comparative effectiveness research in critical care, without always relying on a randomized design. The correct answer, then, is that neither RCTs nor observational research is always “right”. Instead, we should approach each individual study with a critical eye weighing it’s strengths and limitations in the context of the previous literature and our prior beliefs. This is the idea of “Bayesian reasoning”—rather than providing irrefutable evidence, new clinical research should serve to inform and improve our decisions, not completely dictate our practice.(11) Indeed, up to one third of the most cited clinical research trials are refuted or found to overestimate treatment effects, including many in critical care.(12–14) Rather than expect perfection out of research, both clinicians and researchers must sharpen their interpretations and be wary of the strengths and weaknesses of randomized and non-randomized study designs. We must move away from labeling studies “right” or “wrong”. As long as they are well-conducted, conflicting studies provide complimentary findings that build a patchwork of evidence with which to guide our decisions. Of course, in the case of rhAPC this discussion is purely academic. Now off the market, the drug will not be given to patients in septic shock. Yet the conflicting evidence between early RCTs and post-approval observational studies is a warning for when we encounter the next rhAPC. The only “wrong” action is to be overly optimistic about a single RCT or to overly dismiss observational studies because patients are not randomized. Just like at the bedside in the ICU, our interpretation of comparative effectiveness research should consider the entire body of evidence.