Using Design Thinking to Differentiate Useful From Misleading Evidence in Observational Research.
(2017)

Few issues can be more important to physicians or patients than that treatment decisions are based on reliable information about benefits and harms. While randomized clinical trials (RCTs) are generally regarded as the most valid source of evidence about benefits and some harms, concerns about their generalizability, costs, and heterogeneity of treatment effects have led to the search for other sources of information to augment or possibly replace trials. This is embodied in the recently passed 21st Century Cures Act, which mandates that the US Food and Drug Administration develop rules for the use of “real world evidence” in drug approval, defined as “data...derived from sources other than randomized clinical trials.”1 A second push toward the use of nontrial evidence is based on the perception that the torrent of electronic health-related data—medical record, genomic, and lifestyle (ie, “Big Data”)—can be transformed into reliable evidence with the use of powerful modern analytic tools. The use of nontrial evidence puts experts who weigh medical evidence, no less a practicing clinician, in a quandary, as the evaluation of such information requires skills well beyond those of basic critical review. The didactic tutorial by Agoritsas et al2 in this issue of JAMA partly addresses that problem, providing a clear, concise, and accessible introduction to several statistical approaches to adjust analyses of observational studies to minimize or eliminate confounding: matching, standard regression, propensity scores, and instrumental variable analyses. While understanding these technical tools is valuable, valid causal inference from nonrandomized studies about treatment effects depends on many factors other than confounding. These include whether the causal question motivating the study is clearly specified, whether the design matches that question and avoids design biases, whether the analysis matches the design, the appropriateness and quality of the data, the fit of adjustment models, and the potential for model searching to find spurious patterns in vast data streams. Some of these issues are recognizable and remediable, whereas others may defy solely analytic solutions, leading to the rejection of some nonrandomized studies to guide treatment. The hallmark of a well-posed causal question is that one can describe an RCT that would answer it. An RCT requires specifying at minimum an eligible population, an intervention, a comparator, and a start time. As posed by Hernán and Taubman,3 an example of a noncausal question is “Does obesity shorten life?” This is not causal because “obesity” is not a randomizable intervention, nor is weight loss. Randomizable interventions—the goal of which might be weight loss—include diet, exercise, gastric bypass surgery, liposuction, or drugs. Once the causal question is specified, the experimental design must mirror it, creating the foundation for a “like vs like” causal contrast on which a valid analysis is based. When observational data not collected for research are analyzed to estimate treatment effects, sometimes they are fed into statistical software without an explicit design and with a causal question that is either imprecise or not stated. This sets the stage for invalid results, even when the statistical methods appear to be properly implemented.4,5 The example of postmenopausal hormone therapy (HT) illustrates the criticality, when analyzing observational data, of posing a precise causal question, with a design and analysis to match. Observational studies, including one from the Nurses’ Health Study, using some of the adjustment techniques described by Agoritsas et al, showed a strong cardiovascular protective association with HT. This result was famously contravened by a subsequent RCT, the Women’s Health Initiative, which showed cardiovascular harm from HT. A subsequent reanalysis of the Nurses’ Health Study data conformed with the RCT results.6 The reanalysis was structured to mirror the clinical question asked in the trial; viz, how does initiating HT in a postmenopausal woman affect her cardiovascular risk? The original observational analysis, using standard regression, effectively excluded women with coronary heart disease events that occurred shortly after initiation of HT, a remediable design bias known in pharmacoepidemiology as “depletion of susceptibles”7 or in other fields as “survivorship bias.” No amount of adjustment for baseline factors between ongoing HT users and nonusers can overcome that bias, but by refining the question, and then using the appropriately tailored design, an appropriate analysis can be crafted. Following the causal thinking reflected in RCT design, most successful examples of observational research that produce findings similar to those from RCTs have compared new drug users with new users of a comparator agent.8,9 The disproven nonrandomized studies referenced at the end of the article by Agoritsas et al all compared users with nonusers. In an RCT, a nonuser is always someone who was eligible and considered for treatment, but that is not guaranteed in an observational study. If instead 2 active treatments indicated for the same condition are compared, it is likely Related article page 748 Opinion