Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences.

Health care delivery in the United States is changing at a dramatic pace. Millions of uninsured citizens, unsustainable cost growth, and uneven quality of care have prompted numerous policy responses at the state and national level. The 2010 Patient Protection and Affordable Care Act includes landmark expansions to health insurance and the introduction of delivery system reforms to reduce costs and improve quality. Evaluating the effectiveness of these efforts is essential in understanding what works—and what doesn’t—so policy can be redesigned accordingly. Randomized control trials, the gold standard for understanding causal relationships, are impractical or impossible in many circumstances, and are rarely used to evaluate public policies or large-scale delivery system interventions. To maximize learning from this rapidly changing environment, observational studies are needed. There are many challenges, however, in drawing valid conclusions from observational study designs. Difference-in-differences methods have emerged as a powerful tool to address confounding in observational studies and evaluate the impact of health care policies. Although the implementation of difference-in-differences models can be complex, the idea is simple. First, identify an intervention, an outcome of interest, and two groups—one that was exposed to the intervention (the treatment group) and one that was not (the comparison group). Then, take the difference in the outcome between the treatment and comparison groups at baseline (difference 1) and again after the intervention (difference 2) has occurred. The policy effect is estimated as the difference-in-differences (difference 2—difference 1). We can conclude that the intervention had an impact if the outcome changed more for the treatment group than the comparison group. If the differences are the same between the two groups, then there was no effect of the intervention (see Figure​Figure1).1). For example, a recent study used difference-in-differences methods to evaluate the impact of Medicare’s bariatric surgery coverage decision, which limited Medicare reimbursement to “Centers of Excellence.” The study compared rates of surgical complications in Medicare patients (whose care was subject to the policy) and commercially insured patients (whose care was not subject to the intervention) before and after the policy was initiated in 2006. Prior studies, using a simple “pre-post” study design found a beneficial impact of the Centers of Excellence program, but had failed to account for secular trends toward improved outcomes (Nguyen et al. 2010; Flum et al. 2011). Using difference-in-differences, we found similar declines in complications for both the Medicare and commercially insured patients, suggesting that the program did not lead to a reduction in complications (Dimick et al. 2013). Figure 1 Conceptual Depiction of Difference-in-Differences AnalysisNote: Difference-in-differences estimate is equal to: (T2-C2)—(T1-C1). The increasing popularity of difference-in-differences in health policy and medicine (Figure​(Figure2)2) is a result of both the credibility of these designs and their ease of implementation and estimation (Angrist and Pischke 2010). Recent difference-in-differences studies have evaluated the impact of health insurance expansions (Cunningham, Hadley, and Reschovsky 2002; Zhu et al. 2010; Long and Stockley 2011; Blum et al. 2012; Cantor et al. 2012; Graves and Gruber 2012; Dhingra et al. 2013; Joynt et al. 2013), high deductible health plans (Wharam et al. 2012; Kozhimannil et al. 2013), changes in payment policy and providers’ financial incentives (Shen and Zuckerman 2005; Dusheiko et al. 2006; Mitchell 2008; Scanlon et al. 2008; Song et al. 2011; Jha et al. 2012; Sutton et al. 2012; Kantarevic and Kralj 2013; Werner, Konetzka, and Polsky 2013), malpractice reform (Kessler, Sage, and Becker 2005), behavioral health parity laws (Goldman et al. 2006; Azrin et al. 2007), resident work hour reform (Volpp et al. 2007), public quality reporting (Dranove et al. 2003; Chen and Meinecke 2012; Joynt et al. 2012; Ryan, Nallamothu, and Dimick 2012b), changes in clinical practice (Baxter, Ray, and Fireman 2010; Zivin et al. 2010; Leonhardt et al. 2011; Suehs et al. 2014), smoking laws (Anger, Kvasnicka, and Siedler 2011; Nguyen 2013), electronic medical record implementation (Jones et al. 2010; McCullough, Christianson, and Leerapan 2013), and conflict of interest policies (Epstein et al. 2013). Figure 2 Health Policy and Medicine Articles Using Difference-in-Differences Analysis, 1997–2012Source: PubMed database. Search term “difference-in-differences” or “difference-in-difference.”