Evaluation of Sutter Health Programs Utilizing the Synthetic Control Method: A Secondary Analysis of Electronic Health Record Data

Background: Health programs and interventions designed to improve patient outcomes within health care systems are implemented widely; however, many are implemented without a plan for evaluating the impact of the program on relevant and meaningful outcomes. Program evaluations are important for providing context, assessing whether the intervention is having its desired effect, deciding whether the program should be continued, identifying any unintended consequences, and highlighting areas for improvement. Many health programs fall under the category of observational studies, and thus methods such as matching, difference-in-differences (DiD), regression discontinuity, and pre-test/post-test are often used for evaluation. Recently, the synthetic control method (SCM) has surfaced as an important tool for program evaluation and has been described as “arguably the most important innovation in the policy evaluation literature in the last 15 years” 1. SCM is motivated by the common difficulty in identifying a single control unit that approximates the most relevant characteristics of the treated unit. The central idea of SCM is that a combination of control units may provide a better “counterfactual” for the treated unit than any one single control unit alone. A data-driven approach is used to assign weights to potential control units to create a “synthetic” version of the treated unit that closely approximates the time series for the actual treated unit in the pre-intervention period. With this, predictions about what counterfactual trends would look like in the post-intervention period, had the intervention never been implemented, can be made./nMethods: In this dissertation, I apply SCM to evaluate hospital-level effects of three programs recently implemented within the Sutter Health system: (1) the Advancing Health Equity (AHE) asthma program at Alta Bates Summit Medical Center that brings culturally appropriate community-based care to African American/Black patients, and provides high-touch and high-tech counseling services to educate patients about disease and medication self-management; (2) the ETOH-P program at Eden Medical Center that implements two protocols for treating patients who are at risk for developing alcohol withdrawal syndrome; and 3) a group-based lifestyle change program implemented within several Sutter clinics for diabetes management. For each program, we compare the hospital-level results from the SCM analysis to individual-level results from a propensity score matched analysis./nSignificance: This dissertation illustrates the application of SCM to evaluate the impact of health care programs implemented within an open health care system such as Sutter Health. SCM has previously been applied to study a wide range of topics including political and economic effects following terrorist conflict2, state-level policy changes3. health systems reforms4,5, nutritional interventions6, climate events such as drought7, and most recently COVID-19 mitigation strategies and mandates8–14. To our knowledge, SCM has never been applied in such a setting, in which individuals choose to participate in the program or intervention. Lessons learned from this exercise provide valuable insight into the utility of this evaluation tool for health care systems research and offer both data and methodological considerations for future applications.

Advisor: Colford, John M
Reingold, Arthur L