Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care

Although recent reports on the quality of U.S. health care document improvement in many clinical areas (1, 2) quality deficits remain (35). Studies suggest that the U.S. health care system often fails to deliver effective (3), safe (69), and patient-centered care (10, 11)3 areas that the Institute of Medicine has identified as important for a well-functioning health care system (12). Lack of a transparent, explicit, systematic, data-driven performance measurement and feedback mechanism for health care providers has contributed to these failures (13, 14). Unlike information on quality of services and products, such as universities (15) and restaurants (16), U.S. consumers have limited access to information on health care providers, even though studies suggest an interest in comparative information (1719). The public release of performance data has been proposed as a mechanism for improving quality of care (2022) by providing more transparency and greater accountability of health care providers (13). According to Berwick and colleagues' framework for quality improvement (23), public reporting can improve performance through 2 pathways, which are interconnected by a provider's motivation to maintain or increase market share (Figure 1). In the selection pathway, patients or their intermediaries compare publicly released performance data and reward the better-performing providers by selecting (rewarding, recognizing, punishing, or paying) the provider. In the change (or quality improvement) pathway, performance data help providers identify areas in which they underperform and improve their performance (23, 24). Figure 1. Two pathways for improving performance through release of publicly reported performance data. Information from reference 23. Over the past decade, reporting systems that summarize publicly released performance data have proliferated (25, 26). In 2000, Marshall and coauthors (27) performed a systematic review on the use and impact of publicly released performance data. They noted that studies on the topic were scant and found that several reporting systems were evaluated repeatedly, and they concluded that hospitals seem to be most responsive to the data. In 2001, Schauffler and Mordavsky (28) concluded that publicly reported performance data did not meaningfully affect decision making, quality improvement activity, or competition. Since the publication of these 2 systematic reviews, many studies of publicly reported performance data have been published. We therefore performed a systematic review 1) to synthesize the evidence for using publicly reported performance data to stimulate quality improvement activity, affect selection of providers, and improve clinical outcomes (effectiveness, patient safety, and patient-centeredness), and 2) to assess the evidence for unintended consequences. Methods Data Sources Data sources included Web of Science, MEDLINE, EconLit, and Wilson Business Periodicals Abstracts. A professional librarian started with a forward search of Web of Science (Science and Social Science Citation Indexes) database on 2 seminal articles on public reporting that have extensive reference lists: Marshall and coauthors' systematic review (27) and Schneider and Lieberman's review (29). Additional searches of MEDLINE, EconLit, and Wilson Business Periodicals Abstracts by using keywords (Appendix Table 1) complemented this initial forward search strategy. We also searched the references of 3 published review papers (28, 3032) and 1 unpublished working paper by experts in the field (Romano PS, Rainwater J, Marcin JP. Review of the literature on the impact of public reporting of CABG surgery outcomes. In preparation.). Because Marshall and coauthors' review (27) included articles published between January 1986 and October 1999, we limited our database search to articles published between January 1999 and March 2006. Appendix Table 1. Search Strategies For articles published before 1999, our sampling frame was the reference list of Marshall and coauthors' systematic review (27). We judged this sampling frame to be adequate because the scope of their review was broader than that of our paper in that it covered not only the topic we sought to assess but also use of publicly reported performance data and the impact on costs in diverse populations. We also examined references cited in Schauffler and Mordavsky's review (28) and contacted experts on public reporting. Only English-language articles were included. Study Selection Three investigators discussed the inclusion criteria until consensus was achieved, refining coding conventions and the types of data that would be abstracted. Two authors independently evaluated articles for potential inclusion. We included articles that provided empirical evidence on the impact of public reporting on outcomes (effectiveness, patient safety, and patient-centeredness) and unintended consequences, as well as the selection and quality improvement activity (change) pathways (23). Table 1, Table 2, and Appendix Table 2 show the designs of eligible studies. We excluded opinion and theory articles, review articles, nonEnglish-language articles, historical descriptions, and articles on awareness or comprehension of publicly reported performance data that did not also measure a change in the 2 pathways, outcomes, or unintended consequences. Table 1. Effect of Public Release of Performance Data on Health Plan Quality* Table 2. Effect of Public Release of Performance Data on Individual Provider Quality* Appendix Table 2. Effect of Public Release of Performance Data on Hospital Quality Health's Comprehensive Assessment of Reform Options Initiative, which receives funding from RAND's corporate endowment, individual donors, corporations, foundations, and other organizations, supported this project. Only RAND Health staff played a role in the design, conduct, and reporting of the study. Results Identification of Evidence Our literature searches identified 2543 titles, from which we selected 143 articles for more detailed review. From these, we identified 14 articles that met our inclusion criteria. For articles published before 1999, we selected 18 of the 31 articles retrieved from Marshall and coauthors' review (27). The other 13 articles focused on comprehension and perceived usefulness of report cards and impact on costs, which were not the focus of our review. We identified an additional 13 articles through review of reference lists or contacting experts. Thus, we identified 45 pertinent articles (of which 27 were not included in Marshall and coauthors' review) (Figure 2). Figure 2. Study flow diagram. Table 1, Table 2, and Appendix Table 2 summarize our findings and ratings of each article. We organize our findings by reporting level. Within each level, we describe the effect on selection of providers, quality improvement activity, outcomes (effectiveness, patient safety, and patient-centeredness), and unintended consequences. Overall, health planlevel studies had higher global ratings than hospital-level or individual providerlevel studies. Quality improvement activity studies, on average, had low global ratings. Outcome studies overall had medium global ratings. Several reporting systems, most notably the New York State Cardiac Surgery Reporting System (NYS CSRS) and the Cleveland Health Quality Choice (CHQC) Program, were evaluated multiple times. Health Plans Selection Eight studies published after 1999 provide mixed conclusions about the effects of public reporting on selection of health plans. Three studies focused solely on data from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey, which describes patients' experiences with care (36). Two randomized, controlled trials (37, 38) of Medicaid beneficiaries' plan choices found no effect of CAHPS data on health plan selection overall. In a subgroup analysis, participants who read the report and chose an HMO with dominant market share selected plans with higher CAHPS scores compared with the control group (37). Spranca and associates (39), who used an experimental design involving hypothetical performance ratings, found that consumers in general preferred more expensive plans with better coverage but were willing to accept plans with less generous coverage if the health plan had high CAHPS ratings. Also using hypothetical plan performance ratings, Harris (40) found that people were willing to trade access restrictions for higher quality. Four studies used longitudinal observational data and econometrics models to examine the impact of providing health plan performance data. Beaulieu (41) found that Harvard University employees in plans with lower reported quality were more likely to switch plans than those enrolled in higher-quality plans. In Wedig and Tai-Seale's study of federal employees (42), a 1-SD increase in quality-of-care score was associated with an increased likelihood (>50%) of plan selection. Jin and Sorenson (43) found that among federal employees, a 1-SD increase in a quality score was associated with a 2.63percentage point increase in the likelihood of plan selection, although employees overall did not switch plans. Scanlon and coworkers (44) found that General Motors employees avoided plans with below-average ratings but were not strongly attracted to plans with superior ratings. Quality Improvement Activity We identified no studies of health plan performance data and quality improvement activity. Clinical Outcomes and Unintended Consequences In a retrospective cohort study using unadjusted data, Bost (45) found that health plans that voluntarily report performance data outperformed nonpublicly reporting plans in technical (Health Employer Data and Information Set) and patient experience (CAHPS courtesy and customer service measures) domains. In a retrospective cohort study, McCormick and colleagues (46) found that plans with lower quality-of-care scores are more likely than