Evidence-based Synthesis Program Department of Veterans Affairs

Health Services Research & Development Service Evidence-based Synthesis Program 

Public Presentation of Health 

System or Facility Data 

about Quality and Safety: 

A Systematic Review

Department of Veterans Affairs

Health Services Research & Development Service

October 2011

Investigators:

Principal Investigator: 

Paul G. Shekelle, M.D., Ph.D.

Co-Investigator: 

Isomi M. Miake-Lye, B.A.

Annette M. Totten, Ph.D.

Mary E. Vaiana, Ph.D.

Research Associates:

Jessica M. Beroes, B.S.

Prepared for:

Department of Veterans Affairs

Veterans Health Administration

Health Services Research & Development Service

Washington, DC 20420

Prepared by:

Evidence-based Synthesis Program (ESP) Center

West Los Angeles VA Medical Center

Los Angeles, CA

Paul G. Shekelle, M.D., Ph.D., Director
ii

Public Presentation of Health System or Facility Data

about Quality and Safety: A Systematic Review Evidence-based Synthesis Program

PREFACE

Health Services Research & Development Service’s (HSR&D’s) Evidence-based Synthesis 

Program (ESP) was established to provide timely and accurate syntheses of targeted healthcare 

topics of particular importance to Veterans Affairs (VA) managers and policymakers, as they 

work to improve the health and healthcare of Veterans. The ESP disseminates these reports 

throughout VA.

HSR&D provides funding for four ESP Centers and each Center has an active VA affiliation. The 

ESP Centers generate evidence syntheses on important clinical practice topics, and these reports 

help:

• develop clinical policies informed by evidence,

• guide the implementation of effective services to improve patient outcomes and to 

support VA clinical practice guidelines and performance measures, and 

• set the direction for future research to address gaps in clinical knowledge.

In 2009, the ESP Coordinating Center was created to expand the capacity of HSR&D Central 

Office and the four ESP sites by developing and maintaining program processes. In addition, 

the Center established a Steering Committee comprised of HSR&D field-based investigators, 

VA Patient Care Services, Office of Quality and Performance, and Veterans Integrated Service 

Networks (VISN) Clinical Management Officers. The Steering Committee provides program 

oversight, guides strategic planning, coordinates dissemination activities, and develops 

collaborations with VA leadership to identify new ESP topics of importance to Veterans and the 

VA healthcare system.

Comments on this evidence report are welcome and can be sent to Nicole Floyd, ESP 

Coordinating Center Program Manager, at nicole.floyd@va.gov.

Recommended citation: Totten AM, Miake-Lye IM, Vaiana ME, Beroes JM, Shekelle PG. 

Public Presentation of Health System or Facility Data about Quality and Safety: A Systematic 

Review. VA-ESP Project #05-226; 2011

This report is based on research conducted by the Evidence-based Synthesis 

Program (ESP) Center located at the West Los Angeles VA Medical Center, Los 

Angeles, CA funded by the Department of Veterans Affairs, Veterans Health 

Administration, Office of Research and Development, Health Services Research 

and Development. The findings and conclusions in this document are those of the 

author(s) who are responsible for its contents; the findings and conclusions do not 

necessarily represent the views of the Department of Veterans Affairs or the United 

States government. Therefore, no statement in this article should be construed as 

an official position of the Department of Veterans Affairs. No investigators have any 

affiliations or financial involvement (e.g., employment, consultancies, honoraria, stock 

ownership or options, expert testimony, grants or patents received or pending, or 

royalties) that conflict with material presented in the report.
1

Public Presentation of Health System or Facility Data

about Quality and Safety: A Systematic Review Evidence-based Synthesis Program

EXECUTIVE SUMMARY

BACKGROUND

The public presentation of quality and safety data is essential to the Department of Veterans Affairs 

(VA) commitment to transparency. By making data available VA hopes to engage veterans and 

families in care, promote informed choice, and stimulate performance improvement activities. 

The objectives of this project are: 1) to update a recent systematic review of the evidence that 

making performance data publically available leads to improvements in quality of care and 

safety; and 2) to summarize current research about patients’ and families’ use of performance 

data and how the presentation and distribution of these data could be designed to maximize their 

use by veterans and family members. 

The Key Questions were:

1. What is the most effective way of displaying quality and service information so that it is 

understandable?

2. How do patients prefer to receive or access this information?

3. What is the evidence that patients or their families use publicly reported quality and safety 

information to make informed health care decisions?

4. What is the evidence that public reporting of quality and safety information leads to 

improved quality of safety? 

METHODS

We searched Web of Science through 2010 using standard search terms. We limited the search to 

peer-reviewed articles published in the English language. Additional citations were identified from 

reference mining and content experts. Titles, abstracts, and articles were reviewed in duplicate by 

reviewers trained in the critical analysis of literature. All data were narratively summarized.

Study characteristics and key findings were extracted by trained research associates under 

the supervision of the Principal Investigator. We assessed study quality according to criteria 

developed by Fung and colleagues, and used AMSTAR grading criteria for systematic reviews.

DATA SYNTHESIS

We constructed evidence tables showing study objective, subject of public reporting, whether the 

article discusses public reporting of hospital or health plan data, location, sample, study design, 

key findings and ratings, organized by key question. We analyzed studies to compare their 

characteristics, methods, and findings. We compiled a summary of findings for each question 

based on qualitative synthesis of the findings. 

PEER REVIEW

A draft version of this report was reviewed by seven technical experts, as well as by clinical 

leadership. Reviewer comments were addressed and our responses were incorporated in the final 

report.
2

Public Presentation of Health System or Facility Data

about Quality and Safety: A Systematic Review Evidence-based Synthesis Program

RESULTS

We screened 370 titles and rejected 261, and performed a more detailed review on 117 articles. 

From these, we identified 55 articles that addressed one of the key questions. 

Key Questions #1 and #2

We identified reports commissioned by AHRQ and the Robert Wood Johnson Foundation 

regarding how to best produce and disseminate public reports. Their conclusions about solutions 

for the design of public reports are three-fold. To make the information more relevant to what 

consumers already understand and care about, public reports should give an overall definition 

of quality, define the elements of quality and use them as the reporting categories, and include 

information about the sponsor and methods. To make it easy for consumers to understand and 

use the comparative information summarize, interpret, highlight meaning, narrow options and 

help bring the information together in a choice by using summary measures and meaningful 

symbols. Finally, testing reports with consumers during development will help identify areas of 

misunderstanding and assess users’ perceptions of the report’s value.

Key Question #3

Conclusions from the studies of public reporting are mixed, but most studies found the use of 

publicly available data to be modest at best. Although consumers may show interest in public 

reports, in most cases interest does not seem to translate into actual use. The studies that do 

show use suggest that consumers may avoid low performers, but higher performers may not reap 

comparable positive benefits of public reporting.

Key Question #4

We identified relatively few new studies within our scope in the peer reviewed literature during 

the five years since the search was conducted for Fung et al. Two of the newly identified studies 

addressed the impact of reporting on quality improvement activities. Some empirical evidence 

and the conclusion of the prior review support the theory that public reporting stimulates quality 

improvement activities. Five new studies identified address a variety of outcomes (patient or 

consumer experience, obtaining performance targets, rates of caesarean and mortality) and four 

of the five are national studies. All five conclude that public reporting has a positive impact 

on quality or safety outcomes; however, the effect was small and two studies were time series 

studies in a single country, where all providers were subject to public reporting and the change, 

each could have been due to other changes that impacted all providers. 

This small and varied amount of additional evidence is not sufficient to change the conclusion 

of the Fung et al. review that “the effect of public reporting on effectiveness, safety, and patientcenteredness remains uncertain.” However, the CHOP assessment from 2005 provides some 

encouragement that this may be changing.