Evidence-based Synthesis Progra

(ESP)

Public Presentation of Health S

or Facility Data about Quality or Facility Data about Quality 

Safety: A Systematic Revie

Annette M. Totten, PhD

Isomi M Miake Isomi

M. Miake

-Lye BA ,

Paul G. Shekelle, MD, PhD, Directo

West Los Angeles VA Medical CentWest Los Angeles VA Medical Cent

Los Angeles, CA

January 26, 2012 
Evidence-based Synthesis Progra

(ESP)

Acknowledgements

CoAuthors/Collaborators

Co-Investigators: 

Mar

y E. Vaiana, PhD

Research Associates:

Jessica M. Beroes, BS

This project was nominated by the Transparency Initiative 

Office of Quality and Performance, Maris Norwoo
Evidence-based Synthesis Progra

(ESP)

Disclosure

This report is based on research conducted by the Evidence This report is based on research conducted by the Evidence

-b

Synthesis Program (ESP) Center located at the West Los AngMedical Center, Los Angeles, CA funded by the Department o

Affairs Veterans Health Administration Office of Research anAffairs, Veterans Health Administration, Office of Research an

Development, Health Services Research and Development. Tand conclusions in this document are those of the author(s) w

responsible for its contents; the findings and conclusions do nresponsible for its contents; the findings and conclusions do n

necessarily represent the views of the Department of Veteranthe United States government. Therefore, no statement in th

should be construed as an official position of the Department should be construed as an official position of the Department 

Affairs. No investigators have any affiliations or financial invo(e.g., employment, consultancies, honoraria, stock ownership 

expert t ti t t t i d di 

t

testimony, gran

ts or pa

ten

ts receive

d or pending, or r

conflict with material presented in the report.
Evidence-based Synthesis Progra

(ESP)

VA Evidence-based Synthesis (E

Program Overview Program Overview

• Sponsored by VA Office of R&D and QUERI.• Established to provide timely and accurate Established to provide timely and accurate

syntheses/reviews of healthcare topics identified clinicians, managers and policy-makers, as they wimprove the health and healthcare of Veterans improve the health and healthcare of Veterans. 

• Builds on staff and expertise already in place at th

Evidence-based Practice Centers (EPC) designate

Four of these EPCs are also ESP Centers: 

o Durham VA Medical Center; VA Greater Los Angeles 

System; Portland VA Medical Center; and MinneapoliSystem; Portland VA Medical Center; and Minneapoli

Center.
Evidence-based Synthesis Progra

(ESP)

• Provides evidence syntheses on important clinica

topics relevant to Veterans, and these reports hel

o d l li i l li i i f d b id develop cli

nical poli

cies in

forme

d

by evidence, 

o the implementation of effective services to impro

outcomes and to support VA clinical practice gui

performance measures, and 

o guide the direction for future research to addres

clinical knowled

ge.

• Broad topic nomination process – e.g. VACO, VISN

facilitated by ESP Coordinating Center (Portland) 

online process: online process:

http://www.hsrd.research.va.gov/publications/esp/TopicNom
Evidence-based Synthesis Progra

(ESP)

• Steering Committee representing research and op

(PCS, OQP, ONS, and VISN) provides oversight an

program di ti rec on.

• Technical Expert Panel (TEP)

o Recruited for each to

pp p ic to 

provide content ex

pe

o Guides topic development; refines the key questo Reviews data/draft report.

• External Peer Reviewers & Policy Partners External Peer Reviewers

& Policy Partners

o Reviews and comments on draft report

• Final reports posted on VA HSR&D website and di

widely through the VA. 

http://www.hsrd.research.va.gov/publications/esp/repor
Public Presentation of Health Syst

Public Presentation of Health Syst

Facility Data about Quality and S

A Systematic Review A Systematic Review

Published October 2011

Full

-length report available on ESP website: length report available on ESP website:

http://www.hsrd.research.va.gov/publications/esp/repor
Background

o The public presentation of quality and safety data is 

the Department of Veterans Affairs (VA) commitmentransparency.

o By making data available VA hopes to engage vetera

families in care, promote informed choice, and stimuperformance improvement activities performance improvement activities. 

o The Department of Veterans Affairs (VA) “Open Gov

Plan” outlines the agency’s commitment to transparedefines transparency as both increasing access to puinformation and enabling better engagement and adbehalf of Veterans. 
Prior Reviews Prior Reviews

• Th h h d h l f 

is report is the third systematic review with a similar focus on 

of public reporting on performance. (Marshal, 2000 and Fung, 2

• Fung et al. identified 45 articles evaluating the impact of public 

quality-- 10 studies focused on public reporting of health plan d

focused on hospital data, and 11 focused on individual provider 

• These categories were not mutually exclusive, but we include o

articles examining public reporting of health plans or hospitals i

present report. 

• Theyg p categorized their data in two steps:

o First, by the level of data: health plan, hospital, or individuao Then by outcome: whether the public reporting targeted the 

pathway for improving performance influenced quality imprpathway for improving performance, influenced quality impr

activity, affected clinical outcomes, or had unintended conse(see Figure 2).
Publicly Reported 

P f Dt er

formance 

D

a

t

a

Publicly reported performance data

Knowledge

Selection Change

Motivation

Performance:

Effectiveness of care

Safety

Patient-centeredness

Unintended consequences

Figure 2. Two pathways for improving performance through release of publicly reported performance 
Objectives

• To update a systematic review of the evidence t

makingp p y erformance data publically available lea

improvements in quality of care and safety

o (Fung et al, published 2008, searched literature throu

• To summarize current research about patients’ a

families’ use of performance data and how the 

presentation and distribution of these data could presentation and distribution of these data could 

designed to maximize their use by veterans and 

members.
The Key Questions The Key Questions

• Key Question #1. What is known about the most effe

displaying quality and safety information, comparative dhealth sy , ,p stem structure, services, and 

performance so t

understandable?

• Key Question #2. How do patients prefer to receive 

information?

• Key Question #3. What is the evidence that patients 

families use publicly reported quality and safety informa

informed health care decisions? 

• Key Question #4. What is the evidence that public re

quality and safety information leads to improved quality quality and safety information leads to improved quality 
Search Strategy Search Strategy

• We searched Web of Science through 2010 for articles cit

seminal articles.

• We limited the search to peer We limited the search to peer-reviewed articles published 

reviewed articles published 

• Additional citations were identified from reference mining 

content experts. 

• We also conducted a web search by entering the terms “p

reporting of quality information healthcare” into Google a

the top 30 hits. the top 30 hits.
Study Selection Study Selection

• Two reviewers assessed for relevance the abstracts of cita

identified from literature searches. Full-text articles of po

relevant abstracts were retrieved for further review relevant abstracts were retrieved for further review. 

• Specific exclusion criteria were as follows:

We excluded 

o Studies regarding quality and safety information abou

homes, physicians or other individual providers Becauonly anticipating public reporting for facilities only anticipating public reporting for facilities

o Studies that did not address a key question o Non-sy , p

stematic reviews, commentaries or news repor

articles with no original data that provided only backginformation
Data Synthesis Data Synthesis

• We constructed evidence tables showing study 

subject of public reporting, whether the article dpubl f h l h l h l d 

blic reporting of hospital or health plan data, 

sample, study design, key findings and ratings o

quality of studies organized by key question quality of studies, organized by key question. 

• We analyzed studies to compare their character

methods and findings We compiled a summary methods, and findings. We compiled a summary 

findings for each key question based on qualitatsynthesis of the findings. 
Quality Assessment Quality Assessment

• We assessed study designs and stud

We assessed study designs and stud

importance in the literature accordin

criteria developed by Fung and collecriteria developed by Fung and colle

and used AMSTAR grading criteria f

systematic reviews. 
Fung Criteria Fung Criteria

Study design ratings:• 4 stars indicate a randomized trial or experimental trial;• 3 stars indicate a controlled trial, pre-post trial with contr

(controlled before-after trial);

• 2 stars indicate a pre 2 stars indicate a pre-post without control, observational 

post without control, observational 

study without multivariable adjustment, cross-sectional stwithout multivariable adjustment, analysis of time trends 

control or well control, or well-designed qualitative study; and designed qualitative study; and

• 1 star indicates a case series, other qualitative study, or s

(descriptive) study.

Global ratings:• 3 indicates great weight in the stratum’s body of evidence• 2 indicates moderate weight; and 2 indicates moderate weight; and

• 1 indicates little weight. 
Peer Review Peer Review

• Draft reviewed by seven technical expert

well as by clinical leadership. well as by clinical leadership.

• Reviewer comments were addressed and 

responses were incorporated in the final responses were incorporated in the final 
Literature Flow Literature Flow

Literature search

N= 370

Rejected at title sc

Content Experts

N=7

R f Mi i N 1

Articles Retrieved

N= 117

N= 261

Rejected at full screenN=99

Reference Mining N=1

N 117

Included in Fung revieOnly addressed indivi

Does not address a KQ

Design: 9

Background: 28Other: 18

Included: 18 Fung Review Google Search Review

N=37 (See Table 1)

KQ1&2:

9 New

KQ3:

17 in Fung review

6 New

KQ4:

25 in Fung review

7 New
Results: Key Questions 1 and 2 Results: Key Questions

1 and

2

• We identified reports commissioned by AHRQ and the Ro

Johnson Foundation regarding how to best produce and d

public reports. public reports.

• Their conclusions about the design of public reports were

o To make the information more relevant to what consumers 

and b t bli t h ld i ll d fi it

d care a

bou

t, public repor

ts s

hould give an overall

d

efi

nit

quality, define the elements of quality and use them as the categories, and include information about the sponsor and m

o

To mak f d dd h ke it easy 

for consumers to un

derstan

d an

d use t

he c

information reports should summarize, interpret, highlight mnarrow options and help bring the information together for 

usi d i fl bl in

g summary measures an

d meaningful sym

b

ols. 

o Finally, testing reports with consumers during development 

identify areas of misunderstanding and assess users’ percepreport’ l s va ue.
Results: Key Question 3 Results: Key Question

3

• Conclusions are mixed

o Most studies found the use of publicly available data t

mod b est at 

best. 

o Although consumers may show interest in public repo

cases interest does not seem to translate into actual ucases interest does not seem to translate into actual u

o The studies that do show use of publicly available dat

that consumers may avoid low performers, but higher performers may not reap comparable positive benefits performers may not reap comparable positive benefits 

reporting.
Results: Key Question 4 Results: Key Question

4

• We identified relatively few new studies within our scope that h

appeared in the peer reviewed literature during the five years ssearch was conducted for Fung et al. 

• Five new studies identified address a variety of outcomes (patie

consumer experience, obtaining performance targets, rates of cand mortality) and four of the five are national studies. 

o All five conclude that public reporting has a positive impact on qual

outcomes; however, the effects were small. 

o Two studies were time series studies in a single country, where all 

were subject to public reporting and the change, each could have bother changes that impacted all providers. 

• This small and varied amount of additional evidence is not suffic

ch th l i f th F t l i th t “th ff t 

han

ge the conclusion o

f the 

Fun

g e

t al. review th

a

t “the effect 

reporting on effectiveness, safety, and patient-centeredness remuncertain.” However, the CHOP assessment from 2005 provides 

encouragement that this may be changing encouragement that this may be changing. 
Limitations

• Principal limitation: the review is the limited number of p

reporting systems that have been subjected to critical, puevaluations. 

o Possibility of publication bias: additional evaluations may ha

conducted but the results are not easily available in the pubviewed literature or available at all. 

• Other potentially relevant evaluations and studies probab

but cannot be identified and synthesized based on an exa

of databases that are easily searchable For example entof databases that are easily searchable. For example, ent

“public reporting of quality information” into Google prod19,000,000 hits, a number that is impractical to review. 

o Even using limited search terms produces tens of thousand

did incorporate a limited Google search, but did not identify studies in the top 30 hits. 
Reporting Systems Represented Reporting Systems Represented

NYS CSRS

15

CAHPS

7

CHQC

5

PA

3

HEDIS

3

CA

4

222

11111

NYS CSRS=New York State Cardiac Surgery Reporting System; CAHPS=Consumer Assessment of Health PlansCHQC= Cleveland Health Quality Choice program; HEDIS= Healthcare Effectiveness Data and Information Set; 

CA= Public re

portin

g s

ystems in California, includin

g the California Hos

pital Outcomes Project (CHOP); PA=Penn

p gy g p j ( ) 

Two mentions: Federal Employee Health Benefit guide; QualityCounts; public reporting from the UK, including th

One mention: Health Care Financing Administration (HCFA); Buyers Health Care Action Group (BHCA); I

nstitute for Clinical Evaluative Sciences (ICES) Ontario; Missouri Department of Health’s obstetrics consumer repo

Dutch national survey of consumer experience measures (based on CAHPS); national caesarean rates in South K
Conclusions

• Even with these limitations....• The evidence is consistent that most consumers 

know about or make little use of publicly availabperformance data when selecting health serviceproviders. 

• Attention to design, presentation and dissemina

more fully engage consumers more fully engage consumers. 

• In addition, public reporting furthers the VA’s go

transparency.
Bill Clinton Bill Clinton
Recommendations 

f Ft R h for 

F

u

ture 

Researc

h

• As VA pursues its transparency goals and continues to ex

quality and safety information made available to Veterans 

stakeholders, there is an opportunity to increase the impastakeholders, there is an opportunity to increase the impa

public reporting on the wellbeing of veterans and to contrthe knowledge related to public reporting and quality imp

in health care in health care. 

• Examples of specific questions that could be answered by 

appropriate research include: 

o What health care decisions do veterans and their fam

and what kinds of information needs do they have? Hwant to receive or access data about quality? want to receive or access data about quality?

o Are veterans aware of the VA’s public reporting websi

often have they accessed the website? Do they underinformation being presented? information being presented?

o How well is VA’s public reporting meeting Veteran nee
Additional Resources Additional Resources

• AHRQ ‘Talking Quality Talking Quality’ website website

o Quality Initiatives that include Public Reportin

3⁄4 https://www talkingquality ahrq gov www.talkingquality.ahrq.gov

/default aspx default.aspx

• AHRQ EPC Review

bl f8 2

d

o Public Reporting 1 o

f

8 topics in 

2

f

n

d series o

f 

“Closing the Quality Gap”

o I di id l t t f k Indi

vidual repor

t: nex

t

few wee

k

s

o Series overview in Juneo htt // p:

eff ti h lth h ffective

healthcare.a

hrq.gov

/
Evidence-based Synthesis Progra

(ESP)

Questions?

If you have further questions, 

feel free to contact:

Coordinator: Jessica M. Beroes

(310) 478-3711 ext. 48354

Jessica.Beroes@va.gov

The full report and The full report and cyberseminar cyberseminar presentation is available on the 

presentation is available on the 

http://www.hsrd.research.va.gov/publications/esp/