![](media/image1.png){width="2.1770833333333335in" height="0.3125in"}

**Guidance for Using the AHRQ Quality Indicators for Hospital-level
Public**

**Reporting or Payment**

![](media/image2.jpeg){width="5.375in" height="4.555555555555555in"}

+-----------------------+-----------------------+----------------------+
| **Prevention**        | **Inpatient**         | **Patient Safety**   |
|                       |                       |                      |
| **Quality             | ##                    | ###                  |
| Indicators**          | ## Quality Indicators | # Quality Indicators |
+-----------------------+-----------------------+----------------------+

![](media/image3.png){width="2.1354166666666665in"
height="0.6145833333333334in"}

**\[This page intentionally left blank\]**

**Guidance for Using the AHRQ Quality Indicators**

**for Hospital-level Public Reporting or Payment**

**Prepared by**

**Denise Remus, Ph.D., R.N.**

**Irene Fraser, Ph.D.**

**U.S. Department of Health and Human Services**

**Agency for Healthcare Research and Quality**

Rockville, Maryland

[http://www.qualityindicators.ahrq.gov](http://www.qualityindicators.ahrq.gov/)

AHRQ Publication No.04-0086-EF

August 2004

**Acknowledgments**

In preparing this document, we relied very heavily on the experiences
and suggestions of the many researchers, quality experts, and reviewers
and users of the Quality Indicators and other quality measures who have
talked with AHRQ and with our user support center. Earlier drafts of
this document were shared broadly for comment as well, and we are most
appreciative of the time and advice of these early reviewers. In
particular, we gratefully acknowledge the assistance of Chris Queram,
Cheryl DeMars and John Bott (The Alliance); Cheryl Damberg and
colleagues (the Pacific Business Group on Health); Ken Kizer, Ellen
Kurtzman, and Robyn Nishimi (the National Quality Forum); Sheila Roman
and colleagues (Centers for Medicare & Medicaid Services); Patrick
Romano, Kathy McDonald, Sheryl Davies, and Jeff Geppert (Stanford
University and UC-Davis); Mark Gritz and Greg Hubert (Battelle,
International); Arnold Milstein (Pacific Business Group on Health and
Mercer HR Consulting); and Carolyn Clancy, Dan Stryer, Jean Slutsky,
Christine Williams, Peggy McNamara, Anne Elixhauser, and other AHRQ
colleagues. Special thanks to Donna Rae Castillo for document editing
and formatting; and Jan de la Mare, Bob Isquith, and Farah Englert for
their assistance with internal reviews and dissemination. Any errors in
interpretation or incorporation of their suggestions are strictly those
of the authors.

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

This report is in the public domain and may be reproduced without
permission. AHRQ appreciates citation as to source and the suggested
citation is as follows:

Remus D, Fraser I. Guidance for Using the AHRQ Quality Indicators for
Hospital-level Public Reporting or Payment. Rockville, MD: Department of
Health and Human Services, Agency for Healthcare Research and Quality;
2004. AHRQ Pub. No. 04-0086-EF

**Preface**

State data organizations, public payers, employers, health plans, and
others seeking to improve health care quality have a growing interest in
using public reporting and quality-based payment strategies, and have
begun to include AHRQ's Quality Indicators (QIs) as measures in these
efforts. The perceived advantage of the QIs is that they measure
outcomes that consumers care about; these indicators also use
administrative data, which are relatively inexpensive, readily
available, and already used for hospital quality improvement. Because
AHRQ originally developed and refined these indicators for use in
quality improvement and national tracking, there has been some question
about how and if to use them in these new ways. The purpose of this
document is to provide guidance on this question.

This guide briefly examines the fit between criteria used for developing
the Quality Indicators and those used by the National Quality Forum as
part of its national consensus process on public reporting. It concludes
that the fit is quite close. It then illustrates some typical uses of
the QIs and identifies the factors organizations should consider in
selecting which specific QIs to include in their particular public
reporting or payment efforts. The intent is not to endorse a national
approach or provide a one-size-fits-all assessment of which set of
measures to use. Local areas and particular users are likely to have
different uses (public reporting vs. paying for quality vs. paying for
tracking the data), priorities (greater or lesser emphasis on cardiac
care, for example), markets (a large or small number of high-volume
hospitals) and data situations (variations in data completeness and
coding quality). Rather, the goal is to enable organizations considering
use of the Quality Indicators to use the best science in making and
implementing their decisions. The guide also includes some
recommendations on how to use the QIs most productively, for example by
involving hospitals early in the measurement efforts, using the QIs as
part of a broader dashboard, presenting outcomes with process measures,
pairing mortality and volume indicators, using multiple years of data,
and evaluating the use of composite measures.

This document will not answer all of the technical questions a potential
user might have. It is one of many tools we are producing to support use
of the QIs. Other resources include the availability of user support;
development of a QI training curriculum with modules specific to
providers, purchasers and policymakers; and development of QI software
which will incorporate data quality edits and will not require access to
statistical software packages.

All QI software and documentation is publicly available at no charge
from the AHRQ QI Web site
[http://www.qualityindicators.ahrq.gov](http://www.qualityindicators.ahrq.gov/).
For further information or answers to technical questions, users should
feel free to contact our Support for Quality Indicators contract team at
<support@qualityindicators.ahrq.gov>. We would also welcome suggestions
of further guidance materials that might be helpful. Thank you for your
interest in the AHRQ QIs.

+-----------------------------------------+----------------------------+
| Irene Fraser, Ph.D., Director           | Denise Remus, Ph.D., R.N.  |
|                                         |                            |
| Center for Delivery, Organization,      | Senior Research Scientist, |
|                                         | CDOM                       |
| and Markets (CDOM)                      |                            |
+-----------------------------------------+----------------------------+

Contents

[The Changing Climate of Quality Measurement
1](#the-changing-climate-of-quality-measurement)

[Quality Measurement Efforts 1](#quality-measurement-efforts)

[National and Regional Tracking 1](#national-and-regional-tracking)

[Accreditation 1](#accreditation)

[Comparative Public Reporting 2](#comparative-public-reporting)

[Payment (Provider Selection and Pay-for-Performance)
2](#payment-provider-selection-and-pay-for-performance)

[Current Challenges 3](#current-challenges)

[Overview of the AHRQ Quality Indicators
3](#overview-of-the-ahrq-quality-indicators)

[Development Process 4](#development-process)

[AHRQ QI Modules 4](#ahrq-qi-modules)

[Comparison of Criteria of Existing Indicator Evaluation Frameworks
5](#comparison-of-criteria-of-existing-indicator-evaluation-frameworks)

[Using the AHRQ QIs for Public Reporting or Payment: Factors in
Decisionmaking
9](#using-the-ahrq-qis-for-public-reporting-or-payment-factors-in-decisionmaking)

[Organizational and Program Needs 9](#organizational-and-program-needs)

[Availability of Data 9](#availability-of-data)

[Area-level Factors 11](#area-level-factors)

[Other Factors 11](#other-factors)

[Selecting AHRQ Quality Indicators for Public Reporting or Payment
12](#selecting-ahrq-quality-indicators-for-public-reporting-or-payment)

[Current Uses of the AHRQ QIs 12](#current-uses-of-the-ahrq-qis)

[Considerations for Hospital-level Reporting or Purchasing
14](#considerations-for-hospital-level-reporting-or-purchasing)

[Post-Selection Suggestions on *How* To Use the Indicators
18](#post-selection-suggestions-on-how-to-use-the-indicators)

[Conclusion 19](#conclusion)

[References 20](#references)

Appendix A: Current Uses of AHRQ Quality Indicators and Considerations
for Hospital-level
Reporting\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\....A-1

Appendix B: Public Reporting Evaluation Framework---Comparison of
Recommended Evaluation Criteria in Five Existing National
Frameworks\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...B-1

**Tables**

[Table 1. Comparison of the characteristics of existing indicator
evaluation frameworks 6](#__RefHeading___Toc82589508)

[Table 2. Quality indicator evaluation framework criteria: Comparison of
National Quality Forum and AHRQ Quality Indicators
8](#__RefHeading___Toc82589509)

[Table 3. Characteristics and uses of hospital administrative data
10](#__RefHeading___Toc82589510)

[Table 4. Current uses of the AHRQ QIs and considerations for
hospital-level reporting 17](#__RefHeading___Toc82589511)

# The Changing Climate of Quality Measurement 

The climate for quality tracking, measurement, reporting, and linking
payment to quality has changed dramatically in the past few years.
Several major landmark studies and reports have identified significant
gaps and variations in the quality and safety of health care.[^1]^,^[^2]
^,^[^3] ^,^[^4] ^,^[^5] ^,^[^6] ^,^[^7] These reports, in turn, have
accelerated efforts by government, accrediting bodies, large purchasers
and employer coalitions, and others to track quality at the national,
statewide, and provider level; publish comparative quality reports;
launch quality improvement efforts; and use public and private
purchasing power to reward better quality. Leaders of these quality
efforts often consider using administrative data because they are
readily available and inexpensive relative to other data sources.

Some leaders involved in efforts focused on hospital quality have
expressed a specific interest in the AHRQ Quality Indicators. Although
these indicators---specifically the Inpatient Quality Indicators (IQIs)
and Patient Safety Indicators (PSIs)---were originally developed for
quality improvement purposes, some public and private purchasers and
data organizations have begun to use them for hospital-level public
reporting and pay-for-performance
initiatives;[^8]^,^[^9]^,^[^10]^,^[^11] and many others are considering
doing so. Current and potential users have asked AHRQ for guidance in
evaluating the QIs and administrative data for these expanded purposes.
This document is designed to provide such guidance.[^12]

# Quality Measurement Efforts

Activities focused on quality assessment and implementation of quality
measures have exploded in the last few years. Some of these efforts
focus on aggregate tracking at the national and regional levels, whereas
others focus on provider or hospital-level activities including
accreditation, public reporting, purchasing, or payment.

## National and Regional Tracking 

In 2003, the Agency for Healthcare Research and Quality published the
*National Healthcare Quality ReportError: Reference source not found*
(NHQR) and *National Healthcare Disparities Report* (NHDR),Error:
Reference source not found which provide a comprehensive picture of the
level and variation of quality within four components of health care
quality--- effectiveness, safety, timeliness, and patient
centeredness---drawn from a conceptual framework identified by the
Institute of Medicine.[^13] These reports do not include provider level
data but rather national and regional baselines to support statewide,
community, and provider quality improvement efforts.

## Accreditation 

For many years, organizations have incorporated quality measures into
their accreditation process. The National Committee for Quality
Assurance (NCQA) monitors health plan performance using measures in
their Health Plan Employer Data and Information Set (HEDIS^®^),[^14] and
the Joint Commission on Accreditation of Healthcare Organizations
(JCAHO) has incorporated measures of hospital quality in their
accreditation process.[^15] Although these measures initially were used
for quality improvement, there has been a trend towards public reporting
of some or all of these measures. For example, the JCAHO will be
replacing hospital performance reports with quality reports in
2004.[^16]

## Comparative Public Reporting 

A major change in the past several years has been an acceleration of
public reporting efforts, particularly for hospitals. While several
State data organizations including California,[^17] New York,[^18] and
Pennsylvania,[^19] have been involved with hospital quality reporting
efforts for years, others, such as Texas,Error: Reference source not
found have just recently entered the reporting arena. Thirty-six
different hospital quality reporting efforts are summarized in a recent
publication by the IPRO.[^20] The National Quality Forum (NQF) was
established in late 1999 to facilitate widespread health care quality
improvement and public accountability.[^21]

An early NQF effort involved creating a comprehensive measurement
framework for hospital performance.[^22] The framework included more
than 20 specific evaluation criteria organized within the following
categories: 1) importance, 2) scientific acceptability[^23], 3)
usefulness, and 4) feasibility. The NQF framework and measure evaluation
criteria underwent an extensive consensus process involving
representatives of the public and NQF membership including Federal and
State agencies, providers, purchasers, health plans, and consumers. The
efforts of the NQF resulted in endorsement of 39 voluntary consensus
standards for hospital care to inform consumer selection, drive
improvement of care, facilitate performance-based quality improvement
initiatives, and create incentives for value-based purchasing.[^24]
There is a commitment by NQF members to evaluate the endorsed measures
for potential inclusion in their reporting programs.

The Centers for Medicare & Medicaid Services (CMS) worked with a variety
of provider associations to identify an initial core set of voluntary
measures for public reporting; CMS is now continuing to expand these
quality measurement efforts by augmenting the initial list of ten
measures.[^25] The build-out will be accomplished through a partnership
with public and private entities to identify candidate measures across
the spectrum of care and will involve NQF to identify those measures
around which consensus exists.

## Payment (Provider Selection and Pay-for-Performance) 

Several organizations are already using quality information to guide
their provider selection and payments. For example, an increasing number
of large employers and coalitions are using a common Request for
Information (eValue8[^26]) to solicit information about quality from
health plans seeking to do business with them. Through the Leapfrog
Initiative,[^27] alliances of large employers and business coalitions
are asking hospitals to provide data on three safety practices: computer
physician order entry, evidence-based hospital referral, and physician
staffing in the intensive care unit. In addition, both private and
public purchasers are establishing programs basing payment amounts
and/or contractual referral relationships on provider quality
information. In some cases payment is linked to mere provision of the
quality data, such as a component of the Anthem Blue Cross and Blue
Shield of Virginia Pay for Performance Initiative,Error: Reference
source not found whereas in others it is linked to the score itself,
such as the CMS Premier Hospital Quality Incentive Demonstration.Error:
Reference source not found CMS has several projects underway that
require submission of a core set of hospital quality measures.[^28]
Under the Prescription Drug, Improvement and Modernization Act of 2003,
hospitals who do not report quality data will experience a 0.4%
reduction of their Medicare payment levels.[^29]

A recent AHRQ Evidence-based Practice Center (EPC) technical review
examined the evidence supporting quality-based purchasing
strategies.[^30] The researchers' review of the literature and ongoing
research identified considerable variances in the methodologies used to
evaluate quality-based purchasing and only one randomized controlled
trial. They expanded their evaluation by developing stimulations to
determine how often hospitals would be mislabeled in public reports.
This work enabled the examination of the role of random variation versus
true quality differences in assessing reported hospital outcomes. Based
on their modeling work, the researchers concluded that, with appropriate
caution such as using multiple years of data, outcome measures can be
included among the performance indicators used for quality-based
purchasing.[^31]

## Current Challenges

This proliferation of quality measurement activity has led to many
challenging questions:

-   How do we construct a robust, yet parsimonious measure set?

-   How do we ensure that measures span different domains of care,
    different diseases, and conditions affecting all population groups?

-   How do we cover all this ground and still retain a manageable number
    of measures?

-   How do we enable particular stakeholders or localities to choose
    those measures that most concern them from a broader menu of
    measures?

-   How do we make sure that solid measures are populated with reliable
    and verifiable data?

National organizations including AHRQ, NCQA, CMS, JCAHO, and NQF are
working to resolve these and other pressing questions. This report
focuses on one particular issue receiving considerable attention: use of
the AHRQ QIs and other administrative data for public reporting and/or
pay for performance. The next section describes the genesis,
development, major features, and current use of the QIs.

# Overview of the AHRQ Quality Indicators

The AHRQ Quality Indicators (QIs) are measures of health care quality
that make use of readily available hospital inpatient administrative
data. The initial version, developed in the early 1990s, came out of a
data partnership among the Agency, health care industry, and several
States---the Healthcare Cost and Utilization Project or HCUP.[^32] The
original HCUP Quality Indicators represented an effort to enable States
and their hospital associations to use their administrative data for
quality improvement and tracking, and the indicators were extensively
used for these purposes.[^33]

## Development Process

In preparation for the *National Healthcare Quality Report*, AHRQ asked
its Stanford--University of California, San Francisco (UCSF) EPC to
refine, expand, and risk adjust these measures.[^34]^,^ [^35] The
research team followed an extensive multi-step process which included
the following:

1.  Development of an evaluation framework with six criteria including
    face validity, precision, minimum-bias, construct validity,
    fostering real quality improvement, and application experience.

2.  Structured literature review, Web searches and contact with quality
    measurement experts.

3.  Evaluation of risk adjustment methods available for use with
    administrative data.[^36]

4.  Consultation with ICD-9-CM[^37] coding experts.

5.  Structured review of each proposed indicator by clinical panels (for
    the Patient Safety Indicators), focusing on the usefulness of the
    proposed indicator as a quality measure, its preventability, and its
    likelihood of resulting from a medical error.

6.  A series of empirical analyses designed to test precision, minimum
    bias, and construct validity; each indicator was assigned a summary
    score for empirical performance.

7.  Examination and comparison of multiple options for risk adjustment,
    and selection and incorporation of a risk adjustment methodology.

The AHRQ QIs are the result of this extensive evaluation and development
effort and currently include three modules---the Prevention Quality
Indicators (PQIs), Inpatient Quality Indicators (IQIs), and Patient
Safety Indicators (PSIs).[^38] These are briefly described below.

## AHRQ QI Modules

The AHRQ Quality Indicators are a tool to assist health care
decisionmakers in using administrative data to highlight potential
quality concerns, identify areas that need further study and
investigation, and track changes over time.

-   **Prevention Quality Indicators** focus on ambulatory care sensitive
    conditions, hospital admissions that evidence suggests could have
    been avoided, or conditions that could be less severe if treated
    early and appropriately. These measures reflect quality of health
    care within a community, primarily outpatient services, and are not
    measures of hospital quality.

-   **Inpatient Quality Indicators** reflect quality of care inside
    hospitals and include inpatient mortality for medical conditions;
    inpatient mortality for procedures; utilization of procedures for
    which there are questions of overuse, underuse, or misuse; and
    volume of procedures for which there is evidence that a higher
    volume of procedures is associated with lower mortality.

-   **Patient Safety Indicators** also reflect quality of care inside
    hospitals, but these indicators focus on surgical complications and
    other iatrogenic events.

The QIs can be used with any inpatient claims database in the United
States, including not only the HCUP data sets[^39]^,^[^40] but also data
available directly from statewide data collection efforts of State
governments and hospital associations or from individual hospitals. For
purposes of tracking, hospital self-assessment, or reporting of
hospital-specific quality, only the IQIs and PSIs are pertinent. (The
prevention module focuses on community care using hospital data.)

The AHRQ QIs were designed with the goal of creating tools for quality
tracking and improvement and have been extensively used for these
purposes. For example, the recently released NHQR and NHDR provide
national and regional data using the PQIs and PSIs. In addition, States,
State hospital associations, and individual hospitals have been using
them extensively for quality improvement purposes.[^41]^,^ [^42]

While the focus of the initial QI development work was not on
hospital-level comparative reporting or for uses such as health care
purchasing or payment, the increased demand for standardized
hospital-level comparative data in a time of growing quality concerns
has led to their adoption for these new purposes---what might be called
an "off-label" use of the indicators. For example, Texas, and New York
are now using the IQIs to publicly report quality for all acute care
hospitals in their respective StatesError: Reference source not
found^,Error:\ Reference\ source\ not\ found^ and two
pay-for-performance programs mentioned earlier have incorporated the
PSIs.Error: Reference source not
found^,Error:\ Reference\ source\ not\ found^ Through these and similar
efforts, new knowledge has been gained of potential uses and
applications of the QIs and the challenges of balancing the need for
information with access to data and measures. (See Appendix A for
further details on examples of uses of the QIs.)

As a result of the evolution of quality measurement and the changing
reporting landscape, potential users of the QIs and other indicators
based on administrative data have been asking the following question:
How do the criteria used in developing the AHRQ Quality Indicators
relate to recently published national frameworks that were intended to
increase public accountability for hospital performance? This question
is addressed in the next section.

# Comparison of Criteria of Existing Indicator Evaluation Frameworks

One very significant outgrowth of the quality measurement and reporting
movement in the past few years has been the formulation of
evidence-based and consensus-based evaluation frameworks for identifying
and prioritizing measures. These frameworks focus on different
institutions or units of analysis---hospitals vs. health plans, for
example---and were developed for different purposes---accreditation,
quality reporting, quality improvement, etc. Some invited all data
sources, whereas others (such as the EPC report on the AHRQ QIs) focused
just on administrative data.

Table 1 summarizes the characteristics of five frameworks developed by
different organizations to guide their respective quality measure
selection efforts: the NQF,Error: Reference source not found the AHRQ
QIs,Error: Reference source not
found^,Error:\ Reference\ source\ not\ found^ the NHQR,Error: Reference
source not found the JCAHO,[^43] and the NCQA.[^44] The five frameworks
differ by unit of analysis, intended purpose, and data sources. The
differences in the framework characteristics appear to influence the
focus and details of individual criteria as well as their implementation
and emphasis.

As a first step toward guiding the potential use of the AHRQ QIs for
hospital public reporting, AHRQ asked the Support for Quality Indicators
(SQI) team at the Stanford--UCSF EPC to align all of the criteria used
by the various organizations into a common template as all five
frameworks are currently in use (although modifications may occur in the
future based on the new work of NQF); each has been referenced in work
describing selection of performance measures. This was a somewhat
complex undertaking because each of the frameworks use slightly
different terminology.

The next task was to compare the frameworks, to identify which criteria
appeared in which framework, and to identify key similarities and
differences. The QI framework was updated by adding in criteria which
were considered in development of the QI software, efforts subsequent to
the original EPC report. For example, the QI software offers the ability
to generate confidence intervals to support statistical significance
testing---a criterion not explicitly stated in the initial
framework.Error: Reference source not found (See Appendix B for a
side-by-side comparison of all five frameworks.)

[]{#__RefHeading___Toc82589508 .anchor}Table 1. Comparison of the
characteristics of existing indicator evaluation frameworks

  ------------- ----------------- ---------------- ----------- ---------------- ----------------
                **NQF**           **AHRQ Quality   **NHQR**    **JCAHO**        **NCQA**
                                  Indicators**                                  

  **Unit of     Hospitals         Hospitals        National,   Hospitals        Health plans
  analysis**                                       State,                       
                                                   region                       

  **Purpose**   Public            Quality          Quality     Accreditation,   Accreditation,
                accountability,   improvement      reporting   mandated, moving no mandate,
                comparative                        -- health   toward public    voluntary
                quality reporting                  care        disclosure       disclosure
                                                   quality in                   
                                                   America                      

  **Data        All feasible      Administrative   All         All feasible     All feasible
  sources**                       only             feasible                     

  **Measure     Hospital Care     AHRQ Quality     NHQR        ORYX and ORYX    HEDIS^®^
  set**         Performance       Indicators       measure set core measures    
                Measurement Set                                                 
  ------------- ----------------- ---------------- ----------- ---------------- ----------------

The comparison between the NQF measures and the AHRQ QIs is particularly
important for several reasons. First, as shown in Table 1, the NQF
framework is the only one that focuses solely on public reporting as the
goal and hospitals as the unit of analysis. Second, the NQF framework is
the most recent of those described above. Finally, the NQF framework was
endorsed as a national voluntary consensus standard by NQF membership
and Board which included representatives from AHRQ, JCAHO and NCQA as
well as providers, purchasers and consumers. For this reason, the
following discussion pays particular attention to the comparison between
the national voluntary consensus standard framework of NQF, which
concentrated on provider public reporting, and the framework applied to
the AHRQ QIs.

Table 2 compares the criteria from the NQF and the QI frameworks. As
noted earlier, the evaluation framework used for the AHRQ QIs and
phrasing of the individual criteria varies slightly from those in the
NQF framework; but, in a side-by-side comparison, it is clear there are
very few substantive differences. Even the NQF criteria that were not
included in the original EPC report for the AHRQ QIs were later
addressed through incorporation in subsequent work such as the QI
software. So the answer to the question "How do the criteria used to
develop and support the QIs compare with those of the NQF?" is that the
criteria are almost exactly the same.

Based on this comparison, it appears that the AHRQ QIs were selected
through a process that was reasonably aligned with the NQF criteria, and
thus are *potentially* appropriate for use in hospital comparative
reporting projects. The next section of this report turns to the second
question most commonly asked by would-be users of the QIs for reporting,
purchasing, or payment: What factors should potential QI users consider
when deciding which, if any, AHRQ Quality Indicators to use for public
reporting or payment efforts?

[]{#__RefHeading___Toc82589509 .anchor}Table 2. Quality indicator
evaluation framework criteria:\
Comparison of National Quality Forum and AHRQ Quality Indicators

+----------------------------------------------------------+----+------+
| Quality indicator evaluation framework criteria          | *  | **   |
|                                                          | *N | AHRQ |
|                                                          | QF | Q    |
|                                                          | ** | Is** |
+==========================================================+====+======+
| **1. Importance**                                        |    |      |
+----------------------------------------------------------+----+------+
| -   Assesses an important leverage point for improving   | !  | ![]  |
|     quality; significant to target audiences; impact on  | [] | (med |
|     health                                               | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Opportunity for improvement, considerable variation  | !  | ![]  |
|     in quality of care exists                            | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Aspect of quality is under provider or health system | !  | ![]  |
|     control                                              | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 6. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 16 | 3334 |
|                                                          | 66 | in"} |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 6i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Should not create incentives or rewards to improve   |    | ![]  |
|     without truly improving quality of care              |    | (med |
|                                                          |    | ia/i |
|                                                          |    | mage |
|                                                          |    | 5.pn |
|                                                          |    | g){w |
|                                                          |    | idth |
|                                                          |    | ="0. |
|                                                          |    | 1562 |
|                                                          |    | 5in" |
|                                                          |    | hei  |
|                                                          |    | ght= |
|                                                          |    | "0.1 |
|                                                          |    | 4583 |
|                                                          |    | 3333 |
|                                                          |    | 3333 |
|                                                          |    | 3334 |
|                                                          |    | in"} |
+----------------------------------------------------------+----+------+
| **2. Scientific acceptability**                          |    |      |
+----------------------------------------------------------+----+------+
| -   Relationship to quality is based on scientific       | !  | ![]  |
|     evidence                                             | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Well defined and precisely specified                 | !  | ![]  |
|                                                          | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Valid, measures the intended aspect of quality;      | !  | ![]  |
|     accurately represents the concept being evaluated;   | [] | (med |
|     data sources are comparable                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Adequate proportion of total variation is explained  | !  | ![]  |
|     by provider performance and amount of variation in   | [] | (med |
|     measurement is small after provider performance and  | (m | ia/i |
|     patient characteristics are taken into account       | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Reliable, producing the same results a high          | !  | ![]  |
|     proportion of time in the same population            | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Precise, adequately discriminating between real      | !  | ![]  |
|     differences in provider performance and reasonable   | [] | (med |
|     sample size exists to detect actual differences;     | (m | ia/i |
|     captures all possible cases and bias related to case | ed | mage |
|     exclusion or limited data is minimal.                | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Risk adjustment is adequate to address confounding   | !  | ![]  |
|     bias                                                 | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| **3. Usability**                                         |    |      |
+----------------------------------------------------------+----+------+
| -   Effective (understandable and clear) presentation    | !  |      |
|     and dissemination strategies exist                   | [] |      |
|                                                          | (m |      |
|                                                          | ed |      |
|                                                          | ia |      |
|                                                          | /i |      |
|                                                          | ma |      |
|                                                          | ge |      |
|                                                          | 5. |      |
|                                                          | pn |      |
|                                                          | g) |      |
|                                                          | {w |      |
|                                                          | id |      |
|                                                          | th |      |
|                                                          | =" |      |
|                                                          | 0. |      |
|                                                          | 15 |      |
|                                                          | 62 |      |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Statistical testing can be applied to communicate    | !  | ![]  |
|     when differences in performance levels are greater   | [] | (med |
|     than would be expected by chance                     | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Has been used effectively in the past and/or have    | !  | ![]  |
|     high potential for working well with other           | [] | (med |
|     indicators currently in use                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 6. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 16 | 3334 |
|                                                          | 66 | in"} |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 66 |      |
|                                                          | 6i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Compelling content for stakeholder decision making   | !  | ![]  |
|                                                          | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 5.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1562 |
|                                                          | pn | 5in" |
|                                                          | g) | hei  |
|                                                          | {w | ght= |
|                                                          | id | "0.1 |
|                                                          | th | 4583 |
|                                                          | =" | 3333 |
|                                                          | 0. | 3333 |
|                                                          | 15 | 3334 |
|                                                          | 62 | in"} |
|                                                          | 5i |      |
|                                                          | n" |      |
|                                                          | h  |      |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| **4. Feasibility**                                       |    |      |
+----------------------------------------------------------+----+------+
| -   Consistent construction and assessment of the        | !  | ![]  |
|     measure                                              | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Feasible to calculate; benefits exceed financial and | !  | ![]  |
|     administrative burden of implementation              | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Confidentiality concerns are addressed               | !  | ![]  |
|                                                          | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| -   Audit strategy can be implemented, quality of data   | !  | ![]  |
|     is known                                             | [] | (med |
|                                                          | (m | ia/i |
|                                                          | ed | mage |
|                                                          | ia | 7.pn |
|                                                          | /i | g){w |
|                                                          | ma | idth |
|                                                          | ge | ="0. |
|                                                          | 5. | 1770 |
|                                                          | pn | 8333 |
|                                                          | g) | 3333 |
|                                                          | {w | 3333 |
|                                                          | id | 4in" |
|                                                          | th | hei  |
|                                                          | =" | ght= |
|                                                          | 0. | "0.1 |
|                                                          | 15 | 7708 |
|                                                          | 62 | 3333 |
|                                                          | 5i | 3333 |
|                                                          | n" | 3334 |
|                                                          | h  | in"} |
|                                                          | ei |      |
|                                                          | gh |      |
|                                                          | t= |      |
|                                                          | "0 |      |
|                                                          | .1 |      |
|                                                          | 45 |      |
|                                                          | 83 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 33 |      |
|                                                          | 34 |      |
|                                                          | in |      |
|                                                          | "} |      |
+----------------------------------------------------------+----+------+
| ![](media/image5.png){width="0.15625in"                  |    |      |
| height="0.14583333333333334in"} = Indicates the          |    |      |
| criterion is in the evaluation framework.                |    |      |
|                                                          |    |      |
| ![](media/image6.png){width="0.16666666666666666in"      |    |      |
| height="0.14583333333333334in"} = NQF applied scope and  |    |      |
| priority thresholds to measures prior to the measure     |    |      |
| evaluation phase. All NQF hospital care consensus        |    |      |
| standards meet this criterion as part of the established |    |      |
| thresholds.                                              |    |      |
|                                                          |    |      |
| ![](media/image7.png){width="0.17708333333333334in"      |    |      |
| height="0.17708333333333334in"} = Indicates the          |    |      |
| criterion was not in the initial evaluation framework    |    |      |
| but has been addressed through incorporation in          |    |      |
| subsequent work such as the QI software.                 |    |      |
+----------------------------------------------------------+----+------+

# Using the AHRQ QIs for Public Reporting or Payment: Factors in Decisionmaking 

## Organizational and Program Needs 

The path from measure evaluation to actual implementation can be
difficult to negotiate regardless of data source. There are often
significant challenges associated with applying administrative or
clinical data, no matter how "good" the measure is (i.e., precisely
defined with significant opportunities for improvement and substantial
value for stakeholder decisionmaking).

Depending on organizational or programmatic needs, different entities
will approach implementation questions at different stages in the
process, but they need to be addressed at some point. If a program's
goal is to identify a menu of measures meeting certain criteria,
implementation issues can be dealt with later as users select from the
menu. The NQF consensus process evaluated measures for fit against
criteria with the emphasis on defining a measurement set and did not
specifically address implementation issues at the outset; an
implementation committee was subsequently formed to consider these
issues. In contrast, the NHQR/NHDR work followed a one-step, integrated
process. The immediate objective was to produce national reports on
quality and disparities. This meant implementation issues had to be
resolved before a measure was selected for inclusion in the measurement
set: No matter how perfect the measure, it could not be used without
national data. This is usually the case with reporting or purchasing
programs at the State or local level: measures are being assessed for
immediate use. Implementation issues, including data availability and
data quality, need to be addressed *during the measure selection
process* because the immediate goal is to produce comparative data, not
just a measure set.

## Availability of Data

For hospital-level measurement, the following data sources are typical:

-   Clinical data (e.g., medical record abstraction, laboratory data,
    pharmacy data, electronic medical record)

-   Administrative data (refers to UB-92, billing, or claims data)

-   Survey data (e.g., patient experience with care, employee
    satisfaction)

-   Operational data (e.g., licensure, ownership, staffing levels, type
    of staff)

Each of these data sources has strengths and weaknesses. National
reports such as the NHQR often draw from several different types of
data. The most commonly used sources are clinical data and
administrative data. Each of these is discussed in more detail below.

**Clinical data**. For provider-level reporting purposes, clinicians
typically prefer measures of quality derived from clinical data sources,
all things being equal. However access to clinical data can be a
challenge. Medical record abstraction is usually required and increases
the cost of collection. There is often inconsistency in documentation
and abstraction across hospitals with complex audit processes required
to assure comparability of sources. The primary benefit associated with
the use of clinical data is the greater number of data elements that can
be abstracted resulting in enhanced measure definition, risk adjustment,
and linkage to care processes. Efforts are underway at the national
level to expand access to clinical data and to integrate selected
clinical data elements into administrative data systems,[^45]^,^[^46]
but this work is not yet complete.

**Administrative data.** At this time, administrative data are the most
widely available source of information about hospital services, patient
care, and patient outcomes; for this reason there is considerable
interest in using administrative data for public reporting and payment.
All hospitals generate administrative data as part of billing operations
and all payers have access to administrative data (Table 3). The data
are inexpensive to collect, are easily accessible, and incorporate
clinical information through the use of standardized diagnosis and
procedure codes. Administrative data have been shown to be useful in
quality assessment[^47] and medical research,[^48] as well as for other
measurement tasks including screening for complications,[^49]
identifying mortality rates,[^50]^,^ [^51] and tracking health system
utilization.[^52]^,^ [^53]^,^ [^54]^,^ [^55]

Use of administrative data can make quality measurement more accessible,
and hospitals and health care systems can use the same data for internal
evaluations and improvement. Most State data organizations rely on
administrative data for their hospital data programs and often make the
data available for public use.Error: Reference source not
found^,Error:\ Reference\ source\ not\ found,^[^56] But administrative
data, like clinical data, also have limitations. Because administrative
data are collected principally for billing and related administrative
purposes, they lack the depth of clinical detail that can be helpful in
quality measurement; variations in coding practices may create
challenges for quality evaluations; and there can be data validity
issues.[^57]^,^[^58]^,^[^59]^,^[^60]^,^[^61]^,^[^62]

[]{#__RefHeading___Toc82589510 .anchor}Table 3. Characteristics and uses
of hospital administrative data

+-----------------------------------------------------------------------+
| -   Used to bill and pay for hospital services, contains the          |
|     information from the discharge claim.                             |
|                                                                       |
| -   Standardized format, available electronically from all hospitals  |
|     who bill for services (few paper bills).                          |
|                                                                       |
| -   Used for health care quality research, evaluation of services     |
|     (including utilization, outcomes, charges/cost), and quality      |
|     improvement.                                                      |
|                                                                       |
| -   Typical data elements include patient gender, age, diagnoses,     |
|     procedures, length of stay, admission source, discharge status,   |
|     total charges, primary payer, and hospital identifier.            |
|                                                                       |
| -   Data elements that may be available depending on data source      |
|     include patient race, county or ZIP Code of residence, secondary  |
|     payer, detailed charges, and identifier of primary physician and  |
|     surgeon.                                                          |
|                                                                       |
| -   Data format and quality may differ across hospitals or data       |
|     organizations such as the number of diagnosis and procedure codes |
|     available (typically 9, but may be as high as 15 or 30 per        |
|     patient claim) and the sequencing of the codes, the audits or     |
|     edits applied to the data before and after submission, and the    |
|     data values accepted.                                             |
+-----------------------------------------------------------------------+

Use of either administrative data or clinical data for reporting or
purchasing requires clarity about goals and deliberate strategies to
maximize strengths and minimize limitations of the data. For example,
indicators developed for quality screening tend to emphasize capturing
all cases that might reflect a quality concern (i.e., sensitivity) to
provide maximal opportunities for quality improvement. In contrast,
publicly reported indicators may be more desirable if they are designed
to minimize the false positive rate and therefore the potential for
unfair damage to hospitals\' reputations.

## Area-level Factors

Most important, different stakeholders in different market areas are
likely to value various criteria differently, so the leaders of
different State- or community-level public reporting, purchasing, or
payment initiatives may choose differently from the menu of nationally
available measures for programmatic reasons.Error: Reference source not
found^,^[^63] For example, in their initial screen of potential
measures, NQF sought to identify a balanced national measure set that
could work for comparing hospitals across the country. In doing so, they
established priorities (both condition-specific and cross-cutting) that
included identification of the most prevalent conditions and related
outcomes/processes within these areas to determine the extent they were
relevant to the majority of the Nation's hospitals. In a review of the
AHRQ Inpatient Quality Indicators, NQF found, among other issues, that
while cerebrovascular diseases represented a highly prevalent condition
among inpatients, the majority of hospitals perform relatively few
carotid endarterectomies making mortality and volume for this procedure
less appealing than other measures.[^64] But for a variety of reasons,
carotid endarterectomy mortality and volume may be of great concern in
particular markets; and prevalence of the procedure among hospitals in
other markets may be irrelevant to them. As a result, those designing
report cards in Texas and New York included this indicator in their
State-level reporting initiatives.Error: Reference source not
found^,Error:\ Reference\ source\ not\ found^

## Other Factors

Another example of different parameters driving indicator selection
relates to the question of risk adjustment. In decisions to date, the
NQF has only supported measures whose risk adjustment system is
transparent, publicly available, and free of charge. The mortality
Inpatient Quality Indicators (IQIs) are risk adjusted using age, gender,
and the 3M^TM^ All Payer Refined-Diagnosis Related Group (APR-DRG)
system, which is transparent and publicly available, but is not
free.[^65] This was not a concern for others, such as Texas and New
York, which incorporated the mortality IQIs in their public reports.
These organizations supported use of the APR-DRG system---it was
familiar to their stakeholders, had been evaluated in the scientific
literature, and had a developer committed to continued enhancement and
support of the risk adjustment system.

None of the AHRQ Inpatient Quality Indicators has yet been adopted
through the NQF process for hospital reporting. The NQF did review
selected Patient Safety Indicators as part of the Nursing Care Consensus
Standards initiative but concluded the EPC report of evidence on the
link between the indicators and the nursing component of care did not
coincide with the NQF criteria for identifying such a linkage. The
Patient Safety Indicators are risk adjusted using publicly available
methods, including the AHRQ comorbidity system, so the risk adjustment
issue is not applicable. The NQF also has not reviewed any of the
indicators in terms of their use for purchasing and payment strategies.
AHRQ will continue to work with NQF to harmonize measures and will be
monitoring the impact of current and future use of the QIs for reporting
and payment as part of this process. In particular, since the issue of
cost for users of the APR-DRGs has been a hurdle in this harmonization
process, AHRQ will be exploring the option of developing alternative
cost-free risk adjustment approaches for later versions of the mortality
indicators.

# Selecting AHRQ Quality Indicators for Public Reporting or Payment 

The remaining sections of this document focus on selection of particular
QIs for public reporting or payment, including provider selection and
pay-for-performance. When evaluating indicators, it is helpful to
consider how they are currently being used. Table 4 presents an
abbreviated summary, using alpha and numeric characters, of the current
(known) uses of AHRQ's Inpatient Quality Indicators[^66] and Patient
Safety Indicators as well as the major technical and methodological
factors users should examine when evaluating the measures and adapting
them for use in hospital-level reporting or purchasing programs. A
description of the category represented by the symbol is noted below
followed by a brief discussion. (Appendix A presents the same
information as contained in Table 4 but in greater detail.)

## Current Uses of the AHRQ QIs

The current uses of the AHRQ QIs are summarized into seven categories
represented by the alpha characters A through G. The category headings
along with a brief description are noted below.

A.  **State and regional hospital associations' member only reports**. A
    number of State and regional hospital associations across the Nation
    have integrated the IQIs and PSIs into their quality programs and
    performance measurement systems.[^67] Typically the associations
    generate comparative reports on the indicators that are shared with
    members only. Some associations include hospital-specific data while
    others share only blinded reports, providing comparative data
    without identifying the individual hospitals.

B.  **Internal hospital quality improvement**. Many hospitals have run
    the QIs on their internal data and incorporated the indicators into
    their organization's quality improvement programs. The majority of
    inquiries into the QI User Support System are from representatives
    of individual providers (34%) while hospital association
    representatives comprise the second largest group (14%).

C.  **National quality reports (not hospital level)**. The 2003
    *National Healthcare Quality Report* and *National Healthcare
    Disparities Report* included the PSIs. A subset of the mortality
    IQIs are under consideration for addition to the measurement set
    used in the next reports.[^68]^,^ [^69] The IQIs and PSIs were
    included in a recent report on the quality of care received by
    Medicare beneficiaries submitted to Congress by the Medicare Payment
    Advisory Commission (MedPAC).[^70]

D.  **State agencies' safety monitoring and confidential/internal
    analyses**. The majority of States have access to hospital discharge
    data through State data organizations, the State's Department of
    Health, or other State agencies. Several States have run the PSIs
    and are using them in internal evaluations. A few States have shared
    confidential reports with hospitals as part of their safety
    programs.

E.  **Public reporting by hospital**. The IQIs have been publicly
    reported, with hospitals identified, by two organizations. The
    first, in Texas, is the Texas Health Care Information Council
    (THCIC). THCIC's primary purpose is to provide data that will enable
    Texas consumers and health plan purchasers to make informed health
    care decisions.[^71] The second organization, in New York, is a
    collaboration of the Alliance for Quality Health Care (AQHC) and the
    Niagara Health Quality Coalition, a coalition of businesses,
    consumer education groups, and health plans created to improve
    health care quality region-wide while ensuring its cost
    effectiveness.[^72]

F.  **Private pay-for-performance initiative**. An example of a private
    pay-for-performance initiative is the Anthem Blue Cross and Blue
    Shield of Virginia Quality-In-Sights^®^ Hospital Incentive Program
    (Q-HIP^sm^) which began in late 2003.Error: Reference source not
    found The 3-year program is designed to align financial incentives
    with achievement of specific performance objectives. The patient
    safety component includes compliance with six JCAHO patient safety
    goals, adoption of Leapfrog Group recommendations, and selection of
    two (of nine) PSIs for monitoring and root cause analysis when
    appropriate.

G.  **National (public) pay-for-performance demonstration project**. CMS
    launched the Hospital Quality Initiative in 2003 which includes the
    Premier Hospital Quality Incentive Demonstration, a 3-year project
    to recognize and provide financial rewards to hospitals that
    demonstrate high quality performance.Error: Reference source not
    found^,\ Error:\ Reference\ source\ not\ found^ CMS seeks
    significant improvement in the quality of inpatient care by awarding
    bonus payments to hospitals with high quality as measured by
    multiple performance measures in the acute care area, including two
    of the PSIs.

## Considerations for Hospital-level Reporting or Purchasing

When evaluating indicators for hospital level public reporting,
purchasing, or pay-for-performance projects, there are data and
measurement issues that need to be considered and addressed. Problems
typically encountered when using administrative data and hospital-level
reporting are also summarized in Table 4. These issues are represented
by the numeric characters 1 through 7. The seven potential problems are
listed below along with a brief discussion of potential solutions or
actions that can minimize their impact.

1.  **Very low or low volume (small cell size) could impact patient
    confidentiality and also limit the ability to reliably identify
    quality differences**. Small cell size is a frequent problem in
    performance measurement, especially when using measures of rare
    events such as mortality or foreign body left after procedure. Small
    cell size refers to the occasion when there is a small number of
    cases within any individual unit of analysis. For example, a single
    hospital (location unit of analysis) may only have one death (small
    cell size, number of patients who died = 1) in a year (time unit of
    analysis). It would be difficult to ensure protection of patient
    confidentially in this instance. Small cell size also challenges the
    application of statistical significance tests. The volume of cases
    or sample size can impact both the sensitivity and specificity of an
    indicator,[^73] or true positives and true negatives. Typical
    solutions to managing low volume or small sample size are
    suppressing data when the cell size is below a predetermined number
    (e.g., 5 or 30)[^74] or presenting the raw number without an attempt
    to risk adjust or present any analysis of differences. Another
    option is to combine 2 or 3 years of data.[^75] A fourth option may
    be to combine several measures into a composite index.[^76] This
    method does offer advantages in dealing with small cell sizes and
    simplifying data presentation, but there is not yet agreement on the
    best methodology for creating such an index.[^77] A user may also
    want to present selected QIs along with other measures---such as
    those related to process of care---to provide a broader view of
    quality for a specific patient population (for example, mortality
    rates associated with CABG and AMI procedures could be combined with
    the JCAHO or CMS measuresError: Reference source not
    found^,Error:\ Reference\ source\ not\ found^ specific to those
    conditions). In addition to the methods already described, the IQI
    software includes an option for the user to calculate smoothed
    estimates of hospital performance which reduces "noise" in the
    measurement or potential differences due to random factors. This is
    especially helpful when true differences in performance may be
    obscured due to small sample sizes or relatively rare events.[^78]

2.  **Indicator may not be applicable to the majority of hospitals or
    applicable only to hospitals with specific services (e.g., cardiac
    surgery, obstetrics)**. The QIs include priority populations and
    areas such as child health, women's health (pregnancy and
    childbirth), diabetes, hypertension, ischemic heart disease, stroke,
    asthma, and patient safety. However not all hospitals provide all
    the patient care services covered by the QIs. To obtain comparative
    data for specialty services, a broader geographic region may be
    selected or the comparison may be limited to hospitals with similar
    services.

3.  **Volume is a proxy measure; volume may be manipulated leading to
    concerns about appropriate utilization**. Proxy indicators do not
    directly measure patient outcomes but, rather, an aspect of care
    that may be associated with the outcome. Volume indicators are not
    intended as stand-alone measures and are best used in addition to
    other indicators that measure similar aspects of care or when paired
    with an outcome measure (e.g., mortality).

4.  **Potential confounding bias or the impact may be impaired by skewed
    distribution not completely eliminated by risk adjustment or
    carefully constructed operational definitions**. In some situations
    the indicator may be measuring patient populations that are highly
    concentrated at a small number of facilities (e.g., pediatric heart
    surgery). This type of highly skewed volume distribution may have an
    adverse effect on the precision of this measure. In other situations
    patient characteristics may substantially affect performance on a
    measure and may vary systematically across areas. The QI software
    incorporates risk adjustment to minimize these issues. However, the
    QIs are limited to those data elements present in administrative
    data which include patient age, gender, comorbidities,[^79] and the
    3M APR-DRG[^80] system. Additionally, the definitions of the QIs
    were meticulously delineated with the goal being to limit the
    denominator to comparable patient populations rather than expand to
    include a broader base of patients with potentially dissimilar
    characteristics.

5.  **Benchmark or the "correct rate" may not be clear**. For some
    indicators the benchmark rate may not be clear. For other indicators
    (such as vaginal birth after cesarean section) recent scientific
    evidence may challenge established benchmarks. Under these
    circumstances, the national, regional, or peer group averages may be
    the best benchmark available.

6.  **Many, perhaps even the majority, of these procedures are currently
    done on an outpatient basis or observation status**. The health care
    setting in which some procedures are performed may change over time
    (e.g., cholecystectomy), shifting from inpatient admissions to
    observation status, outpatient basis, or even physician offices. The
    QIs are based on inpatient discharge data only. Differences in care
    settings should be considered when evaluating indicators for which
    the setting of care delivery may be shifting or vary by community in
    which case only a partial picture of the provision of this service
    may be provided.

7.  **The indicator may require data not present in all administrative
    data sets, or risk adjustment may be inadequate when based only on
    data available from ICD-9-CM codes**. The QIs were developed using
    data from HCUP. Indicator definitions and risk adjustment variables
    are limited to the readily available data elements but may include
    data not routinely coded such as external-cause-of-injury codes
    (E-codes). The HCUP data have been edited for accuracy and validity,
    both from the originating source (HCUP Partners) and during HCUP
    processing; and this edited version is returned to each HCUP
    Partner. However, not all administrative data undergo such an
    extensive series of audits. State data organizations may have
    different data requirements; and coding practices may vary across
    hospitals. QI users should ask about the validation process used for
    assuring the data quality and integrity of their input data file, be
    aware of coding practices, and remain cognizant of the potential
    limitations of risk adjustment. Further suggestions for evaluating
    data quality will be incorporated into the QI Curriculum currently
    being developed by RTI International under an Integrated Delivery
    System Research Network contract with AHRQ. Information on the
    training sessions associated with the QI curriculum development is
    posted on the AHRQ QI Web site
    ([http://www.qualityindicators.ahrq.gov](http://www.qualityindicators.ahrq.gov/))
    and will be distributed to QI Listserv^®^ subscribers.[^81]

[]{#__RefHeading___Toc82589511 .anchor}Table 4. Current uses of the AHRQ
QIs and considerations for hospital-level reporting

  ----------------------------------------------------------------------------------------
  **AHRQ Quality Indicator**           **Current Uses**                 **Considerations
                                                                        for hospital level
                                                                        reporting**
  ------------------------------------ ---------------- --------------- ------------------
                                       **Quality        **Public        
                                       improvement and  reporting and   
                                       national         pay for         
                                       surveillance**   performance**   

  Esophageal resection volume (IQI 1)  A, B             E               1, 2, 3

  Pancreatic resection volume (IQI 2)  A, B             E               1, 2, 3

  Pediatric heart surgery volume (IQI  A, B             E               1, 2, 3
  3)                                                                    

  Abdominal aortic aneurysm (AAA)      A, B             E               1, 2, 3
  repair volume (IQI 4)                                                 

  Coronary artery bypass graft (CABG)  A, B             E               2, 3
  volume (IQI 5)                                                        

  Percutaneous transluminal coronary   A, B             E               2, 3, 6
  angioplasty (PTCA) volume (IQI 6)                                     

  Carotid endarterectomy (CEA) volume  A, B             E               1, 2, 3
  (IQI 7)                                                               

  Esophageal resection mortality (IQI  A, B             E               1, 2, 4
  8)                                                                    

  Pancreatic resection mortality (IQI  A, B             E               1, 2, 4
  9)                                                                    

  Pediatric heart surgery mortality    A, B             E               1, 2, 4
  (IQI 10)                                                              

  AAA repair mortality (IQI 11)        A, B             E               1, 2, 4

  CABG mortality (IQI 12)              A, B             E               1, 2, 4

  Craniotomy mortality (IQI 13)        A, B             E               1, 2, 4

  Hip replacement mortality (IQI 14)   A, B             E               1, 4

  Acute myocardial infarction (AMI)    A, B             E               1, 4
  mortality (IQI 15)                                                    

  Congestive heart failure (CHF)       A, B             E               1, 4
  mortality (IQI 16)                                                    

  Stroke mortality (IQI 7)             A, B             E               1, 4

  Gastrointestinal (GI) hemorrhage     A, B             E               1, 4
  mortality (IQI 18)                                                    

  Hip fracture mortality (IQI 19)      A, B             E               1, 4

  Pneumonia mortality (IQI 20)         A, B             E               4

  Cesarean section (C-section) (IQI    A, B             E               2, 4, 5
  21)                                                                   

  Vaginal birth after cesarean section A, B             E               2, 4, 5
  (VBAC) (IQI 22)                                                       

  Laparoscopic cholecystectomy (IQI    A, B             E               4, 5, 6
  23)                                                                   

  Incidental appendectomy among        A, B             E               1, 4
  elderly (IQI 24)                                                      

  Bi-lateral cardiac catheterization   A, B             E               1, 2, 5, 6
  (IQI 25)                                                              

  PTCA mortality (IQI 30)              A, B                             1, 2, 3, 6

  CEA mortality (IQI 31)               A, B                             1, 2, 3

  Complications of anesthesia (PSI 1)  A, B, C, D       F               1, 7

  Death in low mortality DRGs (PSI 2)  A, B, C, D       F               1, 7

  Decubitus ulcer (PSI 3)              A, B, C, D                       1, 7

  Failure to rescue (PSI 4)            A, B, C, D                       1

  Foreign body left during procedure   A, B, C, D                       1, 7
  (PSI 5)                                                               

  Iatrogenic pneumothorax (PSI 6)      A, B, C, D       F               1

  Selected infections due to medical   A, B, C, D       F               1, 7
  care (PSI 7)                                                          

  Post-operative hip fracture (PSI 8)  A, B, C, D                       1, 7

  Post-operative hemorrhage or         A, B, C, D       F, G            1, 7
  hematoma (PSI 9)                                                      

  Post-operative physiologic and       A, B, C, D       G               1, 7
  metabolic derangements (PSI 10)                                       

  Post-operative respiratory failure   A, B, C, D                       1, 7
  (PSI 11)                                                              

  Post-operative pulmonary embolism    A, B, C, D                       7
  (PE) or deep vein thrombosis (DVT)                                    
  (PSI 12)                                                              

  Post-operative sepsis (PSI 13)       A, B, C, D       F               7

  Post-operative wound dehiscence (PSI A, B, C, D                       1, 7
  14)                                                                   

  Accidental puncture or laceration    A, B, C, D                       1, 7
  (PSI 15)                                                              

  Transfusion reaction (PSI 16)        A, B, C, D                       1, 7

  Birth trauma (PSI 17)                A, B, C, D                       2, 7

  Obstetric trauma -- vaginal with     A, B, C, D       F               2, 7
  instrument (PSI 18)                                                   

  Obstetric trauma -- vaginal without  A, B, C, D       F               2, 7
  instrument (PSI 19)                                                   

  Obstetric trauma -- cesarean section A, B, C, D       F               2, 7
  (PSI 20)                                                              
  ----------------------------------------------------------------------------------------

**Note**: See text section above, "Selecting AHRQ Quality Indicators for
Public Reporting or Payment," for key to alpha and numeric characters in
Table 4.

# Post-Selection Suggestions on *How* To Use the Indicators

Although identification of individual QIs is clearly a critical process,
decisions about *how* to use them can be equally important to achieving
a successful reporting or purchasing effort. Users are referred to a
growing body of literature on use of hospital report cards, public
accountability, and pay-for-performance.Error: Reference source not
found^,^[^82]^,^[^83]^,^[^84]^,^[^85]^,^[^86]^,^[^87]^,^[^88]^,^[^89]^,^[^90]^,^[^91]
Following are several suggestions that focus on the use of AHRQ QIs with
administrative data, drawing from the referenced literature and the
experience of early users of the QIs.

-   **Clearly delineate the purpose** of the program or measurement
    project. Is it to drive quality improvement? Public accountability?
    Inform consumer decisions? Pay for performance? Are there certain
    patient populations of interest to the program? For example, a
    project may want to begin by focusing on indicators associated with
    high cost hospital admissions (e.g., surgical procedures such as
    coronary artery bypass graft procedures (CABG); craniotomy
    procedures), admissions representing high volume (e.g., obstetrics),
    admissions representing high mortality (e.g., acute myocardial
    infarction (AMI); sepsis), or other criteria. The program purpose
    will drive all subsequent decisions.

-   **Identify stakeholders** involved and their level of
    responsibilities. Assess expectations and determine to what extent
    the available data and measures can meet different stakeholder
    interests.

-   **Involve hospitals** early in the measurement program efforts,
    preferably in the planning stages. Provide them an opportunity to
    understand the methodology including indicator definitions, any risk
    adjustment used, and calculation of measures. Provide an opportunity
    for them to review their own data in order to identify any potential
    issues with coding or characteristics of their patient population
    that may present a measurement challenge.

-   **Arrange for audits** or similar mechanisms to assure accuracy and
    completeness of reporting, to make sure that individual hospitals
    are not penalized for being more accurate in their reporting or
    providing a larger number of codes than others. This is particularly
    important if the indicator definition is based on ICD-9 codes not
    routinely used for payment purposes (such as E-codes). State data
    organizations vary in their data submission requirements and their
    capability to accept all ICD-9 codes available in hospitals' health
    information or billing systems. For example, hospital data program
    requirements in Texas limit the ICD-9 codes to those available in
    the electronic billing format (9 diagnoses and 6 procedures) while
    the data program in California will accept up to 30 ICD-9 diagnosis
    codes and 30 procedure codes.[^92]

-   **Evaluate both data quality and content.** Before finalizing
    indicator selections it is helpful, especially with new measurement
    programs, to begin with data explorations focusing on overall data
    quality and content. This may be simple frequency distributions on
    key variables. Among questions that might be asked are: If the
    program includes the objective to evaluate access or outcome by
    patient race, is the data element present for each case? Are data
    missing in a consistent manner---for example, present in all cases
    from some hospitals but missing in all cases from other hospitals?
    Is a selected procedure, e.g., esophageal resection, performed so
    infrequently in any single year that examining mortality rates would
    be best accomplished by combining data from several years or by
    presenting the IQI smoothed rate?

-   **Consider incremental implementation** of indicators. For example,
    an initial stage in a pay-for-performance program may be to reward
    participation in a measurement effort, e.g., hospital use of the
    PSIs, rather than individual PSI rates.

-   **Utilize comparative benchmarks** (national, regional, and peer
    group) whenever possible. Examples of sources for comparative data
    include the NHQR and NHDR, HCUPnet, and other State-level or
    hospital system reporting efforts.

-   **Use AHRQ's SQI program** for help along the way. Detailed
    information on the AHRQ QIs is available at the AHRQ QI Web site
    ([http://www.qualityindicators.ahrq.gov](http://www.qualityindicators.ahrq.gov/)).
    The QI Support Team is available to assist users with a range of
    issues including running of the software, clarification of indicator
    definitions, theoretical questions on the indicators, and
    interpretation of results. The QI Support Team can be reached via
    e-mail at <support@qualityindicators.ahrq.gov>.

-   **Link reporting formats** to the program purpose and intended
    audiences. Pay early attention to how data will be eventually
    displayed. Different formats may be required for different
    audiences. Examples of data presentation formats may be obtained
    from many of the same organizations that make comparative data
    available.

-   **Include an evaluation component** so that other users can benefit
    from your experience. The SQI program can help with this evaluation
    component.

# Conclusion

As noted earlier, continuous assessment and evaluation by those using
the QIs for reporting, purchasing or payment will be essential. AHRQ
continues to refine and enhance the QIs and plans to expand the QIs in
the near future. In addition, State agencies, hospital associations, and
others are engaged in efforts to improve the accuracy and usefulness of
administrative data. The Pennsylvania Health Care Cost Containment
Council[^93] already requires that hospitals collect and submit selected
clinical data elements to supplement the administrative data and the
UB-02 committeeError: Reference source not found is considering adding
some of these to the minimum data set. AHRQ has funded a project to
describe the value of administrative data[^94] and is considering future
projects on integrating clinical data elements into administrative data.

In addition, use of the QIs in an array of initiatives will provide
continuing lessons on how to improve the data sets against which they
are applied. The history of the quality measurement and improvement
movement shows that new use of an existing data set inevitably reveals
opportunities to improve the data and at the same time provides
incentives to do so. In the arena of hospital reporting and payment, one
often hears that we should not let the perfect be the enemy of the good.
If these efforts are carefully crafted, monitored and
evaluated---whether they use administrative or clinical data---the good
can in fact be the friend and harbinger of the perfect.

# References

[^1]: Kohn LT, Corrigan JM, Donaldson M, Eds. Institute of Medicine,
    Committee on Quality of Health Care in America. To Err Is Human:
    Building A Safer Health System. Washington, DC: National Academy
    Press; 2000.

[^2]: Institute of Medicine, Committee on Quality of Health Care in
    America. Crossing the Quality Chasm: A New Health System for the
    21^st^ Century. Washington, DC: National Academy Press; 2001.

[^3]: Adams K, Corrigan JM, Eds. Institute of Medicine, Committee on
    Identifying Priority Areas for Quality Improvement. Priority Areas
    for National Action: Transforming Health Care Quality. Washington,
    DC: National Academies Press; 2003.

[^4]: Agency for Healthcare Research and Quality. National Healthcare
    Quality Report. Rockville, MD: U.S. Department of Health and Human
    Services, Agency for Healthcare Research and Quality; December 2003.

[^5]: Agency for Healthcare Research and Quality. National Healthcare
    Disparities Report. Rockville, MD: U.S. Department of Health and
    Human Services, Agency for Healthcare Research and Quality; July
    2003.

[^6]: Chassin M, Galvin RW. The urgent need to improve health care
    quality. Institute of Medicine National Roundtable on Health Care
    Quality. *JAMA* 1998 Sept 16;280(11):1000-5.

[^7]: McGlynn EA, Asch SM, Adams J, et al. The quality of health care
    delivered to adults in the United States. N Engl J Med. 2003 Jun
    26;348(26):2635-45.

[^8]: Texas Health Care Information Council. Indicators of inpatient
    care in Texas hospitals, 1999-2001.
    \<<http://www.thcic.state.tx.us/IQIReport2001/IQIReport2001.htm>\>
    Accessed February 2004.

[^9]: Centers for Medicare & Medicaid Services. The Premier Hospital
    Quality Incentive Demonstration: Rewarding superior quality care.
    Centers for Medicare & Medicaid Services Fact Sheet, February 18,
    2004.
    \<<http://www.cms.hhs.gov/quality/hospital/PremierFactSheet.pdf>\>
    Accessed February 2004.

[^10]: Niagara Health Quality Coalition. Alliance for Healthcare
    Quality: Indicators of inpatient care in New York hospitals, 2004.
    \<<http://www.myhealthfinder.com/newyork/doc/interpret.php>\>
    Accessed August 2004.

[^11]: Anthem Blue Cross and Blue Shield. Anthem Blue Cross and Blue
    Shield joins with nine hospitals and the American College of
    Cardiology to reward quality. \[press release\]. February 18, 2003.
    \<<http://www.anthem.com/jsp/antiphona/bcbs/int_primary.jsp?content_id=PW_035971>\>
    Accessed August 2004.

[^12]: This document focuses on public reporting or payment efforts;
    however, the indicators may be used by a variety of stakeholders in
    many contexts to encourage provider engagement in quality
    improvement efforts.

[^13]: Hurtado MP, Swift EK, Corrigan JM, Eds. Institute of Medicine,
    Committee on the National Quality Report on Health Care Delivery.
    Envisioning the National Health Care Quality Report. Washington, DC:
    National Academy Press; 2001.

[^14]: National Committee for Quality Assurance. The State of Health
    Care Quality: 2003. Washington, DC: NCQA; 2003.
    \<[http://www.ncqa.org/communications/State%20Of%20Managed%20Care/SOHCREPORT2003.pdf](http://www.ncqa.org/communications/State Of Managed Care/SOHCREPORT2003.pdf)\>
    Accessed August 2004.

[^15]: Joint Commission on Accreditation of Healthcare Organizations.
    Performance measurement in health care.
    \<<http://www.jcaho.org/pms/index.htm>\> Accessed February 2004.

[^16]: Joint Commission on Accreditation of Healthcare Organizations.
    Setting the standard: The Joint Commission & health care safety and
    quality*.*
    \<<http://www.jcaho.org/general+public/patient+safety/setting_the_standard.pdf>\>
    Accessed August 2004.

[^17]: Damberg CL, Chung RE, Steimle A. The California report on
    coronary artery bypass graft surgery: 1997-1998 hospital data.
    Technical report. San Francisco, CA: Pacific Business Group on
    Health and the California Office of Statewide Health Planning and
    Development; July 2001.
    \<<http://www.oshpd.cahwnet.gov/HQAD/HIRC/hospital/Outcomes/CABG/199798report/ccmrp_technical.pdf>\>
    Accessed August 2004.

[^18]: New York State Department of Health. Coronary artery bypass
    surgery in New York State, 1997-1999. Albany, NY: New York State
    Department of Health; September 2002.
    \<<http://www.health.state.ny.us/nysdoh/heart/1997-99cabg.pdf>\>
    Accessed August 2004.

[^19]: Pennsylvania Healthcare Cost Containment Council. Interactive
    hospital performance report. \<<http://www.phc4.org/idb/HPR>\>
    Accessed February 2004.

[^20]: IPRO. 2003 Review of Hospital Quality Reports for Health Care
    Consumers, Purchasers and Providers. Lake Success, NY: IPRO; 2003.
    \<<http://company.ipro.org/dox/legFINAL10_14_03R.pdf>\> Accessed
    August 2004.

[^21]: National Quality Forum.
    \<<http://www.qualityforum.org/about/home.htm>\> Accessed February
    2004.

[^22]: National Quality Forum. National Quality Forum releases new
    consensus report on framework for measuring and reporting hospital
    quality. \[press release\]. Washington, DC: National Quality Forum;
    May 22, 2003.
    \<<http://www.qualityforum.org/prhospframereleaseFINAL5-21-03.pdf>\>
    Accessed August 28, 2003.

[^23]: Elements of scientific soundness include precise specifications,
    reliability, validity, adaptability, adequacy of risk adjustment,
    and inclusion of explicit conditions of use. (For more details, see:
    McGlynn EA. Selecting common measures of quality and system
    performance. Med Care 2003;41(1 Suppl):I39-47.)

[^24]: National Quality Forum. National voluntary consensus standards
    for hospital care: an initial performance measure set. Washington,
    DC: National Quality Forum; 2003.
    \<<http://www.qualityforum.org/txhospmeasBEACHpublicnew.pdf>\>
    Accessed August 2004.

[^25]: Centers for Medicare & Medicaid Services. Frequently asked
    questions (FAQs). CMS activities related to the National Voluntary
    Hospital Reporting Initiative; February 19, 2004.
    \<<http://www.cms.hhs.gov/quality/hospital/HQI_FAQ.pdf>\> Accessed
    March 2004.

[^26]: National Business Coalition on Health. 2004 Health Plan Request
    for Information. \[NBCH eValue8 Version 1.0\]; February 2004.
    \<<http://www.leapfroggroup.org/toolkit/V8Questions2004.pdf>\>
    Accessed August 2004.

[^27]: The Leapfrog Group. About us.
    \<<http://www.leapfroggroup.org/toolkit/V8Questions2004.pdf>\>
    Accessed February 2004.

[^28]: Centers for Medicare & Medicaid Services. Hospital Quality
    Initiative (HQI). \<<http://www.cms.hhs.gov/quality/hospital>\>
    Accessed February 2004.

[^29]: Centers for Medicare & Medicaid Services. CMS announces FY 2005
    payment increases and policy changes to improve quality and access
    for acute care hospitals. \[press release\]. Baltimore MD: Centers
    for Medicare & Medicaid Services; August 3, 2004.
    \<<http://www.cms.hhs.gov/media/press/release.asp?Counter=1150>\>
    Accessed August 2004.

[^30]: Dudley RA, Frolich A, Robinowitz DL, Talavera JA, Broadhead P,
    Luft HS. Strategies to Support Quality-based Purchasing: A Review of
    the Evidence. Technical Review 10. (Prepared by Stanford-University
    of California San Francisco Evidence-based Practice Center under
    Contract No. 290-02-0017.) Rockville, MD: Agency for Healthcare
    Research and Quality, 2004. AHRQ Pub. No 04-0057.

[^31]: The evidence technical review summary and full report,
    S*trategies To Support Quality-based Purchasing: A Review of the
    Evidence*, will be available on the AHRQ Web site in late Summer
    2004. See: <http://www.ahrq.gov/clinic/epcix.htm#techreviews>.

[^32]: Agency for Healthcare Research and Quality. Healthcare Cost and
    Utilization Project, 1988-2001: A Federal-State-industry partnership
    in health data. \<<http://www.hcup-us.ahrq.gov/overview.jsp>\>
    Accessed February 2004.

[^33]: Agency for Healthcare Research and Quality. HCUP Quality
    Indicators archive.
    \<<http://www.qualityindicators.ahrq.gov/hcup_archive.htm>\>
    Accessed July 2004.

[^34]: Davies S, Geppert J, McClellan M, McDonald KM, Romano PS,
    Shojania KG. Refinement of the HCUP Quality Indicators. Technical
    Review 4. (Prepared by UCSF-Stanford Evidence-based Practice Center
    under Contract No. 290-97-0013). Rockville, MD: Agency for
    Healthcare Research and Quality; 2001. AHRQ Pub. No. 01-0035.

[^35]: McDonald K, Romano P, Geppert J, Davies S, Duncan B, Shojania K.
    Measures of Patient Safety Based on Hospital Administrative Data:
    The Patient Safety Indicators. Technical Review 5. (Prepared by
    UCSF-Stanford Evidence-based Practice Center under Contract No.
    290-97-0013.) Rockville, MD: Agency for Healthcare Research and
    Quality; 2002; AHRQ Pub. No. 02-0038.

[^36]: Additional information on their evaluation of the available risk
    adjustment methods is available in *Refinement of the HCUP Quality
    Indicators* (pp. 56-62 and 171-77).Error: Reference source not found

[^37]: ICD-9-CM refers to the *International* *Classification of
    Diseases, 9^th^ Revision, Clinical Modification*, the diagnosis and
    procedure coding system required for Medicare claims.

[^38]: Agency for Healthcare Research and Quality. AHRQ Quality
    Indicators.
    \<[http://www.qualityindicators.ahrq.gov](http://www.qualityindicators.ahrq.gov/)\>
    Accessed February 2004.

[^39]: Agency for Healthcare Research and Quality. Nationwide Inpatient
    Sample (NIS). Powerful database for analyzing hospital care.
    \<[[http://www.hcup-us.ahrq.gov/nisoverview.jsp]{.underline}](http://www.hcup-us.ahrq.gov/nisoverview.jsp)\>
    Accessed February 2004.

[^40]: Agency for Healthcare Research and Quality. State Inpatient
    Databases (SID). Powerful databases for analyzing hospital care.
    \<[[http://www.hcup-us.ahrq.gov/sidoverview.jsp]{.underline}](http://www.hcup-us.ahrq.gov/sidoverview.jsp)\>
    Accessed February 2004.

[^41]: Georgia Hospital Association. CARE™: Collaborative Approach to
    Research Effectiveness.
    \<<http://www.gha.org/care/care2/care2.html>\> Accessed February
    2004.

[^42]: Illinois Hospital Association. COMPdata: Comprehensive
    Comparative Health Care Information.
    \<<http://www.ihatoday.org/compdata/index.htm>\> Accessed August
    2004.

[^43]: Joint Commission on Accreditation of Healthcare Organizations.
    Attributes of core performance measures and associated evaluation
    criteria.
    \<<http://www.jcaho.org/pms/core+measures/attributes+of+core+performance+measures.htm>\>
    Accessed July 2004.

[^44]: National Committee for Quality Assurance. HEDIS 2001. Washington,
    DC: NCQA; 2000.

[^45]: National Uniform Billing Committee. NUBC meeting summary, May 7 &
    8, 2003, Chicago, Illinois.
    \<<http://63.107.122.208/standardsdev/dsmo/nubcsummpdfs/summay2003.pdf>\>
    Accessed June 2004.

[^46]: Iezzoni LI, Greenberg MS. Capturing and classifying functional
    status information in administrative databases. Health Care Financ
    Rev 2003 Spring;24(3):61-76.

[^47]: Romano PS. Can administrative data be used to compare the quality
    of health care? Med Care Rev 1993 Winter;50(4):451-77.

[^48]: Laine C. Coming to grips with large databases. Ann Intern Med
    1997 Oct 15;127(8 Pt 1): 645-7.

[^49]: Romano PS, Geppert JJ, Davies S, Miller MR, Elixhauser A,
    McDonald KM. A national profile of patient safety in U.S. hospitals.
    Health Aff (Millwood) 2003 Mar-Apr;22(2):154-66.

[^50]: Luft HS, Romano PS. Chance, continuity, and change in hospital
    mortality rates. Coronary artery bypass graft patients in California
    hospitals, 1983 to 1989. JAMA 1993 Jul 21;270(3):331-7.

[^51]: Hannan EL, Kilburn H Jr, Lindsey ML, Lewis R. Clinical versus
    administrative data bases for CABG surgery. Does it matter? Med Care
    1992 Oct;30(10):892-907.

[^52]: Roos NP, Roos LL, Mossey J, Havens B. Using administrative data
    to predict important health outcomes. Entry to hospital, nursing
    home, and death. Med Care 1988 Mar;26(3):221-39.

[^53]: Ashton CM, Petersen NJ, Souchek J, Menke TJ, Pietz K, Yu HJ, Wray
    NP. Rates of health services utilization and survival in patients
    with heart failure in the Department of Veterans Affairs medical
    care system. Am J Med Qual 1999 Jan-Feb;14(1):55-63.

[^54]: Johantgen M, Elixhauser A, Bali JK, Goldfarb M, Harris DR.
    Quality indicators using hospital discharge data: state and national
    applications. Jt Comm J Qual Improv 1998 Feb;24(2):88-105.

[^55]: Asch SM, Sloss EM, Hogan C, Brook RH, Kravitz RL. Measuring
    underuse of necessary care among elderly Medicare beneficiaries
    using inpatient and outpatient claims. JAMA 2000 Nov
    8;284(18):2325-33.

[^56]: National Association of Health Data Organizations. Health Care
    Data Standards Reporting Guide. 837 101 Tutorial.
    \<<http://www.nahdo.org/project/837101Tutorial.htm>\> Accessed
    August 2004.

[^57]: McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel
    MB, et al. Does clinical evidence support ICD-9-CM diagnosis coding
    of complications? Med Care 2000 Aug;38(8):868-76.

[^58]: Lawthers A, McCarthy E, Davis R, Peterson L, Palmer R, Iezzoni L.
    Identification of in-hospital complications from claims data: is it
    valid? Med Care 2000 Aug;38(8):785-95.

[^59]: Romano PS, Chan BK, Schembri ME, Rainwater JA. Can administrative
    data be used to compare postoperative complication rates across
    hospitals? Med Care 2002 Oct;40(10):856-67.

[^60]: Weingart SN, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel
    MB, et al. Use of administrative data to find substandard care:
    validation of the complications screening program. Med Care 2000
    Aug;38(8):796-806.

[^61]: Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD.
    Judging hospitals by severity-adjusted mortality rates: the
    influence of the severity-adjustment method. Am J Public Health 1996
    Oct;86(10):1379-87.

[^62]: Iezzoni LI, Foley SM, Daley J, Hughes J, Fisher ES, Heeren T.
    Comorbidities, complications, and coding bias: Does the number of
    diagnosis codes matter in predicting in-hospital mortality? JAMA
    1992 Apr 22-29;267(16):2197-203.

[^63]: Rainwater J, Romano P. Feasibility of use of quality indicators
    for public reporting: review of public reporting systems. In
    preparation.

[^64]: During its deliberation, NQF also considered the strengths and
    weaknesses of the measures as a set. Particular interest was paid to
    balancing measures within and across priority areas -- as well as
    ensuring that measures within each priority area sufficiently
    represented quality in any one area. In one instance, because a
    single priority area (acute coronary syndrome) had many measures,
    several were excluded in favor of others in another priority area.
    In another case, because only a single measures was retained after
    evaluation (gastrointestinal hemorrhage mortality), NQF concluded
    that this single measure did not adequately represent quality in the
    area of gastrointestinal disease and excluded the measure
    altogether. Application of the NQF-endorsed framework assures that
    the composition of the measure set as a whole, and how measures
    within priority areas and across priority areas are balanced, be
    considered when selecting measures (personal communication, Ellen
    Kurtzman, Senior Program Director, NQF, March 26, 2004). For a
    complete summary of deliberations and concerns, see *National
    Voluntary Consensus Standards for Hospital Care: An Initial
    Performance Measurement Set*.Error: Reference source not found

[^65]: Information on the 3M^TM^ APR-DRG system is available at
    <http://www.3m.com/us/healthcare/his/products/coding/refined_drg.jhtml>.

[^66]: The IQI list presented in Table 4 is based on IQI version 2.1,
    revision 2 and does not include three new IQIs added to version 2.1,
    revision 3 released on July 23, 2004. Information on the latest IQI
    update is available at
    <http://qualityindicators.ahrq.gov/iqi_download.htm>.

[^67]: Hospital associations that have publicly acknowledged their use
    of the AHRQ QIs for quality improvement include the Healthcare
    Association of New York State; the Missouri Hospital Association;
    the Illinois Hospital Association CompData subsidiary that also
    serves hospitals affiliated with the Association of Kentucky
    Hospitals and Health Systems, the Montana Hospital Association, the
    Oregon Association of Hospitals and Health Systems, the West
    Virginia Hospital Association; the Georgia Hospital Association; and
    the Dallas-Fort Worth Hospital Council.

[^68]: Agency for Healthcare Research and Quality. National Healthcare
    Quality Report: 2004 proposed final measure set.
    \<<http://www.ahrq.gov/qual/nhqr04/premeasures.htm>\> Accessed
    August 2004.

[^69]: Agency for Healthcare Research and Quality. National Healthcare
    Disparities Report: 2004 proposed final measure set.
    \<<http://www.ahrq.gov/qual/nhdr04/premeasures.htm>\> Accessed
    August 2004.

[^70]: Medicare Payment Advisory Commission. Report to the Congress:
    Medicare Payment Policy. Washington, DC; MedPAC; March 2004.

[^71]: Texas Health Care Information Council. About THCIC.
    \<<http://www.thcic.state.tx.us/AboutTHCIC.htm>\> Accessed March
    2004.

[^72]: Niagara Health Quality Coalition. History of the coalition.
    \<<http://www.myhealthfinder.com/About_Us/products.htm>\> Accessed
    March 2004.

[^73]: Sensitivity refers to the ability to identify a condition that is
    really present; that is, the proportion positive of those who truly
    have the condition. Specificity is the ability to identify the
    absence of a condition when it is really not present; that is, the
    proportion negative of those who do not have the condition.

[^74]: Massell P. Optimization models and programs for cell suppression
    in statistical tables. Washington, DC: U.S. Census Bureau.
    \<<http://www.census.gov/srd/sdc/Massell.JSM2002.v4.pdf>\> Accessed
    July 2004.

[^75]: Dimick JB, Welch HG, Birkmeyer, JD. Surgical mortality as an
    indicator of hospital quality. The problem with small sample size.
    JAMA 2004 Aug 18;292(7):847-51.

[^76]: Centers for Medicare & Medicaid Services. CMS HQI Demonstration
    Project. Composite quality score methodology overview; March 26,
    2004.
    \<<http://www.cms.hhs.gov/quality/hospital/CompositeQualityScoreMethodologyOverview.pdf>\>
    Accessed July 2004.

[^77]: Kaiser Permanente Institute for Health Policy. Facilitating use
    of clinical quality measures through aggregation. \[roundtable
    meeting\]. April 21-22, 2004, Lansdowne, Virginia.

[^78]: Additional information on smoothed rates is available in the AHRQ
    IQI User Guide available for download from the AHRQ Quality
    Indicators Web site at
    <http://www.qualityindicators.ahrq.gov/iqi_download.htm>.

[^79]: Agency for Healthcare Research and Quality. Comorbidity Software,
    Version 2.
    \<<http://www.hcup-us.ahrq.gov/toolssoftware/comorbidity/comorbidity.jsp>\>
    Accessed July 2004.

[^80]: 3M^TM^ Health Information Systems. 3M^TM^ All Patient Refined DRG
    (APR-DRG) Software.
    \<<http://www.3m.com/us/healthcare/his/products/coding/refined_drg.jhtml>\>
    Accessed July 2004.

[^81]: Instructions on signing up for the QI Listserv^®^ are provided on
    the AHRQ QI Web site at:
    <http://qualityindicators.ahrq.gov/signup.htm>

[^82]: Hibbard JH, Stockard, J, Tusler M. Does publicizing hospital
    performance stimulate quality improvement efforts? Health Aff
    (Millwood) 2003 Mar-Apr;22(2):84-94.

[^83]: Clancy CM, Scully T. A call to excellence. Health Aff (Millwood)
    2003 Mar-Apr;22(2):113-5.

[^84]: Naylor CD. Public profiling of clinical performance. JAMA 2002
    Mar13;287(10):1323-5.

[^85]: Krumholz HM, Rathore SS, Chen J, Wang Y, Radford MJ. Evaluation
    of a consumer-oriented Internet health care report card: The risk of
    quality ratings based on mortality data. JAMA 2002 Mar
    13;287(10):1277-87.

[^86]: Mehrotra A., Bodenheimer T, Dudley RA. Employers' efforts to
    measure and improve hospital quality: Determinants of success.
    Health Aff (Millwood) 2003 Mar-Apr;22(2): 60-71.

[^87]: Tide rises on pay for performance with voluntary reporting
    initiative. Healthcare Financial Management 2004;58(3):120-1.

[^88]: Hibbard JH, Peters E. Supporting informed consumer health care
    decisions: Data presentation approaches that facilitate the use of
    information in choice. Annu Rev Public Health 2003;24:413-33.

[^89]: Pronovost PJ, Nolan T, Zeger S, Miller M, Rubin H. How can
    clinicians measure safety and quality in acute care? Lancet 2004 Mar
    27;363(9414):1061-7.

[^90]: Halvorson G, Isham GJ. Public performance. \[book excerpt\].
    Health Forum Journal, 2003 Summer;46(3):6.

[^91]: Tu JV, Cameron C. Impact of an acute myocardial infarction report
    card in Ontario, Canada. Int J Qual Health Care. 2003
    Apr;15(2):131-7.

[^92]: Agency for Healthcare Research and Quality. SID/SASD Application
    Kit.
    \<[http://www.ahrq.gov/downloads/pub/hcup/sidsasdapp.pdf](http://www.ahrq.gov/downloads/pub/hcup/sidsasdapp.pdf )
    \> Accessed August 2004.

[^93]: Pennsylvania Health Care Cost Containment Council. Info for
    providers-general info/data collection.
    \<<http://www.phc4.org/dept/dc/default.htm>\> Accessed February
    2004.

[^94]: Contract No. 233-02-0089 to the National Opinion Research Center.