***Technical Review 5***

**Measures of Patient Safety Based on Hospital Administrative Data. The
Patient Safety Indicators**

**Prepared for:**

> **Agency for Healthcare Research and Quality**
>
> Department of Health and Human Services
>
> U.S. Public Health Service
>
> 2101 East Jefferson Street
>
> Rockville, MD 20852
>
> *www.ahrq.gov*

**Contract No. 290-97-0013**

**Prepared by:**

> **University of California San FranciscoBStanford Evidence-based
> Practice Center**
>
> Kathryn M. McDonald, M.M. (Stanford)
>
> *Principal Investigator*
>
> Patrick S. Romano, M.D., M.P.H. (UC-Davis)
>
> Jeffrey Geppert, J.D. (Stanford)
>
> Sheryl M. Davies, M.S. (Stanford)
>
> Bradford W. Duncan, M.D. (Stanford)
>
> Kaveh G. Shojania, M.D. (UCSF)
>
> *Investigators*
>
> Angela Hansen, B.A. (Stanford)
>
> *EPC Research Assistant*

AHRQ Publication No. 02-0038

August 2002

> **Acknowledgments**
>
> The investigators would like to acknowledge the following individuals
> and organizations for their aid in this report: Doug Staiger, Dept. of
> Economics, Dartmouth College for his help in developing statistical
> methods; Ros McNally, National Primary Care Research and Development
> Centre, University of Manchester, who aided in the EMBASE^7^
> literature search; American Health Information Management Association
> for providing ICD-9-CM coding expertise; the various professional
> organizations listed in this report and their staff for submitting
> nominations for the clinical review panels; the clinical panelists,
> listed in Appendix B. The investigators also acknowledge the
> contributions of Anne Elixhauser, Ph.D., and Marlene Miller, M.D.,
> M.Sc., Agency for Healthcare Research and Quality (AHRQ), and Margaret
> Coopey, R.N., M.G.A., M.S.A., AHRQ Task Order Officer.
>
> This document is in the public domain and may be used and reprinted
> without permission. AHRQ appreciates citation as to source, and the
> suggested format is provided below:
>
> McDonald K, Romano P, Geppert J, et al. Measures of Patient Safety
> Based on Hospital Administrative DataCThe Patient Safety Indicators.
> Technical Review 5 (Prepared by the University of California San
> Francisco*B*Stanford Evidence-based Practice Center under Contract No.
> 290-97-0013). AHRQ Publication No. 02-0038 . Rockville, MD: Agency for
> Healthcare Research and Quality. August 2002.

ISBN 1-58763-121-0

ii

Summary

# Introduction {#introduction .Level1}

The longstanding cornerstone of medicine "first, do no harm" exists
because of the fragility of life and health during medical care
encounters, and represents the medical profession's understanding that
patient safety has always been an important part of quality health care.
Recently, however, concerns and evidence have mounted about the
complexities of the health care system potentially causing patient
deaths and significant unintended adverse effects. With a major national
interest in addressing patient safety issues, a wide spectrum of
individuals and organizations are working toward developing methods and
systems to detect, characterize, and report potentially preventable
adverse events. These activities are crucial precursors to prioritizing
areas for action and for studying the effects of approaches to reduce
sources of medical error.

As part of this activity, the Evidence-based Practice Center (EPC) at
the University of California San Francisco and Stanford University
(UCSF-Stanford), with collaboration from the University of California
Davis, was commissioned by the Agency for Healthcare Research and
Quality (AHRQ) to review and improve the evidence base related to
potential patient safety indicators (PSIs) that can be developed from
routinely collected administrative data. For the purposes of this
report, PSIs refer to measures that screen for potential problems that
patients experience resulting from exposure to the health care system,
and that are likely amenable to prevention by changes at the level of
the system.

# Reporting the Evidence {#reporting-the-evidence .Level1}

The primary goal of this report is to document the evidence from a
variety of sources on potential measures of patient safety suitable for
use based on hospital discharge abstract data. The approach to
identification and evaluation of PSIs presented in this report serves as
the basis for development of a third module for the AHRQ QI tool set
(referred to as the HCUP II in previous work by the UCSF-Stanford EPC
reporting on the research underpinning the refinement of the initial
AHRQ HCUP QIs, available on AHRQ's web site at
http://www.achq.gov/data/hcup/qirefine.htm). This third module will be
the *Patient Safety Indicators (PSIs)*, which focus on potentially
preventable instances of harm to patients, such as surgical
complications and other iatrogenic events. The two other modules are the
*Prevention Quality Indicators*, based on hospital admissions that might
have been avoided through high-quality outpatient care; and the
*Inpatient Quality Indicators*, consisting of inpatient mortality,
utilization of procedures for which there are questions of overuse,
underuse, or misuse; as well as volume of procedures for which higher
volume is consistently associated with lower mortality.

# Purpose of the PSIs {#purpose-of-the-psis .Level2}

Like the companion AHRQ Quality Indicators (QIs) screening tool set
refined by the UCSF-Stanford EPC, the PSIs are a starting point for
further analysis to reduce preventable errors through system or process
changes. Additionally, these measures are likely to support the public
mandate for aggregate statistical reporting to monitor trends over time,
as planned for the National Quality Report.

# Scope of the Project {#scope-of-the-project .Level2}

This report reviews previous studies and presents new empirical evidence
for identifying potential patient safety problems based on one
potentially important source of data: computerized hospital discharge
abstracts from the AHRQ Healthcare Cost and Utilization Project (HCUP).
Therefore, the measures considered needed to be defined using variables
that are available from most state-level hospital administrative data.
Data elements in these sets include International Classification of
Disease, Clinical Modification (ICD-9-CM) discharge diagnosis and
procedure codes; dates of admission, discharge and major procedures;
age; gender; and diagnostic related group (DRG). Data from outside the
hospital stay (e.g., post-hospital mortality or readmissions) were not
used because most state databases do not accommodate linkages between
datasets. The HCUP State Inpatient Databases (SID) is an example of such
a common denominator hospital discharge dataset, and was used for the
development of the AHRQ PSIs, reported here. The PSIs presented in this
report therefore relate to inpatient care, and the adverse events that
have either a high likelihood or at least a reasonable possibility of
being iatrogenic. These two constraints -- the data source and the
location of care---guided the development and evaluation of a promising
set of patient safety indicators.

Following from these constraints, the PSIs by necessity capture adverse
events that may, but possibly are not, related to medical care. They do
not capture "near misses" or other undocumented adverse events. They
also do not include adverse events related to a number of important
patient safety concerns that are not reliably specified using ICD-9-CM,
the official codes assigned to diagnoses and procedures associated with
hospital utilization in the United States. Based on previous validation
work and the limitations inherent in the data source, PSIs derived from
discharge data capture a mixture of adverse events, including those that
are almost certainly preventable and those that current best practices
and error-mitigating systems of care have not been able to prevent.
However, the evidence is presented for their promise as a low-cost
screen for potential quality concerns to guide further investigations
with additional data gathering and information collection.

# Methodology {#methodology .Level1}

Following the previous refinement of quality indicators described in a
companion technical report from the EPC, and published by AHRQ, an
evaluation framework for validity testing (i.e., face validity,
precision, minimum bias, and construct validity) was applied to each
candidate PSI. Specifically, a four pronged strategy to collect
validation data and descriptive information included two aspects of the
previous work: a background literature review, and empirical analyses of
the potential candidate PSIs using the HCUP SID. In addition to these
approaches of the previous project, expert coders from the American
Health Information Management Association (AHIMA) were consulted, and
clinical panel reviews of potential indicators were conducted based on a
process adapted from the RAND organization and University of California
Los Angeles (RAND/UCLA) Appropriateness Method.

Evidence from these four sources was used to modify and select the most
promising indicators for use as a screening tool to provide an
accessible and low-cost approach to identifying potential problems in
the quality of care related to patient safety. The methods applied
provide baseline information on the ability of a fairly broad range of
discharge-based PSIs to identify systematic differences across
hospitals, and potentially to monitor trends on a national or regional
basis.

# Results {#results .Level1}

A review of previously reported measures in the literature (e.g.
Complications Screening Program by Iezzoni et al, Patient Safety
Indicators by Miller et al), and of medical coding manuals, resulted in
identification of over 200 ICD-9-CM codes representing potential patient
safety problems. Most of these codes were grouped into clinically
meaningful indicators either based on previous indicator definitions or
on clinical and coding expertise. Based on literature review of the
published evidence related to their validity, several potential PSIs
were eliminated. Because of the limited validation literature available
on PSIs and complications indicators from which many PSIs were derived,
the research team conducted a clinical panel review process to assess
the face validity and to guide refinements to the initial definitions of
the 34 most promising PSIs. Response to a questionnaire by clinicians
(i.e., physicians from a number of specialties, nurses, and pharmacists)
for each indicator, augmented by coding review and initial empirical
testing, provided the basis for selecting the indicators expected to be
most useful for screening for potentially preventable adverse events.
Tables 1S and 2S summarize the strength of the evidence literature,
definitions, and key findings for the set of 20 *hospital level* PSIs
that are recommended for implementation as the initial AHRQ PSI set
(designated Accepted indicators).

**Table 1S. Strength of Evidence Literature for PSIs**

+---------------------------------+--------+--------+--------+--------+
| **Indicator**                   | **Co   | *      | *      | *      |
|                                 | ding** | *Const | *Const | *Const |
|                                 |        | ruct** | ruct** | ruct** |
|                                 |        |        |        |        |
|                                 |        | **Expl | **Impl | **Staf |
|                                 |        | icit** | icit** | fing** |
|                                 |        |        |        |        |
|                                 |        | **Pro  | **Pro  |        |
|                                 |        | cess** | cess** |        |
+---------------------------------+--------+--------+--------+--------+
| Complications of anesthesia     | 0      | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| *Death in low mortality DRGs*   | \+     | 0      | \+     | 0      |
+---------------------------------+--------+--------+--------+--------+
| Decubitus ulcer                 | \-     | 0      | 0      | ±      |
+---------------------------------+--------+--------+--------+--------+
| *Failure to rescue*             | \+     | 0      | 0      | ++     |
+---------------------------------+--------+--------+--------+--------+
| Foreign body left in during     | 0      | 0      | 0      | 0      |
| procedure                       |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| Iatrogenic penumothorax         | 0      | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| Infection due to medical care   | 0      | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| *Postoperative hip fracture*    | \+     | \+     | \+     | 0      |
+---------------------------------+--------+--------+--------+--------+
| Postoperative hemorrhage or     | ±      | ±      | \+     | 0      |
| hematoma                        |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| Postoperative physiologic and   | \-     | 0      | 0      | \-     |
| metabolic derangements          |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| *Postoperative respiratory      | \+     | ±      | \+     | ±      |
| failure*                        |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| *Postoperative PE or DVT*       | \+     | \+     | \+     | ±      |
+---------------------------------+--------+--------+--------+--------+
| Postoperative sepsis            | ±      | 0      | 0      | \-     |
+---------------------------------+--------+--------+--------+--------+
| Technical difficulty with       | ±      | 0      | 0      | 0      |
| procedure                       |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| Transfusion reaction            | 0      | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| Postoperative wound dehiscence  | 0      | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| Birth trauma                    | \-     | 0      | 0      | 0      |
+---------------------------------+--------+--------+--------+--------+
| Obstetric trauma -- vaginal     | \+     | 0      | 0      | 0      |
| delivery with instrumentation   |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| Obstetric trauma -- vaginal     | \+     | 0      | 0      | 0      |
| delivery without                |        |        |        |        |
| instrumentation                 |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+
| Obstetric trauma -- cesarean    | \+     | 0      | 0      | 0      |
| delivery                        |        |        |        |        |
+---------------------------------+--------+--------+--------+--------+

^a^ Level of evidence

(-) Published evidence suggests that the indicator lacks validity in
this domain (i.e., less than 50% sensitivity or predictive value;
explicit or implicit process failure rates no more frequent than among
control patients).

\(0\) No published evidence regarding this domain of validity.

(±) Published evidence suggests that the indicator may be valid in this
domain, but different studies offer conflicting results (although study
quality may account for these conflicts).

(+) Published evidence suggests that the indicator IS valid, or is
likely to be valid, in this domain (i.e., one favorable study).

(++) There is strong evidence supporting the validity of this indicator
in this domain (i.e., multiple studies with consistent results, or
studies showing both high sensitivity and high predictive value).

*^b^ Coding:* Sensitivity is the proportion of patients who suffered an
adverse event, based on detailed chart review or prospective data
collection, for whom that event was coded on a discharge abstract or
Medicare claim. Predictive value is the proportion of patients with a
coded adverse event who were confirmed as having suffered that event,
based on detailed chart review or prospective data collection.

*Construct, explicit process:* Adherence to specific, evidence-based or
expert-endorsed processes of care, such as appropriate use of diagnostic
modalities and effective therapies. Our construct is that hospitals that
provide better processes of care should experience fewer adverse events.

*Construct, implicit process:* Adherence to the "standard of care" for
similar patients, based on global assessment of quality by physician
chart reviewers. Our construct is that hospitals that provide better
overall care should experience fewer adverse events.

*Construct, staffing:* Our construct is that hospitals that offer more
nursing hours per patient day, better nursing skill mix, better
physician skill mix, or more experienced physicians, should have fewer
adverse events.

^c^ Note that when content validity is exceptionally high, as for
transfusion reaction or iatrogenic pneumothorax, construct validity
becomes less important.

Table 2S. Summary of Evidence for Accepted Hospital Level PSIs

<table>
<colgroup>
<col style="width: 10%" />
<col style="width: 34%" />
<col style="width: 3%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 3%" />
<col style="width: 4%" />
<col style="width: 4%" />
<col style="width: 3%" />
<col style="width: 4%" />
<col style="width: 5%" />
<col style="width: 4%" />
<col style="width: 4%" />
</colgroup>
<thead>
<tr class="header">
<th></th>
<th></th>
<th colspan="9">Panel concerns of validity<sup>a</sup></th>
<th colspan="3">Empirical performance</th>
</tr>
<tr class="odd">
<th>Indicator name</th>
<th>Definition</th>
<th>Rare</th>
<th>Condition definition varies</th>
<th>Under-reporting/ screening</th>
<th>Adverse consequences</th>
<th>Stratification suggested</th>
<th>Unclear preventability</th>
<th>Heterogeneous severity</th>
<th>Case mix bias</th>
<th>Denominator unspecific</th>
<th>Rate (per 1000 population at risk)<sup>e</sup></th>
<th>Standard deviation<sup>e</sup></th>
<th><blockquote>
<p><strong>Bias detected<sup>b</sup></strong></p>
</blockquote></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Complications of anesthesia</td>
<td>Cases of anesthetic overdose, reaction, or endotrachial tube
misplacement per 100 surgery discharges. Excludes codes for drug use and
self-inflicted injury.</td>
<td></td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>0.80</td>
<td>7.15</td>
<td></td>
</tr>
<tr class="even">
<td>Death in low mortality DRGs<sup>d</sup></td>
<td>In-hospital deaths per 100 patients in DRGs with less than 0.5%
mortality.<sup>c</sup> Exclude trauma, immunocompromised and cancer
patients.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td>1.14</td>
<td>11.94</td>
<td><strong>X+</strong></td>
</tr>
<tr class="odd">
<td>Decubitus ulcer</td>
<td>Cases of decubitus ulcer per 100 discharges with a length of stay
greater than 4 days. Exclude patients with paralysis or in MDC
9,<sup>d</sup> or patients admitted from a long term care facility.</td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td></td>
<td>20.5</td>
<td>20.7</td>
<td><strong>X+</strong></td>
</tr>
<tr class="even">
<td>Failure to rescue</td>
<td>Deaths per 100 patients having developed specified complications of
care during hospitalization. Exclude patients admitted from long term
care facility and patients transferred to or from other acute care
facility.</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
<td>170.3</td>
<td>80.9</td>
<td><strong>X+</strong></td>
</tr>
<tr class="odd">
<td>Foreign body left during procedure</td>
<td>Discharges with foreign body accidentally left in during procedure
per 100 discharges.</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>0.08</td>
<td>0.18</td>
<td><strong>N/A</strong></td>
</tr>
<tr class="even">
<td>Iatrogenic pneumothorax</td>
<td>Cases of iatrogenic pneumothorax per 100 discharges. Exclude trauma,
thoracic surgery, lung or pleural biopsy or cardiac surgery
patients.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>0.86</td>
<td>1.35</td>
<td><strong>X</strong></td>
</tr>
<tr class="odd">
<td>Infection due to medical care</td>
<td>Cases of secondary ICD-9-CM codes 999.3 or 996.62 per 100
discharges. Exclude patients with immunocompromised state or
cancer.</td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>1.37</td>
<td>1.75</td>
<td><strong>X</strong></td>
</tr>
<tr class="even">
<td>Postoperative hemorrhage or hematoma</td>
<td>Cases of hematoma or hemorrhage requiring a procedure per 100
surgical discharges. Excludes obstetric admissions.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td>1.83</td>
<td>3.66</td>
<td></td>
</tr>
<tr class="odd">
<td>Postoperative hip fracture</td>
<td>Cases of in-hospital hip fracture per 100 surgical discharges.
Exclude patients in MDC 8, with conditions suggesting fracture present
on admission.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td>1.12</td>
<td>5.94</td>
<td><strong>X</strong></td>
</tr>
<tr class="even">
<td>Postoperative physiological and metabolic derangement</td>
<td>Cases of specified physiological or metabolic derangement per 100
elective surgical discharges. Exclude patients with principle dx of
diabetes and with diagnoses suggesting increased susceptibility to
derangement. Exclude obstetric admissions.</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>0.92</td>
<td>11.1</td>
<td><strong>X</strong></td>
</tr>
<tr class="odd">
<td>Postoperative PE or DVT</td>
<td>Cases of deep vein thrombosis or pulmonary embolism per 100 surgical
discharges. Exclude obstetric patients.</td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>6.95</td>
<td>12.3</td>
<td><strong>X+</strong></td>
</tr>
<tr class="even">
<td>Postoperative respiratory failure</td>
<td>Cases of acute respiratory failure per 100 elective surgical
discharges. Exclude MDC 4 and 5 and obstetric admissions.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td>2.68</td>
<td>5.01</td>
<td><strong>X+</strong></td>
</tr>
<tr class="odd">
<td>Postoperative septicemia</td>
<td>Cases of septicemia per 100 elective surgery patients, with length
of stay more than 3 days. Exclude principle dianosis of infection, or
any dx of immunocompromised state or cancer, and obstetric
admissions.</td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>10.0</td>
<td>29.6</td>
<td><strong>X+</strong></td>
</tr>
<tr class="even">
<td>Postoperative wound dehiscence</td>
<td>Cases of reclosure of post-operative disruption of abdominal wall
per 100 cases of abdominopelvic surgery. Excludes obstetric
admissions.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>2.43</td>
<td>8.77</td>
<td><strong>X</strong></td>
</tr>
<tr class="odd">
<td>Technical difficulty with procedure</td>
<td>Cases of technical difficulty (e.g. accidental cut or laceration
during procedure) per 100 discharges. Excludes obstetric
admissions.</td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>2.42</td>
<td>2.64</td>
<td><strong>X+</strong></td>
</tr>
<tr class="even">
<td>Transfusion reaction</td>
<td>Cases of transfusion reaction per 100 discharges</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>0.01</td>
<td>0.06</td>
<td><strong>N/A</strong></td>
</tr>
<tr class="odd">
<td>Birth trauma – injury to neonate</td>
<td>Cases of birth trauma per 100 liveborn births. Excludes some preterm
infants, and infants with osteogenic imperfecta.</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
<td>9.36</td>
<td>31.4</td>
<td><strong>N/A</strong></td>
</tr>
<tr class="even">
<td>Obstetric trauma – cesarean delivery</td>
<td>Cases of obstetric trauma (4<sup>th</sup> degree lacerations, other
obstetric lacerations) per 100 cesarean deliveries.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td>6.13</td>
<td>16.12</td>
<td><strong>N/A</strong></td>
</tr>
<tr class="odd">
<td>Obstetric trauma – vaginal delivery with instrument</td>
<td>Cases of obstetric trauma (4<sup>th</sup> degree lacerations, other
obstetric lacerations) per 100 instrument assisted vaginal
deliveries.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td>203.6</td>
<td>142.4</td>
<td><strong>N/A</strong></td>
</tr>
<tr class="even">
<td>Obstetric trauma – vaginal delivery w/o instrument</td>
<td>Cases of obstetric trauma (4<sup>th</sup> degree lacerations, other
obstetric lacerations) per 100 vaginal deliveries without instrument
assistance.</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td>75.6</td>
<td>57.9</td>
<td><strong>N/A</strong></td>
</tr>
</tbody>
</table>

a Concerns raised by panels included the following:

*Rare*: Some events are relatively rare, and thus may not have adequate
statistical power for some providers.

*Condition definition varies*: Conditions covered by this indicator
include conditions for which diagnosis may be subjective, depending on
the threshold of the physician. Thus patients with the same clinical
state may not have the same diagnosis.

*Under-reporting/screening*: These conditions may not be systematically
reported leading to an artificially low rate, or may be routinely
screened for, leading to a higher rate in facilities that screen as
compared to those that do not.

*Adverse consequences*: Use of these indicators may have undesirable
effects, such as increasing inappropriate antibiotic use.

*Stratification suggested*: Indicator includes some high risk patient
groups which should be stratified when examining rates.

*Unclear preventability*: As compared to other PSIs these conditions may
be less subject to the control of the health system, and thus less
preventable.

*Heterogeneous severity*: These indicators include codes that encompass
several levels of severity of that condition that cannot be ascertained
by the codes.

*Case mix bias*: These indicators were felt to be particularly subject
to systematic bias due to the case mix of the provider. DRG and
comorbidity risk adjustment may or may not adequately address the
concern.

*Denominator unspecific*: The denominators for these indicators are less
than ideal, because the true population at risk could not be identified
completely clearly using ICD-9-CM codes, and thus some patients are
likely included that are not truly at risk, or some patients that are at
risk are not included.

^b^ Bias ratings are based on a series of tests of bias using DRG and
comorbidity risk adjustment. Those indicators flagged with 'X+'
demonstrated substantial bias, and should be risk adjusted. Those
indicators flagged with 'X' also demonstrated some bias. Those without a
flag did not demonstrate substantial bias in empirical tests, but may
nonetheless be substantially biased in a manner not detectable by the
bias tests. Those with marked with N/A did not undergo empirical testing
of bias due to lack of systematic variation.

^c^ DRGs that are divided into "with complications and comorbidities"
and "without complications and comorbidities" are only included if both
divisions have mortality rates below 0.5%.

^d^ DRG: Diagnostic Related Group; MDC: Major Diagnostic Category

^e^ Rates represent the average rate of indicator for a nationwide
sample of hospitals. Standard deviation is reported between providers.

Several accepted patient safety indicators were also modified into *area
level indicators*, which were designed to assess the total incidence of
the adverse event within geographic areas. For example, the transfusion
reaction indicator can be specified at both the hospital and area level.
Transfusion reactions that occur after discharge from a hospitalization
would result in a readmission. The area level indicator includes these
cases, while the hospital level restricts the number of transfusion
reactions to only those that occur during the same hospitalization that
exposed the patient to this risk. The five hospital level indicators
that have area level analogs are Iatrogenic Pneumothorax, Transfusion
Reaction, Infection Due to Medical Care, Wound Dehiscence, Foreign Body
Left in During Procedure, and Technical Difficulty with Medical Care.

In addition to the accepted PSIs, another 17 indicators show promise,
though have more concerning limitations. These were designated
"experimental" and examined empirically. They performed empirically
somewhat less well than the accepted indicators empirically. In
addition, the concerns raised about various aspects of these indicators
during the clinical panel discussions limit their potential usefulness.
However, with possible further refinements to the underlying coding of
data and to the indicator definitions, these indicators have the
potential to measure what they purport to identify. For example,
Reopening of Surgical Wound, while conceptually a useful PSI, requires
further information to exclude cases that are planned during staged
operations for example, and requires coding changes in order to capture
only similarly serious reopening procedures.

# Conclusions {#conclusions .Level1}

This project took a four pronged approach to the identification,
development and evaluation of PSIs that included use of literature,
clinician panels, expert coders and empirical analyses. For the
best-performing subset of PSIs, this project has demonstrated that rates
of adverse events differ substantially and significantly across
hospitals. The literature review and the findings from the clinical
panels combined with data analysis provide evidence to suggest that a
number of discharge-based PSIs may be useful screens for organizations,
purchasers, and policymakers to identify safety problems at the hospital
level, as well as to document systematic area level differences in
patient safety problems.

Few adverse events captured by administrative data are unambiguous
enough for a great deal of certainty that every case identified reflects
medical error. Most adverse events identified by the PSIs have a variety
of causes in addition to potential medical error leading to the adverse
event, including underlying patient health and factors that do not vary
systematically. Clinician panelists rated only two of the accepted
indicators as very likely to reflect medical error: 1.) "Transfusion
reaction" and 2.) "Foreign body left in during a procedure." As is
expected for indicators of this case-finding type, these indicators
proved to be very rare with less than 1 per 10,000 cases at risk. All
other accepted indicators identify adverse events which represent a
spectrum of likelihood of reflecting either medical error or potentially
preventable complications of care, but cannot be expected to identify
only cases in these categories.

# Potential Uses of PSIs {#potential-uses-of-psis .Level2}

Because the PSIs are intended for use as an initial, efficient screen to
target areas for further data exploration, the primary goal is to find
indicators that guide those interested in quality improvement and
patient safety to areas where there are systematic differences between
hospitals or geographic areas. These systematic differences may relate
to underlying processes or structures that an organization could change
to improve patient care and safety. These errors may be attributed to
human error on the part of physicians or nurses, or system deficiencies.
On the other hand, the systematic differences will sometimes correspond
to coding practices, patient characteristics not captured by
administrative data, or other factors. These will be dead ends to some
degree. In the application of these PSIs, users will be determining how
well patient safety problems are identified at the level of groups of
patients. Sharing experiences about application of these PSIs,
researchers and health care practitioners will build on the information
highlighted in this report about each indicator, as well as the set of
PSIs.

At the national or state level, these indicators could be used to
monitor the frequency of potential patient safety problems, to determine
whether the rates are increasing or decreasing over time, and to explore
large variations among settings of care. While the indicators were
primarily developed at the hospital level, some were also implemented to
provide an analogous area level measure, and analyses show that
additional cases are in fact identified that correspond to care received
at one institution, and the potentially iatrogenic complication
addressed in another hospital. Clearly, the locus of control and the
ability to study the potential underlying causes for an adverse event is
simpler in the case of the hospital level PSIs. However, trends over
time in area rates, as well as aggregations of the hospital level rates
are likely to reveal points of leverage outside of individual
institutions. No measure is perfect. Each is suited to its designed
purpose. Methods of aggregating across groups of PSIs still need to be
tested. This report provides the background for "safe" use of a tool
that has the potential to guide prevention of medical error, reductions
of potentially preventable complications, and quality improvement in
general. Table 3S provides examples of potential uses and potentially
inappropriate uses.

Table 3S. Use of patient safety indicators

+---+---------+--------------------------+------------------------------+
|   | User    | Potential Uses           | Potential Inappropriate Uses |
+---+---------+--------------------------+------------------------------+
| C |         |                          |                              |
| a |         |                          |                              |
| s |         |                          |                              |
| e |         |                          |                              |
| - |         |                          |                              |
| f |         |                          |                              |
| i |         |                          |                              |
| n |         |                          |                              |
| d |         |                          |                              |
| i |         |                          |                              |
| n |         |                          |                              |
| g |         |                          |                              |
| i |         |                          |                              |
| n |         |                          |                              |
| d |         |                          |                              |
| i |         |                          |                              |
| c |         |                          |                              |
| a |         |                          |                              |
| t |         |                          |                              |
| o |         |                          |                              |
| r |         |                          |                              |
| s |         |                          |                              |
+---+---------+--------------------------+------------------------------+
|   | P       | Identification of events | Identification of cases for  |
|   | rovider | for further              | disciplinary action.         |
|   |         | investigation.           |                              |
|   |         |                          | Comparison of rates.         |
+---+---------+--------------------------+------------------------------+
|   | Public  | Surveillance of events.  | Use of indicators in formal  |
|   | Health  |                          | evaluation of providers.     |
+---+---------+--------------------------+------------------------------+
|   | R       | Flagging of cases for    | Comparison of rates.         |
|   | esearch | use in research studies. |                              |
+---+---------+--------------------------+------------------------------+
| R |         |                          |                              |
| a |         |                          |                              |
| t |         |                          |                              |
| e |         |                          |                              |
| - |         |                          |                              |
| b |         |                          |                              |
| a |         |                          |                              |
| s |         |                          |                              |
| e |         |                          |                              |
| d |         |                          |                              |
| i |         |                          |                              |
| n |         |                          |                              |
| d |         |                          |                              |
| i |         |                          |                              |
| c |         |                          |                              |
| a |         |                          |                              |
| t |         |                          |                              |
| o |         |                          |                              |
| r |         |                          |                              |
| s |         |                          |                              |
+---+---------+--------------------------+------------------------------+
|   | P       | Surveillance of rates    | Physician-level              |
|   | rovider | for internal quality     | investigation.               |
|   |         | improvement              |                              |
|   |         | investigations.          | Use of rates for             |
|   |         |                          | disciplinary action or       |
|   |         |                          | formal evaluation.           |
+---+---------+--------------------------+------------------------------+
|   | Public  | Surveillance of rates.   | Public reporting of provider |
|   | Health  | Examination of area      | level rates.                 |
|   |         | rates over time, by      |                              |
|   |         | region, by hospital      |                              |
|   |         | type.                    |                              |
+---+---------+--------------------------+------------------------------+
|   | R       | Use with other measures  | Use in research as a         |
|   | esearch | of quality to determine  | definitive measure of        |
|   |         | relationships of PSIs    | quality of care.             |
|   |         | with structural, process |                              |
|   |         | or other aspects of      |                              |
|   |         | care.                    |                              |
+---+---------+--------------------------+------------------------------+

# Limitations and Future Research {#limitations-and-future-research .Level2}

Many important concerns cannot currently be monitored well using
administrative data, such as adverse drug events. Just as administrative
data limited specific indicators chosen, the use of administrative data
tends to favor specific types of indicators. The PSIs evaluated in this
report contain a large proportion of surgical indicators, rather than
medical or psychiatric. Medical complications are often difficult to
distinguish from comorbidities that are present on admission. In
addition medical populations tend to be more heterogeneous than
surgical, especially elective surgical populations, making it difficult
to account for case-mix. Panelists often expressed that indicators were
more applicable to patient safety when limited to elective surgical
admissions.

The initial validation evaluations reviewed and performed for the PSIs
leave substantial room for further research with detailed chart data and
other data sources. Future validation work should focus on the
sensitivity and specificity of these indicators in detecting the
occurrence of a complication; the extent to which failures in processes
of care at the system or individual level are detected using these
indicators; the relationship of these indicators with other measures of
quality, such as mortality; and further explorations of bias and risk
adjustment.

Enhancements to administrative data are worth exploring in the context
of further validation studies that utilize data from other sources. For
example, as with other quality indicators, the addition of timing
variables may prove particularly useful in order to identify whether or
not a complication was present on admission, or occurred during the
hospitalization. While some of the complications that are present on
admission may indeed reflect adverse events of care in a previous
hospitalization or outpatient care, many may reflect comorbidities
instead of complications. A second example area, linking of hospital
data over time and with outpatient data and other hospitalizations,
would allow inclusion of complications that occur after discharge, and
likely would increase the sensitivity of the PSIs.

The current development and evaluation effort will best be augmented by
a continuous communication loop between users of these measures,
researchers interested in improving these measures, and policy makers
with influence over the resources aimed at data collection and patient
safety measurement.