# Presentation: 110444

## AHRQ Quality IndicatorsRecent validation efforts

- Patrick S. Romano, MD MPH
- UC Davis Center for Healthcare Policy and Research
- AHRQ QI Users Meeting
- Bethesda, MD; September 28, 2007

## Overview

- Topics (focus on PSIs and PDIs)
- Validation of prior tools
- Extending face/consensual validity
- Construct/predictive validity based on patient outcomes and resource utilization
- Criterion validity based on present-of-admission (POA) data
- Criterion validity based on data linkages
- Criterion validity based on recoded/abstracted data
- Criterion validity based on clinician case review
- Other approaches
- Questions and answers

## Validation of prior tools based on literature review (MEDLINE/EMBASE)

- Validation studies of Iezzoni et al.’s Complications Screening Program
- At least one of three validation studies (coders, nurses, or physicians) confirmed PPV ≥75% among flagged cases
- Nurse-identified process-of-care failures were more prevalent among flagged cases than among unflagged controls
- Other studies of coding validity
- Very few in peer-reviewed journals, some in “gray literature”

**Notes:**
When the PSIs were originally developed, we relied heavily on work done by Iezzoni and colleagues in validating the Complications Screening Program.  Very few other studies of coding validity were available.

## Validation (%) of Complications Screening ProgramMed Care 2000;38:785-806,868-76; Int J Qual Health Care 1999;11:107-18

| CSP Indicator | PSI | Coder: % Complic
Present | MD: %
Complic
present | RN: % Process problem | MD: %
Quality
problem |
| --- | --- | --- | --- | --- | --- |
| Postprocedural hemorrhage/ hematoma | #9 narrower: requires proc code + dx | 83 (surg)
49 (med) | 57 (surg)
55 (med) | 66 vs 46
13 vs 5 | 37 vs 2
31 vs 2 |
| Postoperative pulmonary compromise | #11 narrower:
includes only resp failure | 72 | 75 | 52 vs 46 | 20 vs 2 |
| DVT/PE | #12 surgical only
Slight changes | 59 (surg)
32 (med) | 70 (surg)
28 (med) | 72 vs 46
69 vs 5 | 50 vs 2
20 vs 2 |
| In-hosp hip frx and falls | #8 surgical only, no E codes | 57 (surg)
11 (med) | 71 (surg)
11 (med) | 76 vs 46
54 vs 5 | 24 vs 2
5 vs 2 |

- Percentage with process/quality problem among flagged cases vs. unflagged controls

**Notes:**
This slide summarizes the results of those previous studies, for four of the AHRQ PSIs that most closely resemble indicators from the Complications Screening Program.  Each column represents a separate study, in which coders, nurses, or physicians reviewed each medical record independently.  Reviewers were asked to indicate whether a complication occurred or whether there was a problem with quality of care that affected the patient.  Surgical records flagged on each indicator were compared with nonsurgical records flagged on the same indicator; these data were used to limit the AHRQ PSI definitions of hemorrhage/hematoma, DVT/PE, and in-hospital hip fracture to surgical cases.

## Validation of prior tools:Construct validity evidence in literature

| Indicator | Explicit process | Implicit process | Staffing |
| --- | --- | --- | --- |
| Complications of anesthesia |  |  |  |
| Death in low mortality DRGs |  | + |  |
| Decubitus ulcer |  |  | ± |
| Failure to rescue |  |  | ++ |
| Foreign body left during procedure |  |  |  |
| Iatrogenic pneumothorax |  |  |  |
| Selected infections due to medical care |  |  |  |
| Postop hip fracture | + | + |  |
| Postop hemorrhage or hematoma | ± | + |  |
| Postop physiologic/metabolic derangements |  |  | – |
| Postop respiratory failure | ± | + | ± |
| Postop thromboembolism | + | + | ± |
| Postop sepsis |  |  | – |
| Accidental puncture or laceration |  |  |  |
| Transfusion reaction |  |  |  |
| Postop abdominopelvic wound dehiscence |  |  |  |

## Extending consensual/face validity:OECD Health Care Quality Indicators Project

- Includes 21 countries, WHO, European Commission, World Bank, ISQua, etc.
- Patient safety is one of five priority areas
- Indicator selection criteria:
- Importance
- Impact on health
- Policy importance (concern for policymakers and consumers)
- Susceptible to influence by the health care system
- Scientific soundness
- Face validity (clinical rationale and past usage)
- Content validity
- Feasibility
- Data availability and reporting burden

**Notes:**
The Organization for Economic Cooperation and Development’s Health Care Quality Indicators Project provides an example of extending face or consensual validity.

## Extending consensual/face validity: OECD Review Process

- Patient safety panel constituted with 5 members (Dr. John Millar, Chair)
- 50 indicators from 7 sources submitted for review (US, Canada, Australia)
- Modified RAND/UCLA Appropriateness Method
- Panelists rated each indicator on importance and scientific soundness (2 rounds with intervening discussion)
- Retained indicators with median score >7 (1-9 scale) on both domains; rejected indicators with median score 5 or below on either domain

## International OECD panel ratings of PSIsMcLoughlin V, et al. Int J Qual Health Care 2006 Sep;18 Suppl 1:14-20 

**Notes:**
This summary table divides all PSIs into two groups based on their rating by the OECD expert panel: recommended and “not recommended.”

## AHRQ panel ratings of PSI “preventability”very similar to OECD ratings

- a Panel ratings were based on definitions different than final definitions. For “Iatrogenic pneumothorax,” the rated denominator was restricted to patients receiving thoracentesis or central lines; the final definition expands the denominator to all patients (with same exclusions). For “In-hospital fracture” panelists rated the broader Experimental indicator, which was replaced in the Accepted set by “Postoperative hip fracture” due to operational concerns.
- b Vascular complications were rated as Unclear (-) by surgical panel; multispecialty panel rating is shown here.

**Notes:**
The OECD panel ratings were actually remarkably similar to the AHRQ panel ratings on “preventability.”  Preventability was one of six domains in which the AHRQ panels rated each potential indicator.  Although the AHRQ indicators were retained or rejected based on panel ratings of “overall usefulness,” the panel ratings of “preventability” may better reflect panelists’ views of whether an indicator can be used to characterize patient safety at a hospital level.  Hence, it is noteworthy that only two indicators (shown in blue) were classified differently by the AHRQ panel and the OECD panel.

## Approaches to assessing construct validity

- Is the outcome indicator associated with explicit processes of care (e.g., appropriate use of medications)? – YOUR STUDY
- Is the outcome indicator associated with implicit process of care (e.g., global ratings of quality)?
- Is the outcome indicator associated with nurse staffing or skill mix, physician skill mix, or other quality-related aspects of hospital structure?
- Is the outcome indicator associated with other meaningful outcomes of care (predictive validity)?

**Notes:**
Analyses of construct validity can be based on several different constructs regarding what adverse outcomes ought to be associated with.  We are not aware of any studies embodying the first approach.  One study using the second approach is summarized on the next slide.  Several studies embodying the third approach are now underway; those published to date have focused only on basic comparisons between teaching and non-teaching hospitals, or between large and small hospitals.  Such comparisons are interesting but provide rather weak tests of construct validity.  Two studies embodying the fourth approach have been published.

## Validation Using Implicit Processes of Care:Regression of PSIs on JCAHO Overall EvaluationMiller MR, et al., Am J Med Qual 2005;20:239-252

| PSI | Regression coefficient | PSI | Regression coefficient |
| --- | --- | --- | --- |
| Complications of anesthesia | 0.029 | Postop sepsis | -0.209 |
| Death in low-mortality DRG | 0.012 | Postop wound dehiscence | -0.098 |
| Decubitus ulcer | 0.004 | Accidental puncture/laceration | -0.212 (p<.01) |
| Failure to rescue | 0.112 | Birth trauma | 0.045 |
| Foreign body left in | -0.102 | Ob trauma, vaginal w/out instrum | -0.114 |
| Iatrogenic pneumothorax | 0.261 (p=.03) | Ob trauma, vaginal w instrum | 0.165 (p=.04) |
| Selected infection 2° to care | -0.037 | Ob trauma, cesarean | -0.027 |
| Postop hip fracture | -0.112 | PSI factor 1 | -0.108 (p=.02) |
| Postop hemorrhage/hematoma | 0.096 | PSI factor 2 | 0.026 |
| Postop respiratory failure | -0.284 (p<.01) | PSI factor 3 | -0.010 |
| Postop DVT/PE | -0.210 (p=.06) |  |  |

**Notes:**
Miller and colleagues examined the correlations between hospital performance on the PSIs (smoothed rates) and JCAHO overall evaluation scores, based on surveys of 2,116 hospitals conducted between 1997 and 1999.  The overall evaluation score is derived from 46 grid elements assessed during each survey.  Most correlations were nonsignificant.  Among those that were significant (p<0.10), 3 were in the expected direction (lower PSI rates associated with higher JCAHO scores) whereas 2 were in the counterintuitive direction.  A factor that draws variability from several postoperative complication PSIs was also significant in the expected direction.  Postop DVT/PE was the only PSI for which worse performance was associated with less favorable categorical accreditation decisions.

## Validation Using Structural Measures:Regression of FTR on Skill Mix MeasuresSilber J, et al., Med Care 2007;45(10):918-925

| Indicator | 
FTR-Silber | FTR-Needleman | 
FTR-AHRQ |
| --- | --- | --- | --- |
| Teaching hospital (COTH member) | 0.89 | 0.86 | 0.85 |
| Medium-large hospital (>200 beds) | 0.92 | 0.94 | 0.92 |
| Bed-to-nurse (RN+LVN) ratio | 1.04 | 1.04 | 1.04 |
| Nursing skill mix
RN/(RN+LPN)  | 0.92 | 0.87 | 0.87 |

**Odds ratios from multivariable logistic regression, adjusted for all patient characteristics and all other specified hospital characteristics, based on 1999-2000 Medicare inpatient claims. **
**Odds ratios further from 1 indicate larger, more clinically important effects.**
**Yellow: 0.0001<p<0.01**
**Red: p<0.0001**

## Predictive validity: Impact of preventing each PSI event on mortality, LOS, charges (ROI)NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74

| Indicator | Δ Mort (%) | Δ LOS (d) | Δ Charge ($) |
| --- | --- | --- | --- |
| Postoperative septicemia | 21.9 | 10.9 | $57,700 |
| Selected infections due to medical care | 4.3 | 9.6 | 38,700 |
| Postop abd/pelvic wound dehiscence | 9.6 | 9.4 | 40,300 |
| Postoperative respiratory failure | 21.8 | 9.1 | 53,500 |
| Postoperative physiologic or metabolic derangement | 19.8 | 8.9 | 54,800 |
| Postoperative thromboembolism | 6.6 | 5.4 | 21,700 |
| Postoperative hip fracture | 4.5 | 5.2 | 13,400 |
| Iatrogenic pneumothorax | 7.0 | 4.4 | 17,300 |
| Decubitus ulcer | 7.2 | 4.0 | 10,800 |
| Postoperative hemorrhage/hematoma | 3.0 | 3.9 | 21,400 |
| Accidental puncture or laceration | 2.2 | 1.3 | 8,300 |

**Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.**

## Predictive validity: Impact of preventing each PSI event on mortality, LOS, VA expenditures (ROI)VA PTF 2001 analysis by Rivard et al., Med Care Res Rev; in press

| Indicator | Δ Mort (%) | Δ LOS (d) | Δ Cost ($) |
| --- | --- | --- | --- |
| Postoperative septicemia | 30.2 | 18.8 | $31,264 |
| Selected infections due to medical care | 2.7 | 9.5 | 13,816 |
| Postop abd/pelvic wound dehiscence | 11.7 | 11.7 | 18,905 |
| Postoperative respiratory failure | 24.2 | 8.6 | 39,745 |
| Postoperative physiologic or metabolic derangement |  |  |  |
| Postoperative thromboembolism | 6.1 | 5.5 | 7,205 |
| Postoperative hip fracture |  |  |  |
| Iatrogenic pneumothorax | 2.7 | 3.9 | 5,633 |
| Decubitus ulcer | 6.8 | 5.2 | 6,713 |
| Postoperative hemorrhage/hematoma | 5.1 | 3.9 | 7,863 |
| Accidental puncture or laceration | 3.2 | 1.4 | 3,359 |

**Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.**

## Predictive validity: Impact of preventing each PSI event on mortality, LOS, VA expenditures (ROI)VA PTF 2001 analysis by Rivard et al., Med Care Res Rev; in press

| Indicator | Δ LOS (d) | Δ Cost ($) |
| --- | --- | --- |
| Postoperative septicemia | 5.7 | $13,395 |
| Selected infections due to medical care | 4.5 | 7,292 |
| Postop abd/pelvic wound dehiscence | 8.3 | 17,281 |
| Postoperative respiratory failure | 4.5 | 9,641 |
| Postoperative physiologic or metabolic derangement |  |  |
| Postoperative thromboembolism | 4.5 | 9,064  |
| Postoperative hip fracture |  |  |
| Iatrogenic pneumothorax | 3.4 | 5,476 |
| Decubitus ulcer | 3.7 | 5,552 |
| Postoperative hemorrhage/hematoma | 4.7 | 10,012 |
| Accidental puncture or laceration | 3.1 | 6,880 |

**Excess mortality, LOS, and charges computed from GEE regression models (logged costs and LOS).**

## Predictive validity questionable based on NIS/VA Zhan & Miller, JAMA 2003;290:1868-74 Rosen et al., Med Care 2005;43:873-84

| Indicator | Δ Mort (%) | Δ LOS (d) | Δ Charge ($) |
| --- | --- | --- | --- |
| Birth trauma | -0.1 (NS) | -0.1 (NS) | 300 (NS) |
| Obstetric trauma –cesarean | -0.0 (NS) | 0.4 | 2,700 |
| Obstetric trauma - vaginal w/out instrumentation | 0.0 (NS) | 0.05 | -100 (NS) |
| Obstetric trauma - vaginal w instrumentation | 0.0 (NS) | 0.07 | 220 |
| Complications of anesthesia* | 0.2 (NS) | 0.2 (NS) | 1,600 |
| Transfusion reaction* | -1.0 (NS) | 3.4 (NS) | 18,900 (NS) |
| Foreign body left during procedure† | 2.1 | 2.1 | 13,300 |

*** All differences NS for transfusion reaction and complications of anesthesia in VA/PTF. **
**† ****Mortality difference NS for foreign body in VA/PTF.**

## Criterion validity: POA coding in NY and CAHouchens, Elixhauser, Romano. Joint Comm J Qual Safety; in press

| PSI | CA cases | %not POA | NY cases | %not POA |
| --- | --- | --- | --- | --- |
| PSI 1: Complications of Anesthesia | 934 | 100 | 284 | 100 |
| PSI 3: Decubitus Ulcer | 17,789 | 11 | 16,425 | 14 |
| PSI 5: Foreign Body Left During Procedure | 258 | 64 | 169 | 76 |
| PSI 6: Iatrogenic Pneumothorax | 1,256 | 73 | 782 | 65 |
| PSI 7: Infection Due To Medical Care | 4,286 | 65 | 2,406 | 65 |
| PSI 8: Postop Hip Fracture | 106 | 21 | 69 | 26 |
| PSI 9: Postop Hemorrhage or Hematoma | 1,800 | 79 | 859 | 71 |
| PSI 10: Postop Physiologic and Metabolic Derangement | 686 | 77 | 228 | 64 |
| PSI 11: Postop Respiratory Failure | 2,374 | 94 | 1,312 | 93 |
| PSI 12: Postop PE or DVT | 6,715 | 46 | 5,318 | 43 |
| PSI 13: Postop Sepsis | 865 | 73 | 453 | 70 |
| PSI 15: Accidental Puncture/Laceration | 9,107 | 87 | 3,743 | 87 |
| PSI 16:  Transfusion Reaction | 12 | 58 | 9 | 78 |

**Notes:**
Finally, here is an analysis of criterion validity based on detailed review of medical records at children’s hospitals participating in the National Association of Children’s Hospitals and Related Institutions (NACHRI).  Reviewers classified each event as preventable, nonpreventable, or unclear.  Although the definition of  preventable was not clearly specified in advance, the PSIs varied widely on this metric.  These data were used by the UC-Stanford team to inform their decision-making about which PSIs should be retained for the Pediatric Quality Indicators.  The PSIs in blue were not retained for children, partially because of the low preventability ratings.

## Criterion validity: POA coding at Mayo hospitalsNaessens et al. Med Care 2007;45:781-788 (ob/birth indicators excluded)

| PSI | # cases | %not POA | AHRQ denomin | Corrected denomin |
| --- | --- | --- | --- | --- |
| PSI 1: Complications of Anesthesia | 16 | 6 | 29,681 | 29,681 |
| PSI 3: Decubitus Ulcer | 285 | 18 | 18,698 | 18,772 |
| PSI 5: Foreign Body Left | 13 | 54 | 58,206 | 58,206 |
| PSI 6: Iatrogenic Pneumothorax | 63 | 78 | 47,809 | 49,605 |
| PSI 7: Infection Due To Medical Care | 137 | 60 | 40,019 | 40,288 |
| PSI 8: Postop Hip Fracture | 9 | 22 | 16,770 | 16,788 |
| PSI 9: Postop Hemorrhage or Hematoma | 143 | 87 | 28,998 | 28,998 |
| PSI 10: Postop Physiologic and Metabolic Derangement | 48 | 46 | 23,654 | 23,669 |
| PSI 11: Postop Respiratory Failure | 123 | 74 | 18,270 | 18,270 |
| PSI 12: Postop PE or DVT | 492 | 40 | 28,876 | 28,949 |
| PSI 13: Postop Sepsis | 63 | 76 | 6,349 | 6,467 |
| PSI 15: Accidental Puncture/Laceration | 891 | 85 | 55,840 | 55,840 |
| PSI 14: Postop Wound Dehiscence | 34 | 100 | 7,637 | 7,637 |

**Notes:**
Finally, here is an analysis of criterion validity based on detailed review of medical records at children’s hospitals participating in the National Association of Children’s Hospitals and Related Institutions (NACHRI).  Reviewers classified each event as preventable, nonpreventable, or unclear.  Although the definition of  preventable was not clearly specified in advance, the PSIs varied widely on this metric.  These data were used by the UC-Stanford team to inform their decision-making about which PSIs should be retained for the Pediatric Quality Indicators.  The PSIs in blue were not retained for children, partially because of the low preventability ratings.

## Impact of POA coding in a hospital report card: postop hemorrhage

## Impact of POA coding in a hospital report card: decubitus ulcer

## Criterion validity: NY data linkageGallagher et al., AHRQ Advances in Patient Safety; Shufelt et al., Am J Med Qual 2005;20:210-8; Weller et al., Joint Comm J Qual Safe 2004;30:497-504

- Linking 30 day readmissions increased overall rate of PSIs:
- Selected Infections from 2.02 to 2.52 per 1,000 eligible discharges (56% dialysis patients)
- Postoperative DVT/PE from 9.3 to 11.3 per 1,000 (45% PE)
- Postoperative Hemorrhage/Hematoma from 1.86 to 2.05 per 1,000
- Relaxing the dx-procedure linking criterion increased the rate of Postoperative Hemorrhage/Hematoma from 1.86 to 2.35 per 1,000
- Based on procedure codes for repair of iatrogenic injuries, the PSI for Accidental Punctures and Lacerations missed:
- 27% of bladder injuries from hysterectomy
- 21% of bowel injuries from cholecystectomy
- 47% of abdominal injuries from lysis of adhesions
- 54% of abdominal injuries from nephroureterectomy
- 20% of spinal injuries from lumbar surgery

**Notes:**
In a series of studies, investigators from New York assessed criterion validity using linked administrative data.  They found that current PSI definitions fail to capture a significant minority of all relevant events, because of the requirement for specific combinations of diagnosis and procedure codes, or because of the inability to identify events requiring readmission.  Efforts to incorporate this information into the AHRQ PSI definitions are now underway.

## Criterion validity based on recoded data: CA Obstetric Validation Study

- Organized to assess validity of various potential measures of adverse events after delivery
- Cases sampled from OHSPD Patient Discharge Data Set (nonfederal acute care hospitals)
- Linked delivery, antepartum, postpartum records using SSN and DOB
- Stratified random cluster sample of 1,662 records from 52 hospitals (30% primary cesarean, 19% repeat cesarean, 51% vaginal)
- 97.1% of records received and reviewed by “expert” coder and obstetric nurse abstractor

**Notes:**
Another criterion validation effort focused on the obstetric indicators.  Once again, chart review by expert coders and obstetric coders was  used as a gold standard.

## Criterion validity in CA hospital discharge dataRomano PS, et al. Obstet Gynecol 2005;106(4):717-725

| 
Indicator | Sensitivity |  | Pos Pred Value |  |
| --- | --- | --- | --- | --- |
|  | Unweighted | Weighted | Unweighted | Weighted |
| FORMER AHRQ PSI: Obstetric trauma, Cesarean | 11% | 5% | 67% | 94% |
| HealthGrades: 
major comps, Vaginal  | 67% | 58% | 91% | 91% |
| HealthGrades: 
major comps, Cesarean | 55% | 47% | 64% | 79% |
| AHRQ/JCAHO: 3rd or 4th degree laceration | 90% | 93% | 90% | 73% |

**Sensitivity = TP/(TP+FN) – are all the real cases captured?**
**PPV = TP/(TP+FP) – are all the flagged cases real?**
**Brubaker L, et al. *****Obstet Gynecol***** 2007;109(5):1141-5 reported sensitivity of 77%, specificity of 99.7%, based on a clinical research data set with 393 positive (3****rd****/4****th**** degree tears) and 383 negative vaginal deliveries.**

**Notes:**
In this case, the AHRQ PSI definitions actually performed poorer than the JCAHO definition of perineal lacerations (one of JCAHO’s pregnancy-related core measures).  As a result, the AHRQ definition was changed to align fully with the JCAHO definition.

## Criterion validity:Linking VA PTF and NSQIP abstraction

- NSQIP is a national project that collects and feeds back data on surgical outcomes from 123 VA facilities
- Trained surgical clinical nurse reviewers collect preoperative, intraoperative, and postoperative data.
- Patients are followed for 30 days after index procedure
- Sampling frame: veterans, FY 2001, mainland US acute care facilities
- VA’s Patient Treatment File (similar to HCUP) was linked with NSQIP using SSN, dates of admission and discharge, facility number
- Final data file included 55,752 hospitalizations, representing 59,838 surgeries and 51,832 patients in 110 hospitals

## Criterion validity of PSIs linked to NSQIP in VA hospitals Romano PS, et al. HSR forthcoming?

| 
Indicator | Sensitivity |  | PPV |  | Positive
likelihood ratio |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Current
Inpatient | Better
Inpatient | Current
Inpatient | Better
Inpatient |  |  |
| Postoperative sepsis | 32% | 37% | 44% | 45% | 123 | 131 |
| Postoperative thromboembolism | 
56% | 
58% | 
22% | 
22% | 
65 | 
64 |
| Postoperative respiratory failure | 
19% | 
67% | 
74% | 
66% | 
194 | 
134 |
| Postop physiologic/  metabolic derangement | 
44% | 
48% | 
54% | 
63% | 
524 | 
744 |
| Postop abdominopelvic wound dehiscence | 
29% | 
61% | 
72% | 
57% | 
160 | 
79 |

**Sensitivity = TP/(TP+FN) – are all the real cases captured?**
**PPV = TP/(TP+FP) – are all the flagged cases real?**
**PLR = Sensitivity/(100-Specificity) – how many times more likely is the event?**

**Notes:**
This slide summarizes the findings from this analysis of criterion validity, for the five accepted PSIs that could be roughly matched to outcome measures in the NSQIP database.  Please note that these NSQIP definitions do not correspond exactly to the ICD-9-CM definitions of these diagnoses; for example, postoperative respiratory failure is defined more specifically in NSQIP than in ICD-9-CM.  We were able to come up with revised AHRQ definitions that enhanced the sensitivity of each indicator.  For 3 of these PSIs (shown in yellow), the revised definition was incorporated into the current PSI logic.  (The revised definition was not adopted for the other two because sensitivity could not be enhanced without significantly compromising predictive value.)

## NACHRI Pediatric Patient Safety Indicator (PSI) Collaborative 

- Ran the AHRQ PSIs on NACHRI’s Case Mix database, containing 3 million discharges from approximately 70 children’s hospitals.
- Developed the NACHRI Pediatric PSI Collaborative, a self-selected group of 20 hospitals interested in further study
- Developed and released a PSI Toolkit with sample press release, op ed, Q&A, and background documents for hospitals to educate their communities on the relevance and utility of PSIs for pediatrics.
- Developed an online, secure chart review tool that allowed Collaborative participants to review the preventability of patients flagged as having any of 11 selected PSI events.
- Fostered a relationship with AHRQ and Stanford/UC Davis to update each other on NACHRI’s findings and the PedQI development work.

## NACHRI Pediatric Patient Safety Indicator (PSI) Collaborative 

**Collaborative Participants**
**AL / Children’s Hospital of Alabama / Dr. Crayton Farguson***
**CA / Lucile Packard CH at Stanford / Dr. Paul Sharek***
**CA / UC-Davis / Dr. James Marcin****
**DC / Children’s National Medical Center / Dr. Tony Slonim***
**CA / Mattel Children’s at UCLA / Ms. Mary Kimball****
**FL / All Children’s / Dr. Jack Hutto***
**KY / Kosair Children’s Hospital / Dr. Ben Yandell***
**LA / Children’s Hospital New Orleans / Ms. Cindy Nuesslein***
**MD / Johns Hopkins Children’s Center / Dr. Marlene Miller***
**MA / Children’s Hospital Boston / Drs. Daniel Nigrin and Don Goldmann**
**MI / C.S. Mott Children’s Hospital – U Mich / Dr. Aileen Sedman***
**MO / Children’s Mercy Kansas City / Dr. Cathy Carroll***
**OH / The Children’s Medical Center Dayton / Dr. Thomas Murphy***
**OH / Cincinnati Children’s Medical Center / Drs. Uma Kotagal, Joseph Luria***
**OH / Children’s Hospital Columbus / Dr. Thomas Hansen***
**OH / Children’s Hospital MC of Akron / Dr. Michael Bird**
**PA / Children’s Hospital of Philadelphia / Drs. James Stevens, Joel Portnoy**
**TX / Texas Children’s Hospital / Dr. Joan Shook***
**TX / Children’s Medical Center of Dallas / Dr. Fiona Levy, Ms. Kathy Lauwers***
**WI / Children’s Hospital of Wisconsin / Dr. Matthew Scanlon***

## Criterion validity based on clinician review:AHRQ PSIs in Children’s HospitalsSedman A, et al. Pediatrics 2005;115(1):135-145

| PSI | No. reviewed
(total events) | Preventable
(PPV %) | Nonpreventable | Unclear
 |
| --- | --- | --- | --- | --- |
| Complications of anesthesia | 74 (503) | 11 (15%) | 37 | 25 |
| Death in low-mortality DRG | 121 (1282) | 16 (13%) | 89 | 16 |
| Decubitus ulcer | 130 (2300) | 71 (55%) | 47 | 10 |
| Failure to rescue | 187 (5271) | 15 (8%) | 148 | 11 |
| Foreign body left in | 49 (235) | 25 (51%) | 14 | 10 |
| Postop hemorrhage or hematoma | 114 (1571) | 40 (35%) | 51 | 23 |
| Iatrogenic pneumothorax | 114 (1113) | 51 (45%) | 42 | 21 |
| Selected infection 2° to med care | 152 (7291) | 63 (41%) | 45 | 39 |
| Postop DVT/PE | 126 (1956) | 36 (29%) | 61 | 29 |
| Postop wound dehiscence | 41 (232) | 19 (46%) | 16 | 6 |
| Accidental puncture or laceration | 133 (4020) | 86 (65%) | 19 | 26 |

**Notes:**
Finally, here is an analysis of criterion validity based on detailed review of medical records at children’s hospitals participating in the National Association of Children’s Hospitals and Related Institutions (NACHRI).  Reviewers classified each event as preventable, nonpreventable, or unclear.  Although the definition of  preventable was not clearly specified in advance, the PSIs varied widely on this metric.  These data were used by the UC-Stanford team to inform their decision-making about which PSIs should be retained for the Pediatric Quality Indicators.  The PSIs in blue were not retained for children, partially because of the low preventability ratings.

## Key findings from NACHRI’s PSI physician case reviews

*“...**while 40% to 50% may seem low for positive predictive value, in terms of real patients, this means that 4 or 5 out of 10 children had a preventable event for this indicator.  This is worth looking at and the things we are finding in some instances, will allow for immediate changes that may impact outcomes for future patients.” [Collaborative physician reviewer]*

## Examples from NACHRI’s PSI physician case reviews

- During removal of non functioning port cath the end of the catheter was noted to be "irregular and not smoooth cut". It appeared the tip had been embolized for an unknown duration...
- During replacement of pacemaker lead, a fragment of the lead broke off, embolized and ended up lodged (puncture) in the anterolateral papillary muscle.
- No notation in original operative note or nursing record that sponge/needle counts were done and correct.
- Count was reported as correct. Sponge discovered on xray due to complaints of abdominal pain by patient.
- Child with bone tumor who had mandible removed with subsequent bone graft and much packing in wound. This was supposedly removed before extubation, but at the time of extubation a remaining pack blocked her airway causing reintubation with pack removal.

## Romano’s Conclusions

- Several studies addressing PSI/PDI validity have been published, and several more are on the way.
- Most PSIs have domestic and international consensual (face) validity.
- Most PSIs have strong evidence of predictive (construct) validity in both VA and non-VA data.
- 3 PSIs have significant “POA problems”: postoperative  DVT/PE, postoperative hip fracture, decubitus ulcer.
- Linked readmissions data may be helpful in ruling in/out early discharge as a cause of low PSI rates.
- “Complications of Anesthesia” may be problematic.
- Coding validity looks strong for obstetric PSIs, and mixed for postoperative PSIs, but very limited data.
- Case review suggests 33-67% of most PSIs are potentially preventable (in children), except lower for death-based PSIs, DVT/PE, and complications of anesthesia.

## Questions?

**Notes:**
These were the 3 original modules of AHRQ Quality Indicators.