# Presentation: 646473

## AHRQ QI Guide to Comparative Reporting

- AHRQ Annual Conference
- September 10, 2008
- Bethesda, MD
- Presented by Sheryl Davies

## Overview of document

- User tool to select indicators
- Indicators assigned to four tiers based on evidence
- Gaps assessed for the following areas:
- Importance
- Face Validity
- Coding/Criterion Validity
- Risk Adjustment

**Notes:**
Importance: A quality indicator must reflect variation in quality or have low levels of overall performance. It should also incorporate a key leverage point and a significant opportunity for improvement. 
Face validity: A quality indicator must have sound clinical and or empirical rationale for its use. It should measure an important aspect of quality that is subject to provider or health care system control. Expert consensus of validity.  
Coding or criterion validity: The cases identified by the quality indicator reflect actual cases of the concept of interest (e.g. outcome of interest). This includes evidence of coding accuracy. 
Construct validity: The quality indicator should be supported by evidence of a relationship to quality, important processes of care and should be related to other indicators intended to measure the same or related aspects of quality.
Risk adjustment: The quality indicator is not subject to systematic bias due to case mix, or adequate risk adjustment can be applied using available data. 

Importance:
Because all indicators had to have substantial evidence for importance and face validity, either from prior use, published literature or clinical panel review, all indicators were assumed to pass this criterion, unless additional evidence has suggested otherwise.
Information gathered since the development of the indicator suggests that the indicator is not viewed as important in quality improvement efforts, or is viewed as having little leverage capability. 
Face validity:
Because all indicators had to have substantial evidence for importance and face validity, either from prior use, published literature or clinical panel review, all indicators were assumed to pass this criterion, unless additional evidence has suggested otherwise.
Information gathered since the development of the indicator suggests that the indicator is not viewed as valid by a majority of clinicians or other stakeholders. 
Criterion validity:
Evidence from published literature or otherwise available chart review indicating poor sensitivity or specificity of coding for the complication, or other coding problems that reduce the accuracy of the indicator. 
Evidence regarding criterion validity from literature and chart review is highly contradictory. 
When indicators definitions differed from those used in studies, judgments regarding the impact of the differences was made, and preferences were given to studies using identical or the most similar definitions. 
Construct validity:
No available evidence establishes a clear link between processes of care and the outcome of interest.  Volume-outcome relationships, or simple structure-outcomes relationships (such as staffing) do not qualify.
Evidence regarding construct validity from chart review or literature is highly contradictory. 
When indicators definitions differed from those used in studies, judgments regarding the impact of the differences was made, and preferences were given to studies using identical or the most similar definitions.
Risk adjustment:
Evidence documents important risk factors which cannot be identified using administrative data, unless those risk factors have been shown not to vary systematically by provider or geographic area. 

## Survey of Evidence

- Literature
- POA, coding validity, preventability, correlation with cost, LOS or other indicators, impact of QI programs
- AHRQ Validation Activities
- Chart review of coding validity
- Clinical Panel Review
- National Quality Forum review
- Examination of potential improvements

**Notes:**
Indicators often mirror NQF endorsement; NQF process was one source of vetting the indicators. However, some do not mirror, especially when we did not include the indicators in our original NQF submission. 

## Tiers

- Tier 1: Minor or no evidence gaps
- 1B Serious reportable events
- Most suitable for comparative reporting/public accountability
- Strongest evidence base
- Established evidence in several areas
- No substantial evidence against use for comparative reporting
- Most endorsed by the National Quality Forum

**Notes:**
Tier 1: Minor or no evidence gaps. Indicators in this tier have the strongest evidence base, with established evidence in several or most evidentiary areas and no substantial evidence suggesting that the indicator may not be useful for comparative reporting purposes. These indicators have all been endorsed by the National Quality Forum. A subset of Tier 1, labeled Tier 1B are serious reportable events, most suitable for public accountability and root cause analysis, but for which comparisons between hospitals may not be useful. 
Tier 2: Moderate evidence gaps. Indicators in this tier have some evidence supporting their use, but all have some evidence gaps. These gaps are not judged to be serious enough to preclude use in comparative reporting, but these indicators would be stronger if highlighted issues are better understood through future validation efforts.  These indicators are not currently endorsed by the National Quality Forum.
Tier 3: Significant, but addressable, evidence gaps. Indicators in this tier have at least one serious evidentiary gap or concern, which requires further development or validation work before the indicator is considered appropriate for comparative reporting. The required improvements and validation efforts are specified and underway, and once completed these indicators will be re-tiered. These indicators are not currently endorsed by the National Quality Forum.
Tier 4: Significant evidence gaps. The indicators in this tier have substantial gaps in the evidence that are unlikely to be addressed with further development or validation work. These indicators are not recommended for comparative reporting, but rather, can be used for internal quality improvement. These indicators are not currently endorsed by the National Quality Forum.

## Tiers

- Tier 2: Moderate evidence gaps
- Users may choose to utilize these indicators for comparative reporting
- Some supporting evidence for use
- Some minor evidence gaps
- Indicator would be stronger with some additional evidence
- Most not currently endorsed by the National Quality Forum

**Notes:**
Tier 1: Minor or no evidence gaps. Indicators in this tier have the strongest evidence base, with established evidence in several or most evidentiary areas and no substantial evidence suggesting that the indicator may not be useful for comparative reporting purposes. These indicators have all been endorsed by the National Quality Forum. A subset of Tier 1, labeled Tier 1B are serious reportable events, most suitable for public accountability and root cause analysis, but for which comparisons between hospitals may not be useful. 
Tier 2: Moderate evidence gaps. Indicators in this tier have some evidence supporting their use, but all have some evidence gaps. These gaps are not judged to be serious enough to preclude use in comparative reporting, but these indicators would be stronger if highlighted issues are better understood through future validation efforts.  These indicators are not currently endorsed by the National Quality Forum.
Tier 3: Significant, but addressable, evidence gaps. Indicators in this tier have at least one serious evidentiary gap or concern, which requires further development or validation work before the indicator is considered appropriate for comparative reporting. The required improvements and validation efforts are specified and underway, and once completed these indicators will be re-tiered. These indicators are not currently endorsed by the National Quality Forum.
Tier 4: Significant evidence gaps. The indicators in this tier have substantial gaps in the evidence that are unlikely to be addressed with further development or validation work. These indicators are not recommended for comparative reporting, but rather, can be used for internal quality improvement. These indicators are not currently endorsed by the National Quality Forum.

## Tiers

- Tier 3: Significant, but addressable, evidence gaps
- Not currently suitable for comparative reporting, pending further development
- At least one serious evidence gap or concern
- Further development and validation work has been identified
- Once further work is completed, indicators will be re-assigned to another tier
- Indicators not currently endorsed by NQF

**Notes:**
Tier 1: Minor or no evidence gaps. Indicators in this tier have the strongest evidence base, with established evidence in several or most evidentiary areas and no substantial evidence suggesting that the indicator may not be useful for comparative reporting purposes. These indicators have all been endorsed by the National Quality Forum. A subset of Tier 1, labeled Tier 1B are serious reportable events, most suitable for public accountability and root cause analysis, but for which comparisons between hospitals may not be useful. 
Tier 2: Moderate evidence gaps. Indicators in this tier have some evidence supporting their use, but all have some evidence gaps. These gaps are not judged to be serious enough to preclude use in comparative reporting, but these indicators would be stronger if highlighted issues are better understood through future validation efforts.  These indicators are not currently endorsed by the National Quality Forum.
Tier 3: Significant, but addressable, evidence gaps. Indicators in this tier have at least one serious evidentiary gap or concern, which requires further development or validation work before the indicator is considered appropriate for comparative reporting. The required improvements and validation efforts are specified and underway, and once completed these indicators will be re-tiered. These indicators are not currently endorsed by the National Quality Forum.
Tier 4: Significant evidence gaps. The indicators in this tier have substantial gaps in the evidence that are unlikely to be addressed with further development or validation work. These indicators are not recommended for comparative reporting, but rather, can be used for internal quality improvement. These indicators are not currently endorsed by the National Quality Forum.

## Tiers

- Tier 4: Significant evidence gaps
- Substantial gaps in evidence for _use in comparative reporting_
- Gaps unlikely to be addressed with further development or validation
- Not recommended for _comparative reporting_
- Indicators are not currently endorsed by NGF

**Notes:**
Tier 1: Minor or no evidence gaps. Indicators in this tier have the strongest evidence base, with established evidence in several or most evidentiary areas and no substantial evidence suggesting that the indicator may not be useful for comparative reporting purposes. These indicators have all been endorsed by the National Quality Forum. A subset of Tier 1, labeled Tier 1B are serious reportable events, most suitable for public accountability and root cause analysis, but for which comparisons between hospitals may not be useful. 
Tier 2: Moderate evidence gaps. Indicators in this tier have some evidence supporting their use, but all have some evidence gaps. These gaps are not judged to be serious enough to preclude use in comparative reporting, but these indicators would be stronger if highlighted issues are better understood through future validation efforts.  These indicators are not currently endorsed by the National Quality Forum.
Tier 3: Significant, but addressable, evidence gaps. Indicators in this tier have at least one serious evidentiary gap or concern, which requires further development or validation work before the indicator is considered appropriate for comparative reporting. The required improvements and validation efforts are specified and underway, and once completed these indicators will be re-tiered. These indicators are not currently endorsed by the National Quality Forum.
Tier 4: Significant evidence gaps. The indicators in this tier have substantial gaps in the evidence that are unlikely to be addressed with further development or validation work. These indicators are not recommended for comparative reporting, but rather, can be used for internal quality improvement. These indicators are not currently endorsed by the National Quality Forum.

## Tier 1 Indicators

- IQIs
- Esophageal Resection Volume (#01)*
- Pancreatic Resection Volume (#02)*
- Abdominal Aortic Aneurysm (AAA) Repair Volume (#04)*
- Esophageal Resection Mortality (#08)*
- Pancreatic Resection Mortality (#09)*
- Abdominal Aortic Aneurysm (AAA) Repair Mortality (#11)*
- AMI Mortality (#15 and #32)
- CHF Mortality (#16)*
- Acute Stroke Mortality (#17)*
- Hip Fracture Mortality (#19)*
- Pneumonia Mortality (#20)*
- Incidental Appendectomy in the Elderly (#24)*
- Bi-lateral Catheterization (#25)*
- *NQF endorsed

## Tier 1 Indicators

- PDIs
- Accidental Puncture or Laceration (#01)*
- Decubitus Ulcer (#02)*
- Iatrogenic Pneumothorax (#05)*
- Pediatric Heart Surgery Mortality (#06)*
- Pediatric Heart Surgery Volume (#07)*
- Postoperative Wound Dehiscence (#11)*
- Blood Stream Infection in Neonates (#02)*
- PSIs
- Death among Surgical Inpatients with Treatable Serious Complications (#04)*
- Iatrogenic Pneumothorax (#06)*
- Postoperative Hip Fracture (#08)
- Postoperative Wound Dehiscence (#14)*
- Accidental Puncture or Laceration (#15)*
- OB Trauma with and without Instrument (#18 and #19)

## Tier 1B Indicators

- PSIs
- Death in Low Mortality DRGs*
- Transfusion Reaction
- Foreign Body Left in During Procedure
- PDIs
- Transfusion Reaction
- Foreign Body Left in During Procedure
- *NQF endorsed

## Tier 2 Indicators

- IQI
- Hip Replacement Mortality (#14)
- PDI
- Iatrogenic Pneumothorax in Neonates
- Postoperative Hemorrhage or Hematoma (#08)
- Postoperative Respiratory Failure (#09)
- Postoperative Sepsis (#10)
- PSI
- Postoperative Hemorrhage or Hematoma (#09)
- Postoperative Physiologic and Metabolic Derangement (#10)
- Postoperative Respiratory Failure (#11)
- Postoperative DVT or PE (#12)*
- Postoperative Sepsis (#13)
- Birth Trauma (#17)*
- *NQF endorsed

## Tier 3 Indicators

- IQIs
- CABG Mortality (#12)
- Craniotomy Mortality (#13)
- GI Hemorrhage Mortality (#18)
- PDIs/NQIs
- Hospital Acquired Vascular Catheter Related Infections (#12)
- Neonatal Mortality
- PSIs
- Decubitus Ulcer (#03)
- Hospital Acquired Vascular Catheter Related Infections (#07)

## Tier 4 Indicators

- IQIs
- CABG Volume (#05)
- PTCA Volume (#06)
- Carotid Endarterectomy Volume (#07)
- Cesarean Delivery (#21 and #33)
- VBAC Delivery (#22 and #34)
- Laparoscopic Cholecystectomy (#23)
- PTCA Mortality (#30)
- Carotid Endarterectomy Mortality (#31)
- PSIs
- Complications of Anesthesia (#01)
- OB Trauma-Cesarean Delivery (#20)

## Applying Guidance to your Own Report

- Guidance document is _a source of information_, users need to weigh their own situation to select indicators
- AHRQ QI Reporting Template
- Know your purpose
- Tier 2 indicators may be more useful for some purposes than others

## Applying Guidance to Your Own Support: Data

- Know your data
- Potential data checks
- Rates and variation in POA
- Procedure dates, Elective vs. Non-elective, E-codes
- “Hospitals Like Mine” Tool – HCUPnet
- AHRQ Tools
- Medical record review
- Audit
- Individual hospitals

**Notes:**
POA: AHRQ has published POA rates for the Patient Safety Indicators based on data from California and New York PSI 1, 3, 5-8, 10-13, 15-20 PDI 1-3, 5, 9-10, 12-13

Check percentage of missing data for 
procedure dates (3, 
elective
E-codes: PSI 1,5, 15, 16, 21, 25, 26 PDI, 1, 3, 13