# Presentation: 062432

## AHRQ IQI Clinical Validation Panels

- Sheryl Davies, MA
- Stanford University

## Outline	

- Purpose
- Composition and Recruitment
- Rating Process
- Overall themes
- Indicator results
- Take home lessons

## Purpose of the Clinical Panels

- IQIs and PQIs
- Developed 1999-2001
- Based on established indicators
- Did not undergo panel review
- Panel review establishes face validity of the indicators
- Standardize available evidence for all AHRQ QIs
- Establish face validity for one stakeholder group
- Update evidence
- Panel review for IQIs considered for NQF endorsement

## Methods

- Modified RAND/UCLA Appropriateness Method (Nominal Group Technique)
- Physicians of various specialties/subspecialties and other health professionals were recruited with the assistance of relevant organizations
- 22 contacted organizations nominated
- 103 clinicians nominated  60 accepted  45 eligible
- Panelists selected in order to form diverse panels

## Panelists

- Female: 18%
- Academic: 71%
- Geographic
- East: 44%
- West: 29%
- Other: 27%
- Practice setting
- Urban: 67%
- Suburban: 42%
- Rural: 18%
- Funding of primary hospital:
- Private: 44%
- Public: 27%
- Underserved patient population: 56%

## Panel methods: Ratings

- Initial ratings
- Packet of information summarizing evidence
- Approx. 10 questions
- Tailored to the indicator type
- 9 point scale
- Overall usefulness for quality improvement, comparative reporting
- Compiled ratings provided to panelists
- Conference call
- Discuss differences
- Consensus on definition changes
- Final ratings
- Empirical analyses provided
- Using same questionnaire as initial ratings

## Results: Overarching themes

- Case mix variability
- Reliability
- Volume measures as indirect measures of quality
- Composite measures

## Cardiac Panel

- Reviewed 5 indicators:
- AAA Volume/Mortality
- Pediatric Heart Surgery Volume/Mortality
- Bilateral Catheterization
- 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, surgical nurse

## Cardiac Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| AAA Mortality | Acceptable (7) | Unclear (6) |
| AAA Volume | Acceptable (7) | Acceptable (7) |
| Bilateral catheterization | Unclear (5) | Unclear with disagreement (5) |
| Pediatric heart surgery volume | Acceptable (8) | Acceptable (8) |
| Pediatric heart surgery mortality | Acceptable (8) | Acceptable (8) |

## Cardiac Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| AAA Mortality | Acceptable (7) | Unclear (6) |
| AAA Volume | Acceptable (7) | Acceptable (7) |

-  Case mix variability
-  Ruptured vs. unruptured; endovascular vs. open
-  Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)
-  Total volume (ruptured and unruptured) best predictor of outcomes
-  Stratify by surgical approach (endovascular vs. open)

**Notes:**
Adjusted AAA mortality rates vary substantially across hospitals.
Endovascular repair has gradually replaced open repair (for anatomically appropriate lesions), without an increase in the total of AAA repairs. In 2004, about half of AAA repairs were endovascular, with no evidence of full diffusion.
Endovascular repair has lower short-term mortality than open repair resulting in the reduction in overall AAA repair mortality. Technique-specific mortality has changed little over time, indicating that substitution of endovascular surgery for open surgery is the primary explanation for reduced overall AAA repair mortality. 
The substitution of endovascular surgery for open surgery has occurred at different rates in different areas. 

## Cardiac Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| AAA Mortality | Acceptable (7) | Unclear (6) |
| AAA Volume | Acceptable (7) | Acceptable (7) |

-  Case mix variability
-  Ruptured vs. unruptured; endovascular vs. open
-  Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)
-  Total volume best predictor of outcomes
-  Stratify by surgical approach (endovascular vs. open)

**Notes:**
Adjusted AAA mortality rates vary substantially across hospitals.
Endovascular repair has gradually replaced open repair (for anatomically appropriate lesions), without an increase in the total of AAA repairs. In 2004, about half of AAA repairs were endovascular, with no evidence of full diffusion.
Endovascular repair has lower short-term mortality than open repair resulting in the reduction in overall AAA repair mortality. Technique-specific mortality has changed little over time, indicating that substitution of endovascular surgery for open surgery is the primary explanation for reduced overall AAA repair mortality. 
The substitution of endovascular surgery for open surgery has occurred at different rates in different areas. 

## Cardiac Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Pediatric heart surgery volume | Acceptable (8) | Acceptable (8) |
| Pediatric heart surgery mortality | Acceptable (8) | Acceptable (8) |

-  Case mix variability
-  Supported use of RACHS
-  Correlations of hospital volume for each RACHS complexity are robust (r = 0.74 – 0.95)
-  Best predictor of outcome is total volume, rather than by complexity

## Cardiac Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Bilateral catheterization | Unclear (5) | Unclear w/ disagreement (5) |

-  Modification: Expand list of appropriate indications for bilateral catheterization
-  Primarily a resource indicator
-  Charting of indications may be poor
-  May result in decrease of appropriate uses

## Surgical Resection Panel

- Reviewed 4 indicators:
- Esophageal Resection Volume/Mortality
- Pancreatic Resection Volume/Mortality
- 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), oncologists, internist, gastroenterologists, surgical nurse

## Surgical Resection Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Esophageal resection mortality | Acceptable (7) | Acceptable (7) |
| Esophageal resection volume | Acceptable (7) | Acceptable (7) |
| Pancreatic resection mortality | Acceptable (7) | Acceptable (7) |
| Pancreatic resection volume | Acceptable (7) | Acceptable (7) |

## Surgical Resection Panel

- Esophageal resection
-  Case mix variability
-  Risk adjustment performs well. Underestimates risk for patient with middle esophageal and unspecified site cancer (22%, 37%).
- Pancreatic resection
-  Case mix variability
-  Over-estimates risk for total pancreatectomy (68%), lesser extent underestimates risk for Whipple (15%). Issue raised with 3M.
-  Low rates

**Notes:**
Total pancreatectomy make up only 7% of numerator and 5% of denominator. 

## Geriatric Panel

- Reviewed 4 indicators:
- Acute Stroke Mortality
- Hip Fracture Mortality
- Hip Replacement Mortality
- Incidental Appendectomy
- 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapist

## Geriatric Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Acute stroke mortality  | Unclear (6.5) | Unclear with disagreement (5) |
| Incidental appendectomy | Acceptable (7) | Unclear (6) |
| Hip fracture mortality | Acceptable (7) | Acceptable (7) |
| Hip replacement mortality | Unclear due to disagreement (7) | Unclear (6) |

## Geriatric Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Acute stroke mortality  | Unclear (6.5) | Unclear with disagreement (5) |
| Incidental appendectomy | Acceptable (7) | Unclear (6) |
| Hip fracture mortality | Acceptable (7) | Acceptable (7) |
| Hip replacement mortality | Unclear due to disagreement (7) | Unclear (6) |

-  Case mix variability: Stroke type (hemorrhagic, ischemic, subarachnoid)
-  Risk adjustment accounts for almost all difference in risk
-  Patient factors such as delay in presenting for care

**Notes:**
(hemorrhagic, ischemic, subarachnoid). 

## Geriatric Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Acute stroke mortality  | Unclear (6.5) | Unclear with disagreement (5) |
| Incidental appendectomy | Acceptable (7) | Unclear (6) |
| Hip fracture mortality | Acceptable (7) | Acceptable (7) |
| Hip replacement mortality | Unclear due to disagreement (7) | Unclear (6) |

-  Exclude patients with hip fracture
-  Case mix variability
-  Risk adjustment somewhat overestimates risk for revision
-  Rates very low, reliability concerns

## Geriatric Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Acute stroke mortality  | Unclear (6.5) | Unclear with disagreement (5) |
| Incidental appendectomy | Acceptable (7) | Unclear (6) |
| Hip fracture mortality | Acceptable (7) | Acceptable (7) |
| Hip replacement mortality | Unclear due to disagreement (7) | Unclear (6) |

-  “Is it still being done?”
-  “If it is still being done, it shouldn’t be done. Then it is a good indicator”
-  “I am having a hard time getting excited about this indicator”

**Notes:**
Rate varies substantially across hospitals, with at least 6% of hospitals having rates in excess of 7.5%. 

## Geriatric Panel

| Indicator | Overall – QI | Overall - Comparative |
| --- | --- | --- |
| Acute stroke mortality  | Unclear (6.5) | Unclear with disagreement (5) |
| Incidental appendectomy | Acceptable (7) | Unclear (6) |
| Hip fracture mortality | Acceptable (7) | Acceptable (7) |
| Hip replacement mortality | Unclear due to disagreement (7) | Unclear (6) |

-  Limit to the elderly
-  Case mix variability
-  Risk adjustment accounts for both repair type and fracture location

**Notes:**
Risk adjustment accounts for almost all differences associated with repair type (e.g. internal fixation, replacement) and fracture location (e.g. femoral neck, subtrochanteric, intertrochanteric, unspecified). 

## Conclusion

- Overall good reception of indicators
- Recommendations considered in context of other validation efforts
- Indicator revisions implemented in February 2008
- Further efforts to improve indicators or develop additional evidence

## Questions?