# Presentation: 752802

## Day 2: Session II

- Presenter: Jeffrey Geppert, Battelle
- AHRQ QI User Meeting
- September 26-27, 2005

## Methods for Creating Aggregate Performance Indices 

- AHRQ QI User Meeting
- September 27, 2005
- Jeffrey Geppert
- Battelle Health and Life Sciences

## Overview

- Project objectives
- Why composite measures?
- Who might use composite measures?
- Alternative approaches
- Desirable features of a composite
- Proposed approach for the AHRQ QI
- Questions & Answers

## Project Objectives

- Composite measures for the AHRQ QI included in the National Healthcare Quality Report and Disparities Report
- Separate composites for overall quality and/or quality within certain domains (e.g., cardiac care, surgery, avoidable hospitalizations, diabetes, adverse events)
- A methodology that can be used at the national, state and provider/area level

## Project Objectives

- Feedback
- Does the proposed approach meet user needs for a composite?
- What analytic uses should the composite address?
- What are the important policy issues?
- How should the composite be incorporated into the AHRQ QI software?

## Goals of National Healthcare Reports

- National Level
- Provide assessment of quality and disparities
- Provide baselines to track progress
- Identify information gaps
- Emphasize interdependence of quality and disparities
- Promote awareness and change
- State / Local / Provider Level
- Provide tools for self-assessment
- Provide national benchmarks
- Promote awareness and change

## Unique challenges to quality reporting by states

- States release comparative quality information in a political environment
- Either must adopt defensible scientific methodology or make conservative assumptions
- Examples of reporting decisions:
- Small numbers issues
- Interpretive issues (better/worse, higher/lower)
- Purchasers demanding outcomes and cost information from states

## Why Composites?

- Summarize quality across multiple measures
- Improve ability to detect quality differences
- Identify important domains and drivers of quality
- Prioritize action
- Make current decisions about future (unknown) healthcare needs
- Avoids cognitive “short-cuts”

## Why Not Composites?

- Mask important differences and relationships among components (e.g. mortality and re-admissions)
- Not “actionable”
- Difficult to identify which parts of the healthcare system contribute most to quality
- Detract from the impact and credibility of reports
- Independence of components
- Interpretation of components

## Who Might Use Them?

- Consumers – To select a hospital either before or after a health event
- Providers – To identify the domains and drivers of quality
- Purchasers – To select hospitals in order to improve the health of employees
- Policymakers – To set policy in order to improve the health of a population

## Examples

- “America’s Best Hospitals” (U.S. News & World Report)
- Leapfrog Safe Practices Score (27 procedures to reduce preventable medical mistakes)
- NCQA, “America’s Best Health Plans”
- QA Tools (RAND)
- Veteran Health Administration (Chronic Disease Care Index, Prevention Index, Palliative Care Index)
- Joint Commission (heart attack, heart failure, pneumonia, pregnancy)
- National Health Service (UK) Performance Ratings
- CMS Hospital Quality Incentive Demonstration Project

## Examples

- Measures

- Components

- Composite

- Goals

- Utility

## Alternative Approaches

| Approach | Goal | Utility |
| --- | --- | --- |
| Opportunity | Appropriate care | Volume of opportunities |
| Burden | Minimize excess death/costs | Measures with most excess |
| Expected quality | Better than reference | Lowest ratio |
| Variation | Better than reference | Outliers |
| Latent quality | Reduce variation | Measures with greatest variation |

## Desirable Features

- Valid -  Based on valid measures
- Reliable – Improve ability to detect differences
- Minimum Bias – Based on unbiased measures
- Actionable – Interpretable metric
- Benchmarks or standards
- Transparent
- Predictive – Should guide the decision-maker on likely future quality based on current information.
- Representative – Should reflect expected outcomes for population

## Proposed Approach

- A modeling-based approach
- Latent quality – observed correlation in individual measures is induced by variability in latent quality
- Individual measures with highest degree of variation have larger contribution to composite
- Theoretical interpretation
- Consistent with goal of reducing overall variation in quality

## Proposed Approach

- Latent Quality

- Component

- Component

- Component

- Component

## Advantages

- Avoids contradictory results with individual measures or the creation of composites that may mislead
- Construction of the composite increases the power of quality appraisals
- Allows for both measure-specific estimates and composites
- Allows for validation with out-of-sample prediction

## Advantages (Continued)

- Hierarchical – for small numbers, the best estimate is the pooled average rate at similar hospitals
- Allows for incorporation of provider characteristics to explain between-provider variability (e.g., volume, technology, teaching status)
- Gives policymakers information on the important drivers of quality

## Overview of AHRQ QIs

- Prevention Quality
- 	Indicators
- Inpatient Quality Indicators
- Patient Safety Indicators

- Ambulatory care sensitive
- 	conditions
- Mortality following procedures
- Mortality for medical conditions
- Utilization of procedures
- Volume of procedures
- Post-operative complications
- Iatrogenic conditions

**Notes:**
Volume are useful quality indicators for low volume providers or infrequent procedures, when mortality rates are unreliable

## Examples

## Examples

## Examples

## Examples

## Examples

## Examples

## Hierarchical Models

- Also referred to as smoothed rates or reliability-adjusted rates
- Endorsed by NQF for outcome measures
- Methods to separate the within and between provider level variation (random vs. systematic)
- Total variation = Within provider + Between provider (Between = Total – Within)
- Reliability (w) = Between / Total
- Signal ratio = signal / (signal+noise)

## Hierarchical Models

- Smoothed rate is the (theoretical) best predictor of future quality
- Provides a framework for validation and forecasting
- Smoothed rate (single provider, single indicator) =
- 		Hospital-type rate * (1 – w) +
- 		Hospital-specific rate * w
- Multivariate versions
- Other Years (auto-regression, forecasting)
- Other Measures (composites)
- Non-persistent innovations (contemporaneous, nonsystematic shocks)

## Outcomes and Process

- Source:  Landrum et. al. Analytic Methods for Constructing Cross-Sectional Profiles of Health Care Providers (2000)

## Hierarchical Models

- Provider Type

- Provider

- Patient

- Policy

- Quality

## Policy and Prediction

- The best predictor of future performance is often historical performance + structure
- The greater the reliability of the measure for a particular provider, the more weight on historical performance
- The less the reliability of the measure for a particular provider, the more weight on structure
- Volume often improves the ability to predict performance for low-volume providers
- Other provider characteristics (e.g. availability of technology) do as well
- Area characteristics (e.g., SES) do as well

## Socio-Economic Status

- The Public Health Disparities Geo-coding Project  - Harvard School of Public Health (PI: Nancy Krieger)
- Evaluated alternative indices of SES (e.g. Townsend and Carstairs)
- Occupational class, income, poverty, wealth, education level, crowding
- Gradations in mortality, disease incidence, LBW, injuries, TB, STD
- Percent of persons living below the U.S. poverty line
- Most attuned to capturing economic depravation
- Meaningful across regions and over time
- Easily understood and readily interpretable

## Socio-Economic Status

## Limitations

- Measures and methods difficult
- Restrictive assumptions on correlation
- Correlations may vary by provider type
- Requires a large, centralized data source

## Expansions

- Flexibility in weighting the components
- Empirical – domains driven entirely by empirical relationships in the data
- A priori – domains determined by clinical or other considerations
- Combination – empirical when the relationships are strong and the measures precise, otherwise a priori

## Welfare-driven Composites

- Measures

- Composites

- Meta-composites

## Welfare-Driven Composites

- Making current decisions about future needs – maximize expected outcomes, minimize expected costs
- Policymaker focus – for a population
- A provider focus – for their patients
- A employer focus – for their employees
- A consumer focus – based on individual characteristics

## Acknowledgments

**Funded by AHRQ**
- Support for Quality Indicators II (Contract No. 290-04-0020)
- Mamatha Pancholi, AHRQ Project Officer
- Marybeth Farquhar, AHRQ QI Senior Advisor
- Mark Gritz and Jeffrey Geppert, Project Directors, Battelle Health and Life Sciences
**Data used for analyses:**
- Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality
- State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality

## Acknowledgements

- We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health Planning & Development; Colorado Health & Hospital Association; Connecticut - Chime, Inc.; Florida Agency for Health Care Administration; Georgia: An Association of Hospitals & Health Systems; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Health; Maine Health Data Organization; Maryland Health Services Cost Review; Massachusetts Division of Health Care Finance and Policy; Michigan Health & Hospital Association; Minnesota Hospital Association; Missouri Hospital Industry Data Institute; Nebraska Hospital Association; Nevada Department of Human Resources; New Jersey Department of Health & Senior Services; New York State Department of Health; North Carolina Department of Health and Human Services; Ohio Hospital Association; Oregon Association of Hospitals & Health Systems; Pennsylvania Health Care Cost Containment Council; Rhode Island Department of Health; South Carolina State Budget & Control Board; South Dakota Association of Healthcare Organizations; Tennessee Hospital Association; Texas Health Care Information Council; Utah Department of Health; Vermont Association of Hospitals and Health Systems; Virginia Health Information; Washington State Department of Health; West Virginia Health Care Authority; Wisconsin Department of Health & Family Services.

## Questions & Answers

- Questions And Answers