# Presentation: 094227

## AHRQ Quality Indicators

- Developed by Stanford-UCSF Evidence Based Practice Center
- Funded by the Agency for Healthcare Research and Quality

- EPC Team (PSI Development)
- PI: Kathryn McDonald, M.M., Stanford
- Patrick Romano, M.D., M.P.H, UC Davis
- Jeffrey Geppert, J.D., Ed.M., Stanford
- Sheryl Davies, M.A., Stanford
- Bradford Duncan, M.D., M.A., Stanford Kaveh G. Shojania, M.D., UCSF

- Support of Quality Indicators
- PI: Kathryn McDonald, M.M., Stanford
- Sheryl Davies, M.A., Stanford
- Patrick Romano, M.D., M.P.H, UC Davis
- Jeffrey Geppert, J.D. Ed.M., Stanford
- Mark Gritz, PhD, Battelle
- Greg Hubert, Battelle
- Denise Remus, RN PhD, AHRQ Project Officer

## Acknowledgements 

**Funded by AHRQ**
- Contract No. 290-97-0013
- Support of Quality Indicators Contract No. 290-02-0007
- Presentation funded by AHRQ
**Data used for analyses:**
- Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality
- State Inpatient Databases (SID), 1997 (19 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality

## Acknowledgements

- We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health and Development; Colorado Health and Hospital Association; CHIME, Inc. (Connecticut); Florida Agency for Health Care Administration; Georgia Hospital Association; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Maryland Health Services Cost Review Commission; Massachusetts Division of Health Care Finance and Policy; Missouri Hospital Industry Data Institute; New Jersey Department of Health and Senior Services; New York State Department of Health; Oregon Association of Hospitals and Health Systems; Pennsylvania Health Care Cost Containment Council; South Carolina State Budget and Control Board; Tennessee Hospital Association; Utah Department of Health; Washington State Department of Health; and Wisconsin Department of Health and Family Service.

## Outline

- Administrative data and quality indicators
- AHRQ Quality Indicators (QI)
- Development of AHRQ QIs
- Risk adjustment & MSX smoothing methods
- Application of QIs to research and quality

## History of AHRQ QIs/PSIs

- Healthcare Cost and Utilization Project (HCUP)
- HCUP discharge data collection (1988)
- HCUP Quality Indicators
- Mortality for Inpatient Procedures
- Complication Rates
- Potentially Inappropriate Utilization
- Potentially Avoidable Hospital Admissions

**Notes:**
HCUP – began as a data collection project in 1988.
QIs developed by AHRQ in the early 1990s in response to States' requests for a quality assessment tool that could be used with hospital administrative data. 
33 Quality indicators. Intended to be used a screening tools. 
Based on existing indicators reported in the literature at the time.
Limitations of QIs: 1.  Limited Risk Adjustment. 2.  Low frequencies in some 3. Primarily surgical indicators. 4. Some less appropriate denominators (not area level).

## Refinement of HCUP QIs

- Refinement commissioned by AHRQ in 1999
- Completed by UCSF-Stanford EPC
- Two related projects
- Two technical reviews
- Refinement of the HCUP Quality Indicators
- Measures of Patient Safety Based on Administrative Data
- Three indicator sets, AHRQ QIs
- Inpatient Quality Indicators (IQIs)
- Prevention Quality Indicators (PQIs)
- Patient Safety Indicators (PSIs)

**Notes:**
Refinement of HCUP QIs completed between 1999 and 2002 by UCSF-Stanford EPC. 
The first project examined mortality, utilization and ambulatory care sensitive conditions, later split into the PQIs and IQIs.
Defined at both an area level, and hospital level. 
Used literature review and empirical analyses to identify and summarize the evidence for use of these quality indicators as screening tools, including face validity, precision of measurement, unbiasedness, construct validity, and fostering true quality improvement. 
Novel empirical methods used mathematical techniques to extract true variations and discard noise or random variation. 
Second project identified complications and patient safety indicators. Methods will be explained in detail later, similar to QI project, with additional clinical review.
Result of the two projects was two separate technical reviews, available for download or from AHRQ.
Three indicators sets and software – IQIs, PQIs, and PSIs. 

## Administrative Data & Quality Improvement

- Opportunities
- Coding practices improving
- Data availability improving (e.g., less truncation)
- More specific codes
- Large data sets improve precision
- Comprehensive: all hospitals
- Quality screening feasible

- Obstacles
- Coding errors introduce noise
- Lack of information on timing, comorbidity vs. adverse events
- Varying number of  secondary diagnoses fields can cause bias
- Heterogeneous severity within single code

## Administrative Data

- State Inpatient Databases
- Includes ICD-9-CM dx and procedure codes, DRG, dates, age, sex, payer, race, discharge disposition, hospital and/or patient zip codes
- 1995-2002
- 33 States
- 80% + of all U.S. hospital discharges
- 18 states available for purchase
- In 27 state sample, approximately 3200 hospitals

**Notes:**
Ranges from $20-$1600 per state per year for academic organizations. 

## Administrative Data

- Nationwide Inpatient Sample (NIS)
- Sampling of State Inpatient Databases
- 1988-2001
- 7.5 million discharges/1,000 hospitals/33 States
- Approximates 20% sample of nonfederal acute care hospitals
- Discharge level weights applied for national estimates
- Available for purchase

**Notes:**
NIS releases for data years 1988-2001 are available from the HCUP Central Distributor. NIS 2001 may be purchased for $200 in a set of two CD-ROMs with accompanying documentation from the HCUP Central Distributor

## HCUPnet

- _[http://www.ahrq.gov/data/hcup/hcupnet.htm](http://www.ahrq.gov/data/hcup/hcupnet.htm)_
- Web-based tool to query NIS and KIDS databases, 1993-2001
- Pre-run tables for 1997-2001
- Query based on ICD-9-CM, DRG or CCS
- Information on hospitalizations, charges, length of stay, mortality, discharge status
- Stratification by age, sex, race, income, insurance, hospital characteristics
- Rank order hospitalizations

## Outline

- Administrative data and quality indicators
- AHRQ Quality Indicators (QI)
- Development of AHRQ QIs
- Risk adjustment & MSX smoothing methods
- Application of QIs to research and quality

## Sample AHRQ QI definition

| DECUBITUS ULCER |  |
| --- | --- |
| Relationship to quality: Identifies cases of decubitus ulcers that develop during hospitalization.  |  |
| Indicator definition: 
           Number of patients with decubitus ulcer (see definition and exclusions below) per 100 eligible admissions (population at risk). |  |
| Definition of decubitus ulcer: | Definition of population at risk:
Patients eligible to be included in this indicator: |
| Secondary diagnosis code of decubitus ulcer:
decubitus ulcer [707.0] | a. All patients (medical and surgical)
c. Patient must have a length of stay of more than 4 days.
d. Patient must not be in MDC 9 (Diseases and Disorders of the Skin Subcutaneous Tissue and Breast), or have any diagnosis of hemiplegia, paraplagia, or quadriplagia.  |

**Notes:**
The indicator definition summarizes the operationalization, but doesn’t include ICD-9-CM codes. The specific ICD-9-CM codes used in the definition of the complication are listed, as well as the denominator (the population at risk). Specific ICd-9-CM codes for the denominator were listed in an appendix. 

This indicator has several examples of exclusion criteria. Some of these exclusions exist because patients in these categories were more likely to have decubiti present on admission (which cannot be discerned with most administrative data), or are more likely to develop a non-preventable decubitus ulcer. 

The clinical rationale briefly explains the face validity assertion of the indicator, the source, and the reasoning for all the details of the operationalization of the indicator.

## Prevention Quality Indicators (PQIs)

- Defined using area population as denominator
- Potentially avoidable hospitalizations or ambulatory care sensitive conditions
- Conditions for which good outpatient care can potentially prevent the need for hospitalization or for which early intervention can prevent complications or more severe disease
- Public health, comprehensive health care systems
- Based on existing, validated indicators set, but modified and updated

**Notes:**

conditions for which good outpatient care can potentially prevent the need for hospitalization or for which early intervention can prevent complications or more severe disease. 
Potentially of interest to public health, researchers or comprehensive health care systems.
Examine the health care system as a whole, access to quality outpatient care. 
Most indicators, but not all, based on Billings et al, and Weissman et al. indicators sets, but have been modified and updated. 

## Prevention Quality Indicators (PQIs)

- Bacterial pneumonia
- Dehydration
- Pediatric gastroenteritis
- Urinary tract infection
- Perforated appendix
- Low birth weight
- Angina without procedure
- Congestive heart failure
- Hypertension
- Adult asthma
- Pediatric asthma
- Chronic obstructive pulmonary disease
- Diabetes short-term complication
- Diabetes long-term complication
- Uncontrolled diabetes
- Lower-extremity amputation among patients with diabetes

**Notes:**
16 measures included in PQI set, several measures for diabetes. Covers acute infection, asthma, and other chronic conditions. 

These measures are all defined as number of admissions per 10,000 (or 1000) population, except perforated appendix (per admissions for appendicitis in an area) and low birth weight (per births in an area). 

## Inpatient Quality Indicators (IQIs)

- Defined using both hospital admissions and area population as denominator
- Inpatient mortality for certain procedures and medical conditions
- Utilization of procedures for which there are questions of overuse, underuse, and misuse
- Volume of certain procedures
- Risk-adjusted using APR-DRGs
- Potential for internal quality improvement purposes
- Based on existing, validated indicators

**Notes:**
Most indicators are defined using hospital level admissions as the denominator (e.g. mortality after CABG, denominator is all CABG surgeries). Included in the IQIs are some area level utilization indicators for potentially overused procedures (e.g., hysterectomy rate). Appropriate denominators cannot be ascertained using administrative data, so population rates are best alternative.

All mortality measures are at hospital level, and reflect mortality for both surgical cases and medical cases (e.g., CHF, hip fx, or AMI). Mortality measures come from literature, previous HCUP QIs, and Clinical Classification System (CCS) developed by AHRQ.

Utilization measures defined at hospital level have appropriate denominators (e.g., bilateral cath, C-section, VBAC) and which hospital have more impact over rates. Most do not have “right rates” with the exception of bilateral catheterization and incidental appendectomy in the elderly, which are likely to be very rarely indicated. Other procedures may be over or under-utilized. 

Volume indicators reflect hospital level volumes for procedures for which volume has a strong association with better outcomes. Proxy measure, not direct, but in absence of other data, the Leap Frog group has recommended these types of measures. Volume definitions taken from the literature where the volume-outcome relationship has been established (Ethan Halm did a large literature review).

Risk adjusted using APR-DRGs from 3M. Difference between APR-DRGs and DRGs: Although both APR-DRGs and DRGs are based on a patient’s International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic and procedure codes, they are different patient classification schemes. The 3M Corporation developed and sells a widely used software product to define risk adjustment categories called APR-DRGs, which has the advantage of subclass groupings for Severity of Illness and Risk of Mortality. DRGs are designated by the Centers for Medicare and Medicaid Services (CMS) and software groupers are available from various vendors to assign DRGs. 

## Inpatient Quality Indicators (IQIs)

**Mortality Rates for Conditions**
- Acute myocardial infarction (2 versions)
- Congestive heart failure
- Gastrointestinal hemorrhage
- Hip fracture
- Pneumonia
- Stroke
**Mortality Rates for Procedures**
- Abdominal aortic aneurysm repair
- Coronary artery bypass graft
- Craniotomy
- Esophageal resection
- Hip replacement
- Pancreatic resection
- Pediatric heart surgery

**Hospital-level Procedure Utilization Rates**
- Cesarean section delivery (primary and total)
- Incidental appendectomy in the elderly
- Bi-lateral cardiac catheterization
- Vaginal birth after Cesarean section (2 versions)
- Laparoscopic cholecystectomy
**Area-level Utilization Rates**
- Coronary artery bypass graft
- Hysterectomy
- Laminectomy or spinal fusion
- PTCA
**Volume of Procedures**
- Abdominal aortic aneurysm repair
- Carotid endarterectomy
- Coronary artery bypass graft
- Esophageal resection
- Pancreatic resection
- Pediatric heart surgery
- PTCA

**Notes:**
See previous slide for information on classes and sources.

A few indicators are paired indicators (and you may want to say less strong). Volume indicators tend to have associated mortality indicators, although some of the associated mortality indicators are not widely used, nor will be applicable to many hospitals (e.g., the resections).

## Patient Safety Indicators (PSIs)

- Defined using hospital admissions as denominator
- Inpatient complications of care and potential patient safety events
- Potential for internal quality improvement purposes, monitoring of patient safety events
- Novel indicators, based on concepts reported in the literature

**Notes:**
Patient safety indicators are also hospital based, but cover complications of care (e.g. deep vein thrombosis after surgery)  and potential patient safety events (e.g., in-hospital hip fracture). 

## Patient Safety Indicators (PSIs)

**Provider-level Patient Safety Indicators**
- Accidental puncture or laceration during procedure
- Complications of anesthesia
- Death in low mortality DRGs
- Decubitus ulcer
- Failure to rescue
- Foreign body left in during procedure
- Iatrogenic pneumothorax
- Selected infection due to medical care
- Postoperative hemorrhage or hematoma
- Postoperative hip fracture
- Postoperative physiologic and metabolic derangements
- Obstetric trauma – vaginal delivery with instrument
- Obstetric trauma – vaginal delivery without instrument
- Obstetric trauma – cesarean section delivery

- Postoperative pulmonary embolism or deep vein thrombosis
- Postoperative respiratory failure
- Postoperative sepsis
- Transfusion reaction
- Postoperative wound dehiscence in abdominopelvic surgical patients
- Birth trauma – injury to neonate
**Area-level Patient Safety Indicators**
- Foreign body left in during procedure
- Iatrogenic pneumothorax
- Infection due to medical care
- Technical difficulty with medical care
- Transfusion reaction
- Postoperative wound dehiscence in abdominopelvic surgical patients

**Notes:**
These indicators were selected using a structured selection process. Covers post-surgical complications, obstetric complications and general complications. Heavy on the surgical. Not the way it was designed, but surgical are often easier to operationalize (less present on admission/case mix concerns, more homogeneous).

For PSIs the denominators are specific to the complication, not generic. Specific inclusion and exclusion criteria are used to refine denominator to those that are truly at risk for a potentially preventable complication (as opposed to not at risk for the complication at all, or likely to have a non-preventable complication).

Area indicators are analogs of the provider-level indicators, but include complications which appear in the principal diagnosis. This means that the preceding condition or procedure occurred in an outpatient setting or in another hospitalization. Unlike other area indicators, the denominator is not population, but it is the same denominator as the provider-level indicators taken at an area level. 

## PQI Rates

- Source: SID, 2000. AHRQ Prevention Quality Indicators SAS Software Version 2.1 Revision 3.

| PQI | Per 100,000 | Area SD |
| --- | --- | --- |
| DIABETES SHORT TRM COMPLICATION | 43.2 | 33.3 |
| PERFORATED APPENDIX | 311.6 | 157 |
| DIABETES LONG TERM COMPLICATION | 100.4 | 83.1 |
| PEDIATRIC ASTHMA | 166.4 | 191.2 |
| CHRONIC OBSTRUCTIVE PULMONARY DISEASE | 371.6 | 342.0 |
| PEDIATRIC GASTROENTERITIS | 128.8 | 189.6 |
| HYPERTENSION | 50.2 | 67.0 |
| CONGESTIVE HEART FAILURE | 526.6 | 488.3 |
| LOW BIRTH WEIGHT | 38.0 | 39.0 |
| DEHYDRATION | 158.9 | 143.6 |
| BACTERIAL PNEUMONIA | 506.8 | 360.0 |
| URINARY INFECTION | 169.0 | 142.1 |
| ANGINA | 108.8 | 128.2 |
| DIABETES UNCONTROLLED | 33.7 | 39.3 |
| ADULT ASTHMA | 102.3 | 95.6 |
| LOWER EXTREMITY AMPUTATION | 27.9 | 32.1 |

## IQI Rates

| IQI | Per 100 | Provider SD |
| --- | --- | --- |
| IN-HOSP MORT ESOPHAGEAL RESECTION   | 11.51 | 28.88 |
| IN-HOSP MORT PANCREATIC RESECTION   | 10.53 | 25.11 |
| IN-HOSP MORT PEDIATRIC HEART SURG   | 4.90 | 11.87 |
| IN-HOSP MORT AAA REPAIR       | 16.87 | 22.97 |
| IN-HOSP MORT CABG             | 3.91 | 4.35 |
| IN-HOSP MORT CRANIOTOMY       | 9.74 | 12.35 |
| IN-HOSP MORT HIP REPLACEMENT  | 0.38 | 2.32 |
| IN-HOSP MORT AMI              | 15.41 | 13.16 |
| IN-HOSP MORT CHF              | 5.03 | 4.39 |
| IN-HOSP MORT STROKE           | 11.19 | 8.35 |
| IN-HOSP MORT GI HEMORRHAGE    | 3.46 | 5.27 |
| IN-HOSP MORT HIP FRACTURE     | 3.45 | 6.52 |
| IN-HOSP MORT PNEUMONIA        | 8.11 | 4.86 |
| CESAREAN SECTION DELIVERY     | 20.33 | 8.59 |
| PRIMARY CESAREAN SECTION | 13.12 | 7.45 |
| VAGINAL BIRTH AFTER C-SECTION  | 26.54 | 15.28 |
| LAPAROSCOPIC CHOLECYSTECTOMY  | 73.25 | 18.65 |
| INCIDENTAL APPENDECTOMY       | 2.83 | 5.08 |
| BI-LATERAL CATHETERIZATION    | 11.19 | 13.96 |

- Source: SID, 2000. AHRQ Inpatient Quality Indicators SAS Software Version 2.1, Revision 3. Release pending.

**Notes:**
Ranges from 0.38% (hip replacement) to 16.87 (AAA repair) for mortality. 

## PSI Rates

- Source: NIS, 2000. AHRQ Patient Safety Indicators SAS Software Version 2.1 Revision 2. Release pending.

| PSI | Rate per 1000 |
| --- | --- |
| Complications Of Anesthesia   | 0.71 |
| Death In Low Mortality Drgs   | 0.60 |
| Decubitus Ulcer               | 21.96 |
| Failure To Rescue             | 143.82 |
| Foreign Body Left In During Proc  | 0.08 |
| Iatrogenic Pneumothorax       | 0.81 |
| Infection Due To Medical Care  | 2.00 |
| Postoperative Hip Fracture    | 0.31 |
| Postop Hemorrhage Or Hematoma  | 2.24 |
| Postop Physio Metabol Derangmnt  | 0.87 |
| Postop Respiratory Failure    | 3.48 |
| Postoperative Pe Or Dvt       | 7.29 |
| Postoperative Sepsis          | 10.75 |
| Postoperative Wound Dehiscence  | 1.89 |
| Accidental Puncture/Laceration | 3.25 |
| Transfusion Reaction          | 0.005 |
| Birth Trauma                  | 6.58 |
| Ob Trauma - Vaginal W Instrument  | 222.63 |
| Ob Trauma - Vaginal Wo Instrument  | 83.69 |
| Ob Trauma - C-Section         | 5.95 |

## Outline

- Administrative data and quality indicators
- AHRQ Quality Indicators (QI)
- Development of AHRQ QIs
- Risk adjustment & MSX smoothing methods
- Application of QIs to research and quality

## Methods

- Evaluation framework
- Literature review
- Identification of indicators
- Gray literature/interviews
- Identification of indicators
- Literature review
- Evidence for indicators
- Empirical analyses
- ICD-9-CM coding review (PSI only)
- Clinical panel reviews (PSI only)

**Notes:**
4 pronged approach, intended to provide information regarding the indicators. All were used to identify the indicators included in the final PSI set. 

Each will be described in details. 

Standard empirical analyses were conducted for descriptive purposes only, however, specified empirical analyses were done to help operationalize indicators (example – iatrogenic complications – try to figure out what complications it is capturing and whether they are meaningful). 

## Evaluation Framework

**Face validity**: does the indicator capture an aspect of quality that is widely regarded as important and subject to provider or public health system control?
**Precision:** is there a substantial amount of provider or community level variation that is not attributable to random variation?
**Minimum Bias:** is there either little effect on the indicator of variations in patient disease severity and comorbidities, or is it possible to apply risk adjustment and statistical methods to remove most or all bias?
**Construct validity:** does the indicator perform well in identifying true (or actual) quality of care problems?
**Fosters real quality improvement:** Is the indicator insulated from perverse incentives for providers to improve their reported performance by avoiding difficult or complex cases, or by other responses that do not improve quality of care?
**Application:**  Has the measure been used effectively in practice? Does it have potential for working well with other indicators?

## Literature ReviewIdentification of Indicators

- Systematic review to identify indicators
- Thousands of articles screened
- Over 200 abstracted
- Only 20 + articles actually described indicators, most of which had overlapping indicators
- Grey literature searched to identify over 200 indicators

## Empirical Analyses

- Used novel statistical methods to measure
- Precision/Reliability
- Bias
- Inter-relatedness of indicators
- Precision criteria of 1.0% or more systematic variation among providers
- Then, literature review conducted

## Literature ReviewEvidence for Each Indicator

- Identified and reported evidence for:
- Face validity
- Precision and reliability
- Potential bias
- Construct validity
- Fosters true quality improvement (gaming)
- Current use

## PSIs Methods Development of Candidate Indicator List

- Background literature review
- Little evidence in peer reviewed journals
- Complications Screening Program
- Miller et al. Patient Safety Indicators
- Review of ICD-9-CM code book
- Codes from above sources grouped into indicators and assigned denominators
- Review of CSP evidence to retain indicators
- Final refinements of indicators

**Notes:**
Complications Screening Program consists of 28 complications, developed by Lisa Iezzoni et al. for the purpose of identifying potentially preventable complications of adult medical and surgical hospital care, using administrative data. Uses ICD-9-CM diagnosis and procedure codes, patient age, sex, DRG, and date of procedure.
CSP was developed using clinical judgement of developers and review of ICD-9-CM codebook. 
CSP uses “risk pools” instead of standard denominators – “major surgery, minor surgery, medical patients, etc.”
Miller et al. PSIs – researchers reviewed ICD-9-CM code book, including CSP utilized codes. Selected indicators based on team judgement. Strict definition of patient safety event. 
75% positive predictive value (3/4 patients identified by indicators actually had complication)
46% PPV for “process failure” (average for non-flagged cases)
Final refinements of indicators.
Changes to the denominator definitions intended to reduce the bias due to the inclusion or exclusion criteria
 Elimination of selected ICD-9-CM codes from numerator definitions, intended to focus attention on more clinically significant complications, or complications more likely to result from medical errors.
Addition of selected ICD-9-CM codes to numerator definitions, intended to capture related complications that could result from the same or similar medical errors.
Division of a single indicator into two or more related indicators, intended to create more clinically meaningful and conceptually coherent indicators.
Stratification or adjustment by relevant patient characteristics (e.g. by procedure type for delivery, or high-low risk patients for decubitus ulcer).  

## PSIs Methods Review of Candidate Indicators

- Literature review of potential indicators
- Coding validity/consistency
- Construct validity
- ICD-9-CM coding review
- Clinical panel review (face validity)
- Results used to define final set of indicators

**Notes:**
Literature review on candidate indicators identified reported validity of similar indicators – specificity and sensitivity of codes in identifying complications, specificity and sensitivity of codes in identifying complications caused by medical errors. 
ICD-9-CM coding review by coding specialist at AHIMA. Reviewed indicators to ensure appropriate use of ICD-9-CM codes, suggested additions and deletions and responded to team inquiries. 
Clinical panel review assessed face validity of the indicators, more detailed methods next slide. 

## PSIs Methods Clinical Panel Review

- Intended to establish consensual validity
- Modified RAND/UCLA Appropriateness Method
- Doctors of various specialties/subspecialties, nurses, specialized (e.g., midwife, pharmacist)
- Initial rating, followed by conference call, followed by final rating
- Rated indicator on:
- Overall usefulness
- Present on admission
- Preventability of complication
- Likelihood due to medical error
- Extent indicator subject to bias
- Eight multispecialty panels, three surgical panels (5-9 members on each panel)

**Notes:**
Consensual validity – “extends face validity from one expert to a panel of experts who examine and rate the appropriateness of each item”

RAND/UCLA appropriateness method designed to assess the appropriateness of a procedure given various indications. We modified the method substantially, but the idea is the same – use a structured process and a panel of experts to assess the appropriateness of the indicators for use for screening tools. 

Each panel had 5-9 members of various specialties. All included and internist, surgeon, at least 1 nurse (surgical or critical care) and critical care doc (except OB panels). Surgical panels included only surgical subspecialists. For some indicators specific specialties were required, such as a pharmacist for dosage complications, or cardiologists for CABG after PTCA. 

Each panel looked at 4-6 indicators.

Surgical panels followed multispecialty panels. 

## Example reviewsMultispecialty Panels

- Overall rating
- Not present on admission
- Preventability (4)
- Due to medical error (2)
- Charting by physicians (6)
- Not biased (3)

**Notes:**
Ratings shown are for multispecialty panel and are on a 9 point scale, with 9 being more favorable (I flipped bias) 

Postoperative pneumonia – 

Interesting concept to panelists, but operationalization not doable. Preventability somewhat shakey given current definition, definitely not likely due to medical error, varied definition of “pneumonia” and case mix lead to bias. Overall wanted to use concept, but couldn’t do it well with data. 

In contrast...

Decubitus ulcer was very attractive. It was an error felt to be very preventable, and given definition was not likely to be present on admission. Did change a few aspects of the indicator to include more patients in the denominator (removed age limit).

## PSIs Methods Final Selection of Indicators

- Indicators for which “overall usefulness” rating was high
- Some changes in indicator set based on coding review and operationalization concerns (e.g., reopening of surgical site)
- Empirical analyses of nationwide rates, variation, impact of risk adjustment, and relationship between indicators

**Notes:**
Final indicator set consists of indicators rated as a 6.5/8 or higher for overall usefullness.

Changes were made due to coding review or other concerns that we believed would have changed the panels rating had they known the information at the time of rating – e.g. reopening of surgical site: 1) coding only includes certain reopenings, when not involving another procedure, including repairs 2) impossible to operationalize planned reopenings. 

Empirical analyses intended as descriptive analyses.

## Outline

- Administrative data and quality indicators
- AHRQ Quality Indicators (QI)
- Development of AHRQ QIs
- Risk adjustment & MSX smoothing methods
- Application of QIs to research and quality

## Risk-Adjustment Criteria

- User-specified criteria for evaluating risk-adjustment systems
-    1) “Open” systems preferred
-    2) Data collection costs minimized and well-justified
-    3) Multiple-use coding system
-    4) Official recognition

**Notes:**
Also conducted interviews with stakeholders and found they preferred the following:

## Evidence on DRG-based Systems

- Open systems
- Widely adopted by state agencies
- Based on existing data collection systems
- Use for reimbursement ensures improved data quality
- Evidence suggests at least equivalent performance across broad spectrum of conditions
- Studies underway to examine alternatives

## 3M APR-DRG

- All-patient refined (956 categories in version 15.0, including pediatrics)
- Severity of illness subclass that reflect presence of co morbidity/complication and level
- Risk of mortality subclass
- Differential impact of secondary diagnosis by condition

## Evidence on 3M APR-DRG

- All-patient refined (956 categories in version 15.0, including pediatrics)
- Severity of illness subclasses that reflect presence of co morbidity/complication and level
- Risk of mortality subclasses
- Differential impact of secondary diagnosis by condition

## Evidence on 3M APR-DRG

- Better empirical performance than DRG-based alternatives on predicting mortality (especially for surgical patients; patients at large, urban, teaching hospitals)
- Better empirical performance than DRG-based alternatives on predicting resource use (especially for medical patients; patients over 65, at children, teaching hospitals)
- Better at reflecting the distribution of patient severity at the extremes

## Risk-Adjustment Conclusions

- No single system based on administrative or clinical data is clearly superior
- DRG-based systems perform as well, and often better, than alternatives
- Data enhancements may improve performance (e.g., condition present on admission, key clinical variables)

## Risk-Adjustment ModelInpatient Quality Indicators

- Direct standardization
- Indirect standardization
- 	RA = (OR / ER) * PR
- (RA – risk adjusted; OR – observed; ER – expected; PR – population)

## Risk-Adjustment Model

- Expected rate – Assuming the hospital’s case-mix and the population rates
- Risk-adjusted rate – Assuming the population’s case-mix and the hospital’s rates

**Notes:**
WILL BE DOING A CRASH COURSE THIS MONTH TO GET MORE FAMILIAR WITH THE MODELS

## Risk-Adjustment Model

- Linear regression model:
-    observed rate = hospital effect +
-                              demographic effect +
-                              condition effect + error
- Model estimated on the SID, 2000 (25 million discharges)

## Risk-Adjustment Model

- IQI – Age, sex, APR-DRG (with risk of mortality or severity of illness subclass)
- (linear with hospital fixed effects)
- PQI – Age and sex
- (linear with area fixed effects)
- PSI – Age, sex, modified CMS DRG and AHRQ comorbidity (logistic)

## How it Works: CABG Mortality

| Covariate | Freq. | Effect | Covariate | Freq. | Effect |
| --- | --- | --- | --- | --- | --- |
| Intercept |  | 0.000 | 1652 | 0.240 | -0.001 |
| Sex | 0.279 | 0.005 | 1653 | 0.131 | 0.020 |
| Age1 | 0.204 | 0.001 | 1654 | 0.036 | 0.301 |
| Age2 | 0.179 | 0.009 | 1661 | 0.127 | 0.000 |
| Age3 | 0.195 | 0.013 | 1662 | 0.134 | -0.001 |
| Age4 | 0.245 | 0.031 | 1663 | 0.086 | 0.014 |
| Sex*Age1 | 0.049 | 0.010 | 1664 | 0.014 | 0.350 |
| Sex*Age2 | 0.049 | 0.011 | 1631 | 0.010 | 0.010 |
| Sex*Age3 | 0.060 | 0.018 | 1632 | 0.013 | 0.013 |
| Sex*Age4 | 0.085 | 0.034 | 1633 | 0.016 | 0.051 |
|  |  |  | 1634 | 0.004 | 0.478 |
|  |  |  | Other | 0.077 | 0.163 |

## How it Works: CABG Mortality

|  | SID
Effect | SID
Freq. | Hosp
Freq. | Diff | Effect*
Diff |
| --- | --- | --- | --- | --- | --- |
| Sex | 0.013 | 0.279 | 0.262 | -0.016 | 0.000 |
| 56 to 63 | 0.005 | 0.204 | 0.256 | 0.052 | 0.000 |
| 64 to 68 | 0.016 | 0.179 | 0.161 | -0.018 | 0.000 |
| 69 to 73 | 0.027 | 0.195 | 0.155 | -0.039 | -0.001 |
| 74 to high | 0.054 | 0.245 | 0.165 | -0.079 | -0.004 |
| Demographic Effect |  |  |  |  | -0.005 |
| Observed Rate |  |  |  |  | 0.047 |
| Risk-adjusted Rate |  |  |  |  | 0.052 |

## MSX Smoothing Model

- Observed quality measure = true quality (signal) + error (noise)
- Smaller hospitals and/or less frequent conditions have more noise
- Difficult to compare hospitals, trend over time, and identify best practices
- Confidence intervals reflect but do not address the problem

## Key Features of MSX Approach

- Removes noise – uses redundancy over time and among measures
- Improves forecasts – predicting current quality based on past performance
- Reduces dimensionality appropriately - allows meaningful summary measures
- Reveals and helps reduce biases, identify best practices

## Outline

- Administrative data and quality indicators
- AHRQ Quality Indicators (QI)
- Development of AHRQ QIs
- Risk adjustment & MSX smoothing methods
- Application of QIs to research and quality

## Caveats of Use

- Validity of data
- Validity of coding
- Present on admission
- Outpatient care
- Linking of admissions and impact of LOS
- Incomplete risk adjustment

## Using the AHRQ QI

- State monitoring of rates
- Hospital quality improvement
- National Healthcare Quality Report
- PQIs and PSIs
- CMS Pay for Performance Demonstration Project
- Postoperative Hemorrhage or Hematoma
- Postoperative Metabolic and Physiologic Derangement
- Romano et al
- PSI National trends, (HA, Mar/Apr ’03)

## Using the AHRQ QI

- Kovner
- QI and nurse staffing
- Miller
- PSI and Costs and LOS
- Alexander/Shortell
- PSI and Quality improvement culture
- Baker
- PSI and Patient safety culture; hospital characteristics
- Rosen
- VA hospitals, QI and other measures (NSQuIP)
- Volpp
- QIs and new resident work hours

## Technical Reports

-  Development of Quality Indicators, risk adjustment and MSX methods documented in:
- Davies S, Geppert J, McClellan M, McDonald KM, Romano PS, Shojania KG. Refinement of the HCUP Quality Indicators. Technical Review Number 4. Rockville, MD: (Prepared by the UCSF-Stanford Evidence-based Practice Center under Contract No. 290-97-0013) Agency for Healthcare Research and Quality; 2001. Report No.: 01-0035
- McDonald KM, Romano PS, Geppert J, Davies S, Shojania KG. Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators. Technical Review Number 5. Rockville, MD: (Prepared by the UCSF-Stanford Evidence-based Practice Center under Contract No. 290-97-0013) Agency for Healthcare Research and Quality; August 2002. Report No.: 01-0038.

## For More Information on AHRQ QIs

- Quality Indicators: Additional information and assistance
- E-mail: _[support@qualityindicators.ahrq.gov](mailto:support@qualityindicators.ahrq.gov)_
- Website: _[http://qualityindicators.ahrq.gov/](http://qualityindicators.ahrq.gov/)_
- QI technical reports, documentation and software is available on the website
- User Support is provided under contract by Battelle Memorial Institute, Stanford University and University of California at Davis