Variation in Quality by Hospital Characteristics: 

True or False?

November 5, 2013

Presentation to the Federal Committee on Statistical Methodology 

Research Conference

Washington, DC

Eric Schone
Acknowledgments

 Dejene Ayele, Swaati Bangalore, Alex Bohl, Alicia 

Haelen, Hali Hambridge, Xiaojing Lin, Dmitriy

Poznyak, Jessica Ross, Eric Schone, Sam Stalley, 

Sheng Wang, Haixia Xu, Frank Yoon, Joe Zickafoose 

(Mathematica)

 Ann Borzecki, Qi Chen, Amy Rosen (BU)

 Sydney Dy, Lilly Engineer (JHU) 

 Pam Owens (AHRQ)

 Funding: HHSA290201200004I Task Order No. 2
Quality Indicators Overview

 The AHRQ QI are a set of more than 90 quantitative 

indicators of health care quality 

 The area level indicators characterize quality of care 

for ambulatory care sensitive conditions by portion of 

the country, e.g., by county

 The provider level indicators characterize quality of 

hospital care

 The indicators are calculated using hospital inpatient 

administrative data

 See www.qualityindicators.ahrq.gov 
Four Modules

 Prevention Quality Indicators (PQI)

– Potentially avoidable hospital admissions – by area

 Inpatient Quality Indicators (IQI)

– Reflect care inside hospitals and by area

– Mortality for medical conditions and surgical procedures

 Patient Safety Indicators (PSI)

– Reflect care inside the hospital

– Potentially avoidable complications and iatrogenic events

 Pediatric Quality Indicators (PDI) 

– Reflect care inside the hospital and by area

– Specific to children and neonates
Software Updates and QI Types

 Software updated frequently

– Fiscal year coding updates

– Changes to reference population

– Changes to numerator and denominator

– Changes to statistical methods

– Currently Version 4.5

 Three types of indicators

– Count and Volume Indicators

– Area-Level Rate Indicators

– Provider- (Hospital-) Level Rate Indicators
Focus on Provider-Level Rate Indicators

 Numerator is count of records that match QI 

specification 

– E.g., hip replacement mortality rate; pressure 

ulcer rate

 Denominator is count of persons at risk in the 

hospital

– E.g., persons who had hip replacement surgery
Provider-Level Rate Indicators - Methods 

 Risk adjusted using several types of covariates 

(customized model for each QI)

 Outcome or risk factor might be present on 

admission (POA); POA sometimes missing and 

imputed

 Smooth rates using shrinkage estimator 

– Weighted average of risk-adjusted rate from hospital and 

nationwide reference population rate

– Weights are calculated using signal variance and noise 

variance
Hospital Characteristics and Hospital Indicators

 Indicators of hospital quality are used in a number 

of high-profile, high-stakes programs to compare 

hospital quality on a national scale across a variety 

of hospital types.

 The use of the indicators in comparative reporting 

has been critiqued in the popular press and 

academic literature.

– Use of administrative data

– Quality indicator specifications

– Methods used to calculate the rates
Example of the Critique – Teaching Status

 Rates for many indicators differ on average between 

hospitals that provide medical education and those that do 

not. 

 Critics argue that teaching hospital rates for some measures 

differ for reasons not related to quality. 

 What can explain differences in rates by teaching status? 

– Differences in coding, data sources, and so on

– Unaccounted for differences in patient risk

– Relationship with volume

– Quality 
Exploratory Data Analysis: Data

 Research focused on IQI, PSI, and PDI individual and composite 

measures

 Data source – State Inpatient Databases (SID), Healthcare Cost 

and Utilization Project (HCUP), Agency for Healthcare Research 

and Quality

 12 states*, ~ 1,500 hospitals

 All inpatient hospital discharges for 2009 and 2010

 Hospital characteristics – American Hospital Association (AHA) 

2010

 AHRQ QI software v4.4

* We would like to thank the HCUP Partners from the following states: AR, AZ, CA, FL, 

IA, KY, MA, MD, NE, NJ, NY, WA (http://www.hcup-us.ahrq.gov/partners.jsp) 
 Structural characteristics

– Bed size

– Teaching status

 Aggregate patient characteristics

– DSH status

– Race

 Market characteristics

– Urban 

– Median income

Exploratory Data Analysis: Hospital Characteristics

11
 Analysis of differences by hospital 

characteristics

– Comparison of means and overall distribution

– In raw, risk-adjusted, and smoothed rates

– In coding frequencies

– In risk

 Differences in relation to volume

– Reliability weights

– Classification and Regression Tree (CART)

– Multivariate regression

Exploratory Data Analysis: Methods

12
 PSI 12: Post-operative Pulmonary 

Embolism/Deep Vein Thrombosis

– Risk-adjusted and smoothed rates of teaching 

hospitals significantly worse than those of nonteaching hospitals, raw rates no different

 IQI 20: Pneumonia mortality

– Risk-adjusted and smoothed rates of teaching 

hospitals significantly better than those of nonteaching hospitals, raw rates no different

Differences by Teaching Status: Examples

13
Do Teaching Hospitals List More Codes on Administrative Data 

Sources? 

Source: HCUP, SID 2009–2010, AHA 2010.
Do Teaching Hospitals Have Riskier Patients?

Source: HCUP, SID 2009–2010, AHA 2010.
What Is the Relationship Between Volume and QI Rates?

Relationship with Volume: Reliability Weights

Source: HCUP, SID 2009–2010, AHA 2010.
What Is the Relationship Between Volume and QI Rates?

 Degree of smoothing, by hospital size

Source: HCUP, SID 2009–2010, AHA 2010.
What Is the Relationship Between Volume and QI Rates?

 Rates grouped by bed size in CART analysis

Source: HCUP, SID 2009–2010, AHA 2010.
What Is the Relationship Between Volume and QI Rates?

 Rates by teaching status, stratified by volume

Source: HCUP, SID 2009–2010, AHA 2010.
What Is the Relationship Between Volume and QI Rates?

 Multivariate model with bed size and teaching status

Hospital 

Characteristic

IQI 20 – 

Coefficients 

(standard errors)

PSI 12 – 

Coefficients

(standard errors)

Bed Size (< 55 omitted)

 55 to 130 - 0.11

(0.08)

- 0.39

(0.15) **

 131 to 271 - 0.35

(0.09) ***

 0.38

(0.16) **

 272 or more - 0.21

(0.08) ***

- 0.43

(0.15) ***

Teaching Hospital - 0.06

(0.08)

0.51

(0.14) ***

 * Significantly different from zero at the 0.10 level, two-tailed test.

 ** Significantly different from zero at the 0.05 level, two-tailed test.

*** Significantly different from zero at the 0.01 level, two-tailed test.

Source: HCUP, SID 2009–2010, AHA 2010.
Quality

 It is difficult to distinguish other factors that 

influence variation by teaching status versus 

variation due to factors we seek to estimate – 

quality.

 Ongoing analyses

– Matching analysis by discharge characteristics

– Simulations of patient populations with known 

characteristics

– IV analysis using travel time and distance to hospitals