Hospital Peer Groups, Reliability, and 

Stabilization: Shrinking to the Right Mean

November 5, 2013

Presentation to the Federal Committee on Statistical Methodology

Washington, DC

Alex Bohl
Agenda

2 

 Including Peer Groups in Hospital Comparisons

– Rationale

– Technical Approaches

 Empirical Example

 Challenges and Next Steps
Stabilizing the Quality Indicators

3 

– Small sample sizes

 Hospital Risk-Adjusted Rates (RARs) are often unstable

– Rare events

 Smoothing stabilizes RARs by using information from the entire sample of 

hospitals

0

5 10 15

Rate per 1,000 Discharges

Group A Group B
 Including hospital characteristics to create peer 

groups is controversial

– Influences hospital ranking (Austin et al. 2004)

– Changes the interpretation (Romano 2004)

 Volume is the most common characteristic considered

– Strong empirical volume-outcome relationship for mortality 

(Silber et al. 2010)

 The ultimate choice of peer group

– Needs conceptual and empirical backing

– Depends on the outcome of interest

– Should be precise

What’s the Correct Smoothing Target?

4 
Technical Approaches to Peer Grouping

5 

Hospital characteristics can enter risk- or reliabilityadjustment models (or both)

 Risk-Adjustment Model

– Peer group fixed effects, and/or

 Reliability-Adjustment Model

– One-part or unified: Smooth to peer group rates

• Peer group random effects

• With or without risk adjustment for hospital-level factors

– Two-part shrinkage model: Standardize to peer group rate

• Estimate reliability as signal-to-noise ratio

• Smooth to the peer group target
Illustrative Example

6 

 Aim: Incorporate peer group targets into the AHRQ QI model

 Peer grouping: Teaching vs. Non-Teaching Affiliation

 Measure: PSI 12 (Postoperative Pulmonary Embolism or Deep 

Vein Thrombosis Rate)

 Approach: 

– Base case: Current QI methodology

– Alternative: Two-part approach smoothing to teaching peer 

group target rates

 Evaluation criteria:

– Change in reliability (signal variance/total variance)

– Correlation of hospital ranking across approaches

– Proportion of hospitals moving above/below national average
Methods

7 

 Calculate reliability weights and shrinkage targets for 

two scenarios:

– Base Case (“Overall”)

– Alternative (“Peer Group”)

 Reliability weights vary for each approach

– Recalculate signal and noise

 Changes in smoothed rate estimates is therefore a 

function of

– The new shrinkage target

– The change in reliability weight
Descriptive Statistics: PSI 12 (DVT/PE)

8 

Overall Non-Teaching Teaching

Hospitals (n) 1,264 944 320

Denominator (mean) 4,605 3,056 9,177

Observed Rate 5.81 4.80 6.81

Expected Rate 5.81 5.52 6.11

Risk-Adjusted Rate 5.81 5.06 6.48

 Rates have units per 1,000 discharges

 Random sample of hospitals from 12 states with Healthcare Cost 

and Utilization Project (HCUP) State Inpatient Databases (SIDs), 

2009 and 2010*

* We would like to thank the HCUP Partners from the following states: AR, AZ, CA, FL, IA, KY, MA, 

MD, NE, NJ, NY, WA (http://www.hcup-us.ahrq.gov/partners.jsp) 
Smoothed Rate Distribution

9 

0 .1 .2 .3 .4 .5

0 5 10 15 20

Smoothed Rate (per 1,000 Discharges)

Overall, Non-Teaching Overall, Teaching

Peer, Non-Teaching Peer, Teaching

PSI-12 Smoothed Rate Distribution
Reliability Estimates – Non-Teaching Hospitals

10

0 .2 .4 .6 .8 1

Reliability Weight

0 5,000 10,000 15,000 20,000

Number of Qualifying Discharges

Overall Peer Group

Non-Teaching Hospital Reliability
Reliability Estimates – Teaching Hospitals

11

0 .2 .4 .6 .8 1

Reliability Weight

0 20,000 40,000 60,000

Number of Qualifying Discharges

Overall Peer Group

Teaching Hospital Reliability
Smoothed Rates

12

0

5 10 15 20

Peer-Group Smoothed Rate

0 5 10 15 20

Overall Smoothed Rate (per 1,000 Discharges)

Non-Teaching Teaching

Reference Line

 Teaching hospitals: 18% move above national average

 Non-teaching hospitals: 15% move below national average
Summary

13

 Using peer group targets changes ranking of smoothed 

rates

– Teaching: 18% move above national average

– Non-teaching: 15% move below national average

– Rank sum correlation of 0.91

 Peer grouping changes the variability in PSI 12 

distribution through reliability weights

– Teaching: Increased variability

– Non-teaching: Decreased variability
Challenges and Limitations

14

Practical, Conceptual, and Technical Questions Remain

 What happens for hospitals on the boundary?

– For example: volume, disproportionate share percentages, 

or nurse staffing ratios

 What about more precise subgroups?

– Major versus minor teaching status

– Subdividing non-teaching hospitals further

 What happens for small peer groups (e.g., two 

hospitals)?

 How do we handle hospitals missing peer group 

information?
Contact Information

15

 Alex Bohl

 Mathematica Policy Research

 abohl@mathematica-mpr.com

 (617) 301-8996
References

16

 Austin et al. Impact of the choice of benchmark on the conclusions of 

hospital report cards. American Heart Journal, 148(6); 2004.

 Romano, P.S. Peer group benchmarks are not appropriate for health care 

quality report cards. American Heart Journal, 148(6); 2004. 

 Silber et al. The Hospital Compare Mortality Model and the Volume-Outcome 

Relationship. Health Services Research, 45(5); 2010.
Appendix: Estimating Noise

17
Appendix: Estimating Signal 

18
Appendix: Estimating Reliability

19

We have assumed a simple linear regression which has a known solution 

found using the least-squares estimate or the maximum likelihood estimate: 

(MLE) 
Future Considerations

20

 Multilevel random effects

– Cross-classification of groups

 Incorporating peer groups (or different peer groups) into 

the risk-adjustment model

 Exploring the impact of historical priors, or priors 

defined outside the analytic population

 Application to patient safety indicators

– Lower event rates

– No consistent relationship with characteristics
Conclusions

21

 Whether to shrink to peer group means depends on 

– Empirical evidence

– Conceptual background

– Precise peer group classification

– Desired interpretation or use