Hospital Profiling With Enhanced Risk 

Adjustment Based On Laboratory Tests 

And Vital Signs Data

Ann Borzecki, MD MPH

Amresh D. Hanchate, PhD

CHOIR

VA Bedford & Boston
Acknowledgements 

• Funding: VA HSR&D - IIR 08-351 (PI: Hanchate)

• Co-investigators:

– Arlene Ash, PhD (University of Massachusetts)

– Aaron Fink, MD (VA Atlanta)

– Mary Jo Pugh, PhD (VA San Antonio)

– Amy Rosen, PhD (CHOIR, VA Boston)

– Michael Shwartz, PhD (CHOIR, VA Boston)

– Hassen Abdulkerim, MS (CHOIR, VA Boston)

– Kelly Stolzmann, MS (CHOIR, VA Boston)

2 
Aims

• Examine impact of adding laboratory tests + 

vital signs to administrative data for hospital 

profiling

• Evaluate whether this leads to improved 

measures of hospital quality

3 
Background

• Adding clinical data to administrative data 

– Pine et al. (JAMA 2007): 188 Pennsylvania hospitals

– Tabak et al. (Med Care 2007): 266 hospitals Cardinal Health 

database 

– Escobar et al. (Med Care 2008): 17 Kaiser hospitals

– Render et al. (2003; 2010): VA hospitals - ICU & AMI, Heart 

Failure & Pneumonia

• Findings 

– Significant increase in risk prediction model (e.g., sizable 

increase in c-statistic)

– Changes in hospital profiles

• Better measure of hospital quality?

4 
Data & Study Population

• Data

– VA National Patient Care Data, FY2006-2010: Inpatient 

Patient Treatment File (PTF) and Outpatient Encounter File

– VA Decision Support System (DSS) National Data Extracts 

(NDE), FY2007-2010: Laboratory Results (LAR)

– Corporate Data Warehouse Vital Signs, FY2007-2010

– VA Vital Status File, FY2011

• Study Population

– CMS/VA Hospital Compare protocol for identifying 

discharges for AMI, HF and pneumonia

– Exceptions: 

a) Age 18 and older

b) Hospitals with at least 50 discharges

5 
Hospital Counts & Outcome

Acute 

Myocardial 

Infarction

Heart Failure Pneumonia

No. of hospitals 91 128 131

No. of discharges 22,608 59,595 62,996

Median no. of 

discharges per 

hospital [range]

193 

[53, 1202]

435 

[55, 1758]

428 

[50, 1586]

Median 30-day 

mortality rate per

hospital [range]

9.9% 

[3%, 26%]

8.0% 

[3%, 15%]

9.7% 

[4%, 19%]

6 
Laboratory Tests & Vital Signs

Laboratory Tests

• Hematology

– Hemoglobin 

– Hematocrit 

– White Blood Cell Count

– Prothrombin Time 

– Partial Thromboplastin Time

Vital Signs 

• Temperature

• Pulse

• Systolic Blood Pressure 

• Diastolic Blood Pressure

• Respiratory Rate

• Pulse Oximetry

• Chemistry 

– Sodium

– Potassium

– Bicarbonate (HCO3/CO2)

– Glucose

– Creatinine 

– Bun

– Albumin

– AST

– Bilirubin 

– Alkaline Phosphatase

– Troponin

7 
Risk Measures from Laboratory Tests & 

Vital Signs

• Identified all laboratory tests and vital signs within 24 

hours of admission time

• In case of multiple tests, most extreme result selected 

• Based on clinical judgment and bivariate association with 

30-day mortality, categorized each test result into 5 

categories: 1) Normal, 2) Low abnormal, 3) Moderate 

abnormal, 4) High abnormal, 5) Missing 

• In final models, excluded measures for which none of 

categories significantly associated with 30-day mortality

8 
Example of Lab Test Measure 

Test Measure * AMI Cohort 

Creatinine (mg/dl) Definition Hosp. Prevalence

Median % (range)

Normal (OR not diff. from 1) (0.81-0.90) 4.7 (0-11.7) 

Abnormal Low (OR >1, ≤2) (0.20-0.80) & 

(0.91-1.20) 

39.2 (8.3-57.1) 

Abnormal Moderate (OR>2, ≤3) (1.21-1.30) 7.1 (0-12.6) 

Abnormal High (OR>3) (1.31-35) 38.2 (6.9-65.3) 

Missing 3.5 (0-81.9) 

9 

• Categories derived by logic regression of 30-day mortality on 

decile indicators and defined based on odds ratios (OR) as shown 
Missingness Rates

• Post-2007, missingness stabilized for most 

laboratory tests and vital signs

• Missingness rates varied across

– Conditions

– Hospitals

10
Final Models

• For each admission cohort

– Administrative data model = CMS/VA Hospital 

Compare model

– Enhanced model = Administrative + laboratory 

test + vital signs

• Hierarchical logistic model with unobserved 

hospital effects

• Risk adjusted hospital mortality rates (RSMRs)

11
Model Performance

Administrative

Data Model

Enhanced Data 

Model

AMI 0.79 0.85

Heart Failure 0.73 0.81

Pneumonia 0.76 0.82

12

* C-statistics are shown
Comparison to Prior VA Study

Current Study Prior Study 

(Render, 2010)

Admin.

Data 

Model

Enhanced 

Data 

Model

Admin.

Data 

Model

Enhanced 

Data 

Model

AMI 0.79 0.85 0.77 0.82

Heart Failure 0.73 0.81 0.71 0.79

Pneumonia 0.76 0.82 0.74 0.80

13
Impact on Hospital Profiles

• Change in hospital risk standardized mortality 

rates (RSMRs)

• Does adding clinical data result in improved 

measures of hospital quality?

14
Absolute Change in Hospital Risk Standardized 

Mortality Rates (RSMR)

% Absolute

Change in 

RSMR after 

Adding Clinical 

Data

Admission Cohort

AMI 

(N=91)

HF

(N=128)

Pneumonia 

(N=131)

<10% 69 (75%) 94 (73%) 96 (73%)

10% to 20% 20 (22%) 29 (23%) 27 (21%)

>20% 2 (2%) 5 (4%) 8 (6%)

15
Relative Change in RSMRs: Heart Failure Cohort

(N=128)

RSMR 

without 

Clinical 

Data

RSMR with Clinical Data

Q1 

(low 

mortality)

Q2 Q3

Q4 

(high

mortality)

Q1 24 8 0 0 

Q2 7 17 16 0 

Q3 1 5 16 10

Q4 0 2 8 22

16

Discordant Proportion (%)

AMI = 48%; HF = 42%; Pneumonia = 42%
Change in Hospital Compare Designation: 

Pneumonia Cohort (N=131)

Without Clinical 

Data

With Clinical Data

Better than VA 

national rate

No different than 

VA national rate

Worse than VA 

national rate

Better than VA 

national rate 5 3 0 

No different than 

VA national rate 1 109 0 

Worse than VA 

national rate 0 10 3 

17

# hospitals with change in designation: 

AMI=5/91 HF=16/128 Pneumonia=14/131
Change in Hospital Compare Designation: 

Pneumonia Cohort

0

5 10 15 20 25

P/E RSMR (%) with 95% Confidence Interval

Hospitals

VA Data VA + (Lab + Vitals) Data National VA 30-day Mortality Rate

18
Does Adding Clinical Data Lead to Better 

Measures of Hospital Quality?

• Concerns about measuring hospital quality 

using administrative data

• Poor correlation 

– Mortality and Process measures

– Mortality and Readmission measures

• Does adding clinical data improve correlation?

19
20
RSMR and ORYX Composite Score: Quartiles

(Matched admission cohort)

Concordance of quartiles

Kappa statistic

AMI 

(N=91)

HF

(N=128)

Pneumonia 

(N=131)

RSMR 

Base Model -0.01 -0.08 0.03

RSMR

Enhanced Model -0.01 -0.03 0.05

21

Source of ORYX Composite Scores: 2012 VHA Facility Quality and Safety Report, Office 

of Quality, Safety and Value, Department of Veterans Affairs , 2012

* Denotes p<0.05
RSMR and Readmission Rates: Quartiles

(Matched admission cohort)

Concordance of quartiles

Kappa statistic

AMI 

(N=91)

HF

(N=128)

Pneumonia 

(N=131)

RSMR 

Base Model 0.03 0.08 0.06

RSMR

Enhanced Model 0.03 0.08 -0.01

22

Source of Readmission Rates: 2012 VHA Facility Quality and Safety Report, Office of 

Quality, Safety and Value, Department of Veterans Affairs , 2012

* Denotes p<0.05
RSMRs in 2007-08 and 2009-10 : Quartiles

Concordance of quartiles

Kappa statistic

AMI 

(N=91)

HF

(N=128)

Pneumonia 

(N=131)

RSMR 

Base Model 0.09 0.06 0.19

RSMR

Enhanced Model 0.03 0.16 0.18

23

* Denotes p<0.05
Correlation between RSMR and Other 

Quality Measures

• RSMRs from clinical data did not result in a 

significant change in concordance with

– RSMRs from other admission cohorts

– VASQIP Surgical Mortality O/E Rates

– VASQIP Surgical Morbidity O/E Rates

– Case volume

24
Change in RSMR after Adding Clinical Data: 

Regression to the Mean

p= <0.001; R-sq=0.51 p=0.41; R-sq=0.005 p= <0.001; R-sq=0.26

-.2

0 .2 .4

5 10 15 5 10 15 5 10 15

Acute Myocardial Infarction Heart Failure Pneumonia

95% CI Fitted values

RSMR change (%)

% RSMR change after model enhancement

RSMR from base model

25
Summary of Findings

• High rates of completeness of data on 

laboratory tests and vital signs, although 

missingness rates vary by facility

• Adding risk measures from laboratory tests

– Improves performance of risk adjustment models

– Changes relative hospital profiles

• Correlation of enhanced mortality 

performance measures with other hospital 

quality indicators remains poor

26
Discussion / Limitations

• No gold standard of hospital quality for 

comparison

• Quality is multifaceted

– Limitations of comparison measures

• Small # hospitals

• However,

– Absence of concordance with range of quality 

indicators remains an issue

– Regression to the mean

27
Implications 

• Need further research on ability of enhanced 

risk-adjusted mortality to distinguish hospital 

quality

28
Conclusions

• Use of data on laboratory tests and vital signs, 

in addition to administrative data, enables to 

better account for differences in patient status 

in measuring risk-adjusted 30-day mortality

• Concerns about lack of concordance of 

adjusted mortality with other hospital quality 

measures remain

29
Thank you

• Ann Borzecki (ann.borzeck@va.gov)

• Amresh Hanchate (amresh.hanchate@va.gov)

30
Missingness Rates

Year Sodium HCO3 Serum 

Creatinine Bilirubin

2005 13% 100% 17% 32%

2006 13% 41% 20% 31%

2007 15% 32% 21% 37%

2008 13% 19% 18% 34%

2009 9% 11% 17% 35%

2010 8% 10% 18% 31%

31
Variation in Missingness

• Considerable missingness across hospitals for 

all tests

• Different hospitals had high missingness in 

different tests

• High missingness was weakly correlated with 

hospital volume

32