Risk prediction models for hospital 

readmission: a systematic review

Devan Kansagara MD, MCR

David Kagen MD

Honora Englander MD

Amanda Salanitro MD

VA HSR&D Cyber Seminar

12/16/2011
POLL QUESTION 1

• What is your interest in this topic?

– A. I am involved in implementing a transitional 

care intervention

– B. I am involved in using readmission rates as a 

quality metric

– C. I am a researcher interested in studying 

readmission risk prediction

– D. I am just curious 
Outline

• ESP program overview

• Why the interest in readmission prediction

– Risk-standardized readmission rates and quality 

reporting 

– Clinical application

• Summary of systematic review methods and 

findings

– Overview of 3 specific models

• Reasons for poor performance

• Lessons learned
Evidence-based Synthesis Program 

(ESP) 

Disclosure

This report is based on research conducted by the Evidence-based Synthesis 

Program (ESP) Center located at the Portland VA Medical Center, Portland, OR 

funded by the Department of Veterans Affairs, Veterans Health Administration, 

Office of Research and Development, Health Services Research and Development. 

The findings and conclusions in this document are those of the author(s) who are 

responsible for its contents; the findings and conclusions do not necessarily 

represent the views of the Department of Veterans Affairs or the United States 

government. Therefore, no statement in this article should be construed as an 

official position of the Department of Veterans Affairs. No investigators have any 

affiliations or financial involvement (e.g., employment, consultancies, honoraria, 

stock ownership or options, expert testimony, grants or patents received or 

pending, or royalties) that conflict with material presented in the report.
Evidence-based Synthesis Program 

(ESP) 

VA Evidence-based Synthesis (ESP)

Program Overview

• Sponsored by VA Office of R&D and HSR&D.

• Established to provide timely and accurate syntheses/reviews of healthcare 

topics identified by VA clinicians, managers and policy-makers, as they 

work to improve the health and healthcare of Veterans. 

• Builds on staff and expertise already in place at the Evidence-based 

Practice Centers (EPC) designated by AHRQ. Four of these EPCs are also 

ESP Centers: 

o Durham VA Medical Center; VA Greater Los Angeles Health Care 

System; Portland VA Medical Center; and Minneapolis VA Medical 

Center.
Evidence-based Synthesis Program 

(ESP) 

• Provides evidence syntheses on important clinical practice topics relevant 

to Veterans, and these reports help:

o develop clinical policies informed by evidence, 

o the implementation of effective services to improve patient 

outcomes and to support VA clinical practice guidelines and 

performance measures, and 

o guide the direction for future research to address gaps in clinical 

knowledge.

• Broad topic nomination process – e.g. VACO, VISNs, field – facilitated by 

ESP Coordinating Center (Portland) through online process: 

 http://www.hsrd.research.va.gov/publications/esp/TopicNomination.cfm
Outline

• ESP program overview

• Why the interest in readmission prediction

– Risk-standardized readmission rates and quality 

reporting 

– Clinical application

• Summary of systematic review methods and 

findings

– Overview of 3 specific models

• Reasons for poor performance

• Lessons learned
Medicare rehospitalization within 30 Days after Hospital Discharge: $17.4 Billion

Jencks, NEJM 2009

Readmissions are common and costly
30 Day readmission rates in VA and 

non-VA hospitals are similar

CHF AMI Pneumonia

VA hospitals 25.2% 20.6% 19.2% 

Non-VA hospitals 24.8% 19.9% 18.4% 

Source: Kaiser Health News, Sep 2011, based on data from 

www.hospitalcompare.hhs.gov
“Hospital readmissions are frequent and costly 

events which...can be reduced by systemic 

changes to the health care system, including 

improved transition planning, quick follow-up 

care, and persistent treatment of chronic 

illnesses.”

www.commonwealthfund.org, accessed 12/6/11
Reasons for interest

• Risk-standardized readmission rates have 

become a quality metric

• Public reporting

• Financial penalties

• Identify high-risk patients for intervention
POLL QUESTION 2

• Do you think hospitals should be publicly 

compared on the basis of 30 day readmission 

rates?

– A. Yes

– B. No

– C. Don’t know/not sure
Rationale for Risk-standardization

Hospital A

Mid-sized in affluent suburb

Hospital B

Urban tertiary care center

Patients

- Few 

comorbidities

- Younger

- Insured

System

-Good access to 

outpatient care

- Track record of 

care coordination

Patients

-Multiple 

comorbidities

- Complex 

illness

- Uninsured

System

-Limited access 

to outpatient 

care

- Limited peridischarge 

services

Is it fair to compare hospitals A and B?
Adjust for patient case-mix
Adjust for patient case-mix

Hospital A

Mid-sized in affluent suburb

Hospital B

Urban tertiary care center

Patients

- Few 

comorbidities

- Younger

- Insured

System

-Good access to 

outpatient care

- Track record of 

care coordination

Patients

-Multiple 

comorbidities

- Complex 

illness

- Uninsured

System

-Limited access 

to outpatient 

care

- Limited peridischarge 

services

Targets for change
Calculation of Risk-Standardized 

Readmission Rates (RSRR)

• Conceptually: compares a hospital’s 

performance, given its case mix, with the 

average hospital’s performance, given the 

same case mix

Number of 30-day readmissions predicted

based on the hospital’s performance with its 

observed case mix

Number of 30-day readmissions expected

based on the nation’s performance with that 

hospital’s case mix

X U.S. national 

readmission rate

www.hospitalcompare.hhs.gov

Readmission: Who Is Counted?

• Patients age 65 or older

• Enrolled in traditional fee-for-service Medicare

• Enrolled for at least 1 year

• Discharged alive from acute care hospital

• Did not leave against medical advice (AMA)

• Principal diagnosis:

– Acute myocardial infarction (AMI)

– Congestive heart failure (CHF)

– Pneumonia

www.hospitalcompare.hhs.gov
Penalty for High Readmission Rates

• CMS penalty begins in FY 2013

– Payment cuts if risk-standardized readmission 

rates for AMI, CHF, or pneumonia are in the worst 

quartile

• Max penalty: 1% of total reimbursement

– Increases to 2% in FY 2014, 3% in FY 2015

• Additional diagnoses to be added (likely):

– Chronic obstructive pulmonary disease, coronary 

artery bypass surgery, peripheral vascular disease
POLL QUESTION 3

• Do you think hospitals should be financially 

penalized for high readmission rates?

– A. Yes

– B. No

– C. Don’t know/not sure
Reasons for interest

• Readmissions as a quality metric

– Hospital comparison based on risk-standardized 

rates

• Public reporting

• Financial penalties

• Identify high-risk patients for intervention
Transitional care

“a set of actions designed to ensure the 

coordination and continuity of health care as 

patients transfer between different locations 

or different levels of care” 

Coleman EA, Ann Int Med, 2004

POLL QUESTION 4

• Is there a transitional care program at your 

facility?

– A. Yes

• IF YES, how are patients identified for the intervention?

– 1) based on disease (CHF, COPD)

– 2) clinician referral

– 3) risk assessment model

– 4) don’t know

– B. No

– C. Don’t know/not sure
Clinical application

Courtesy of David Labby MD
Characteristics of ideal models

• Hospital comparison

– Reliable data that is 

easily obtained

– Deployable in large 

populations 

– Use variables clinically 

related to and validated 

in target population

– Good predictive value

• Clinical application

– Provide data before 

discharge 

– Discriminate very high 

from very low risk 

patients 

– Not overly complex

– Adapted to settings and 

populations in which use 

is intended
Objective: 

Synthesize the available literature on validated 

readmission risk prediction models, describe 

their performance, and assess their suitability 

for clinical or administrative use.

Kansagara, JAMA, 2011
METHODS
Search

• MEDLINE, CINAHL, and the Cochrane Library

– database inception through March 2011

– EMBASE through August 2011
Inclusion and exclusion

• Inclusion

– Studies of statistical models to predict hospital 

readmission risk

– Medical population

– Validated models

• Exclusion

– Non-medical population (pediatric, surgical, 

psychiatric, obstetric)

– Non-english language

– Studies in developing nations 
Model characterization

Administrative Data source Primary (survey, 

chart review)

Timing of 

data 

collection

Real-time 

(available 

before d/c) 

Retrospective 

(available at or 

after d/c) 

Real-time 

(available 

before d/c) 

Retrospective 

(available at or 

after d/c) 

Model 

category

Retrospective 

administrative

Real-time 

administrative

Retrospective 

primary

Real-time 

primary

Use Hospital 

comparison Clinical Clinical ? Clinical
Assessing model performance

• Discrimination

– C-statistic measures model’s ability to discriminate 

between those who get readmitted and those 

who don’t

– C-stat of 0.7 means a model will correctly sort 

high- and low-risk pair of patients 70% of the time

– Range 0.5 (no better than chance) to 1.0 (perfect)

– C-stat 0.5- 0.7  poor 

 0.7-0.8  acceptable/modest 

 >0.8  good
Assessing model performance

• Calibration

– degree to which predicted rates are similar to 

those observed in the population 

– we report the range of observed readmission rates 

from the predicted lowest to highest risk 

groupings
Methodologic assessment

• cohort definition

• follow-up

• adequacy of prognostic and outcome variable 

measurement

• validation method
RESULTS

Findings

• 14 retrospective administrative data models

• 4 real-time primary data models

• 3 real-time administrative data models

• 5 retrospective primary data models

• Only 1 model specifically evaluated 

preventable readmissions

• 30 day readmission most common outcome
Retrospective administrative data 

models (hospital comparison models)

• 9 of these tested in large US populations

– C-stat 0.55-0.65 

– 3 of these were CMS models (CHF, AMI, 

pneumonia)

• C-stat 0.60-0.63

• 3 better performing models (c-stat > 0.70) 

developed in Europe or Australia
Clinical models

• 3 used real-time administrative data

– C-stat 0.69-0.72 

• 4 used primary data available before hospital 

discharge

– C-stat 0.53-0.61 

• 5 used primary data available at or after 

discharge

– C-stat 0.66-0.83 
Calibration

• Though model discrimination was often poor, 

high- and low-risk scores were associated with 

clinically meaningful gradient of readmission 

rate
Examples of 3 different types of 

models 

• Retrospective Administrative Data

• Real-time Administrative Data

• Primary Data collected in Real-Time
Model Using Retrospective 

Administrative Data: CMS CHF model

• 30 day readmits

• Comorbidities from Medicare claims data:

– index admission and 12 months before index 

admission

• 37 variables: age, gender, CV variables and 

comorbidities

• C-Stat 0.60

• Observed readmission rates from lowest to 

highest decile of predicted risk: 15.0 - 37.0%

Krumholz, 2008
Model using Real-time Administrative 

Data

• 30-day readmits among pts with CHF

• Single urban US center

• Real-time EMR data

– Social: # address changes, marital status, SES, anxiety/ 

depression 

– Behavioral: cocaine, missed clinic visits

– Utilization: prior admissions, ED presentation time

• C-stat 0.72 (0.70-0.75)

• Observed readmission rates from lowest to 

highest quintile of predicted risk: 12.2 – 45.7%

Amarasingham, 2010
Model using Primary Data collected in 

Real-Time

• 4-year all-cause readmission

• Medicare, age ≥70 in 1984

• 8 factors: 

– Age, sex, self-rated health, informal caregiver, 

coronary disease, diabetes, hospital admission 

within past year, ≥ 6 visits within past year

• C-Stat 0.61

• Observed readmission rates from lowest to 

highest halves of predicted risk: 26.1 – 41.8%

PRA, Boult, 2003
Studies that compared models within 

a population

• 6 studies compared different models within 

the same population

• In 2 of these instances, addition of social 

determinants and functional status variables 

improved performance

Amarasingham, 2010

Coleman, 2004
Use of Variables

Variable considered Included in 

final model 

(n)

Evaluated, 

not included 

(n)

Variable not

considered 

(n)*

Medical dx or comorbidity index 24 0 3 

Prior hospitalization 14 1 10

SES/ income/ employment 5 7 10

Education 0 4 17

Caregiver availability/ social 

support

2 1 19

Access to care 5 2 14

*6 studies did not report candidate variables
POLL QUESTION 5

• Based on your own observations at your facility, 

which of the following contributes often to 

preventable readmissions (can choose more than 

one answer)?

– A. Lack of access to timely outpatient follow-up 

– B. Poor quality of inpatient care 

– C. Lack of patient self-management training

– D. Lack of access to palliative care/hospice services

– E. Patient factors (lack of social support, compliance, 

mental health/substance abuse issues)
DISCUSSION
Why have most models created to 

date had difficulty in predicting 

readmission risk?
Readmissions: it’s complicated

Comorbidities

Social support

Literacy

Housing

Inpatient care 

quality

Post discharge 

care

Bed supply

Rehospitalization
Social determinants and readmission 

risk

Comorbidities

Social 

determinants

Inpatient care 

quality

Post discharge 

care

Bed supply

Rehospitalization

Access to
Social determinants and readmission 

risk

• Most commonly included

– Diagnoses or comorbidity index

– Prior hospital utilization

– Age, sex, race/ethnicity
Social determinants—less commonly 

utilized

• Illness severity

• Mental health and substance use

• Overall health and function

• Socioeconomic status

• Social support

• Access to care

• Health literacy, numeracy

• Self-management skills
BOOST – 8Ps

• Problem meds (insulin, warfarin, digoxin, ASA)

• Psychological (depression)

• Principal Dx (CA, DM, COPD, CHF, CVA)

• Polypharmacy (> 5 meds)

• Poor health literacy (can’t teach back)

• Patient support lacking

• Prior hospitalizations (last 12 months)

• Palliative care

www.hospitalmedicine.org
Project RED

• No formal model

• Risk factors:

– Depressive symptoms

– Limited health literacy

– Frequent hospital admissions

– Unstable housing

– Substance abuse

www.bu.edu/fammed/projectred
Summary

• Readmission risk prediction models have been 

developed for hospital comparison and clinical 

intervention purposes

• Most models in both categories perform 

poorly

• Most models have relied on comorbidity and 

utilization data

• Few models have examined social 

determinant variables
IMPLICATIONS
Implication 1

Broad-based comparisons of risk-standardized 

rates, especially when tied to reimbursement, 

may be problematic and could be associated 

with unintended consequences
Which hospitals have the highest 

readmission rates?

Joynt, Circ Cardiovasc Qual Outcomes, 

2011
Implication 2

For clinical purposes, the perfect does not 

have to be the enemy of the good. Even 

modest incremental knowledge of risk can 

improve the cost-effectiveness of 

interventions. 
Implication 3

Match the model to intended use

 

– Models designed for measuring quality are 

probably not well suited for clinical use and vice 

versa. 

– Think carefully about the local population to 

which it is being applied. 
Implication 4

Given the lack of an existing risk prediction 

standard, incorporate clinically informative 

variables in your risk assessment that would not 

otherwise be captured.

• Housing status

• Access to care

• Health literacy

• Substance abuse 
Implication 5

Think about workflow and feasibility of data 

collection when adapting risk assessment 

tools

– Avoid overly complex models that impede 

workflow 

– Data must be easily available in real-time 

• ? Incorporate into EMR

• Simple surveys
Implication 6

We do not know how many readmissions are 

preventable. Think about using additional 

metrics to measure peri-discharge care. 
Can we make improvements in an 

already-integrated system?

CHF AMI Pneumonia

VA hospitals 25.2% 20.6% 19.2% 

Non-VA hospitals 24.8% 19.9% 18.4% 

Source: Kaiser Health News, Sep 2011, based on data from 

www.hospitalcompare.hhs.gov
Evidence-based Synthesis Program 

(ESP)

Questions?

If you have further questions, 

feel free to contact:

Devan Kansagara, MD MCR

kansagar@ohsu.edu

The full report and cyberseminar presentation is available on the ESP website: 

http://www.hsrd.research.va.gov/publications/esp/
12/13/2011 12:00pm Risk Prediction Models for Hospital 

Readmission

Spotlight on Evidence-based Synthesis 

Program

Englander, Honora

Kagen, David

Kansagara, Devan

Amanda Salanitro

VA Uses readmission rates on the IPEC Links Dashboard. Are these readmission rates based on the 

models with the .6 C statistic?

No, I believe these are unadjusted rates, though VA did participate in the hospital compare initiative and 

these data are risk-standardized rates using the same CMS methodology we discussed. 

Can you discuss the reliability of the LACE readmission tool?

The LACE index measures Length of stay, Acuity of admission, Comorbidity (Charleson index), and prior 

Emergency room use. Its C-stat was 0.68 which is not great, but better than many other models. 

However, this was only tested in Canada across a broad group of patients who were not that ill. Baseline 

readmission rates were only about 7%. Also, since measure includes LOS, requires waiting until d/c to 

calculate it. (van Walraven, CMAJ, 2010)

There's been a lot of focus on model fit (e.g. C-stat). Assuming that some of the most major predictors 

of 30-day readmission are system-factors such as care transition practices, are standard assessments 

of model fit too strict?

Agree – model discrimination as measured by the c-stat is only one way of evaluating these models. As 

we discussed, it is almost certainly the case that the exclusion (by design) of system-level factors from 

many model s reduces their performance. The question then becomes how good is good enough – the 

answer probably depends on what the model is being used for and the consequences of model use. As 

we said, there are lots of considerations in terms of using models clinically – the c-stat is only one small 

piece of this. Thinking about population of interest, other patient factors that have high face validity 

(social determinant issues), ease of use and so forth all need to be considered. The perfect does not have 

to be the enemy of the good. On the other hand, the potential negative unintended consequences of 

using poorly performing models for risk standardization if these risk-standardized rates are used for 

financial penalty purposes need to at least be considered. 

Predictions either account for hospital specific fixed or random effects and so are meant to essentially 

use the characteristics of the hospital itself to predict the probability of readmission.

Yes, when CMS calculates its RSRR as rate predicted for a given hospital/rate expected for average 

hospital with same case mix, it is both adjusting for patient case mix and taking into account a hospital’s 

baseline readmission risk. “The predicted number of

readmissions (technically called a shrinkage estimate) is calculated by adding the 

hospitalspecific

intercept, representing baseline readmission risk, to the sum of the estimated

regression coefficients applied to the patient characteristics in the hospital, and after

transformation, summing over all patients in the hospital.”
12/13/2011 12:00pm Risk Prediction Models for Hospital 

Readmission

Spotlight on Evidence-based Synthesis 

Program

Englander, Honora

Kagen, David

Kansagara, Devan

Amanda Salanitro

Did you have any information on models that speak to Non-medical populations: Mental Health? 

Especially since the depressive symptoms and MH and substance abuse continues to surface during 

these discussions.

No – we didn’t examine models in non-medical populations. 

Comment: The expected values are essentially the predictions gathered from a "reference" facility, 

that is those from a model which does not use random or fixed effects to predict the probability of 

readmission.

“Operationally, we obtained the expected number of readmissions for each hospital by

regressing the risk factors on readmission using all hospitals in our sample, applying the

subsequent estimated regression coefficients to the patient characteristics observed in the

hospital, adding the average of the hospital-specific intercepts, and after transformation,

summing over all patients in the hospital to get a count.”

Did you look at readmissions from skilled nursing facility?

No, not explicitly – many models would have included patients readmitted from home or a SNF. 

An individual may be hospitalized more than once, would then they be entered twice into the model? 

Was there accounting of the nesting?

Typically, patients would not be counted more than once – they would either have a readmission within x 

period or not. 

Does any model look at hospitals with a transition of care program?

Some of the larger models would have tested patients that had been part of a hospital with a transitions 

program, but we’re not aware of any models specifically tested in hospitals with transitions programs. 

Are there specific biomarkers that predict readmission in cardiac patients?

Ross JS, Archives Int Med, 2008;168(13) is a systematic review of predictors of heart failure readmissions. 

They looked at both statistical models and the literature on patient predictors (including biomarkers) on 

CHF readmission. 
12/13/2011 12:00pm Risk Prediction Models for Hospital 

Readmission

Spotlight on Evidence-based Synthesis 

Program

Englander, Honora

Kagen, David

Kansagara, Devan

Amanda Salanitro

What was the purpose of this study? Was it to examine methods or was it to study findings?

“This systematic review was performed to synthesize the available literature on validated readmission 

risk prediction models, describe their performance, and assess their suitability for clinical or 

administrative use”