Propensity Scores

Todd Wagner PhD Todd Wagner, PhD

July 2012
Obse v o s ud es

Outline

1. Background on assessing causat

– R d i d i Ran

dom

ize

d tr

i

a

ls 

– Obse

r

vati

onal 

s

tud

ies 

2. Calculating a propensity score 3 . Limitations Limitations 
Causality 

 Researchers are often interested in 

understandi l ng causal relationships

– Does drinking red wine affect health– Does a new treatment improve mort

 Randomized trial provides a venu

understanding causation understanding causation 
t n om

Randomization

Treatment Group (A) Ou

R iecrut Ra d Participants 

Random Sorting 

Ou

Comparison Group (B) Group (B) 

Note: random sorting can, by chance, lead to unbalanced groupuse checks and balances to preserve randomization 
Trial analysis

 The expected effect of treatment i

E(Y)=E(YA)-E(YB E(Y) ) E(Y )Expected effect on group A minus eeffect on group B (i.e., mean differeeffect on group B (i.e., mean differe
β

Trial Analysis (II) 

 E(Y)=E(YA)-E(YB) can be analyz

using th e foll owing model

yi = α + βxi + ε yi i i 

Where 

– y i th se outcome 

– α is the intercept 

– x is the mean difference in the outcome between treatment A re

treatment B 

– ε is the error term 

– i denotes the unit of analysis (person) denotes the unit of analysis (person) 
β

Trial Analysis (III)

 The model can be expanded to con

baseli ne characteristi cs

yi = α + βxi + δZi + ε yi i i i 

Where 

– y is outcome 

– α is the intercept 

– x is the added value of the treatment A relative to treatment B – Z is a vector of baseline characteristics (predetermined prior to – ε is the error term 

– i denotes the unit of analysis (person) denotes the unit of analysis (person) 
estimate of the causal effect of the trea

Assumptions

 Classic linear model (CLR) assumes th

– Right hand side variables are measured w

Right hand side variables are measured w

noise (i.e., considered fixed in repeated s

– There is no correl ation between the ri 

ght 

variables and the error term E(xiui)=0 

 If these conditions hold, β is an unbias

estimate of the causal effect of the trea

on the outcome 
e sbe

Observational Studies 

 Randomized trials may be 

– Unethi cal – Infeasible – Impractical – Not scientifically justified 
Sorting without randomiz

Patient characteristics characteristics Observed: health, income, age, gender. 

Treatment group 

Sorting 

Provider 

Comparison 

characteristics 

Observed: staff, costs, congestion, 

group 

IF everything is fully observed; results Never happens in reality.

Based on: Maciejewski and Pizer (2007) Propensity Scores and Selection Bias in Observational StuCyberseminar 
Sorting without randomiz

Patient characteristics characteristics 

Provider Characteristics 

Treatment group 

Sorting 

Unobserved Comparison ch t i ti haracteristics 

group 

Teamwork, provider communication, 

Unobserved factors affect outcome, but sorting; treatment effect is biased. Fixed effects would be potential fix. 

patient education 

Fixed effects would be potential fix. 
Sorting without randomiz

Comparison group Treatment group 

Unobserved factors affect outcome and sorting. Treatment effect is biased. Provides little or no information on caus

Patient characteristics characteristics 

Provider Characteristics 

Sorting Sorting 

Unobserved c

haracteristi cs 

Teamwork, provider communication, patient education 

Provides little or no information on causNo fix. 
Sorting without randomiz

Exogenous factorslaws, programs, “prices” Patient 

characteristics 

Provider Characteristics 

Treatment group 

Sorting 

Comparison group 

Unobserved ch t i ti haracteristics Teamwork, provider communication, patient education Unobserved factors affect outcome and 

sorting. Treatment effect is biased. Instrumental variables is potential fix. Instrumental variables is potential fix. 
Propensity Scores

 What it is: Another way to correct for 

observable characteristics observable characteristics 



Wh i at 

it 

is not: 

A way to a dj ust 

for uno

characteristics 

 If you read wikipedia you will get the 

If you read wikipedia, you will get the 

impression about propensity scores 
Strong Ignorability 

 Propensity scores were not developed 

handle non handle non

-random sorting random sorting 

 To make statements about causation, y

would need to make an assumption thatreatment assi

gnment is stron

g ly i

gnor

– Similar to assumptions of missing at rand– Equivalent to stating that all variables of 

Equivalent to stating that all variables of 

are observed 
Calculating the Propensity

 One group receives treatment and 

group 

doesn’t 

 Use a logistic regression model to Use a logistic regression model to

estimate the probability that a persrecei d t ived 

trea

tmen

t

 This predicted probability is the This predicted probability is the 

propensity score 
a e u e ated to t e

Variables to Include

 Include variables that 

are unrelated to the exposure but related to the outcome 

 This will decrease the 

variance of an variance of an 

estimated exposure 

effect without effect without

increasing bias 

E OuExposure 

Brookhart MA, et al Am J Epidemiol. 2006 Jun 15;163(12):1149-56. Variable selection for propensity scor
Variables to Exclude

 Exclude variables that are related 

exposure b t t t th t b

u

t not 

to the ou

tcome 

 These variables will increase the v

These variables will increase the v

of the estimated exposure effect w

decreas

ing bi as

 Variable selection is particularly Variable selection is particularly 

important in small studies (n<500

Brookhart MA, et al Am J Epidemiol. 2006 Jun 15;163(12):1149-56. Variable selection for propensity scor
Example: Resident Surg

 Do cardiac bypass patients have b

worse ou

tcomes w

hen th 

e

ir surger

conducted b

y a resident? 

 CSP 474 

– Randomized patients to radial artery 

sa

phenous vein 

– Tracked primary surgeon 
v b o es de

Is Resident Assignment Rand

 Assignment may depend on 

– P i at ent risk – Availability of resident – Resident skill – Local culture 

 In CSP 474 23% (167 / 725) of ca

In CSP 474, 23% (167 / 725) of ca

by resident 
Use of Resident Varies by 

Site Resident

%

501 0% Only supplies informa506 81% group. 521 6% No variance within sit

No variance within sit

523 0% are dropped if you us578 89% effects. 580 0%598 37%618 61%629 15%652 0%678 8%
e 

Resident Assignment in CS

OR P value

Age 1.00 0.79

Canadian Functional Class

Class 2 1.93 0.15

Class 3 2.12 0.09

Class 4 4.25 0.02

Urgent priority 0.93 0.89

Artery condition at site

Calcified 0.67 0.25Sclerotic 2.63 0.00

site 2 62.89 0.00

site 3 0.67 0.60

site 5 138.16 0.00site 7 11.66 0.00

site 8 19.85 0.00

site 9 1 76 . 0 43 .endo vascular harvest 0.20 0.01

On pump surgery 1.20 0.75

1‐2 grafts 1.70 0.16

4‐5 grafts 0 79 . 0 46 .

Assiglinked 

Uncle(cultu

patteresid

Bakaeen F, Sethi G, Wagner T, et al. Coronary Artery Bypass Graft Patency: Residents Versus Atten

of Thoracic Surgery. in press 
Resident Assignment in CS

AgeCanadian Functional Class

Class

2

Class

3

Class

4

Urgent priority

Arter

y condition at site

CalcifiedSclerotic

site

2

site

3

site

5

site

7

site

8

site

9

endo vascular harvest

On pump surgery1‐2 grafts

4‐5 grafts

OR

1.001.932.124.25

0.93

0.672.63

62.89

0.67

138.16

11.6619.85

1 76 .

0.20

1.20

1.70

0 79 .

P value

0.79

0.150.090.02

0.89

0.250.00

0.000.60

0.00

0.000.00

0 43 .

0.01

0.75

0.16

0 46 .

As

asor 

Bakaeen F, Sethi G, Wagner T, et al. Coronary Artery Bypass Graft Patency: Residents Versus Atten

of Thoracic Surgery. in press 
Resident Assignment in CS

AgeCanadian Functional Class

Class

2

Class

3

Class

4

Urgent priority

Arter

y condition at site

CalcifiedSclerotic

site

2

site

3

site

5

site

7

site

8

site

9

endo vascular harvest

On pump surgery1‐2 grafts

4‐5 grafts

OR

1.001.932.124.25

0.93

0.672.63

62.89

0.67

138.16

11.6619.85

1 76 .

0.20

1.20

1.70

0 79 .

P value

0.79

0.150.090.02

0.89

0.250.00

0.000.60

0.00

0.000.00

0 43 .

0.01

0.75

0.16

0 46 .

Ass

assang

and and 

har

Bakaeen F, Sethi G, Wagner T, et al. Coronary Artery Bypass Graft Patency: Residents Versus Atten

of Thoracic Surgery. in press 
Sorting 

 Sorting is non-random  If sorting is fully observed, we can est

unbiased effect of resident surgeon eff

 Improbable that we fully observe the s

process process 

– Thus E(xiui)≠0 – M l i i i bi d d d i 

Multivariate is biased and we need instru

variables 
Dimensionality 

 The treatment and non-treatment g

may 

be differen

t on many dimens

i

 The propensity score reduces thes

single dimension 
.2 .4 .6 .8

Common Support

3 2 

kdensity m1

1 0 

0 1 

x 

kdensity m1 kdensity m1 

Common support 

These are the densities of having resident or non-resident surgpropensity score) propensity score) 
Using the Propensity Sco

 Match individuals (perhaps most comm

approach) approach) 

 Include it as a covariate (quintiles of th

the regression model 

 Include it as a weight in a regression (

Include it as a weight in a regression (

place more weight on similar cases) 

 C d on

duct su

bgroup ana

lyses on s

i

mil ar 

(stratification) 
Matched Analyses

 The idea is to select controls that resem

treatment group in all dimensions exctreatment group in all dimensions, exc

treatment 

 You can exclude cases and controls th

match

, which can reduce the sam

ple 

size/power. 

 Different matching methods Different matching methods 
Matching Methods

 Nearest Neighbor: rank the propen

score and h t l th t i l 

d c

hoose con

trol th

at 

is c

l

case. 

 Caliper: choose your common sup

and from within ran

dom

ly 

draw co
PS or Multivariate Regres

 There seems to be little advantage to u

over multivariate analyses in most casover multivariate analyses in most cas

 PS provides flexibility in the functiona

PS provides flexibility in the functiona

 Propensity scores may be preferable if 

sam

ple size is small and the outcome o

interest is rare.2 

1. Winkelmeyer. Nephrol. Dial. Transplant 2004; 19(7): 1671-1673. 2. Cepeda et al. Am J Epidemiol 2003; 158: 280–287 
≠

Silk purse out of sow’s ea

 Propensity scores focus only on observ

on unobserved on unobserved. 

 Improbable that we fully observe the s

process 

– Thus E(xiui)≠0 – Multivariate (including propensity score) 

and we need instrumental variables and we need instrumental variables 
Second Example

 CSP 474 was a randomized trial th

enroll

e

d pati en

ts 

in 11sit es 

 Patients were randomized to two t

Patients were randomized to two t

heart bypass 

 Is the sample generalizable? 

We compared enrollees to non We compared enrollees to non

-enro

enro
Methods

 We identified eligible bypass patie

across VA (2003 across VA (2003 -2008) 

 We compared: 

– participants and nonparticipants wit

participating sites participating sites 

– participating sites and non-participa– participants and all non-participants 
Propensity Scores

 A reviewer suggested that we shou

a propensit t id tif d 

ity score 

to identify 

degr

overlap 

 Estimated a logistic regression for 

parti

c

ipati on 

(pscore an

d ps

tes

t co

in Stata

) 
ern epanec n o an

Group Comparison befor

40 

Kernel density estimate 

30 

ity 

10 20

Densi 

0 

0 .05 .1 

Estimated propensity score 

Participants Non-Participants 

k el h ik b d idth 0 0045 kernel = epanechnikov, bandwidth = 0.0045 
Variable Variable Sample 

Mean 

Treated Control Treated Control %bias 

%reduct 

|bias| t 

t-test 

p>

ms_1 Unmatched 

Matched 

.09729 

.09729 

.10659 

.0986 

-3.1 

-0.4 85.9 

-0.75 0.45-0.22 0.82

ms 3

_

3 Unmatched Unmatched 

Matched 

.35407

.35407 

36275

.

36275 

.35769 

-1 8. 

-0.8 58.3 

-0 45 0 65

0.45 0.65

-0.37 0.71

male Unmatched 

Matched 

.99043 

.99043 

.99069 

.99049 

-0.3 

-0.1 76.6 

-0.07 0.94

-0.03 0.97

aa2 Unmatched 

Matched 

.12919 

.12919 

.09003 

.11989 

12.6 

3.0 76.3 

3.37 0.001 

1.36 0.173 

aa3 aa4 aa5 

Unmatched 

Matched 

Unmatched 

Matched 

Unmatched 

Matched 

.27113 

.27113 

.27751 

.27751 

.10367 

.10367 

.22301 

.26578 

.22921 

.26658 

.1388 

.11048 

11.2 

1.2 

11.1 

2.5 -10.8 

-2.1 

88.9 77.4 

80.6 

2.86 0.004 

0.59 0.554 

2.84 0.005 

1.20 0.230 -2.52 0.01

-1.10 0.27

aa6 Unmatched 

Matched 

.09569 

.09569 

.13058 

.10471 

-11.0 

-2.8 74.2 

-2.57 0.01

-1.51 0.13

aa7 Unmatched 

Matched 

.05104 

.05104 

.10121 

.05918 

-19.0 

-3.1 83.8 

-4.14 0.00

-1.82 0.06

aa8 Unmatched 

Matched 

.01754 

.01754 

.05057 

.0204 

-18.3 -1.6 91.4 

-3.76 0.00-1.07 0.28

Standardized difference >10% indicated imbalance and >20% se
Largest 

Summary of the distribution of the abs(bias) 

BEFORE MATCHING BEFORE MATCHING 

Percentiles Smallest 

1% .0995122 .0995122 5% .2723117 .2723117 

10% 1 809271 1.809271 1 061849 1.061849 Obs 25% 3.781491 1.809271 Sum of Wgt. 50% 10.78253 Mean 10.5

Std. Dev. 9.03

75% 15.58392 18.99818 90% 18.99818 19.16975 Variance 81.5

95% 29.75125 29.75125 Skewness 1.8499% 46.80021 46.80021 Kurtosis 8.09

AFTER MATCHING 

Percentiles Smallest 

1% .0321066 .0321066 

5% .0638531 .0638531 10% .4347224 .332049 Obs 38 

25% .7044271 .4347224 Sum of Wgt. 

50% 1.156818 Mean 1.416819 

Largest Std. Dev. 1.215813 

75% 1.743236 2.848478 90% 2.848478 2.97902 Variance 1.4782 95% 3 083525 3 083525 Skewness 2 524339 95% 3.083525 3.083525 Skewness 2.524339 99% 6.859031 6.859031 Kurtosis 11.61461 
Results

 Participants tended to be slightly 

healthier and younger, but

 Sites that enrolled participants we

different in provider and patient characteristics than non characteristics than non -participat

participat
–

PS Results

 38 covariates in the PS model 

20 variables showed an imbalance 20 variables showed an imbalance 

– 1 showed severe imbalance (quantit

CABG operations performed at site

 Balance could be achieved using t

Balance could be achieved using t

propensity score 

 After matching, participants and c

were similar were similar 
Generalizability 

 To create generalizable estimates 

th RCT i ht th l the RCT, you can we

i

ght the ana

ly

with the 

pro

pensityscore. pp 

Li F, Zaslavsky A, Landrum M. Propensity score analysis with hierarchicB t MA H d U i it 2007 Bos

ton MA: 

Harvard 

U

niversity; 2007. 
RCTs and Propensity Sco

 What would happen if you used a 

propensity score with d

a

t 

a 

from a 
Share Common Suppo

1.5 1

m1 

.5 

kdensity 

0 

0 .2 .4 .6 .8 1 

x

kdensity m1 kdensity m1 
Summary

 Propensity scores offer another wa

adj t f f di b d djust for confounding based on 

observables

 Reducing the multidimensional na

confoundi ng can be helpful 

 Propensity scores do not attempt t

Propensity scores do not attempt tfor unobserved. 
Unrealistic Expectation
Weaknesses 

 Propensity scores are often 

misunderstood 

 While they can help create balanc

observables, they do not control founobservables or selection bias unobservables or selection bias 
Strengths

 Allow one to check for balance be

cont l dt t trol and treatment 

 Without balance, average treatmen

Without balance, average treatmen

effects can be very sensitive to the of th e esti mators. 1 

1. Imbens and Wooldridge 2007 http://www.nber.org/WNE/lect_1_matc
Further Reading

 Imbens and Wooldridge (2007) 

www nber org/WNE/lect 1 match fig www.nber.org/WNE/lect_

1

_

match_fig

 Guo and Fraser 

(2010

) Pro

pensit

y 

Analysis. Sage. 

 B kh Brookhart MA l A J MA, et al 

Am J E id i l E

pidem

i

o

l. 

Jun 15;163(12):1149-56. Variable selepropensity score models.