Research Design

Jean Yoon Jean Yoon

July 11, 2012
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Causality

 Want to be able to understand impact of implement

program or intervention.

 Ideally, wouldestimate causal eff ect of treatmento

by comparing outcomes under counterfactual 

– Treatment effect=Y Treatment effect=Y (1) Y (0) i(1)-Yi(0) – Observe outcome Y when patient gets treatmen

when same patient does not get treatment t when same patient does not get treatment, t0 =

– Compare difference in outcomes to get impact o– In reality we don In reality we don t ’observe same patients with a

observe same patients with a

treatment 
Randomized Experime

 Randomize who gets treatment T Randomize who gets treatment T 

R T O R O 

 Com

pare outcome between treated and untr

groups to get impact of treatment 

 Because treatment was randomized, there ar

Because treatment was randomized, there ar

systematic differences between treated and 

groups. groups. 

 Differences in outcomes can be attributed to 

effect of treatment effect of treatment 
Causality and Observatio

Causality and Observatio

Studies

 Most health services research is observ

and cross sectional 

– Causality difficult to show because of co

Causality difficult to show because of co

also referred to as selection and endogen

 Omitted variables bias Omitted variables bias  Selection  Reverse causality Reverse causality  Measurement error 
Observational Study Exam

 Observe some patients with diabetes in 

primary care clinic participate in phonprimary care clinic participate in phon

disease management program, others d

– Compare A1c, cholesterol, other outcom

between groups of patients at end of prog

– If patients who participated in the progra

better outcomes than those who didn’t, cconclude the program caused the better o
Q&A 

 Please respond using Q&A panel: 

– What other factors could have led th

program participants to have better outcomes than the non-participants?
Bias of Treatment Effe

 Characteristics not be balanced between 

 Enrolled had better outcomes to start with Enrolled had better outcomes to start with 

 Patients selected into treatment 

 Enrolled would have improved over time b/c 

Enrolled would have improved over time b/c 

motivated to improve 

 Changes over time occurred that were un

Changes over time occurred that were un

intervention 



Enroll e

d 

a

lso engaged i ti 

in oth er acti

viti es 

of program), e.g. reading diabetes info on
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Quasi-experimental Meth

 Observational studies do not have rand

treatment so use methods to make like treatment, so use methods to make like

experimental study.

– Identify similar control group – Try to eliminate any systematic diff

Try to eliminate any systematic diff

between treatment and control grou



Compare 

(

c

hange 

i

n

) outcomes 

betwe

treatment and control groups 
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Sample selection 

 Bias arises when outcome is not observable for som

sample selection 

 Form of omitted variables bias  Example: attrition from study collecting outcomes 

patients 

– Health outcomes only observed in respondents – Not possible to make inferences about determin

health outcomes in study population as whole. 

– If reasons why patients don’t respond are correl

unobservable factors that influence outcomes, th

identification problem identification problem. 
Sample selection 

 One solution to sample bias is a selection model ca

Heckman probit or heckit or generalized tobit 

 First stage uses probit model for dichotomous outc

(selection equation) for whole sample 

 From parameters in first stage, calculate inverse mi

selection hazard for each patient 

 Add inverse mills ratio as variable into second stag Second stage uses OLS model for linear outcome (

equati ) f d t i l tion) 

for respon

den

ts 

in samp

le 

 Assumption is that error terms of two equations are 

normally distributed and correlated by normally distributed and correlated by 

ρ
β ρ γ

Sample selection

1) D=Zγ+ε D=1 participation, D=0 no pa2) w*=Xβ+ μ Onl ha e o tcome * hen 

Only have outcome w* when 

Estimate 1) and get φ(Zγ)/Φ(Zγ)= λ inverse m

Estimate 2) adding λ as a parameter 

 E[w|X, D=1]= Xβ + ρδμλ(Zγ)  Coefficient of λ is ρ, if =0 then no sample s Want instruments (included in Z) that are no

Want instruments (included in Z) that are no

to outcome w 
Sample selection 

 If outcome is dichotomous, can use 

bivariat

model. 

 If outcomes data are available for all patient

use treatment effects model. 

 New, semiparametric models do not make j

normality assumptions. normality assumptions. 
Sample selection

 Strengths 

– Generalizes to 

p po

pulation of interest 

– Addresses selection based on unobservab

 Weaknesses Weaknesses 

– Assumes joint normality of errors in two 

equations.

– Can still be run if no instrument (unlike 2

relying on distributional assumptions of nlinearity in inverse mills ratio. 
Sample selection 

 Stata code Heckman 

y x1 x2 x3, select 

(x1 x2 z1 z2

) twoste
Sample Selecti on Examp

 Zhoua A, Gbao J, Xbue Q, Y bang X Y J 

X, Yan J

of rural mutual health care on outpatient ser

utili ti i Chi ill di l i tit 

tilization in Chinese village medical institu

evidence from panel data. Health Econ. 18: 

S136 (2009) S136 (2009) 

 Compare effect of outpatient insurance prog

drug policy to restrict drug prescribing on omedical expenditures. 

 Only some residents had outpatient visits. 
Sample Selecti on Examp

 Zhou et al. results 
Sample Selection Examp

 Zhou et al Zhou et al . results
P ll 1 Poll 1 on sample section: 

a. Sample sel i l li 

ection models are used to elimina

due to unobservable characteristics. 

b. Sample selection models are used to eliminonl d t b bl h t i ti ly due to observable characteristics. c. Sample selecti on models do not assume ji

o n

normality of the selection and outcome equati
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Differences-in-Differences

 Can exploit natural experiment with D N d l it di l d t b t 

Need 

longitudinal 

d

a

ta or o

bserve ou

tc

different time points for treatment and groups 

 Subtract out differences between treatm

Subtract out differences between treatm

control groups and differences over tim
Differences-in-Differenc

25

Average treatment effect=(B-A) 

20

A C 

B 

1510 D 

50 

Pre Post 
β

Differences-in-Differenc

Program P: 0=no, 1=yes Time T: 0=pre, 1=post 

Y= β + β T+ β P + β P*T +ε 0 + β1T + β2P + β3P T +ε

 So β3 is the differences-in-differen

estimate 

β = (ΔY ) − (ΔY ) 3 P=1 P=0 
dd esses o ed v b es b s

Differences-in-Differenc

 Strengths 

– Difference out t

ime trend

– 

Add

resses 

omitted 

varia

b

les 

bia

s if 

unmeasured time invariant factors

 W k ea

knesses 

– If unobserved factors that change ov

If unobserved factors that change ov

can have biased estimates 
β β β

= –

Differences-in-Differenc

 Unobserved factors often cause omitted var Panel analysis with time invariant character

for individual (observed and unobserved) Yit= β0 + β1Tt + β2Pit +δi +εit 

 Difference model 

YY = β + (P P )*β + ε ε i1-Yi0 β1 + (Pi1 -Pi0)*β2+ εi1 εi0 

 β1 is time trend  β2 is treatment effect  Time invariant δi drops out of model 
Diffe e ces erences-in-Diffe e c

erenc

 Fi d ff i f “ i hi i 

Fixed effects estimate of “within estim

same as first differencing with 2 time 

 Cannot estimate effect of time invarian

in FE model in FE model 

 Stata command xtreg will run FE 
e ts O ed cat o d e e ce W t

D-D Example

 Chernew ME et al. Impact Of Decreasing 

Copay 

m

e

nts 

On Med

icat

i

on A

d

h

e

r

e

nce 

W

i

t

Disease Management Environment. Health Jan/Feb 2008

; 27 , 

1

;103-112. 

 Two health plans implemented disease man

programs but only one reduced drug copayprograms, but only one reduced drug copay

 Drug adherence for two groups of patients c

pre

-post implementation of disease manage

post implementation of disease manage

reduction in drug copays 
D-D Example 

 Chernew et al. 
Poll 2 on differences-in-differences 

a. Differences-in-differences eliminates confodue to time trends. b. Differences-in-differences can eliminate omvariables bias from time invariant factors. c. Both a and b d. None of the above 
Outline 

 Causality and study design  Quasi-experimental methods for 

observational studies observational studies 

– Sample selection – Differences-in-differences – Regression discontinuity Regression discontinuity 
Regression Discontinui

 Can do when treatment is not randomly ass

Can do when treatment is not randomly ass

based on a continuous, measured factor Z 

– Z called forcing variable Z called forcing variable 

 Discontinuity at some cutoff value of Z  Individuals cannot manipulate assignment o Onlyjum

pin outcome due to discontinuit

y 

treatment

 Treatment effect Treatment effect 

=the expected outcome fo

the expected outcome fo

above the cutoff minus the expected outcom

units just below the cutoff (otherwise identunits just below the cutoff (otherwise ident
Regression Discontinui

 Strengths 

– Z can have direct impact on outcome (un

Z can have direct impact on outcome (un

instrumental variables) 

 Weaknesses 

– Need to test functional form for effect of 

(e.g. linear, interaction, quadratic terms) biased treatment effects if model is miss

p
e act o ed ca su a ce o t e

RD Example

 Bauhoff, S., Hotchkiss, D. R. and Smith, O. 

Th

e impact 

of med

ical insu

r

a

nce for 

t

he poo 

Georgia: a regression discontinuity approacEcon.,20: 1362

–1378. 

 Effect of medical insurance program for poo

republic of Georgia on utilization republic of Georgia on utilization 

 Eligibility for program limited to residents b

means test score (SSA) means test score (SSA) 

 Compare outcomes for eligible residents ve

income resid ents w

ho are not eli

gibl 

e
RD Example 

 Bauhoff et al 
RD Example

Bauhoff et al 

Y=β0+β1MIP+β2f(score-cutoff)+β3MIP*f(score-c

β1 =treatment effect, discontinuous change at c

β2 =effect of means test on outcomes for non-b

β3 =effect of means test on outcomes for benef
RD E l xamp e 

 Bauhoff et al 
Poll 3 on Regression discontinuity

a. Regression discontinuity can be used when treatment is assigned based on some unknown b. Regression discontinuity can’t be used when treatment assignment is directly related to the outcome. c. Regression discontinuity can be used when treatment is assigned based on cutoff of a conteligibility score 
Poll 4 on final review 

a. Quasi-experimental methods can help address commof bias of treatment effects in observational studies. b Quasi b. Quasi-experimental experimental methods attempt to reduce any 

methods attempt to reduce any 

differences between treatment and control groups. 

c. Quasi-experimental methods provide stronger study order to make inferences about causality. d. All of the above 
-

Review

 Quasi-experimental methods can help 

common sources of bias of treatment eobservational studies. 

 Quasi -experimental methods attempt t

experimental methods attempt t

any systematic differences between treand control groups. 

 Quasi-experimental methods provide s

study designs in order to make inferen

about causality about causality. 
References

 C b ll D T d St l J C E i t l 

Campbell, D. T., and Stanley, J. C. Experimental a

experimental Designs for Research. Chicago: Rand 

1966. 

 Heckman, J. (1979). "Sample selection bias as a sp

error". Econometrica 47 ( ) 1 : 153–61. 

 Wooldridge, J. M.: Econometric Analysis of Cross 

Panel Data. MIT Press, Cambridge, Mass., 2002. 

 Trochim, William M. The Research Methods Know

Base, 2nd Edition. Internet WWW page, at URL: http:// www.socialresearchmethods.net/kb/(version 

of 12/07/10). 
More References

 Zhoua A, Gaob J, Xueb Q, Yangb X, Yan J. Effect

mutual health care on outpatient service utilization village medical institutions: evidence from panel dEcon. 18: S129–S136 (2009) 

 Chernew ME et a

l. 

Impact Of Decreas

ing 

Copaym

Medication Adherence Within A Disease Managem

Environment Environment. Health Affairs; Jan/Feb 2008; 27 1;

Health Affairs; Jan/Feb 2008; 27, 1;

 Bauhoff, S., Hotchkiss, D. R. and Smith, O. The im

medical insurance for the poor in Georgia: a regresmedical insurance for the poor in Georgia: a regres

discontinuity approach. Health Econ. 2011; 20: 136
Next Lectures 

July 25, 2012 

Propensit y 

Scores 

Todd Wa

gner 

August 8, 2012

Instrumental Variables ModInstrumental Variables Mod

Patsi Sinnott