Alternatives to the

Randomized Controlled 

in Implementation Scie

HSR&D QUERI Cb i y

berseminar

February 2012

Alexander S. Young, MD, MSHS

Greater Los Angeles VA HSR&D

VA VISN‐22 MIRECC

UCLA Department of Psychiatry Psychiatry
Overview

• Randomized Controlled Trials (RCTs)

– strengths, weaknesses, and barriers in

Implementation Science

• Observational Methods

– can they produce accurate results?

• What should you consider?• Two examples

– cohort study of the effect of insurance insurance p– instrumental variables analysis of

effectiveness of treatment treatment for depressio

depressio
Poll Question #1

• What is your primary role in VA?

– student, trainee, or fellow– clinician– researcher– manager or policy‐maker– other
Poll Question #2

• Which best describes your research

experience?

– have not done research– have collaborated collaborated on research research– have conducted research myself– have applied for research funding– have led a funded research grant
Why We Use RCTs

• Confounding factors can be associate

both exposure and outcomes

– the associ i at on between al hl co o and lu

cancer is due to the confounding factor 

cigarette smoking

• In RCTs, with a large sample size,

confounding factors are equally prese

exposed and unexposed groups

• RCTs are often blinded

– knowledge of exposure can bias the eva

of outcomes
What about RCTs in Implemen

Research?

• Implementation usually occurs at the organi

practice level

• Many sites are not interested interested in participatin

participatin

implementation, research, plus random assi

– substantial site motivation is required– limiting research to h hl ig y motivated sites redu

generalizability of results

• How many variables affect intervention outc

the organizational level?

– how many sites are needed to balance variable

intervention intervention and control control groups using randomi

randomi

100? 500? 1,000?
What Observational Design

are Available?

• Many!• Associations or correlations• Regression analyses to control for

measured confounders

– propensity scoring

• Instrumental variables to control fo

unmeasured confounders

• Control group: none, historical, be

after, case‐control, cohort
Results from RCTs vs. Observat

Studies

• Concato and colleagues reviewed me

analyses of RCTs and observational st

– 5 cli i l n ca areas, 99 arti l c es

– New England Journal of Medicine, 2000

• Ch t o or or case‐cont l ro st di u es

– excluded studies with historical controls– excluded excluded clinical clinical trials with non‐random

assignment to the intervention

• RCT & observational observational results results were qui

similar
Results from RCTs vs. Observational 

Concato, New England Journal of Medicine, 2000
Non‐RCTs in Implementation Scien

Two Examples

• Cohort with matched control

– evaluate p y arit of mental health insu– policy implementation– 9 implementation implementation plans, 9 control control pl

• Instrumental variables

– eval t ua e the eff ti ec veness of d 

epress

treatment

– quality i t mprovement for d i epression 

primary care

– 938 pati t en s
Cohort with Matched Con

Evaluation of Parity

in the Federal Employees H

B fit (FEHB) P Benefits (FEHB) Progra

Goldman et al New England Journal of Medicin

Sponsored by: Conducted by:

Goldman et al, New England Journal of Medicin

p y

U.S. DHHSU.S. Office of Personnel

M t

Northrop Grumman

Westat

Harvard Medical Sc

Management RAND

University of Maryl
FEHB Program

• Largest employer‐sponsored health in

p g ro ram in the Nation

– 8+ million beneficiaries– over $29 billion billion annually annually in health care b

b

• U.S. OPM administers FEHB Program• Over 250 health plan choices
Mental Health & Substance Abuse

History of B fi ene ts in FEHB

• Beginning in 1975

– MH/SA coverage began to erode with o

diminution of benefits

• Copays, deductibles, limits• From 1980 to 1997

– coverage became much more limited th

other medical treatments

– the share of total claims accounted for 

MH/SA claims declined from 7.8% to 1.

– interest in restoring parity, but at what 

• Similar in private sector insurance ma
Parity Policy

• 1999: President Clinton d d irected O

institute a policy of parity improvin

MH/SA coverage within FEHB

– MH/SA insurance benefit design equ

benefits for general medical services

– equal copays, deductibles, limits

– parity to begin January 2001

• Included funding funding for research research eval

eval

– played major role supporting recent

enactment of national national MH/SA parity
Research Questions

• Did FEHB plans comply with the p

policy?

• How did the FEHB parity policy af

MH/SA benefit design and

management?

• How did parity affect access to M

care?

• How did the parity policy affect co

MH/SA care to the beneficiary an

insurance companies?
Research Design

• P ti rospective, coh t or , mathd c e cont l ro• 9 FEHB plans selected on the basis of

– 500 or more enrollees– geographic location– plan type and structure

– size of enrollee population

– pl ’ an s it t n erest in eval ti ua on

• Matched 9 comparison plans to accoun

secular trends

– matched on location and type of plan

• Difference‐in‐differences analysis
Results

Research Question Research Question Findings

Did all FEHB plans comply with the parity policy? All plans complied. the parity policy? p pHow did parity affect MH/SA Most plans enhanced be

and entered into manag

benefit design and management? and entered into manag

carve-outs.

How did the parity policy affect access to MH/SA care?How did parity affect cost of MH/SA care to the insurance compani d b fi i ? ies and beneficiary?
MH/SA Use and Spending

Difference in Difference by P

Difference in differences in probability of Difference in differenc

Difference

-in

-Difference by P

Plan

Difference-in-differences in probability of

MH/SA use from pre- to post-parity

Difference-in-differenc

spending per user from 

Estimate Significance Estimate 

FFS-NAT -0.12% NS -$68.97 

FFS-MA1 -0.96% p

≤0.05 -$42.13 

FFS-MA2 0.78% p

≤0.05 $27.11 

FFS-NE1 0.23% NS -$5.50 

FFS-NE2 -0.38% NS -$119.26 

FFS-W -0.24% NS -$22.60 

FFS-S -0.35% NS -$201.99 

HMO-W1 0.32% NS $32.96 

HMO-NE -2.73% p

≤0.05 -$77.82 
Results

Research Question Research Question Findings

Did all FEHB plans comply with the parity policy? All plans complied. the parity policy? p pHow did parity affect MH/SA Most plans enhanced be

and entered into manag

benefit design and management? and entered into manag

carve-outs.

How did the parity policy affect access to MH/SA care?

Access increased on pasecular trends.

How did parity affect cost of MH/SA care to the insurance compani d b fi i ? ies and beneficiary?
MH/SA Use and Spending

Difference in Differences by P

Difference in differences: probability of Difference in differenc

Difference

-in

-Differences by P

Plan

Difference-in-differences: probability of

MH/SA use from pre- to post-parity

Difference-in-differenc

spending per user from 

Estimate Significance Estimate 

FFS-NAT -0.12% NS -$68.97 

FFS-MA1 -0.96% p

≤0.05 -$42.13 

FFS-MA2 0.78% p

≤0.05 $27.11 

FFS-NE1 0.23% NS -$5.50 

FFS-NE2 -0.38% NS -$119.26 

FFS-W -0.24% NS -$22.60 

FFS-S -0.35% NS -$201.99 

HMO-W1 0.32% NS $32.96 

HMO-NE -2.73% p

≤0.05 -$77.82 
MH/SA Out of Pocket Spend

Difference in Difference by P

Out-of-pocket spending per user

-

-Difference by P

Plan Estimate SignificFFS-NAT $4 48 . p

≤0 0

.

FFS-MA1 -$15.43 p

≤0.0

FFS-MA2 -$13 82 . p

≤0 0

.

FFS-NE1 -$8.78 NSFFS-NE2 -$48 12 . p

≤0 0

.

FFS-W -$49.80 p

≤0.0

FFS-S -$87 06 . p

≤0 0

.

HMO-W1 $25.16 p

≤0.0

HMO-NE $23 40 . p

≤0 0

.
Results

Research Question Research Question Findings

Did all FEHB plans comply with the parity policy? All plans complied. the parity policy? p pHow did parity affect MH/SA Most plans enhanced be

and entered into manag

benefit design and management? and entered into manag

carve-outs.

How did the parity policy affect access to MH/SA care?

Access increased on pasecular trends.

How did parity affect cost of MH/SA care to the insurance 

i d b fi i ?

Total costs increased in secular trends. In most out-of-pocket costs dec

companies and beneficiary? out-of-pocket costs decl
Instrumental Variable

The Effectiveness of Prim

Care Depression Treatm

on Patients’ Clinical Sta

and E l t d Emp

loymen

t

Schoenbaum et al, Health Services Resea

Health Services Resea
Selection Bias

U d i bl i t d i

Unmeasure

d var

i

able assoc

i

a

t

e

d wi

treatment and outcome

E l l hh d 

Examp

le: peop

le w

ho c

hoose 

depre

treatment have more severe depres

Result: depression treatment is corr

with more severe depression

– so, depression treatment makes depressi
How to Evaluate the Effectiven

D i T t t? Depress

ion Trea

tment?

 Clinical trials problem Clinical trials problem

– depression treatment is well established and ava– cannot ethically deny access to treatment cannot ethically deny access to treatment– who would enroll in a randomized trial? people 

treatment-refractory, mildly ill, or doing it for t

– response rates in trials getting smaller and smal

 Implementation science problem

– quality improvement trial

– organizations agree to participate

– patients & providers choose & providers choose whether or not to u

whether or not to u

treatment

– the effectiveness of treatment cannot be directly 

due to selection bias
Instrumental Variable

 E ti t i t f t t t i E

stima

te 

impac

t o

f

trea

tmen

t, g

iven 

unmeasured selection bias

 Requires instrumental variable (z): affec

Requires instrumental variable (z): affec

key independent variable, but only impaoutcome through the key independent va

– geographic proximity to treatment– assignment to quality improvement program

 Requires larger sample size Requires larger sample size
Partners in Care

 Si i x commun

ity-b d i i i5 base

d organ

izat

ions 

in 

5

states

–

p

ri li i 46 f 48 imary care cli

n

ics: 46 o

f 48 agree

– 27,332 consecutive patients screened

– 1,093 patients with depression enrolled 1,093 patients with depression enrolled– 938 patients completed 6-month follow-up surv

 Clinics randomized to 

q yp ualit

y im

provem

program or usual care

– increase use of antidepressant medication and 

psychotherapy

 Patients assessed

– treatment use, cli i l l li

n

ica

l status, emp

loyment
Quality Improvement Program

Increase

d Appropriate Treatm

(p

s

ychotherapy or medication

Baseline QI Pro

grams

(p y py 

Usual Care

6 Months

0 10 20 30 40 

% of patients receiving appropria% of patients receiving appropria
Effectiveness of Treatm

Appropriate antidepressant medica

or psychotherapy or psychotherapy

Effect on depression, quality of life

empl t oymen

t

ObservationalInstrumental variables

– i t t i t t QI ins

trumen

t: ass

ignmen

t

to QI program

– affected treatment use, but not outcome
Predicted Outcome at 6 Mo

By Receipt of Treatmen
Alternatives to RCTs

 W k ea

k d i es

igns

– no control group– before

- after

– historical controls

 Potentially strong designs

– restricted cohort

– case-control

 Must be alert for possible confounders Must be alert for possible confounders

– carefully match implementation and control gro– anticipate important confounders: policy shock

differences in access to care, differences in patiand illness severity

 Must include robust qualitative methods Must include robust qualitative methods

characterizing the organizations and prov
Are RCTs the Gold Standa

f I l t ti R for Imp

lemen

t

ation Resear

 Evidence reviews often exclude any study 

not an RCT

– Coch i rane rev

iews

 You want to evaluate implementation at 4 

RCT? non RCT? non

-RCT?

– does randomization make any difference?

 Consequences

– do it randomly: appearance of internal validity, 

qualitative study, journal reviewers accept the d

– match intervention and control: generalizable, n

qualitative evaluation, does not fit the clinical r

paradigm journal reviewers may question the dparadigm, journal reviewers may question the d
Are RCTs the Gold Standa

f Eff ti R for Effectiveness Researc

 Treatment guidelines often exclude or mi

the importance of non-RCTs

–

pati h ll i b i l hi 

ients w

ho enroll

in RCTs can 

be atyp

ica

l: hi

g

motivated, little comorbidity

– care is often atypical: patients are paid to be tre

care is often atypical: patients are paid to be tre

intensity of care, expert clinicians, strong treatmfidelity



Consequences

– treatments in guidelines may not be able to be 

implemented into routine practice implemented into routine practice

– treatments in guidelines may not work in routin– guidelines do not include effective treatments, s

“definitive RCT” has not been done
C li onc usions

 RCTs can be convincing with a large num

sites

– resul b l i h ld 

lts may not be relevant to sites that would n

participated

 High quality observational methods are High quality observational methods are

available

– cohort, case-control, , instrumental variables, & 

propensity

– inclusion criteria, outcome measurement

 B h b fi f b li i h 

Both benefit from robust qualitative meth

 Need to consider the implementation stra

avail bl d t d lik l f d ilable data, and likely confounders

References

Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS: 

Im

p g p p pp y g lementin

g chronic care 

princi

ples and app

l

yin

g format

evaluation methods to improve care for schizophrenia: QUImplement Sci. 2008; 3: 9.

Concato J, Shah N, Horwitz RI: Randomized, controlled trial

ob i l di d h hi h f h d i bservat

iona

l studies, an

d t

he hierarchy o

f researc

h

design

England Journal of Medicine. 2000; 342: 1887-92.

Goldman HH, Frank RG, Burnam MA, Huskamp HA, Ridgel

Normand SL Young AS Barry CL Normand SL, Young AS, Barry CL, Azzone V Busch AB V, Busch AB

Moran G, Lichtenstein C, Blasinsky M: Behavioral health parity for federal employees. New England Journal of Med354: 1378-86.

Schoenbaum M, Unutzer J, McCaffrey D, Duan N, Sherbourn

KB: The effects of primary care depression treatment on pclinical status and employment. Health Services Research. 

1145

-58.

Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith 

J, Miranda J, Carney MF, Rubenstein LV: Impact of disse

quality improvement programs for depression in managed quality improvement programs for depression in managed 

care: a randomized controlled trial. JAMA. 2000; 283: 212
2/9/2012 12:00pm Alternatives to the Randomized Controlled Trial in Implementation Science 

QUERI Implementation Research Presented by: Dr. Young, Alex

Since 'sample size' is a challenge in implementation and improvement science, we suggest use of the 

research network, the Improvement Science Research Network. www.isrn.net

Can you elaborate on robust qualitative methods? Qualitative methods can be used to characterize 

the organizations being studied. These data can inform understanding of implementation success, both 

in terms of uptake and outcomes. It is possible to triangulate using qualitative data, to understand why 

and how implementation worked at particular sites. This is useful for strengthening and disseminating 

the intervention. To help readers understand how implementation can generalize beyond the study in 

question. With small sample sizes, this is particularly important, since success or failure in a small 

sample might be related in part to variation in the organizations, variation that cannot be controlled for 

with randomized assignment. To understand the methods themselves, I would refer you to a fellowship 

or professional training program in this area, or to work with someone that specializes this work.

What would you say to a study design where there was one study site with an intervention and one 

control, and they randomized which got the intervention, and call that a RCT - is this design not 

publishable in any journal? Should they call it observational? I think the randomization in this instance 

does not actually accomplish the goals of randomization in RCTs. It will not balance covariates between 

the two sites. I think to call this an RCT is misleading, since it provides inappropriate reassurance. With 

an n of 2, one could be better finding a pair of matched sites. If the sites are well matched, then 

whether the choice is A or B, or a coin-flip for A or B seems not likely to make much difference.

What would you say to a study design where there was one study site with an intervention and one 

control, and they randomized which got the intervention, and call that a RCT - is this design not 

publishable in any journal? Should they call it observational? See response to the question 

immediately above

What about multiple baseline designs or permutation designs? There are very promising designs in 

this area. This is especially helpful if all sites are to receive the implementation. Then other sites can 

serve as controls. There was a very good recent QUERI Cyberseminar on a similar topic by C Hendricks 

Brown that I would refer you to. They focus on “Roll-Out Randomized Implementation Trials.” Within 

this design class, I would caution you that studies with historical controls or a before-after design are 

thought to be easily biased or confounded.

Great presentation! I think the Concato NJM study may overstate the correlation between RCT and 

observational studies. I've seen a big literature on propensity scoring showing it can result in wrongsigned estimates compared to RCTs (using the same intervention and sample). Can you comment? 

Yes. It’s important to consider what sort of observational study is being discussed. Observational can 

mean many things. The Concato study focused on cohort and case-control designs, with no historical 

controls. Propensity analyses, on the other hand, are fundamentally correlation studies, looking for 

associations, controlling for known confounders. The problem is that important confounders are often 

not known, or not well measured. In this instance, Propensity Scoring is not appropriate. I think some 

people do not appreciate that is as easily biased as correlations in this circumstance.

In the partners in care example, how is what you described different from an RCT randomizing at the 

level of the clinic? Partners in Care did assign to intervention or control at the clinic level. However, the 

outcome of interest was at the patient level. And depression treatment was received in both groups. 

Though it was more likely under the intervention. I would refer you to the Schoenbaum or Wells papers 

on this study.

Any comment on relative merit of regression discontinuity designs vs. RCTs? See response above
2/9/2012 12:00pm Alternatives to the Randomized Controlled Trial in Implementation Science 

QUERI Implementation Research Presented by: Dr. Young, Alex

Can you suggest a design to be used for implementation of a QI project on a nursing unit with the 

goal(s) of the intervention to improve team performance and patient safety outcomes? I would 

suggest having a control group that is similar to the intervention group. So if there are temporal trends, 

policy changes, or unknown factors that affect outcomes over time, then this will be controlled for. So, 

a cohort design. Or, you could do this as a pilot study, with before-after or historical controls, knowing 

that this is a pilot and does not produce definitive outcome results, but can inform future work.

On the study of mh coverage in benefits plans-how did the researchers know about out of pocket 

costs if they could not be submitted to the insurance co? This is important since it could it could affect 

the outcome. Yes. Out-of-pocket costs here were measured based on co-pays, deductibles, and 

payment limits. So, if someone was hospitalized, how much was and was not covered. If there was no 

claim made at all for a service, then out-of-pocket costs would not have been detected. Though these 

plans all provided some mental health coverage, so one would expect claims for most services received.

Regarding the depression study, it was originally presented as an RCT of a QI approach, not as an 

instrumental variable study. Can you discuss why this study could be conceived of in either way, and 

whether there is a "right" way to consider this study? When this study was done, there was more 

interest in studying quality improvement than the effectiveness of services. How studies are 

characterized and justified, and the science itself, is often steered by what the priorities are of funding 

agencies.

I would like to do a study of implementation of a case manager for hospitalized patients with severe 

liver disease at 4 medical centers - what specifically would be the options for a cohort type study to 

look at this with respect to effect on re-hospitalization rates at 6 months? Or should it be patient level 

randomization at each site (more expensive though)? These are design choices for which pros and 

cons should be systematically considered and justified. There are advantages to patient-level 

randomization, however, future implementation is often at the clinic or organizational level, so the 

generalizability and future utility of patient-level results could be less. There are also challenges with 

cohort studies, finding good control groups, and minimizing confounders. You may also want to 

collaborate with an expert in study design or implementation research designs, as you consider your 

options.