Randomized Designs for System Level Impleme

?

1

Trials 

C HENDRICKS BROWN

UNIVERSITY OF MIAMI MILLER SCHOOL OF MEDICI

DIRECTOR, CENTER FOR PREV ENTION IMPLEMENTAT

METHODOLOGY (CE-PIM)

DIRECTOR, PREVENTION SC IENCE AND METHODOLO

G OU R 

P

DIRECTOR, SOCIAL SYSTEMS INFORMATICS, CENTER 

COMPUTATIONAL SCIENCE

CHBROWN@MED.MIAMI.EDU

QUERI Implementation Research Cyberseminar
Acknowledgements

2

y Center for Prevention Implementation Methodology (Ce-PIM) for 

HIV Sexual Risk Behavior (P30-DA027828) NIDA/OBSSR) PI C He

y Scaling p u MTFC in California NIMH: R01MH076158, PI Patti Chay Advanced Center to Improve Pediatric Mental Health Care (P30 M

Implementation Methods Research Group (IMRG) PI John Lands

y Methods for MH/DA Prevention and Early Intervention, NIMH/N

MH040859, PI C Hendricks Brown

• Collaborative Synthesis of Adolescent Depression Trials, NIMH R

22 PI C Hendricks Brown

y NIDA R01 DA019984 A Follow-up of Classroom Services to Preven

Jeanne Poduska, 

QUERI Implementation Research Cyberseminar 

y NIDA R21DA024370, Scaling-up Prevention Services for Early Dru

School Systems, PI Jeanne Poduska
Acknowledgements

C A th

3

o-Authors

John Landsverk, Rady Children’s Hospital Patricia Chamberlain,g 

 Ore on Social Learn

Lawrence Palinkas, USCWei Wang University of South Florida Wei Wang, University of South FloridaLisa Saldana, Center for Research to Practi

QUERI Implementation Research Cyberseminar 
Here is One Perspective on Randomize

fo p e e tat o esea c r Imp

l

ementati

on

Resear

ch

4

SCIENTIFIC PERSPECTIVEy Strategies to implement “evidence-based” p

or programs necessarily involve organizatioor programs necessarily involve organizatiocommunity, or system level interventions.

y If we wanted to formally test implementa

strategies, we would at least on we would , at least on scientif

scientif

grounds prefer to randomly assign imp gg lementation strategies at or

ganizationa

since randomized trials provide the strongeevidence. 

QUERI Implementation Research Cyberseminar 
ANOTHER PERSPECTIVE

5

y R d i d t i l d ’t k i l 

Randomized trials don’t answer key implemquestions, too rigid, little generalizeability external validity...

QUERI Implementation Research Cyberseminar 
Your Vote on Scientific Groun

6

R d i d I l t ti T i l t Ran

domized Imp

lement

ation T

r

i

al

s

at 

Organizational Level would be scientificallfor advancing Implementation Science -- if could be carried out

How much do you agree?

1 - Not at All2 - Somewhat3 - Strongly

QUERI Implementation Research Cyberseminar 
Here are some Practical Concerns abo

R

a do ed at O ga at o a e 

ndomized at O

rganizati

on

al

L

e

7

FINANCIAL ETHICAL and ORGANIZATFINANCIAL, ETHICAL, and ORGANIZAT

PERSPECTIVE

y Carrying out a randomized trial at the organiz

l l i ll i i i level i

s generall

y too expens

ive, as i

t requ

ires m

organizations to achieve sufficient statistical p

y Randomizing at the organization/community 

unlikely to be supported by these organizationcommunities who are averse to withholding efinterventions programs or practices interventions, programs, or practices

CONCLUSIONy Randomized design strategies should rarely if 

used for conducting large scale implementatioevaluations.

QUERI Implementation Research Cyberseminar 
Your Second Vote on Practical Gr

8

O ti l ( i tifi d ) O

n practical (or scientifi

c groun

ds), we

rarely if ever use organization – level randomized trials to test/examine implementation strategies

How much do you agree?

1 - Not at All2 - Somewhat3 - Strongly

QUERI Implementation Research Cyberseminar 
Outline

9

1. Introduction

What Methodologies are Important for the Field of Implementation ScienceCenter for Prevention Implementation Methodology (Ce-PIM)What is the role of a randomized implementation trial?

2 Two approaches: Randomize at a lower Level 2. Two approaches: Randomize at a lower Level

Randomize on Time of Implementation

3. Changing a Non-Randomized Design for Implementation Design to 

Randomized: 

“Timecasting”

4. Illustrative Randomized Design for Implementation Research

Example of the CAL-OH Randomized Implementation Trial

5 Roll 5. Roll-O t Randomi ed Implementation Trials Out Randomized Implementation Trials

Statistically Useful?Community Buy-In?Conduct?

6. ConclusionsReferences at the end

QUERI Implementation Research Cyberseminar 
Prevention Science – for Mental, Emoti

B

e a o a ( ) so de s havi

or

al (MEB) Disorders 

10

State of the Sciencey Conclusion: Lots of Effective Preventive InterventioMethodology for Effectivenessy Conclusion: Rigorous Randomized Preventive Trials 

this Evidence

Implementationy Recommendations: NIH charged with developing m

to address maj gp y or 

g

a

ps the stud

y of dissemination an

implementation of successful interventions.

National Academy of Sciences (2009). Preventing Mental, Emo

B h i l Di d A Y P l P d P B

ehavioral Disor

ders Among Young Peop

le: Progress and Po

QUERI Implementation Research Cyberseminar
Center for Prevention Implementation Methodo

PIM) for Drug Abuse and HIV/Sexual Risk B

11

F d d b NIDA d NIH/OBSSR Fun

d

ed by NIDA and NIH/OBSSR

1. Devel hd lop new met

h

o

d

s f l i 

for Imp

lementatio

2. Partner with Implementation Researchers o

of I i M h d i d i f Innovat

ive Met

h

o

d

s i

n

a

dvanc

ing

Implementation Science

3. Ad di i th Advance discovery i

n th

e P ti f Practice

of 

Implementation by Partnering with CommOrganizations Practitioners and Policy MakOrganizations, Practitioners, and Policy Mak

+ Mental Health, Depression, Suicide, Ser

QUERI Implementation Research Cyberseminar 
Our Methods Discussion Today is Lim

Tri

a s o p e e tat o St ateg 

ls of Imp

l

ementati

on Strateg

12

L t f Q lit ti d Mi d M th d QI d L

ots

of Qualit

ative and Mixed M

ethods, QI d

Landsverk, Brown, et al., 2012.

QUERI Implementation Research Cyberseminar 
Limited Use of High Quality Quanti

Designs for Implementation Resea

13

Review of research literature on behavior change rel

healthy diets. { Among 2,872 studies, including 16 sy

revi l fi t di 

iews, on

l

y five

s

t

udies were appro

designed and/or reported on the ranoutcomes so as to influence 

p y olic

y an

practice.

Schillinger, D. (2010). An Introduction to Effectiveness, Dissemina

I l i R h Imp

lementat

ion Research. P Fl i h d E G ld i d F 

P. Fl

ei

s

her and E. G

oldstein,

eds. Fro

UCSF Clinical and Translational Science Institute (CTSI) Resouand Guides to Community-Engaged Research, P. Fleisher, ed. PClinical Translational Science Institute Community Engagemen

QUERI Implementation Research Cyberseminar 
Randomized Trials In Fact Do Occ

Imp e e tat o l

ementati

on

14 

Child welfare/mental health implementaty

9 338 p g p of 338 studies had a comparison 

grou

p

{ 8 of 9 used a randomized trial

Q li I i H l h CQuality Improvement i

n Hea

l

th Care

y Cochrane Collaboration Effective Practice and 

Organization of Care Review Group (EPOC) R{ 57% exclusively Randomized Trials

Landsverk, Brown, Rolls Reutz et al (2011)Landsverk, Brown, Chamberlain et al. (2012)

QUERI Implementation Research Cyberseminar 
Cluster/Group-Level Randomized T

Pub s ed p e e tat o Sc e 

li

s

hed in Imp

l

ementati

on Sci

e

15

~ 44 / 193 Research and Study Protocols 44 / 193 Research and Study Protocols 23% =y Some involve sizeable numbers of organizations

40 counties in California (now 53)Wang, Saldana, Brown, Chamberlain. 2010, Implementation 

y M ll b f i ti Many use small numbers

of organ

izations

6 oncology clinicsBoveldt et al 2011 Implementation Science et al., 2011 Implementation Science

y Many randomize at lower levels than the organization

37 first-grade classrooms in 12 elementary schoolsPoduska, Kellam, Brown et al. 2009 Implementation Science

QUERI Implementation Research Cyberseminar 
Strategy 1: Randomize at a level lower 

o ga at o rganizati

on

16

y When does this work? 3 requirements1. “U it f I l t ti ” t t t 

“U

nit 

of Imp

lement

ation” you want t

o

stu

actually at a level below the organization 

Ex: Coaching of teachers to deal with agg

behavior

Classrooms in a school randomly ass

different coaching conditions

2. Amount of leakage of intervention is acc

QUERI Implementation Research Cyberseminar 
Lower Level Implementation Strate

Rep ese ted 3 e e s resented in 3 

L

evel

s

17

Education Health Health Social 

Servic

School District

Health Organization

AHRQ, VA, SAMHSA

County 

District

Level 3 School Clinic Health

Organization

Agency 

Level 2 Level 2 Classroom: Clinician: Clinical Service Type of Implementation

Classroom: Coaching of an EBP

Clinician: Implement EB Practices

ClinicalPractice: Reminders

Service ProviderSupervisTraining

Level 1 Child Patient Clinician Client 

QUERI Implementation Research Cyberseminar 
Strategy 1: Randomize at a level lower 

o ga at o rganizati

on

18

y Wh d thi k? Thi d i t When does thi

s work? Thi

rd requ

irement

3. Level 2 units (classes) are similar to one a

within Level 3 Groups (schools) OR Implem

is Strong

QUERI Implementation Research Cyberseminar 
When does Randomizing at a Lower Level Affect Really Benefit St

Similar Classrooms / Strong Intervention

19

QUERI Implementation Research Cyberseminar 

Brown & Liao, A J Commun Psychol, 1999
Magnitude of Effect

There are very few Broad Street Pump Handles Left to 

20

John Snow’s Map of London 18 d 49 proposed1854 500 deaths 1855 ~ 0 deathsRemoval of Pump Led to ImmediateReduction in Cholera DeathsQUERI Implementation Research Cyberseminar 
Strategy 2

21

y M ti ti Motivation

QUERI Implementation Research Cyberseminar 
Strategy 2: Using “Roll-Out” to Overcome the Ch

Small Numbers of Organizational Leve

22

y The Problem: Effective Sample Size is # y Implement in State 1 No Implementation 

N 100 = 000 , N 100 = 000 ,

QUERI Implementation Research Cyberseminar 
The Two Major Challenges with 

O ga at o a /Syste e e te e 

rganizati

on

al/System

L

evel Interve

23

y S ll b f O i ti l 

Small number

of Organ

izations means low

power, even with very large individual sam

y High degree of organizational differences l

power

QUERI Implementation Research Cyberseminar 
Randomization can Sometimes Strengthen an 

Design s Inferences, without Losing Anyth

’s Inferences, without Losing Anyth

24

y Ill t ti f C it Illustration

of 

a Community-L l I l 

Level Imp

lem

Initial Design

QUERI Implementation Research Cyberseminar 
Hawkins NG et al., AJPM 2007: “M

Base e, te upted e Se e

line, Interrupted Time Serie

25

QUERI Implementation Research Cyberseminar 
A “Timecasting” Nonrandomized Designs 

Dissemination/Implementation RE-AIM comp

26

1 . Broadcast of an Intervention

Standardize Invitation and See Who Comes

2. N t f I t ti Narrowcast of an Intervention

Social Marketing to a Targeted Audience

3 . “Timecast Timecast” of an Intervention

Multiple Baseline design = Interrupted Time Series

o Repeated measures over time of a community outcomeo Introduce an implementation to a community midway througho Check whether community outcome differs before and after intr

QUERI Implementation Research Cyberseminar 
Big , lan et al., AJCP 2006 

27

QUERI Implementation Research Cyberseminar 
Other Communities

28

QUERI Implementation Research Cyberseminar 
Inferential Limitations with this D

29

y Wh t if d E F t H t 

What if and Exogenous Factor Happens

at one

Times of Transition?

Recession in CA 

– CAL

-40 Randomized Implementatio

Dozens of deaths in Mexico – Implement of HIV Prev S

y What if you Select the Most Promising Commu

Work with First?

y What if there are only a small number of commy Hard to Conclude that Implementation Cause

QUERI Implementation Research Cyberseminar 
Turning a Multiple Baseline Design int

Randomized Experiment: Roll

-Out D

30

ROLL

-OUT DESIGN

y Divide Available Communities into Comparaby S i O f ll C i i Start Measur

ing Outcomes for

All Commun

i

tie

y Randomly Assign Each Comparable Batch to 

th Di i i B i “ he Dissemination Begins “Ti i mecasting”

y At the end, ALL Communities Are Exposedy Analysis Uses All Communities and All Times

Ù Communities Still Serve as Own ControlsÙ Communities Compared by Exposure Status Across Tim

QUERI Implementation Research Cyberseminar 
Roll-Out Randomized Trials for Disse

R h (B t l 6 Research (Brown

et 

al., 2006 200

31

Population Equivalent Subsets 

that are Ordered 

Randomly

1

5

R

2

3

4

5

Time of Transition in Dissemination Randoml

y Deter

QUERI Implementation Research Cyberseminar 
Timingp (

 of Implementation ( 0 t

32

y 1 0 0 x xxxxy 2 0 0 0 x x x xy 3 0 0 0 0 x x xy 4 0 0 0 0 0 x xy 5 0 0 0 0 0 0 x

QUERI Implementation Research Cyberseminar 
Some Reasons to Consider Random

Ass g e t i

gnment

33

y St ti ti l R St

ati

stical Reasons

Statistical Precision/Power

Reduces Bias

Dynamic Wait

-Listed Design (Roll

-Out)

Brown, Clinical Trials , 2006

QUERI Implementation Research Cyberseminar 
Randomized Assignment Can be F

34

y Persony Place/Groupy Time Random Assignment of Schools to Different Tim

Eff i T i l Effectiveness Trial

Brown et al., Clinical Trials, 2006

Random Assignment of Counties to Different T

Implementation 

Brown et al., 2008 Drug & Alcohol Dependence

QUERI Implementation Research Cyberseminar 
How Random Assignment Reduces Bia

L R ong un 35

y Example: mPowerment Community Intervent

Treat Control Treat Contro

Yr 1

R

Yr 2

. R..

Yr 10

R

QUERI Implementation Research Cyberseminar 
Implication of Roll-Out Designs for Com

Resea cr

ch

36

y #Units (communities) to get randomized are Large but few are avaA Single Trial with a Small Number of Communities is Nearly Always USEQUENTIALy Randomize small numbers of communities nowy Randomize small numbers next yeary

...

y Randomize small numbers in following yearsy Combine results across the years y Eventually you obtain sufficient statistical power • Brown et al., Ann Rev Public Health 2009

• B t l Brown

et 

al.,

Prev S i 2011 S

cience 2011

QUERI Implementation Research Cyberseminar 
Randomized Roll-Out Implementat

37

A R d i d T i l W th D i i I l Are Ran

domized T

r

i

al

s Worth D

oing i

n Imp

lem

Probably NOT Without Some Changes in 

Practice.

Main Idea in this Presentation is to 

Out Designs

y Wh t d th l k lik ? What d

o they look like? -- CAL

-OH T i l OH T

r

i

al

y Are They Scientifically Useful?y Can You Get Community Buy-In?y Can They Be Conducted, Adhering to the Proto

QUERI Implementation Research Cyberseminar 
Buy-In: Will Communities Agree t

R

a do ed to o ndomized to 

R

oll-Out a s

Tri

als

38

y Fair Systemy Make Sure there Are Equal Advantages for Each R

Assignment

Eff i T i l Effect

iveness T

r

i

a

l:

Roll-Out Trial: 

Randomly Assign to Time that Intervention Begins, so ALL Communities Do Get Intervention

Advantage of going early: Thought to be usefuAdvantag gg y 

e of 

goin

g later: Intervention may b

Dynamic Wait-List: Brown et al., 2006, Clini

QUERI Implementation Research Cyberseminar 
Will the Idea of Randomized Roll-Out 

Be Accepted for Implementation Rese

Be 

Accepted 

for Imp

l

ementati

on

Rese

Well as for Effectiveness? 39

{F I l t ti f E i i 

For Imp

lement

ation

of an Emp

i

r

ic

Supported Program, All Communithe SAME Evidence-Based ProgramDifferent Implementation

{ Advantage for Waiting -

Wait liste

communities may get better 

implementation

QUERI Implementation Research Cyberseminar 
Cal-OH Imp 

lementation Trial Ex

40

y E id v ence-B d P Based Program – M ltidi i 

Multidimensio

Treatment Foster Care (MTFC)

y Two l i S i i d C 

Implementation Strategies aimed at C

California and Ohio

Ù Communi l ( ) ity Development Team (CDT)Ù Standard County Implementation (Stnd)

QUERI Implementation Research Cyberseminar 
Two-Arm Trials

Effectiveness vs Implementat

41

Existing 

Implementation 

Supports for 

MTFC 

Implementation 

Supports for Standard 

County, Agency, 

Group Home 

Supports for 

County, Agency, 

Clinicians, Parent

Sta da d 

Implementation 

Supports for 

County

MTFC 

Intervention

MTFC 

Intervention 

Control 

Condition

y

Intervention

Youth Youth

Y h out 

QUERI Implementation Research Cyberseminar 
Detailed Illustration of the CAL-OH Ran

I l t ti T i l Imp

lement

ation T

r

i

al

Objective : Test the effectiveness of the Commun

42

Development Team (CDT), a theory driven modpromote the adoption, implementation, and susfor delivering the evidence-based MultidimensioTreatment Foster Care (MTFC) intervention in Ccounties that are not already using MTFC relaticounties that are not already using MTFC, relatiStandard County Implementation (Stnd). 

Method: Randomize counties into 6 equivalent c

which receive CDT, other 3 receive standardimplementation.

Measures: Time it takes to adopt recruit staff t

: Time it takes to adopt, recruit staff, t

place youth in MTFC homes.

QUERI Implementation Research Cyberseminar 
Initial CAL-4

COHORT 1 COHORT 2 COH

4 

43

40 CA Counties

CDT

Stnd

26 Wait 

W it 

13 Wait 

LIsted

QUERI Implementation Research Cyberseminar 

LIsted Wait Listed LIsted
3. Addressing Community Concerns with High Quality Alternatives to the Traditional Random

Issues in t e h CAL-40 Des gi n

44

1. Atccep ance of th D i C l t f the Design was Complete

2. Some Counties Were Not Ready to take Pa

Moved Up Counties from Next Cohort

remained in same implementation conditio

Chamberlain et a

QUERI Implementation Research Cyberseminar 
3. Addressing Community Concerns with High Quality Alternatives to the Traditional Random

45

Consort Diagram

Assessed for eligibility, n=58. 

Excluded n=18.- Already had implemented MTFC, n- Fewer than 6 youth per year, n=8 -Los Angels County, n=1 

Formed 6 equivalent clusters by matching background, n=40. 

Randomized the clusters to 3 cohorts 

and 2 conditions, n=40. 

Cohort 1, n=12. 

CDT, n=6 - Accepted n=4

Cohort 2, n= 14. 

IND, n=6 

Accepted n=4

Cohort 3, n= 14. 

CDT, n=7 

Accepted n=4

I- 

CDT, n=7 - Accepted n=7

IND, n=7 

- Accepted, n=4 - Accepted n=7

Declined, n=2 

- Accepted, n=4

-Declined, n=2 

- Filled by cohort 2 counties, n=0 - Received, n=4 

Accepted, n=4Declined- n=1 

- Pending, n=2

- Expect to be moved to cohort 2, n=2 - Expect to be filled by declined cohort 1

- -- - bc- n

- Accepted, n=7- Moved to cohort 1, n=2 - Expected to be filled by cohort 3 counties, n=2 - Expected to receive n=7

- Accepted, n=7- Moved to cohort 1, n=0 - Expected to receive, n=7 - Invited to ‘go early’, n=3 (no 

2 counties moved up counties accepted by declined cohort 1

counties, n=2 - Expect to receive, n=7 

receive, n 7 n- Invited to ‘go early’, n=3 (2 counties accepted invitation) 

counties accepted

invitation) 

2 counties moved up. 

QUERI Implementation Research Cyberseminar 
Exogenous Factors in a Roll-Out 

R i R d i d T i l D i R 

46

Running Randomized Trials During a RecessSolution: Added 13 

 counties in a second stat

equivalent inclusion/exclusion criteria

QUERI Implementation Research Cyberseminar 
Su a y: a do ed p e e tat o 

mmary: Randomized Implementation 

47

Randomized Imp , p p

lementation Trials in NOT a Panacea, but does have a place esp

beginning

1. Some circumstances allow randomizing at lower level

“Unit of Implementation”

Limit implementation leakageSomewhat comparable lower level units

2, Roll-Out Randomized Implementation Trials

y Community Standpoint

Everyone gets active intervention with implementationFair assignment of when intervention occurs

y Scientific Standpoint

Can reduce bias, protect against external events, and improve statistical power, 

expanded over time

Valuable when there is LARGE Variation, at early stages and with highly variabl

y RANDOMIZE EVEN FROM THE BEGINNING, EVEN IN PAIRS OF 2 BUT CO

TIME

y Need to extend work on statistical properties and conduct of such trials

QUERI Implementation Research Cyberseminar 
Related Papers

48

Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Im

Service Sectors. Administration and Policy in Mental Health and Mental Health Services Resea

Brown, C. H., P. A. Wyman, et al. (2007). "The role of randomized trials in testing interventions for t

suicide." International Review of Psychiatry 19(6): 617-631.

Brown, C. H., P. A. Wyman, et al. (2006). "Dynamic wait-listed designs for randomized trials: new d

youth suicide." Clinical Trials 3(3): 259-271.

Brown, , , g, p , g , , , , p p 

 Kellam, Muthen, Wang, Kaupert, Ogihara, Valente, McManus, Pantin, Szapocznik (accepted

Effectiveness and Implementation Research: Experiences of the Prevention Science and Method

Brown CH. Design principles and their application in preventive field trials. In WJ Bukoski and Z Sl

Drug Abuse Prevention: Theory, Science, and Practice. New York: Plenum Press, pp. 523-540, 

Brown, CH, Berndt D, Brinales JM, Zong X, and Bhagwat D. Evaluating the Evidence of Effectivenes

Interventions: Using a Registry System to Influence Policy through Science. Addictive Behavior

Brown, CH, Ten Have TR, Jo B, Dagne G, Wyman PA, Muthén BO, Gibbons RD. Adaptive Designs i

Annual Review Public Health, 30: 17.1-17.25, 2009.

Brown CH, Sloboda Z, Faggiano F, Teasdale B, Keller F, Burkhart G (Forthcoming). Methods for Synthesiz

Moderation Effects Across Multiple Randomized Trials. To appear in Prevention Science.

QUERI Implementation Research Cyberseminar 
Related Papers

49

Cb , , hamberlain P, Saldana L, Bro C, wnH Leve LD. Imp oo C C 

lementation of MTFC in C

Randomized Trial of an Evidence-Based Practice. In M Roberts-DeGennaro Empirically Supported Interventions for Community and Organizational CBooks, Inc, In Press.

Chamberlain, P., Saldana, L., Brown, H., & Leve, L. (2010). Implementation of M

A Randomized Trial of an Evidence-Based Practice. In M. Roberts-DeGennar(Eds.), Using Evidence to Inform Practice for Community and Organization(pp.218–234). Chicago: Lyceum. 

Chamberlain, P, Marsenich L, Sosna, T, et al. (accepted for publication). Three 

models for scaling up evidence-based programs.

Flay B Biglan A et al (2005) Standards of E Flay, B, Biglan A, et al. (2005). Standards of Evidence: Criteria for Efficacy vidence: Criteria for Efficacy Effe

, Effe

Dissemination, Prevention Sci, 6, 152-175.

Landsverk J, Brown C, Rolls Reutz J, Palinkas L, Horwitz S. Design Elements in 

Research: A Structured Review of Child Welfare and Child Mental Health Stu

lfare and Child Mental Health Stu

Administration and Policy in Mental Health and Mental Health Services Re

QUERI Implementation Research Cyberseminar 
Related Papers

Wang, Saldana, Brown, Chamberlain (2010). Factors that influenced county system leaders to implement an 

50

g, , , y y p 

a baseline survey within a randomized controlled trial. Implementation Science. 

Chamberlain P, Saldana L, Brown CH, Leve LD. Implementation of MTFC in California: A Randomized Trial 

Practice. In M Roberts-DeGennaro & SJ Fogel (Eds.) Empirically Supported Interventions for Commun

Change. Lyceum Books, Inc, In Press.

Aarons, Horwitz, Hurlburt, Landsverk (accepted for publication). Advancing a Conceptual Model of Evidenc

Implementation in Public Mental Health and Child Welfare Sectors

Landsverk, Brown, Chamberlain, Palinkas, Horwitz (2012). Desigy p n and Analysis in Dissemination and Imple

R Brownson, G Colditz and E Proctor (Eds.), Dissemination and Implementation Research in Health: TPractice, Oxford University Press.

Brown CH, Kellam SG, Kaupert S, Muthén BO, Wang W, Muthén L, Chamberlain P, PoVey C, Cady R, Valent

Pantin H, Szapocznik J, Czaja S, McManus J. Partnerships for the Design, Conduct, and Analysis of Effe

I l i R h E i f h P i S i d M h d l G (I ) S Implementation Research: Experiences of the Prevention Science and Methodology Group. (In press). Spartnerships, Administration and Policy in Mental Health.

BooksValente ( 0 0), o o o , o 2010), Social Networks and Health: Models, Methods 

Applications

Palinkas and Soydan (2010). Translation and Implementation o

Based Practice in Social Work: A Strategy for Research 

QUERI Implementation Research Cyberseminar 
1/12/2012 12:00pm Randomized Designs for System Level Implementation 

Trials

QUERI Implementation 

Research

Brown, Hendricks

1) Regarding your comments on slide 35: can you explain why the left-hand-side method 

"confounds implementation with community readiness"?

If we only select those communities that are "ready" to offer the intervention first, then the less ready 

communities will always serve as controls, therefore perfectly confounding intervention status with 

readiness.

2) In the time cast design, how do you deal with a group that doesn't have a stable baseline when 

it's their turn to get the intervention?

This should go in the modeling, for example by modeling the developmental trajectory at the 

organization level in growth models, allowing a separate parameter for the variation in this baseline 

period, in pieces, for example, pre-intervention, during intervention, and after intervention. In growth 

modeling there are 3 standard ways the implementation can change outcome, the simplest is that there 

is just a difference in the mean of the growth trajectory, i.e., intervention causes more rapid change. The 

second is that the intervention affects the slope differently as a function of baseline, e.g., it may be that 

only those organizations that have a climate more inducive to implementation are ones that benefit from 

an implementation strategy. Third, there may be changes not just in the mean change or slope, but also 

in the variance, as this question suggests (i.e., variability may be important not just in baseline).

3) Is stepped wedge RCT another term for your roll-out RCT? 

Yes. This idea was apparently first used in Cook and Campbell, 1979, but there are lots of times this 

design has been used without the name stepped wedge. Sometimes people use the name "stepped 

wedge design" even if timing is not randomized, which to me might better be called a multiple 

baseline study.

Here are a couple of relevant papers:

Hussey MA, Hughes JP: Design and analysis of stepped wedge cluster randomized trials.

Contemparary Clinical Trials 2006. doi: 10.1016/j.cct.2006.05.007

Brown CH, Wyman PA et al. (2006). Dynamic wait-listed designs for randomized trials: new designs for 

prevention of youth suicide. Clinical Trials, 3, 259-271.

4) How should one determine the time period between additional rollouts, and how does this 

affect the analysis. Does it have anything to do with the time it takes for the intervention to be 

fully implemented? Or is that an independent issue? 

In the work that I have done, the timing depends more on logistical issues, i.e. how long it takes to train 

or implement, but it clearly does affect power. In the work of Brown et la. (2006) Clin Trials and 

following, we have actually used every new training that occurs as a factor in the analysis, even if there 

are multiple trainings that occur at a single school once randomized.
1/12/2012 12:00pm Randomized Designs for System Level Implementation 

Trials

QUERI Implementation 

Research

Brown, Hendricks

5) Also, what are the implications of a larger number of rollouts with smaller number of 

organizations in each wave, versus a smaller number of roll-outs with a larger number of 

organizations per roll-out?

Our calculations in Brown et al. (2006) Clinical Trials show pretty clearly that there is a large gain going 

from a standard wait-list (2 cohorts), to even 3 or 4 cohorts. But an additional gain with lots of time 

points that get randomized is the opportunity to control bias that might come from external factors.

6) So in summary, is it correct that having lower level randomization has more power than higher 

level randomization (as long as the variability between lower levels is low)

Yes, the statistical power for lower level randomization is very often much higher, even with a small 

number of lower level units per higher level. Also see Brown & Liao, 1999.

7) Do you have any suggestions for determining on which criteria to pair experimental and control 

groups (e.g., socio-economic status, urban vs. rural setting)? How do you decide which are the 

most important factors for determining similarity?

I would start with theoretically driven factors, and then add practical ones; the ones we used in the 

suicide roll-out effectiveness trial were high school vs. middle school and number of referrals in the 

previous year regarding suicide, the variable we wanted to change through our intervention. It would be 

possible to include some measures of context such as "readiness" but probably this is not really helpful 

unless you are confident you can measure this precisely enough.

8) Implementation studies in healthcare delivery often involve organization-level interventions, 

such as introduction of a new care model or other reorganization of care delivery processes, or 

delivery of group-level education and attitude-change strategies. The desired practice change is 

at the level of the individual clinician, however. Do we gain anything in statistical power by 

analyzing at the clinician level but adjusting for the organization-level clustering?

There are some important limitations about this clinical level analysis that adjusts for organize level 

clustering.

This won’t allow for evaluation of the implementation components aimed at the organizational level.

If there is randomization at the level of the organization but analyzed at the level of the clinician, there 

can be a big cost in power relative to a design that randomized clinicians within organizations. But that’s 

not always possible.
1/12/2012 12:00pm Randomized Designs for System Level Implementation 

Trials

QUERI Implementation 

Research

Brown, Hendricks

9) In your experience are grant reviewers aware of these issues and accepting of “weaker” designs 

(such as roll-out or wait-list designs) or do they display more of a knee-jerk reaction that 

anything other than an RCT is flawed, despite the fact that randomized designs are essentially 

infeasible for the types of research questions you are addressing? How can we educate 

reviewers?

Good point, I think as implementation scientists we need to develop a strategy for this. One of the things 

I am trying to do is have the IOM commit do a new committee on Implementation Science. This might be 

of help. I think we need some key publications that lay these out for the field. Sound like a debate that 

would be interesting for Implementation Science?

10) Your work and many of the examples you cited involve communities, schools and other 

entities. Are there any differences to consider as we plan studies within a large healthcare 

system like VA, consisting of hundreds of hospitals and clinics?

I do think there are some major differences, but don’t feel fully clear about them. One of the things that I 

found that distinguishes the settings that Patti C uses for foster care (which in my untutored view may be 

somewhat like the VA), and prevention in schools, for example, is that the programs that get 

implemented have different roles: for prevention the programs are often brought in to address distal 

outcomes that are not the primary mission of the organization; for social and health service systems we 

are often focused on the primary mission. This means the strategies for keeping such programs 

sustained over time, as well as adoption and fidelity, are often different.