r 

re 

VA HSR&D Center fo VA HSR&D Center fo

the Study of HealthcaProvider Behavior 

Pre -Implementation and Implementation and

Hybrid Effectiveness-Implemen

Study Di es gns

Alison B Hamilton Ph D M P H Alison B. Hamilton, Ph.D., M.P.H

VA Center for Implementation Practice & Resear

VA Greater Los Angeles Healthcare Syst

Enhancing Implementation Science in VEnhancing Implementation Science in V

May 17, 2012
Acknowledgements Acknowledgements 

Many thanks for content: 

• Mark Bauer • Amy Cohen • Deborah Delevan • Geoff Curran • Hildi Hagedorn • JoAnn Kirchner • Brian Mittman • Cheryl Stetler • Alex Young 
Today s ’Presentation Presentation 

 Review characteristics of efficacy tria

effectiveness trials, and implementatio

 Describe pre-implementation (QUERI 

designs and methods 

 Provide examples of Step 3 studies 

 Describe hybrid designs 

 Provide examples of each type of h

design

 Raise hybrid design considerations 

 Questions 
Today s ’Presentation Presentation 

 Review characteristics of efficacy tria

effectiveness trials, and implementatio

 Describe pre-implementation (QUERI 

designs and methods 

 Provide examples of Step 3 studies 

 Describe hybrid designs 

 Provide examples of each type of h

design

 Raise hybrid design considerations 

 Questions 
Clinical Efficacy Trials Clinical Efficacy Trials

Address whether a treatment improves outco

under controlled conditions 

 Outcomes: clinical 

(e.g., symptoms, side effects, hospitalizatio Process measures not considered 

 Levels of anal

ysis: 

patient, clinical unit 

 Favor internal validity: are changes attri

Favor internal validity: are changes attri

the intervention and nothing else? 
Clinical Effectiveness Tr

Clinical Effectiveness Tr

Typically follow efficacy research trials 

 Outcomes: typically clinical 

( g , e.

g., s

y

m

ptoms, side effects, hos

pitaliz

 Process measures considered secondary 

 Levels of analysis: patient, clinical un Favor external validity: “real” clinics; 

and more diverse sam

ples 

From Curran, G., Bauer, M
Implementation Resear

Implementation Resear

Enhance uptake of established clinical 

interventions 

 Outcomes: process measures 

(e.g., rates of adoption, utilization of 

context) 

 Clinical outcomes data may not be of pr

interest since intervention is established 

 Levels of analysis: provider, clinical 

facility 

From Curran, G., Bauer, 
Today s ’Presentation Presentation 

 Review characteristics of efficacy tria

effectiveness trials, and implementatio

 Describe pre-implementation (QUERI 

designs and methods 

 Provide examples of Step 3 studies 

 Describe hybrid designs 

 Provide examples of each type of h

design

 Raise hybrid design considerations 

 Questions 
Pre-implementation 

(QUERI Step 3) 

Implementation Effectiveness Studies Efficacy 

Studies p Research Studies 

Studies 

Clinical Research-Implementation P
QUERI Step 3 QUERI Step 

3

 Measure and diagnose quality/perfo

gaps 

 Importance of systems-thinking 

(i.e., understanding social systems) 



“Diagnosis” 

 Results in the identification of actionab

contributing to performance gaps and contributing to performance gaps and 

reasons for failures in implementing in

(QUERI website) (QUERI website) 
QUERI Step 3 QUERI Step 

3

3A. Measure existing practice patterns 

outcomes across VA and identify vaoutcomes across VA and identify va

from evidence-based practices (“qu

performance gaps performance gaps ) ”

)

3B. Identify determinants of current pr3C. Diagnose quality/performance gap3D Identify barriers and facilitators to 3D. Identify barriers and facilitators to 

improvement 
Pre-Implementation Stud

Implementation Stud

 Ob ti l Observational, cross-secti l d ona , and 

longitudinal studies 

 Methods to consider: 

 Measurement of practice variation via 

Measurement of practice variation via 

review, administrative database analy

 Modeling determinants of clinical prac

Modeling determinants of clinical prac

 Semi-structured interviews, focus gro Surveys Surveys  Policy/archival review Expert panels Delphi consensus Expert panels, Delphi consensus 
t t t t t 

QUERI Step 3 Example: 

Diabetes Mellitus-QUERI

Purpose of St ep 3 study: To understand the circumstances of preve

loss among patients with diabetes; focused on timing of laser ekey issue in preventing visual loss (Krein et al., 2008) 

 Physician reviewers examined and analyzed medical records us

criteria regarding optimal timing of photocoagulation 

 Identified patients whose visual loss was considered preventabl

treatment. 

 Found that two-thirds of cases were associated with problems re

surveillance of those with identified disease, including inadequadelays in treatment scheduling, or unexpectedly rapid disease p

 Results identified a lack of close follow-up of those with known d

potentially important gap in quality of care. 

 High priority issue (Step 1) + evidence to support a change 

performance measure (Step 2) + identified gap in quality of performance measure (Step 2) + identified gap in quality of 

3)ÆIMPLEMENTATION (Steps 4/5/6) 
t

QUERI Step 3 Example: 

HIV/Hepatitis QUERI

Purposes of Step 3 study: To identify p ossible gaps in HIV testing in 

understand the source of gaps in care and discern facilitators thcurrent practice (Goetz et al., 2008) 

 Reviewed VA policies regarding HIV testing  Surveyed providers' practices and attitudes regarding HIV testin

facilities 

 Conducted systematic review of 62 studies of HIV testing practic

b i arr ers, etc. 

 Note: some evidence for Step 3 can come from prior/publish

 Found organizational barriers to testing (policies)  Found provider barriers to ordering HIV tests (survey)  Found that higher acceptance rates were associated with confid

protections, and the provider's belief that testing would be benefreview) 

 High priority condition (Step 1) + evidence-based guideline

(Step 2) + identified gap in quality of care (Step 3)Æ IMPLEM(Step 4) (Step 4) 
Today s ’Presentation Presentation 

 Review characteristics of efficacy tria

effectiveness trials, and implementatio

 Describe pre-implementation (QUERI 

designs and methods 

 Provide examples of Step 3 studies 

 Describe hybrid designs 

 Provide examples of each type of h

design 

 Raise hybrid design considerations 

 Questions 
Hybrid Designs Hybrid Designs

Concept introduced in 2008; presented subsequenConcept introduced in 2008; presented subsequen

published in 2012 

From 2008: Hybrid study design should: 

▪ Involve the most realistically rapid timeline given 

complexity of the implementation program complexity of the implementation program, 

▪ Focus during the study on progress and identify b

and actual influences on the progress and effec

i l t ti ff t th h th f f implemen

t

ation effor

ts through the use of 

forma

evaluation, and

▪ Plan action durin

g the stud

y, as needed based on 

data, to refine the change intervention, resolve mbarriers, and enhance available facilitators, in or

optimize: optimize: 
a r

Hybrid Designs (2008 coHybrid Designs (2008 co

 P li re minary use of th e implementati on str

achieve or at least assess its potential; 

 Th l f li i ll i f l t j t 

The goal of clinically meaningful, not just 

significant, evidence-based practice; 

 U d t di f th bl k b f i l 

Understanding of the black box of implem

including cost-benefit; 

 Id tifi ti f t t di h Identification of outstanding research que

and 

 De elopment of eplicable implementat

Development of a replicable implementat

program. 
Implementation Effectiveness Studies Efficacy 

Studies p Research Studies 

Studies 

Spatially speaking, our Hybrids “go” i

From Curran, G., Baue

“Newer ” Clinical Research Clinical Research-Impleme

Impleme

Pipeline
Some D iti efi nitions

• Clinical Intervention: Clinical initiative, 

manipulation, change to be introduced into 

h lth ea care venue

– e.g., collaborative care for depression– May include health promotion or delivery system interv

• Implementation Intervention: “A single meth

technique to facilitate change” (QUERI Glos

– e.g., automated clinical reminder, performance feedbac

• Implementation Strategy : “An integrated se

bundle, or package of [implementation]interventions” (QUERI Glossary) 
Clinical 

Eff ti Impleme

Effectiveness 

Research 

Impleme

Rese

Hybrid 

Type I 

Hybrid 

Type II 

Hybr

Type 

Types of Hybrids Types of Hybrids

Hybrid Type I: Hybrid Type II: 

test clinical test clinical intervention, intervention, observe/gather study information on implementation 

implementati on i t ti intervention 

From Curran, G., Bauer, M., Stetler, C., Mittman, B. 

Hybrid Tyimplemeninterventioobserve/ginformatio

i t n erventi o

outcomes 
Alternative Look at Hybrid Ty

Alternative Look at Hybrid Ty

Focus Implementation 

Clinical 

Intervention 

Yes 

Yes Hybrid Type II Hybrid Typ

No Hybrid Type III Hybrid Type III Observation

Observation

From Curran, G., Baue
And one more alternativ

And one more alternativ

Study Characteristic 

Hybrid Type 1 Hybrid Type II Hybr

Research Research Primary Question Primary Question: Primary Questions Primary Questions: Prim

Prim

Questions Will a clinical Will a clinical treatment Whic(examples) treatment work in this 

setting/these patients? Secondary Question: What are the potential barriers/facilitators to a treatment’s implementation? 

work in this setting/these patients? Does the implementation method show promise? 

betteimple

cli i 

li

nic

WhiccompcriticSecoIs the 

eff 

ffec

settin

From Curran, G., Baue
Some Important Questions to be Ad

Some Important Questions to be Ad

• What clinical intervention or implementation barrie

problems emerge early on? problems emerge early on? 

• In what ways are clinical intervention effects sensit

implementation process factors? 

• What changes to the implementation strategy or th

What changes to the implementation strategy, or th

clinical intervention, could be made to improve upt
A Critical Hybrid Component: 

Evaluating the Trial Process

• Process Evaluation: 

– Identify influences on process of implementation 

clinical intervention prior to, during, and/or after 

– No data fed back during study 

– Typical of Type 1 designs 

• Formative Evaluation: 

– Identify influences on process of implementation 

clinical intervention prior to, during, and after stu

– Data used to optimize implementation or clinica

intervention processes during study

– T i l fT Typical of 

Types 2 & 3 d i & 3 

designs 
Hybrid Type I Designs Hybrid Type I Designs

• Definition: 

– Test clinical intervention, observe/gather information 

implementation implementation 

• Description: 

– Clinical effectiveness trials with added 

process evalu

implementation 

• I di ti Indications: 

– Some effectiveness data available, clinical intervent

t t l ti idl to move 

toward i mplemen

t

ation more rapidly if k 

key 

implementation factors identified
H b id T I E Hybrid Type I Example 

The Rewarding Early Abstinence and Treatment Participation The Rewarding Early Abstinence and Treatment Participation (Hagedorn et al.) • Clinical Intervention: Incentive intervention in SUD treatme• Why yp T e I? 

• Few effectiveness trials and none with a large sample • Obtaining clinical funds for incentives was not feasible 

evidence specific to VA. 

• Main aim of this study was to demonstrate effectivene

population 

• Why not just an effectiveness trial? 

• Main goal of research agenda is to support broad imp• Inclusion of process evaluation would inform future im

• Process evaluation measures 

• R h T Ob ti Research Team Observation Log: 

• Record details of interactions with staff particular

on reactions of staff to the intervention, barriers t

recommendations for improvements recommendations for improvements. 

• Data NOT used to optimize implementation 
t

Hybrid Type Hybrid Type II Designs

• Definition: 

– Test clinical intervention and study implementation s

• Description: 

– Effectiveness and implementation trial with formative 

throughout to: throughout to: 

 Identify contextual influences on clinical inte

and implementation throughout 

 Collect and analyze data that will maximize 

the intervention throughout the study (tailori

• I di i Indications: 

 Robust clinical intervention data available Barriers and facilitators data available Barriers and facilitators data available 
Hybrid Type Hybrid Type II Example

Enhancing Enhancing QUality QUality of care In Psychosis (EQUIP of care In Psychosis (EQUIP -2(Young, Cohen, Hamilton, et al.; VA HSR&D QUER• Clinical Intervention: Chronic care model • Why Type II? 

• Evidence-based practices for patients w

mental illness are known but mental illness are known but 

• No multisite studies have substantially impro

care for schizophrenia within the context of ueffecti veness study)

• Known barriers and facilitators to uptake • Need to study our implementation appro

Need to study our implementation approincrease uptake of EBPs 

• Our approach: evidence-based quality impro
Hybrid Type Hybrid Type II Example (cont 

II Example (cont

Design 

 Clustered, clinic-level controlled trial  Enrollment 

 4 VISNs, 8 clinics  201 staff (clinicians + administrators)  801 patients 801 patients 

• Formative evaluation measures 

• Organizational readiness and staff burno

pre/post pre/post 

• Semi-structured interviews pre-, mid-, an

implementation

• Fieldnotes, logs, minutes • Evaluation data WAS used to optimize im

Evaluation data WAS used to optimize im
Hybrid Type III Designs Hybrid Type III Designs

• Definition: 

• Test implementation strategy, observe/gather 

information on clinical intervention and outcomes

• Description: 

• Implementation trial with formative evaluation 

throug

hou

t, plus eval ti ua on of h lth f 

health out comes

• Indications: 

• Robust clinical intervention data available but eff

Robust clinical intervention data available but eff

suspected to be “vulnerable” during implementat

( i.e., most of the time

)

• High level need for clinical action despite limited

evidence base
Hybrid Type III Example Hybrid Type III Example

Blended Facilitation to Enhance PCMH Program IBlended Facilitation to Enhance PCMH Program I

(Kirchner, Curran, et al.) 

• Controlled trial of an implementation strategy (in

external facilitation) to support adoption of three integrated primary care and mental health 

• 16 matched sites with comparison sites r

16 matched sites with comparison sites r

“standard” dissemination plan supported clinical 

p g ro

gram office 

• Multiple uptake and fidelity measures acr

and sites 

• Patient-level analysis of depression outcomes 

• Depression symptoms • Hospitalization Hospitalization 
Which framework?

a s e sa a e

Hybrid Design ConsideratiHybrid Design Considerati

• Which hybrid type? • Which implementation plementation framework? • Randomization or quasi-experimental • What is the sampling frame? • What are the unit(s) of analysis? • What are the domains of interest / mea

gather? 

• Study tasks (& duration): 

– Pre-implementation – During im plementation – Post-implementation 
More Considerations More Considerations

 Challenge: concurrent data collection a

analyses 

 Not parallel data (effectiveness and 

implementation); need to address both implementation); need to address both

simultaneously; mutually informative

 Team expertise/size needed to accompl

Team expertise/size needed to accompl

type of interpretation 

 Manuscripts throughout Manuscripts throughout  Where to publish? 
Questions?

For further information:

alison hamilton@va gov alison.hamilton@va.gov