Evaluation Methods in Implemen

Implemen

Research:

An Introd i uct on

Jeffrey L. Smith, PhD(c)

Implementation Research Coordinat

VA Mental Health QUERI

Email: Jeffrey.Smith6@va.gov
Audience Poll Question

• What exposure have you had to the

presentation for today?

– Attended the EIS training last July in Den– Listened Listened to the presentation presentation online– Reviewed only the slides online– This will be my first time hearing about i
Audience Poll Question

• How confident do you feel about eval

methods and measures to use in cond

an implementation study?

– Not at all confident confident– Somewhat confident– Moderately Confident– Very confident– I’m ready to dive in!
Audience Poll Question

• Have you ever adapted or tailored an

intervention using formative evaluatio

techniques?

– Yes– No
Objectives

• Offer perspectives on what constitutes ‘ fl successful

implementation’

• Define formative and summative evaluation in

implementation research (IR)

• Describe 4 stages of formative evaluation (FE) in IR• Provide Provide overview overview of summative summative (or ‘impact )’ evalu

evalu

approaches and methods in IR

• P id rov e overview on eval ti ua ng usef l u ness of theo

to inform or guide implementation
What is Successful Successful Implementa

Implementa

• SI metrics and indicators from ‘PARIHS Guid

Appendix 5 (see handout)

– Implementation plan and its realization– EBP innovation uptake: i.e., clinical intervention

delivery system interventions

– Patient and organizational outcomes achievem
How do we define implementation success

(from Lukas and Hall ‘Think Tank Summary Summary • Implementation Implementation success success is the presence presence 

the innovation as intended in the deliver

context

• Measuring implementation implementation success success is for

for

determining that presence

Adapted from: Lukas CV, Hall C. Challenges in Measuring Implementation SuccessConference on the Science of Implementation and Dissemination: Methods and MeaMarch 15-16, 2010. Bethesda, MD. 
What is implementation strategy v. innovat

outcomes?• Examples:

Study Implementation Innovation Process outc

strategy 

Innovation Process outc

Bipolar CCM Replicating effective 

programs (REP) 

Collaborative chronic care model (CCM) 

Guideline contreatment for disorder

Organizational collaboration for hand-hygiene 

External facilitation Organizational model 

for strengthening implementation of 

Hand-hygiene compliance 

evidence-based practices 

Adapted from: Lukas CV, Hall C. Challenges in Measuring Implementation SuccessConference on the Science of Implementation and Dissemination: Methods and MeaMarch 15-16, 2010. Bethesda, MD. 
Does the concept concept of implementation implementation success success ap

implementation processes as well as to the

i ti ? nnovation?

Measures of implementation success 

?

Process 

Intervention 

Implementation Clinical 

?

outcomes o

p

strategies innovation 

Other factors affecting progress and success 

Adapted from: Lukas CV, Hall C. Challenges in Measuring Implementation SuccessConference on the Science of Implementation and Dissemination: Methods and MeaMarch 15-16, 2010. Bethesda, MD. 
Against what shared standards standards should

implementation success be judged?

• Fidelity ‐‐ Adherence, Scope/reliabil

Intensity/dose, Quality Quality of program, program, Pa

Pa

responsiveness, Sustainability

• RE‐AIM – Reach, Effectiveness, Adop

Implementation, Maintenance
Definitions

• Formative evaluation

– Rigorous assessment process designed to identify potentia

influences on the progress and effectiveness of implemen

† (Stetler et al, 2006†)

• Summative (impact) evaluation evaluation

– Systematic process of collecting and analyzing data on

impacts, outputs, products, outcomes and costs in an imp

study

– Used to assess success, effectiveness, or goal achievemen

of an intervention

† St tl CB L MW W ll CM B C G ih M H d H Ki 

Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimm

Smith JL. The role of formative evaluation in implementation research and the QUexperience. Journal of General Internal Medicine 2006; 21 (Suppl 2):S1 2006; 21 (Suppl 2):S1-8.
Need for FE in

Implementation Research

• Capture information on factors that hinder 

facilitate successful implementation

• Address interpretive weaknesses

– Avoid ‘implementation assessment failure’

• erroneous study results occur because intervention not im

planned (Type III error)

– Avoid explanation and outcome attribution failure

• FE helps identify what did / did not happen in implementa• Identify factors in setting, anticipated and unanticipated, t

implementation

– Enhance understanding of study outcomes to more accu

interpret project findings and inform future implementa
Four Stages of FE

• Developmental• Implementation‐Focused• Progress‐Focused• Interpretive
Developmental FE

• aka “local needs assessment”, “organizational • I l nvo ves d t a a coll ti ec on on...

– Actual degree of less‐than‐best practice (need for improvem– Determinants of current practice– Potential barriers / facilitators to practice change or implem

proposed adoption strategy

– Strategy Strategy feasibility, feasibility, including including perceived perceived utility utility of project project

• Goals

– Identify determinants and potential problems and try to ad

in i l t ti implementation st t ra egy; refine proposed st t ra egy as f 

ea

– Avoid negative unintended consequences

– Engage clinical stakeholders in defining the problem and po

innovative solutions may emerge from within
Tension... What is Appropriate Use o

Developmental FE• Stand‐alone developmental FE for diagnost

assessment of process / d i eterm nants of cu

practice prior to implementation trial

(use it later)

vs.

• Developmental FE used SOLELY within the c

an ong g oin implementation trial

(use it now)
Implementation‐Focused F

• Occurs during implementation of p j ro ect plan• Focuses on assessing discrepancies between

implementation plan and actual execution

• Enables researchers to...

– Avoid ‘Type III’ errors, where intervention not implemented 

designed

– Understand nature and implications of local adaptation

– Describe and understand major barriers to implementation 

t ka es to achieve did es re change

– Identify and implement new intervention components or re

strategy to optimize potential for success

– Identify critical details necessary to replicate implementatio

other clinical settings
Critical Measures of Implemen

• Integrity of innovation

– Fid lit Fidelity to pl d anne i l t ti mplementation st t ra egy

– Dose of intervention delivery, when variability is possible

– Requires clear operational definitions of intervention co

• Exposure to innovation

– Degree to which intervention is experienced by targeted – Dose of exposure, when variability is possible

– Requires clear operational defs for measuring interventio

• Intensity of implementation implementation

– Eg, implementation or intensity scores for multifaceted i– Eg, ‘goal attainment scaling’ when strategy allows local a

choice of alternative interventions across sites
Progress‐Focused FE

• Occurs during implementation of project plan

(concurrent with implementation implementation‐focused focused FE)

• Focuses on monitoring impacts and indicators of pro

implementation or clinical clinical QI goals

– audit/feedback of clinical performance data– p g ro ress in relation to pre‐determined timelines for

implementing intervention components

• Can be used to inform need for modifying or refining 

implementation strategy

• Can also be used as positive reinforcement for high p

sites; negative reif t n orcement for low performers
Interpretive Evaluation

• Uses data from other FE stages; data on sta

experiences experiences can also be collected collected at end of 

• Obtain stakeholder views on:

– Usefulness Usefulness / value of intervention intervention (as a whole, or individ

individ

components)

– Barriers / facilitators to implementation success or failur

– S ti f ti Satisfaction with i l t ti mplementation process

– Recommendations for further refinements to interventio

• Can provide provide working working hypotheses hypotheses on implementatio

implementatio

failure, particularly when implementation and eva

grounded in a conceptual framework

(for theory building building / refinement) refinement)
Stages of FE

Imple

Pre- Implementation Implementation

Developmental Implementation-Focused

• Assess discrepancies between

In

• Asse

p Implementation p

• Identify determinants of 

current practice• Identify potential barriers / 

facilitators

• Assess discrepancies between

implementation plan and execution, exploring issues of fidelity, intensity, exposure

Asse

usefulnstakeh• Elicit 

facilitators• Assess feasibility of proposed intervention• Integrate findings into

• Understand and document nature and implications of local adaptation

recom

further refinem

Integrate findings into •Asses

intervention design and refinement prior to implementation

Progress-Focused

• Monitor impacts and indicators of progress toward project goals

Asses

with inimplem• Ident

b i 

• Use data to inform need for modifying or refining original strategy

P id iti i f t

barrier

• Provide positive reinforcement

to high performers; negative reinforcement to low performers 
FE Assessment Assessment Methods Methods / To

• Quantitative

– Structured surveys / tools

• Instruments assessing organizational culture, readiness to cha

receptivity to EBPs (eg, ORCA, OCM, others)

• Intervention fidelity measures... including dose, intensity mea

– A dit u / f db k ee ac of ad i i t ti dministrative d t a a on cli i l n ca performa

measure(s) of interest

• Qualitative

– S i em ‐sttd ruc ured it i n erv ews w/ cli i l n ca st k h ld takeholders (pre‐/ 

po

– Focus groups– Direct observation of clinical structure/processes in site visi– Document review

• Mixed Methods (ie, Quantitative + Qualitative)

– Can provide richer explanation of study results to enhance 

of key factors in il i mp ementation success / f il a ure
Summative Evaluation in Impleme

Impleme

Research

• Outcomes Assessment

– A priori measures defined at outset of project that will b

cli i l/ linical/system i i ntervention eff i ect veness

– May be defined at patient‐, provider‐, clinic‐, facility‐, an

level... depending on clinical/system intervention target

– I l nvo ves use of ad ii i m n strative data, chart review,

and/or primary data collection

• Cost Assessment

– Assess costs associated with implementation strategy

to inform decision decision makers on value and feasibility feasibility of imp

imp

intervention
Summative Evaluation –

Outcomes Assessment

• Process of care measures measures

– Assess intervention effect on targeted clinical pract

measures

• Increase use of targeted EBP?• Decrease use of ineffective, inefficient practices?

• Clinical outcome measures

– Assess intervention effect on patient outcomes

• Symptom Symptom reduction/remission reduction/remission, functioning functioning / health‐related related qu

qu

• Particularly important to assess in QUERI Phase 1 and 2 studie

• Are outcomes outcomes observed observed similar similar in magnitude magnitude t

effects in earlier efficacy / effectiveness trials?
Summative Evaluation –

Cost Assessment

• Budget Impact Analysis (BIA)

– Purpose Purpose: Estimate Estimate financial financial consequences consequences of ad

ad

diffusion of an intervention within a specific he

setting setting or system given inevitable inevitable resource resource con

con

– Document and assess costs to system associate

intervention design and implementation implementation

• Include costs of provider time, utilization (eg, clinic v

costs) associated with intervention implementation

• Utilize administrative data, chart review

– See HERC guidelines for cost analysis in Q
RE‐AIM Evaluation Framewo

Framewo

PURPOSES• To broaden and standardize criteria used to ev

programs

• To evaluate issues relevant to program adoptio

implementation, and sustainability

• To help close the gap between between research research studie

studie

and practice by

 Informing Informing design of interventions interventions and se

se

key outcome measures

 Suggesting standard standard reporting reporting criteria criteria

Glasgow, et al. Ann Behav Med 2004;27(1):3-12
RE‐AIM TO HELP PLAN, EVALUATE, AND 

STUDIES

R Increase ReachE Increase EffectivA Increase AdoptiI Increase ImplemM Increase Mainte
RE‐AIM DIMENSIONS DIMENSIONS AND DEFINITI

DEFINITI

DIMENSION DIMENSION DEFINITION DEFINITION

REACH

1. Participation rate amo

individuals

2. Representativeness of Representativeness of

participants

1 Effects on primary out

EFFICACY / EFFECTIVENESS

. Effects Effects on primary primary out

out

interest

2. Impact on quality of li

Impact on quality of li

negative outcomes negative outcomes

www.re-aim.org
RE‐AIM DIMENSIONS DIMENSIONS AND DEFINIT

DEFINIT

DIMENSION DIMENSION DEFINITION DEFINITION

1. Participation rate amon

ADOPTION

p 

possible settings

2. Representativeness of 

participating

IMPLEMENTATION

1. Extent to which interve

delivered as intended

2. Time and costs of inter

Time and costs of inter

inter

1. Long-term effects of in

term effects of in

2 Impact of attrition on o

MAINTENANCE

(sustainability)

2. Impact of attrition attrition on o

o

3. Extent of continuation 

modification of treatme

www.re-aim.org 
Evaluate Usefulness of The

In terms of...• Planning the implementation strategy

• Conducting evaluations

• Identifying unanticipated elements critical to SI bu

unexplained or poorly described by selected theor

• Gaining additional insights about the framework / 

• Helping to understand findings, including relations

between dimensions or constructs; complexity of

implementation.
Summary

• Formative evaluation can be key to effective interventio

success success, interpretation interpretation and replication replication of results results in imp

imp

projects

• S ti umma ve eval ti ua on is used to assess rel t evan outcom

associated with implementation of evidence‐based pra

• Together, formative and summative evaluation are com

developing, implementing, evaluating and refining imp

interventions

• Important to evaluate usefulness of theory as routine p

as well
Questions? Comments?

• Please open the Q&A tab located at th

left of your screen, to submit a questi

comment.
Audience – Poll question question (after 

• Think back to the level of confidence you

indicated at the beginning of this presen

Now, how confident do you feel about e

methods and measures to use in conduc

implementation study??

– Less confident confident than before hearing hearing this pres

pres

– I’m feeling about the same– Slightly more confident confident– Much more confident
Jeffrey L. Smith, PhD(c)

Implementation Research Coordina

VA M tl en a H lth ea QUERI

Email: Jeffrey.Smith6@va.go