Research to help

spread improvements 

more eff i l ect

ively

This PPT and other resources from This PPT and other resourcesfrom

http://public.me.com/johnovr Thanks 

John Øvretveit,

Director of Research, Professor of Health Innovation and 

Evaluation, Karolinska Institutet, Stockholm, Sweden

3/24/2011
Poll Question:

Ever been in serious pain in the ho

been in serious pain in the ho

Yes.

Definitely

a bit

No.
"Over the past 20 years there have been huge 

developments in treatment of pain” (Details - APS “Pain in hospital is avoidable 

- patients reporting extended high level p

indicator ofp q y oor 

quality care”

Variations and opportunities to improve Variations, and opportunities to improveDoes “throw over the wall” research change enservi f t h? ices 

fas

t enough?
Messages

 Needed: Faster spread of proven improvements 

t eat e ts reatments, se v ce de ve y ode s, s (c a 

service delivery models, PHIs (chan

...and of proven methods for enabling changeimplementation methods (change methods (change processes) processes)

 Needed: Better evaluation of promising improvem Relevant research can help research can help – poorly communicated 

poorly communicated 

 Designs and practices for this research & how we n Spread research the frontier Spread research the frontier of implementation scie

of implementation scie

issues of adaption, attribution, generalisation
Outline

Analogy anchor Analogy anchor – spreading 

spreading 

Examples IssuesResearch designs and practic Implications
Spreading plants to different setting

Seed Gardener/planting & nurture Clim

Modify seed for Can th

or othe

Modify gardening techniques for the setting? 

Can an ordinary gardener do this?

or othe

parts ocontex

S t i 

techniques for the

situation?

(to those where the seed grew before?)

3/24/2011

gardener do this?

(Change content) grew before?)

Sus

t

ai

(Change process)
Spread example 1- VHA TIDES PHC depr

 Researchers gave practitioners evidence about ef

changes, drawing on proven chronic care model.

+ frameworks for using quality improvement im

plement and test these chan

ges. 

(gave proven change content and change-adaption p

 Evaluated the changes and developed Evaluated the changes, and developed

“packages packages

others copy and test changes

 Hand

-o er to operations hierarch for top o

ver to operations hierarchy for top

-do n 

do

w

n 

supported by central development unit

 N h d i ti f l t t f 

No researc

h

descr

i

ptions o

f

l

a

ter s

tages o

f sprea

documents tracking results)

3/24/2011
Spread example 2 – CLABSI at JH and 

 Historical context: Death, mothers campaign, H

leaders, charismatic clinician

 CLABSI “light” version of CDC guidelines wit

safet

y-chan

ge- friendl

y culture

 Evidence of “bundle” effectiveness  P bli i d h k P

ubli

c

ise

d, t

hen ta

ken-up b l d i Mi hi 

by 

lea

ders 

in Mi

chiga

 Collaborative breakthrough Later, combined with campaign approach, for n

international s

prea

d

3/24/2011
Points about the 2 examples

1 CLABSI Clinical researchers Clinical researchers did QIR studies did QIR studies Spread description (Watson et al 2009)2 VHA TIDES PHC depression care Pilot stages described and evaluated using qu

Pilot stages described and evaluated using quexperimental designs

 Collaborative research practice research practiceBoth need more research into later spread – descripof i ti f i l t ti d lt b t 

f var

i

ations o

f

imp

lemen

t

ation an

d results 

b

e

tween 

explanations
VHA HIV/AIDS collaborative Asc

“Substantial variation across 

f y acilities on virtually all meas

Analogy: growing oranges in california, texas, florida and vi

Substantial variation in quantity and quality of fruit from see

why? The seed plant, or the gardening technique or both? Is there one orange seed plant which will grow in these di

environments? Or environments? Or, 

 Gardeners use a modified seed plant proven for the enviro Or gardeners Or gardeners themselves modify themselves modify a seed plant a seed plant

– how?

Or does success depend on gardener’s techniques:

 Follow exactl

y the 

proven methods for 

pg 

lantin

g and nurtu

 Or modify both the seed plant and gardening techniqu

situation?
Issues – fidelity or adaption

“Substantial variation across facilities on vi

measures...

”

? Some sites lower performance because: à Did not copy exactly the proven change co Did b h d d b 

Did copy, 

but c

hange content nee

d

e

d to 

be 

 Did ada

p, p g p t

, but did not use 

proven chan

ge 

p

make adaption 
Example 1 – research found fidelity e

 Better outcomes with Hi Fi implem

of these of these

à Employment for people with mental h

problems (Resnick et al 2003). 

à Substance abuse (Noel 2006) Substance abuse (Noel 2006)à Smoking prevention (Thomas et al 20à Parent training (Forgatch et al 2005). 
Example 2 – research found adaption e

Treatment effective in outpatients for adolescents w

substance use disorders, adapted for a residential th

community. For 24hr setting, with “positive reinforcement c

senior therapists made small modifications to 1procedures

 Part of intervention used by less trained staff & 

audit modified

Without modifications nothing could have been imso possible degredation of effectiveness is irreleva
Wanted – answers from research to customers quest

When is When is adaptation needed adaptation needed, and of which aspects which aspects?

1 Adaption of some components of the change cont

 Eg Rapid response team Rapid response team

 Do not adapt call criteria to trigger team Adaption latitude for team composition ( Adaption latitude for team composition (eg some with 

some with 

without)

 Bundles for preventing stroke and health desease? (kar

2 Adaption of the implementation method? Chang

Eg is effective treatment for heart disease best im

through academic detailing, or active CME, or pocomputer reminders? (PHC Grol)

D d fi th i t ti l th h t Do you define the intervention as only the change contmethod of implementation? 

3/24/2011
Questions

3 Which 3 Which methods methods are best for adapting a 

are best for adapting a 

intervention?

Probably: 

 “a sy pp g stematic approach”, using ideas from 

adaptions (“evidence-based adaption”)

 change barrier analysis change barrier analysis or change or change-readine

readine

 to assess and address likely hindrances to change befo

implementation (Flottorp & Oxman 2003, Baker et al 

 quality improvement methods, 

 which p p rovide the implementers with feedback on out

they can check if their modifications are effective (Lan

3/24/2011
Questions

4. Which approach to spread is best for 

interventions in which situations

What is “an approach to spread” any

Implications for researchersWhich research design and research praWhich research design and research pra...to answer the implementation/spread 

questions? 

3/24/2011
How close are we to answering these qu

C li onc

lus

ions f ii d 

from rev

iew

ing spread re

3 studies noted differences between 3 studies noted differences between

“spread app

spread app

- McCannon et al 2009 : 8 “approaches” Natural diffusio

Series Collaborative model; g ; g

 Extension a

gents; Emer

genc

Grass roots organizing; Wave sequence; Campaign modemodels. (see also Massoud et al 2010)

- Øvretveit 2011: 3 underlying assumptions Hierachical 

participatory adaption, facilitated evolution

- S l p

lunteren et al 2011: 6“ i scenar

ios

” C l entra

l di i 

direct

ion; p

institutional; supporter; consumer; insurer scenarios.
Conclusions from reviewing spread 

Limitations give great opportunities for VA re Descriptions Descriptions

“add hoc add hoc : ”

“hindsight over hindsight over

-co

co

 Stronger evidence from studies of breakthr

collaboratives

à more standardised and understood than o

methods.

 Few s

p pg read 

pro

grammes based on evidence 

effective spread from research, 

à or designed to test such evidence prospec

or designed to test such evidence prospec

3/24/2011
Conclusions from reviewing spread 

Little exchange between knowledge domainParadigm alienation & allergic alienation

& allergic reactions reactions

 Diffusion of innovation Knowledge translation E perimental E

xperimental clinical research applied to clinical research applied to Q

 Programme evaluation/case study... between like-minded researchers in simil(eg healthcare education, public health) healthcare education, public health)
Why customers don’t use spread re

Literature confusing and dispersedNo economics of implementation 

programmes

Patient adherence not viewed as part of implementation

No comparative effectiveness research on spread approacheoriented rather than user-uptakeSome case study descriptions ...but Russian novels:Cases cannot be compared – different terminology and data 

Poor cumulation of knowledge and duplication of studies

Early stages of description and classification 
“One of these things is not like the o

Sesame Street Sesame Street song

Rewiew started considering dimens

Comp

3/24/2011
Research needs to answer wider range of 

P titi ’ ti b t “i Practitioner

’s questions abou

t “improve

1 Is this “improvement” change efficacious

?

à (in treatment, service delivery model, or metho

improvement)

2 Is this change effective in most routine servic

3 How do we implement it quickly here

?

à What “implementation intervention” is effectiv

achieving this change locally?

 (eg training, payment incentives, measurement and 

4 How do we implement it quickly, everywhere5 How much does it cost/are the outcomes wort
Answers from review and personal obser

 Directives from above slow for changing cor

Directives from above slow for changing cor

practice and introducing service delivery mo

 S h h di t t Suc

h c

hanges nee

d

inves

tmen

t

à Support infrastructures to provide

 Expertise, information, measurement, feedback on per

reports

 Infrastructure can be research-based, or development u

line (generic) or clinical entrepreneur (specific))

à Incenti (j ) ves 

(project grant, revenue

)

à Comparisons with others to make laggards visible(assuming other basics covered eg credible evidence o
Answers -2

1. For intervention proven effective in different settings - 

Eg CLABSI content and implementation process

 Spread activity and infrastructure focus on measuremen

supervision/audit

2. For intervention proven efficacious for one patient gro

Eg some disease management models 

 Spread infrastructure to adapt-test-implement: 

support and require reports of adaption and results and use this kn

later implementers

3. Same for promising interventions (eg some care transit

handover models)
Recommendation for research-partner

VHA initiative to build knowledge about appropriat

comparative effectiveness of fidelity vs adapted imp

of the same intervention of the same intervention Choose an intervention proven efficacious for one patient 

or where we know ada

ption is essential e

g

y gg ou suggest on

 Support implementers (with finance and skills) 

à to document their adaption à evaluate results à make a report to a VHA website, or do deal with AHRQ for innov

section for this VHA adaption experiment.

 Analyse project reports, and provide guidance to later imp

about when and how to adapt for different patients/setting
Better answers from research to help spee

M hi h i Matching researc

h to quest

ion

Change ourselves firstAccept 

- these questions cannot all be answered by experimentadesigns - other research designs can provide answers which are n- we may need to work ith h k with researc

h users t 

o agree ques

answerable by research, and to answer them

To provide more actionable research To provide more actionable research

- we and our units need skills in a variety of research me

- and in research practices research practices
Petticrew, M & Roberts, H Evidence, hierarchies, and typologies: h

J Epidemio

lCommunity Health 2003;57;527

-5

H

b

d

3/24/2011
Implications for researchers -2

Match research practice to nee

research practice to nee



Traditional detached objective study repoaft 1 2 3 fter 

1

,

2

,

3 yrs

At other extreme

 Collaborative engaged action research, ag

questions data gathering responsibilities questions, data gathering responsibilities, 

interim findings

( d ti t f h (an

d sometimes suppor

t

for c

hange – “ 

con

research” or “clinical sociology”))

Don’t stop the research at the spread stage! 
3 More use of Theory-informed s

p

programme evaluation (TISPE

Environment influences which help and 

3/24/2011
TISPE 

For documenting and evaluating spread strategy/strwhere effectiveness of intervention known First make theory of influences expected to affec

à intermediate and end outcomes, including the interven

andi l i mp ementation process

 Use theory to decide which data to gather, and to 

h h ypot eses abh hh out how much each i fl ff influence affec

 Use theory to enable research users to generalise 

setting:

à can we reproduce that principle or causal process and 

does our context enable us to do this? does our context enable us to do this?



3/24/2011
Implications 3 - Research Capacity De

 For funders 

à to assess non to assess non-traditional designs in relatio

traditional designs in relatio

questions

 For researchers 

à awareness range of convenience of diffe

awareness, range of convenience of diffe

methods, some gain extra skills in other m

 F hi or research units 

à mix of exp ,q p p ertise, or quick partnership-for

proposals or JIT expertise 

3/24/2011
Challenge : show 1$ on spread resea

brings more health improvement brings more health improvement

than 10$ on most drug treatment rese

 R hf d esearch funders recognise il i mp ementation 

and variation

 Beginning to recognise potential value of s

research

 Not convinced that research designs propos

answer questions or provide help to practitianswer questions or provide help to practiti

à When to adapt, attribution and generalisation

 Are you?
Poll

1) 1$ on spread research could

bring more health improvement than 10$ g on most drug treatment re

Yes No Don’t know
Poll

2) Research methods and practice not 

sufficiently developed to begin to Which approach to spread is best f

approach to spread is best f

which interventions in which situa

3) Yes No Don’t kno
Conclusions: Research to answer practical 

aboutspreading improvements to health improvementsto health 

Needed: 

à Faster and better implementation & spread of p

interventions 

à Better testing of promising ideas and changes

 Researchers can help can help practitioners do practitioners do this

à knowledge about when and how to adaptà research practice to generate knowledge while 

research practice to generate knowledge while 

 Match research design & research practice 

needs and questions needs and questions
Conclusions - 2

 Increase range of skills in different m

espy Q ecially collaborative QIR

 More use of theory-informed program 

evaluation designs

 Spread research the frontier of implem

science – central issues of adaption, 

attribution, generalisation
Your views

1 Must some changes be exactly copied to get the sam

Which ones?

2 Must some be adapted to be implemented to have an

causing improved outcomes? – Better dilution than n

à Is th d i l i h h d fh h h i i l 

he adaption only in the method of how the change is impleme

change content the same?

 Eg guideline same but implementation method different in different 

3 Best methods for answering these and other practic

questions? Different views to John’s 4 implications- 

4 Changes in research skills and practice needed and l
http://public me 

public.me

m/j ho novr

3/24/2011
 American Pain Society Quality of Care Committee. Quality improve

for the treatment of acute p p (

ain and cancer pain. JAMA. 1995;274(23

 Cleeland et al 2003 Rapid Improvement in Pain Management: The V

Administration and the Institute for Healthcare Improvement CollabJournal of Pain: September/October 2003 - Volume 19 - Issue 5 - pp 

 Godley, S Kenney, M 2010 Implementing Outpatient Evidence- Ba

Residential Program, Counselor 27 May 2010, pp 2-7 

http://www.counselormagazine.com/comp 

onent/content/article/1068

Accessed March 7th 2011.

 Vedel I De Stampa M Bergman H Ankri J Cassou1 B Blanchard F L

Healthcare professionals and managers Healthcare professionals and managers' participation in developing a

participation in developing a

pre-intervention study in the elderly care context, Implementation Sc1-9.

3/24/2011
 Øvretveit, J 2011 Implementing, Sustaining, and Spreading Improv

.

Berman, S (ed) 2011, From Front Office to Front Line: Essential issCare Leaders, Second Edition, Joint Commission Resources, Oak B

 Øvretveit 2011b Non-experimental and naturalistic research me

evaluating and understanding quality improvement interventioat The Health Foundation International Colloquium on the EpistemImproving Quality, 12-16 April 2010, Cliveden House Hotel, UK, fhttp://public.me.com/johnovr in folder <evaluation> in folder <th I f d l ti theoryInformedevaluation> file <O tD i N t li ti HF5A 2

<OvretDesignNaturalisticHF5Ap2

 Donaldson, S Gooler L 2003 Theory-driven evaluation in action: le

million statewide Work and Health Initiative, Evaluation and Progra(2003) 355 (2003) 355–366

 Bickman, L ed 1987 Using program theory in program evaluation, v 

Directions in Program Evaluation, San Francisco: Jossey-Bass, 198

3/24/2011
.  Burgers, Splunteren, Ouwens, Boerboom, & Øvretveit 2011 New p

Effective Imp qy lementation, IQ scientific institute for quality of healUniversity and Trimbos Institute, Utrect, Netherlands. 

Spread - More conclusions 

 For action For action – see Øvretveit Øvretveit 2008 2010 2011 2008,2010,2011 For research – WIP (CIPRS/IHI project & San Diego heart/stroke n

3/24/2011