Research methods for understandin

Research methods for understandin

improvements & helping sprea

This PPT and other resources from:

http://homepage.mac.com/johnovr/FileSharing2

John Øvretveit,

11/10/2009 

Director of Research, Professor of Health Innovation and 

Evaluation, Karolinska Institutet, Stockholm, Sweden
What this is about

• Experimental evaluation good for 

– high certainty about effects of change on patie

high certainty about effects of change on patie

outcomes.

• Don’t answer decision makers 

questions ab

– Would it work here?

• What was their situation – helping (or hindering)

– How did they implement the change?

• VA research can help make more progress 

improving safety and quality

• Non-experimental methods to help answer 

and assist spread.
This seminar overview

1 S bj t 1

S

ubjec

t

– Methods for studying improvements in the VHA systnot involving researchers or QUERI – local unit and sy– Methods for studying spread 

2 Problems and questions

– Some improvements catch Some improvements catch

-on or are implemented be

on or are implemented be

– Intentional spread slow and patchy– Need to improve our “spread technology” – can resea

3 Solutions

– Understand which improvements are “successful” an– Mixed method naturalistic research methods Mixed method naturalistic research methods– Researcher involvement in spread programmes differ

QUERI

4 Your views and questions 4 Your views and questions
Setting the scene - My p ex erience and 

1) Effective QI =1. Clinical change: single subject, single site, single proj

– Eg Improve beta blocker or diabetes care guideline adhera

2. Short-process improvement

– eg wi hi t n-l ba oratory process

3. Micro-system improvement

– eg multi l h ithi b d d li i l ltiple changes within bounded clinical area

2) The real innovation is in implementation – local adap

invention 

How resourceful management and teams make improvement

constrained settings

With little outside assistance little time With little outside assistance, little time divertable divertable from clini

from clini

few change skills 
3) g p g ( 

 Challenges spreading (and sus

To get organisations to do changes 1,2,3 – la

spread programmes or administrative dire

or administrative dire

–Limited evidence (Mittman 2004, Shoja

Grimshaw 2005, Schouten et al 2008)

– Variable implementation and outcomes 

2005 2009) 2005, 2009)

(eg collboratives 10/20/30/40 rule; large sca

–What about others not in the programm
If we know a change is effective

• Either QUERI-formed, or locally made-up (how to finHow get better and wider implementation?4) Need a “step change” in “spread technology”•

Better VS i w

i

t: 

Va S id i l i h 

System wide 

imp

lementat

ion tec

hno

• “Throw over the wall” efficacy research of limited hel• Can research and researchers help find and spread effe

chan

ges?

• How best? • What needs to change?
VHA NODs – where were the resea

• Practice/leader-led improvements not researcher-le• Limited research description, evaluation or assistan

– EHR development and implementation – where were 

and the publications? and the publications?

– Primary care programs system-wide– Increased preventive care & evidence preventive care

& evidence

-based guideline

based guideline

– VA performance measurement and incentives

– New clinical programs: integrated care for returning s

integration into primary care of mental health services 

• National program assists, but VISNs and facilities 

t t th f d li i to carry ou

t the new ways of

d

eliver

ing care.
Va Pressure ulcer 

postedAt AHRQ InnovatiAt AHRQ Innovati
Observations

1 Difficult to evaluate large scale collaborative or other 

programmes

2C t d an study i t di t lt d d ib l 

intermediate results and describe causal 

but uncertainty about effects on patient outcomes.

3 These programmes t t i l t t ti 

target social systems not patien

systems, facilities, larger organisations or regional 

(VISNs)

– Change mechanism works through how people perce– Not biopy p p h sical natural causal processes Perceptions 

mediators 

– Can find out perceptions to help understand change
Observations

4 Quality and safety are systemic properties

on how the parts of each system works toon how the parts of each system works to

• One aim is to develop improvement capac

5 The programmes themselves are complex 

systems of actors and actions changing ov

in a changing context.
Part 2: Problems and questio

Practical • N d t d d f h hi h i Nee

d

to spee

d sprea

d o

f c

hanges whi

c

h

im

quality 

• Can use experimental to evaluate many 

improvements, to check if they do.

• But decision-makers want to know other th
Decision-makers questions

• Would it work here locally? • How did they implement this change? (descri

• Which context factors helped and hindered factors helped and hindered

implementation (attribution)

• I hi h n whi

c

h range of tti se ngs and diti did 

d conditions did 

(generalisation certainty)

• How do I adapt it, or the context, to impleme

(adaption)
Can researchers hel

p

?

• Find and describe successful “locally-made u

im

p y rovements or administrativel

y directed on

(diffusion)

• Describe intentional-spread programmes (ISP

• Evidence of what helps and hinders successfu

• Give “s

preaders” feedback for real time im

pro

in doing spread.

Need spread research assistance programme (SR

help develop VHA System-wide implementat

thl ec

hno

logy
Research-oriented 

questio

• What helps and hinders successful spread (eg some “b

research)?

• What explains successful natural or administrative dif

some improvements?

• How do we conceptualise and describe lar

ge scale spr

programmes? 

– Describin

g pp p what happens between the 

plan and results

• How do we detect variation in sites,

– adaptation of the intervention (e.g., guideline adaption)?– i l t ti th d ( imp

lemen

t

ation meth

o

ds 

(e.g., sh t t i i EMR i 

hor

ter 

tra

i

n

ing, EMR rem

i

n

– implementation fidelity and extent?– intermediate and end results?

• How do we s

ynthesise the data to create valid and use
Research challen

ges

For studying system-wide intention

natural d f diff l sprea

d o

f different 

im

provements

e.g., national collaborative, or 

administrative directive to im

plem

chronic care guideline
Can you grow pineapples in Swed

Seed Gardener/planting & nurture Clim

Your change?

Change ideaE id

+ Context 

L l

+ Implementation actions

0 5?

Your change?

11/10/2009 

Evidence

0-5?

- Local

- Wider

0-5?
Distinguish

INTERVENTIONSeed

IMPLEMENTATIONSTRATEGIESPlanting

CONTEXT

Seed Planting Soil and cli

cli

Effective treatment

EducationGuidelines

Audit and feedback

Organisational Culture

Audit and feedback SystemsAcademic detailing

SystemsFinancial syste

Effective organisation Breakthrough collaborative Organisational 

Eg team organisation

Care manager

Or A change idea

g

Implementation network

g 

supportCultureFinancial incen

g

Which effective for which intervention?

Classification ofstrategies?

Which features helstrategies/support 

interventions?
Emergg p g ing evaluation paradigm for 

• Qualitative data on outcomes (participants valuat

– Multiple stakeholder perspectives of “effectiveness”

• Observational description of programme implem

real setting (multiple data methods & who did wreal setting (multiple data methods, & who, did w

• Assessment of influence of context – which facto• U d di f i fl d b f Understanding of outcomes influenced by many f

of which is the CSI

• Description of CSI/ Description of CSI/programme programme-context interactio

context interactio

dependencies

• Bias towards objective of improvement rather tha

Bias towards objective of improvement rather tha

attribution and generalisation
Observations: the intervention / contex

• Experimental separates intervention from c• Focus on the intervention Focus on the intervention – effect if presen

effect if presen

absent?

• But disconnect with Systems thinking

– Safety a result of combination of conditions, n

intervention 

– Improvements through changes at multiple lev

levels create the context for lower levels) levels create the context for lower levels)

• Need methods to study multiple changes at 

levels over time and relate to safety performlevels over time and relate to safety perform
Ways forward

1 Match method to question2 C bi d d i 2 Combined design

– Experimental and – Parallel process evaluation 

3 Naturalistic mixed method with programm

(case study evaluation)

4 Chang g, p , es in understanding, respect, skills a

funding

FOCUS on 3) and examples
Question matched to design

Q1 In which ran

ge of settin

gs and c

did it work? (generalisation certadid it work? (generalisation certa

Experimental testing & add descriptions• selection of settin

gs and conditions:

• selection is theory-informed eg pre-assess

do

“change ready change ready

” settings implement bet

settings implement bet

get better results?)
Question matched to desig

Q2 How did they implement this chan

(description)

Which context factors helped and hindered imple

(attribution)

• Multi method case study - examples• EMR implementation Kaiser and EMR implementation Kaiser and Karolinska Karolinska• 12 innovations in Swedish healthcare• UK PSI t d UK PSI s

t

u

dy 

• Context which helps and hinder patient safety 

improvements - Reviews of research Reviews of research
EMR implementation Kaiser and K

• Kaiser first try: 2 years, stopped, cost 1.2m$• Karolinska: 1 year, under budget, generally • Discovered explained by different implemen

cont t ex s

• Modified Rogers diffusion of innovation the

predictive

• Some value to implementers and science: em

descriptions and theory descriptions and theory.

Øvretveit,J Scott, T Rundall, T Shortell, S Brommels, M Impleme

electronic medical records in hospitals: two case studies, He

84(2007)181–190
11/10/2009 
One, of 12 innovations in Swedish h

Integrated system: All services for 300k population 

H it l PHC d i l i bli Hospital, PHC and social care in one public com

• Corresponding merged Health and social care p

organisation

• Combined county and local political governing • Case study described:

– steps in change, conditions helping and hindering– before and after structures and clinical processes– eff t l k ti f ti ffects on personnel work satisfaction

Cannot attribute changes in patient outcomes to the cha

This case evaluation used in decisions of future changeThis case evaluation used in decisions of future change
– .
Case studies

Programme theory and concepts for desc

change

Programme theory ideas about sequence of actions and situation factors leading to 

changes and output and outcome changes (their theory, our 

Cretin et al 2004 model

11/10/2009 
MRC CSI safety research model (Brow

MRC CSI safety research model (Brow

11/10/2009 
VHA advanced access – evaluation model by C

11/10/2009
Naturalistic methods PSI study (UK 100k col
Naturalistic methods PSI study (UK 100k col
Tool are available for research describin

assessing context

Organisational readiness for change 

- shared subjective sense of change comm

and chan

g y( ) e efficac

y

(Weiner 2009

)

FOR Diabetes registry in community health centerORC high ORC high, 

- staff skillfully and persistently take action to put a diabetes regi

and demonstrate more consistent, high-quality use of the regist

ORC low 

staff resist initiating change, put less effort into implementation, p

and hibit li t i t t b t R i t lik l t b 

d exhibit complian

t reg

i

s

try use, a

t

bes

t. Reg

i

s

try lik

e

ly 

to 

be 

scattered, and uneven
Tool box for research - describin

asse i te t ess

i

n

g conte

x

t

O i ti l Organ

isationa

l readiness f h or c

hange 

- Different from

Context receptiveness for change

– objective factors which influence chan

external

(Pettigrew et al 1992, Dopson et al 2002
But does the best context for one inter

diff f b f h i differ 

from 

best context 

for anot

her 

inte

E.g., cont t f h d h i i t ti tex

t

for 

han

d

hyg

iene 

i

n

tervention 

vs context for com

p pp uter decision support 

intervention?

• Initial findings yes Initial findings yes

– from AHRQ PSP stu

from AHRQ PSP stu
Contexts which helps and hinder 

safet

y improvements - Reviews of 

AHRQ funded Karolinska/RAND/Stanford/

Hopkins

• Five PSPs• Evidence of context influences on implem

Evidence of context influences on implem

• Designs and measures for researching thi
Summary so far

• Most research looks only at effectiveness of • Uses methods exclude context methods exclude context as confounder

as confounder

• At best give description of context (eg report

g Q) uidance SQUIRE)

• No theory of intervention influence pathway• No theory of context theory of context• Both help decision makers and implementer • Some general frameworks of possible contex

general frameworks of possible contex

influences on change implementation

• But likely different contexts likely different contexts for different cha

for different cha

interventions
Common features of this rese

• Intervention or change not a treatment

– b t a ne ser ice deli er model s stem or t pe of b

u

t

a new ser

vice deli

very model, 

s

ystem or

t

ype of 

• Intervention or change not fixed and standardise

– but evolvin

gy y intentionall

y and unintentionall

y

• Sometimes no control group or comparitor

• Context influences studied, not excluded by desi

– S i fl b Some 

i

nfluences can 

be operati li d tionalise

d and 

 measur

– Some need qualitative exploration and understandin

change participants interpret the influence

• Theory

of “h i k ” f “how 

it wor

k

s

”

– mod l e

l

s

• Weak on patient outcomes but shows pathway of 

• Multiple data and perspectives Multiple data and perspectives about effects about effects
Summary points

We have traded certainty about efficacy 

for less knowledge about adaption, 

generalisation, implementation 

and explanation of causal pathways

Other research needed• N t i t l t ti f ti h N

o

t exper

imen

t

a

l

testing o

f practice c

han

• But observational documentation of succe

developments
Other disciplines 

• Developed evaluation methods to answer 

questions

– Public health and health promotion: causal pa

and theory g of change

– Programme evaluation, education and crimin

study, realist evaluation

– Social sciences and Business studies: case stu

evaluation and action research

• H ld h h d b li d How would these methods be applied to a

improvers questions and build theory abou

improvement implementation in ordinary improvement implementation in ordinary 
Ways forward?

1)Parallel implementation study to the experi

QUERI evaluation.

(QUERI – high certainty effective but widespread 

implementation less good)

2) Non experimental research of “naturally oc

developments” (NODs)

– The 3F approach: Find, Follow and Forward w

– Observational “detective” research

Less objective certainty of effectiveness but good Less objective certainty of effectiveness but good

implementation lessons

Next – research methods for 2) and their uses
Summary

• Local developments – find evaluate and he• Large scale programmes – spread technolo• Match method to question Match method to question• Some improvements can be evaluated usin

experimental And parallel process evaluatexperimental. And parallel process evaluat

• Others, and large scale changes need other 

– causal d l d i d th d l models and mixed methods
Summary

• Need Better VaSwit: Va System wide 

implementation technology

• Research to study pp and explain spread a

spread

• Match methods to users question and th

Match methods to users question and th

programme

• Cutting edge developments in implemen

Cutting edge developments in implemen

research 

• Mi d di i d th d Mixed paradigm, mixed methods 
Ways forward 

1) Expand QUERI with Implementation studies parrall

experimental.

2) Spread research assistance programme (SRAP) Start a Find, Follow, Forward programme

Fi dn successf l i ( d f il ) ful improvements (and failures)Follow: describe steps in development, context which helped 

and different results

F d orwar to spread d df i h i d and study fate in other settings

tools to assess if others have the right conditions, tools for impleme

study examples, experiential reports, implementation network

– U C d i d hd i 

Use Case study mixed methods, using context assessmen

• Virtual QUERI centre for identifying evaluating and 

service innovations VQSIs
Implications

• More flexible and tolerance. But know go

bad research for the question bad research for the question.

• Develop researchers to understand range 

• More to specialise in mixed methods and 

implementation research.

• Methods to find local improvements and 

research
QUERI limitations

• Strong evidence of effectiveness – at resea• Spread requires destination services get as

desigy p n and carry out their own implementa

• QUERI evaluations give limited info on ef

different situations or explanations for this different situations or explanations for this

• Experimental methods not designed to
Reality check

Valid Assumptions?• Little research into practice lead transformat

Little research into practice lead transformat

VHA?

• Research not useful to decision-makers imp

in other areas?

• QUERI research g p ives little info about impl

and conditions needed? Controls to assess e

• QUERI changes – limited spread?

Evidence for or against any of the above?
Comments on this view:

As researchers we too often try (and too often fail) t

care through rigorous research through rigorous research, 

while at the same time VA's clinical and administrat

are conducting numerous QI programs that succeeare conducting numerous QI programs that succee

least appear to succeed. 

We 

( ) p p yp as researchers

) need to hel

p the 

polic

y

/practice 

understand if, and how, their programs succeed, b(researchers) also need to learn from their strategisuccesses. 

This requires observational research that QUERI res

(and too many grant reviewers) do not (and too many grant reviewers) do not respect nor 

respect nor 
Your reactions and question

1. Any surprises Any surprises

...

2. Not certai b n a

bout...

3. This could be useful...11/10/2009 
11/10/2009
Background: Note the French 2009 

• Subject: implementing evidence based pr• Approach: not individual clinician behav

change, but social context and networks tpart of.

• Intervention highly dependent on social c

Intervention highly dependent on social c

– both context and intervention adjust 

• Model they suggest for studying how con

influences whether clinicians adopt EBPc

11/10/2009

French 2009 review

Categories and measures• Climate:, e.g., openness, respect, trust

• OL culture

• Vision• Leadership• Knowledge need• Acquisition of new knowledge Acquisition of new knowledge• Knowledge sharing• Knowledge use Knowledge use
11/10/2009
.

11/10/2009
Pain medication . Does it work? Standardise the treatment to eac

patient

Compare to no treatment/placeb

reatment/placeb

Exclude other factorsMeasure outcomes – are changes these measures associated withp / resence/absence of thetreatment?

Pain medication improvement

Does it work Are systems and supports change

Do providers change their

bh i ?

improvement behaviour? 

Are patient outcomes better

Implementing a change 

What did they do?Was this strategy effective

to get the change?

Details of actions taken at differe

times (method/strategy for the

change)

Conditions which help and hinder

Could we do this? Which influences help and hinder

ordinary services to make the

change

( ti t i il i ti

11/10/2009

hinder (patients similar, organisations

different)
Questions and criteria

EfficacyDoes it work (anywhere)?

Certainty about effects 

validity)

Better than.... Key issue

=Attribution

Effectiveness

Wh t th ff t i til

Certainty about effects 

Wh

a

t are th

e

effec

t

s in

typical i t di t

d lti 

settings?

in

termedi

a

t

e an

d

ultima

(external validity)Key issue

=Generalisati

ImplementationSellStart

What were the actions t

the changes?Which conditions hel

pe

SustainSpread externally

p

hindered?Key issue

= Description 

ex

planation
The MMC-U

pati t f t i t 

tient safety inte

study

11/10/2009
Summary points

• QUERI focus - Does it work?

–

but not 

g q ood answers to local 

questions.

• Tendency to require exact replication• But know adaption needed and likely not to 

But know adaption needed, and likely not to 

without some conditions 

• Few changes context insensitive and exactly 

Few changes context insensitive and exactly 

• Information lacking for implementers

– If use

“evidence evidence

”

 – mostly practitioner reports (

mostly practitioner reports (

or JC Journal,

– But limitations in items reported and validity But limitations in items reported and validity
Summary points

We have traded certainty about efficacy for less knowledge about adaption, generalis

im

plementation 

and explanation of causal pathwaysOther research needed•

Not experimental testing of practice chang

• But observational documentation of succe

developments
But does the best context for one int

differ from another interventi

•

E.g., cont t f h d h i i t ti tex

t

for 

han

d

hyg

iene 

i

n

terventio

vs context for com

p pp uter decision support 

intervention?

• Initial findings yes: Initial findings yes:

• Four types of interventions – each require

different context
Four types of interventions – each requires differ

w poss b y d e e p e e o pp o c es d d e e 

ith poss

i

b

l

y

diffe

r

ent im

p

l

e

m

entati

on app

r

o

a

c

hes an

d

diffe

r

ent a

• Complexity and dynamism

– Single simple intervention (to patient, to provider: Single simple intervention (to patient, to provider: eg w

w

guide)

– Multiple simple interventions (eg surgery and meds; Q

– Mul it

i

-l li i ( leve

l

intervent

ions 

(eg

NDL team, manager, reg

– Evolving adaptive interventions (eg QI programmes in 

s

y ) stems

)

Other grouping for interventions:• Focus of intervention: 

– individual (patient or provider); team; unit; organisatio

system.

11/10/2009