Researching the conte

of programmes to impr

health and health car

This PPT and other resources from This PPT and other resources from

http://public.me.com/johnovr

or https://www.idrive.com ‐ see references at e

John Øvretveit,

Director of Research, Professor of Health Innovation and 

3/8/2012 

Evaluation, Karolinska Institutet, Stockholm, Sweden
Purpose: methods for discovering ho

aff i l i d 

ffects 

imp

lementat

ion an

d outcom

Aft 1 h After 

1

hour 

– b bl t l i be able 

to exp

l

a

i

n

1 Three desi

g g ns for evaluatin

g interven

implementation2 F k id d h i 2

Framewor

ks to guide 

data gat

her

ing

3 When and why to collect data about 3 When and why to collect data about 4 How to present data to customers for 

3/8/2012
Three designs for complex progr

1 Experimental- & Process- Evaluaparallel2 The i f ed te t e iti e 2 Theor

y

i

n

formed conte

x

t s

ensiti

v

e

observational evaluation (TI-CSOE)

3 Action evaluation 

(for developing interventions iterativel

3/8/2012
Experimental Outcome Evaluat

Findings

• Guideline implementation 1 sore throat and 2 wo

urinary tract infection (Norway 142 PHC Centres) Flotto

• Little change overall in these measured ou

– use of antibiotics (sore throat only)

– use of laboratory tests

– use of telephone consultations

• Great variation between practices in amou

change:

• Next slide shows: % patients receiving antibiotics 

after intervention for controls and Exp PHCs

3/8/2012 
•

These reduced use aft

(practices below the l

3/8/2012 
Why? – The parallel parallel process process evalua

evalua

At same time as trial:Investigated factors that might explain lack of change 

variation:

• Data from 120 practices: practices:

– observations, semi‐structured telephone interv

postal survey and data from electronic medical 

• F d oun

– agreement with guidelines; degree of participation in th

– taking time to discuss the guidelines and their impleme

– use of the components of the interventions;

– procedures for telephone consultations;

– communication within each practice.

3/8/2012 
What do differently?

• Use more active interventions (eg outreach visit).

• Use a longer time period for intervention and follow‐up.

• Focus more on organizational arrangements to change ro

involve not only physicians.

• On‐line access to communicate with the practices, install 

collect data online.

(last two are conditions) conditions)(“these make it easier and potentially more cost‐effective to 

and run quality improvement and guideline implementat

primary care”)

i.e. add to the intervention, other interventions to change the cont

context enfolded into the intervention (CLABSI)

3/8/2012 

– context enfolded into the intervention (CLABSI)
Approach 2 Theory‐informed context sens

evaluation

Example: Zapka et al 2004

Intervention Intervention to promote promote Smoking Smoking Cessation Cessation 

Community Health Centers

“demonstrates the importance of process‐monimethods methods to examine examine context context factors factors of collabor

collabor

research,

outlines outlines lessons lessons learned learned and the challenges challenges of r

r

carried out in CHCs,

and summarizes implications for researchers an

practitioners”.
4 Formulate programme the

• Assumptions about which actions lea

which intermediate and later results

• and which context factors critical to

allow/enable actions

• EXAMPLE programme theory model c

Intervention to promote Smoking CessaCommunity Health Centers

3/8/2012
Smoking Cessation in 

Community Health Centers

3/8/2012
Smoking Cessation in 

Community Health Centers

3/8/2012
Zapka et al 2004

Summary report

of context
Approach 3 Action evaluation for interventions requ

One version: One version: Integrated Integrated research research‐implementation implementation 

Purpose not to control,...but to describe and improve the interventi

results.

R h k ih ii i Researchers work with practitioners to assi

implementation, 

get and feedback feedback data on progress and inter

data on progress and inter

results.

Eg Rubenstien et al study of PCMH demos 

et al study of PCMH demos 

VHA & TIDES

. 
Rubenstein et al 2010

• TIDES collaborative collaborative action research research project project

• (using evidence

‐based quality improvement (EBQI) 

Intervention (s)

• Researchers helped regional leaders to select, ad

implement proven depression depression care models

Research role

• Provide Provide evidence evidence of EB practices/models practices/models. Discour

Discour

ineffective ones.

• Initiate Initiate use of QI methods methods (setting (setting targets, targets, data c

c

PDSA, iteration)

• Evaluated results

(Trained nurse de

pression care

‐

collected data on patient adherence and outcom
Rubenstein et al 2010

• No randomisation randomisation

, standardisation standardisation (fidelity (fidelity asses

asses

comparison group

Research products

• Examples of models and how implemented by lo

teams

• Description of implementation structure steps an

• Outcome Outcome data (mean PHQ

‐

9 scores from 15 1. to

Result

• “TIDES achi d eve

d excell t en

t overall pati t en

t o

and the program is undergoing national spre

(without researcher or other support?)
Summary: 3 approaches to study 

• Parallel process evaluation 

– to explain results of trial if little average change 

variation – context factors discovered inductivelinterviews and researcher observations

• Theory informed case evaluation

– Initial theory frames data collection about interv

Initial theory frames data collection about interv

context (theory modified in some)

• Action evaluation Action evaluation

– Later versions more explicit theory – enfold con

vers

ions

3/8/2012
Which aspects of context do I gather da

• Previous evaluations of similar type of interv

– Which context factors at different levels do they 

critical to success?

– Leadershi

p

– which level, yp p, 

 type of leadership, in 

measures?

– Culture 

– which as

pect and how measure?

• Failing this, use a generic frameworks suited 

intervention

– Eg for implementing evidence based practices in 

PARHIS f k PARHIS

framewor

k.
Context for EBP (PARiHS) y R croft-Ma
Understanding context for quality improvemen

(p ) Ka

plan et al 2011

)

Context influences from

4 leve

• External environment• Organisation• Microsystem• QI team
Kaplan et al 2010

• Reviewed Reviewed research research for evidence evidence of con

con

influences over QI success

–Notes limitations in the research reviewe

Appeared to important to success:

• leadership from top management, organiza

culture culture, data infrastructure infrastructure and informatio

informatio

and years involved in QI.

• Other potentially potentially important important factors factors were: p

p

involvement in QI, microsystem motivation 

resources resources for QI, and QI team leadership leadership.

3/8/2012
MUSIQ (Kaplan et al 2010)
French et al 2009 synthesis of 30 instruments meas
Measures of aspects of context – safety, Shekelle 

• Patient safety culture culture

,

– AHRQ surveys;– Patient Patient Safety Climate Climate (Pronovost (Pronovost et al 2003

, K

2005);

– Safet

y Climate Surve

y

(Kho et al 2005

)

• Teamwork,

– ICU Nurse

‐Ph

ysician

Questionnaire

(Shortell et 

• Leadership,

– ICU Nurse

‐ Physician Physician Questionnaire Questionnaire (Shortell (Shortell e

– Leadership Practice Inventory (Tourangeau

& M

2004)

– Practice Environment Scale (Lake 2002).
Garden analogy framework (GAF, Øvretve

S di prea ng plants to different setti 

ng

Seed Gardener/planting & nurture Clim

Modify seed for C Modify seed for setting? 

Can an ordinary gardener do this? 

Can 

or otparts 

Modify gardening techniques for the situation?(to those where the seed grew before?)

3/8/2012 

(Change content) cont

Sust

before?)(Change process)
3 Why study context?

1 Any evidence context affects outcomeintervention?

Why spend $m doing RCTs?Drugs affect patients differently (body is the contDrugs affect patients differently (body is the contTraining affects providers differently (receptivenet t d d t t) to ac

t

depen

ds on con

text)

QI interventions to organisation changes some, noBut not much evidence of how and which contexinfluential (safety review (Øvretveit et al 2011))

3/8/2012
Vote: which of these context influences 

success?

• leadership from top management,• organizational culture,

• data infrastructure infrastructure and information information sy

sy

• years involved in QI

•

wh h et

her CMO supports

D ll a as

C b ow

boy

(Kaplan et al 2010 review)

• But how do you operationalise? (LATE

– define presence/absence presence/absence or measure measure or 

amount of these?
Review of evidence of context influen

patient patient safety practices practices (Øvretveit (Øvretveit et 

• Falls in institutions:

– No strong evidence for or against context fac

– either helping or hindering implementation o

interventions

• Medication reconciliation tool and pr

redesign

– “Blocking functions” in electronic systems to 

compliance with medication reconciliation st

(context or part of intervention? ( ) Vote?More in Details section later in PPT
3 Why study context?

• Wh d i hi h l f 

Why not use a 

des

ign whi

c

h contro

ls 

for c

– Comparison group of patients, or providers, o

organi ti sa ons

– Randomise or match characteristics which m

more likel to be affected b the inter ention more likely to be affected

by the inter

vention

• Expensive• Time consuming• Difficult to arrange good comparison grou

Difficult to arrange good comparison grou

organisational units

• (also controlling intervention may be diffi

(also controlling intervention may be diffi

inappropriate) 

3/8/2012
3 Why study context?

If i k t wor

ks 

–

wh b o cares a

bout contex

t

?

Research customers have other 

questions 

– especially if we say it works sometimes

Was it failure of implementation? Was it failure of implementation?

Partially implemented, or should be iterative adapimplementation?

Or was it implemented the same, but a differeWhat was needed to allow implementation? Should this be 

part of future interventions?

How much did it cost etc 

3/8/2012
3 Why study context?

1 Knowing which context influences help aimplementationEnables our customers to decide whether thim

plement

Or what they need to change to implement2 Hav g a t eo y o ode o t e te ve t 

in

g

a

t

heo

r

y

or model 

of 

t

h

e inte

rve

n

t

Enables our customers to replicate the princrather than digitally copy but miss the activrather than digitally copy but miss the activ

3/8/2012
4 Final points – then resources for s

context

• More implementation research is showing succ

depends depends on facilitative facilitative organisational organisational and exter

exter

• Soft features: leadership, culture, project mana

“interdisciplinariness

”.

• Hard: IT support for data gathering and feedbac

• Success more likely if “the intervention” include

influence context favourably.

• Theorising and data collection about change ste

evaluation of “fidelity” of implementation acco

or d i ocumentation

o

f

a

d i apt ons
Summary Poll – agree or disagree or 

1 Only implementation implementation actions actions are sensitive sensitive to 

2 Some interventions are more sensitive to co

others.3. There are different different context context influences influences for di

di

interventions

4.Different context influences are more and le

at different sta

ges in sellin

g

, startin

g

, pg 

ro

gress

sustaining and spreading an improvement.

3/8/2012
Summary

5. Some context influences may need ttogether together to have maximum maximum effect in en

en

implementation. Coordinated multi‐lev

strategies may be needed.6. We need to categorise categorise quality quality interv

interv

into groupings, according to which gro

context influences are most important i l t ti implementation.

3/8/2012
3/8/2012
Resources on Johns web site f

1) References References in

*O retConte t21Janfor14Feb

O

vretConte

xt21Janfor14Feb

1 http://public.me.com/johnovr till summer 2) After summer 2012

‐ Download files from id

to web site: http://www.idrive.com/;

•

Log in user

=

jovr pass= anna. THEN use th

e s

on the right to enter in

a word realated to th

You will see files on this sub

ject

– click on the 

want to download, after entering anna and 

download to your computer.

.

3/8/2012
Ovretveit 2012 Guide for research into conte

1. When do I need to collect collect data about context? context?2 Why do I need to decide the research user, questions and 

3 How do I match the research design to the question?

4 Which data to collect?

5 How to collect the data?

6 How do I analyse data to find how much the context influ

implementation?

7 How do I report context context data?8 Checklist9 Terms used in this guidance guidance10 Appendices: frameworks for deciding which data to gath

intervention implementation and context

11 References and resources

3/8/2012
Ovretveit 2012 Guide for research into conte

Checklist1. Who is the primary research user and their questions, and 

the research deliver answers?

2. Which are the potential research designs and why the chos

3 Which data are needed for a) others to replicate the interve

to assess

how

si il m ar an

d different

th i e

r context is,

c) to

b

e la

explain the degree of implementation observed and what acc

and the outcome.

4 Which data sources and types might provide these data?

5 What is the data gathering plan and methods and the times 

the data

6 How will you analyse data to find how much the context inl

implementation?7 How will you present the context data? 

3/8/2012
References.

• See list in PPT in “presenters notes” view be• Also see in Implementation f ld o er on web si

• Ovretveit 2012

– Guidance for research into the context of qualit

improvement, implementation or innovation in 

services

– Guidance for case study research and publicatio

– Practical Tips for Context‐Sensitve Implementat

3/8/2012
Conclusions

1. These were the main points were the main points...2. Thi i i f his was new or surprising, for me...3. The most useful idea for my work was...4. What I would like to find out more abou3/8/2012 
3/8/2012
7 Steps‐guide for context‐sensitive e

of QI i t ti n ervention

• How would you carry out a context context 

evaluation of one of these interven

– Medication reconciliation

– SMAs for disadvantaged groups ‐ diabetes

– Remote monitoring of chronic diseases using bi

sensors

3/8/2012
Steps‐guide for context‐sensitive 

• 1 Define primary customer for the r

their questions, questions, timescale timescale and budg

– Does it work? (for whom & what outcomes/measures?)– Costs? (for whom and saving )s– How best to implement?

– Can we do it here?

– What results to expect?

• Evaluation and certainty? Generalisation maximiza

All of above, 6 months

Estimate time and cost to answer each and negotiate

3/8/2012
Steps‐guide for context‐sensitive 

2 Define the type of intervention

– si l mp e/ l comp ex, replicate or confi bl gura e (d i a apt on 

and target subject of the intervention

– human body, behaviour, group, multi‐group, health s

Eg Medication reconciliation

– Complex; replicate development process, but adapt c

– Intervention directed at: behaviour, group, multi‐gro
Step 3 Find previous research int

“t ” ype of i t ti n ervention

• Type: Complex, some adaptability, dire

behaviour behaviour, group and multi‐group

• Did it report or suggest context influe

affecting impl ? ementation? ( ) reviews).

• If not, use one from EBM

– eg PARHIS or other (eg innovation mode

Greenhalgh Greenhalgh 2004) – these shown later in 

3/8/2012
4 Formulate programme the

• Assumptions about which actions lea

intermediate and later results

• and which context factors critical to

allow/enable actions

• EXAMPLE programme theory model c

Intervention to promote Smoking CessaCommunity Health Centers

3/8/2012
4 Formulate programme Smoking Cessation in the

Community Health Centers

• Assumptions about which actions lead 

intermediate and later results

• and which context factors critical to allo

actions

• EXAMPLE programme theory model co

Intervention to promote Smoking CessatCommunity Health Centers

3/8/2012
4 Formulate programme Smoking Cessation in the

Community Health Centers

• Assumptions about which actions lead 

intermediate and later results

• and which context factors critical to allo

actions

• EXAMPLE programme theory model co

Intervention to promote Smoking CessatCommunity Health Centers

3/8/2012
Formulate Formulate programme programme theory using previous previous re

re

or implementers assumptions of links

– (use framework framework to help them articulate articulate the steps 

• define intermediate stage outcomes and indic• Include context factors factors.

5 List other explanations for any changes in ind

may find6 Decide measures or interview questions for gdata about context context and how to analyse analyse.

7 Collect, analyse, report: examine and report d

lik l e y i fl n uence of b th o the i t ti n ervention and co

factors on outcomes. 
Why should we follow these s

• Presentation later shows evidence for 

influences

• Customers need to know if it will work 

setting

& what they need to modify to increa

of successFirst – what do we mean by “context” oi t ti ? intervention?
Vote: which are definitions of c

1 the environment environment or setting setting in which the pr

pr

change is to be implemented

2...physical environment in which practice ta[with] boundaries boundaries and structures structures that togeth

togeth

the [setting] for practice

3 willingness, preparedness and capacity for

implementation of

a specific, discrete practi

4 everything which is not the intervention.Next: Initial points about context of inte

3/8/2012
Example of one aspect of context

Part of Part of

“leadership support leadership support

” context assessment (Damschroder 20

context assessment (Damschroder 20

To what extent do leaders show active and visible supp

change or this type of EBP and implementation?

• Is the leader willing to engage with the study team f

• Is the leader willing to provide connections/entrees f

team?

• Does the leader have experience/comfort in this role

l d ea

der h ld o serv

ice di t rec

tors account bl a

e

for coll b 

ll

a

bor

coordination in such change efforts/this change effo

To what extent are appropriate appropriate stakeholders stakeholders or teams 

accountable and incentivized or rewarded to carry out 

implementation?

• What about past experiences with this type of chang

3/8/2012
1 Research finds some interventio

effective in some contexts contexts.

• Statins may be less effective in women 

( ibl possibly diff t eren “b do y cont t” ex of

intervention)

• Guideline implementation interventio

effective effective in some primary primary health care 

• Re‐engineering and process improvem

eff ti ec ve in some h it l ospitals

3/8/2012
2 Levels or layers of context

• The body is the context for drug,

– but wider context for “implementation” 

patients lifestyle, wealth, social group.

• Context for many QI’s is

– group, facility facility, and system or regional/n

regional/n

contexts

• E h ac l l eve of cont t ex is the source of i 

affecting implementation

3/8/2012
3 Multiple level interventio

• Some chronic care interventions are m

and mul it ‐l l eve :

• target the body and social g p rou

– (eg medications, & lifestyle changes, so

group to encourage encourage adherence adherence, and fina

fina

incentives)

• P di ti Prediction: the more cont t ex i fl n uence

are aligned with the intervention,

..... then the more effective it will be.

3/8/2012
Context only affects outcomes th

influencin

g implementation

H i l CEO id f QHosp

ita

l CEO provides resources 

for Q

after little progress 

Helps and hinders the intervention actiintermediate stages in pathway to outcointermediate stages in pathway to outco

3/8/2012
More

.

on Øvretveit et al 2011 review

Prevention Prevention of central central line ‐associated associated blo

blo

infections (CLABSI)

– Lead h ers ip involvement, teamwork, nursing staff empow

interdisciplinary rounds, and training resources (11)

– Barriers: Barriers: insufficient insufficient time or resources, resources, organizational organizational a

barriers, and lack of a quality improvement infrastructu

organization (12).

– Involvement of hospital leadership, project leadership, q

improvement experience, education, and motivation (1

– Hand washing campaigns campaigns (14).

– Safety culture (15)

– Previous education, teamwork and culture intervention

leadership, feedback and support of outside quality imp

expertise (16). 

3/8/2012
Wrong site surgery uni l versa prot l oco

• Participation of the surgeon in p p reo 

verification, participation of all surgicmembers members in the “time out”, and the s

s

explicitly empowering team member

up if concerned concerned and acknowledging acknowledging c

c

when expressed (17). Strong correlat

between techi l n ca error and teamwo

(18).

3/8/2012
.

• Computer physician order entry (CPOE) 

computer d ii ec s on support system (CDS

– Regulation (100% of the 23 papers review

incentives (100%), organizational size and 

(100%), teamwork (74%), leadership (30%

(9%), training (61%), internal incentives (5

and feedback (35%), and quality improvem

consultants (13%).

3/8/2012
Summary - Review found

• Little empirical evidence evidence• Weak stren

g (g y th of evidence 

(

e

g surve

y o

implementers)

• B t u

t some evidence th t a

t cont t ex

t f t ac

tors 

implementation

• These factors vary between organisatio

intervention

, and stage of implementat

implementat

• This for these 5 PSPs – probably so for 

CSIs

3/8/2012
Research challenges (Øvretveit 

• Patient safety could be speeded and costs sa

recognition that many “interventions” are no

time changes but evolve over time in interac

their context – perhaps better described as “

vol ” utions

”.

• Studies would be more useful to implemente

defined more clearly what is the intervention 

not (“context”), and which aspects of contex

may be important to implementation and oueffectiveness.

3/8/2012
Guidance for research into the context of quality improvement, implementation or innovation in 

health services

V2, 15mar2012

John Ovretveit, jovret@aol.com, 

Director of Research, Professor of Health Innovation Implementation and Evaluation, 

Medical Management Centre, The Karolinska Institutet, Stockholm, 

With thanks to Brian Mittman, CIPRS, Veterans Health Administration, USA.

Introduction

Context is everything which is not the intervention

Only some elements of the context affect implementation 

The purpose of context-sensitive research is to discover which elements most influence 

implementation of the intervention, and hence the outcomes.

Purpose of the guide:

To help decide which context data to collect, for different types of research, and how to 

analyse and report it to answer the research user’s questions.

Who for:

Primarily quality and safety improvement researchers, but also researchers studying any 

change or intervention to patients, providers or services.

Improvers and decision makers, to understand more about conditions necessary for 

successful implementation, and what research can and cannot provide to help.

What the guidance covers:

1. When do I need to collect data about context?

2. Why do I need to decide the research user, questions and purpose?

3. How do I match the research design to the question?

4. Which data to collect?

5. How to collect the data?

6. How do I analyse data to find how much the context inlfuenced implementation?

7. How do I report context data?

8. Checklist

9. Terms used in this guidance

10. Appendices: frameworks for deciding which data to gather about intervention 

implementation and context

11. References and resources

1. When do I need to collect data about context?

In all evaluation research. To help users assess generalisablity.

For controlled experimental trials: context includes characteristics of patients selected for 

the trial. Most aspects of context are controlled for their influence on outcomes by having a 

comparison group, who are the same apart from the intervention, and ensuring full 

implementation of the standardised intervention. 

For observational studies (eg case studies, process evaluations): more details about certain 

context elements needs to be collected and about how these may change, depending on 

who the user of the research is, and as described below.

For action research or quality improvement reports: the same level of detail and rigor will 

not be possible, but some description is needed if others are to assess how similar and 
different theor context is, and their likely success in implementing the intervention, as 

described below.

2. Why do I need to decide the research user, questions and purpose?

Why - To save collecting data you do not need, and to decide the best design and the data 

needed.

Does it work? If the purpose is to achieve high certainty that the intervention is associated 

with certain changes in measurable outcome variables, then a ranomised or controlled 

experimental design is best, if this is feasible.

If the research users are primarily decision-makers or implementers, and if a controlled trial 

is not feasible, then their main question is: what should decide the design – the next item.

3. How do I match the research design to question?

Know the range of designs you could use, and their costs and timescales, and the one or two 

answers they are most well designed to answer. Best reference: Grol et al 2003, Craig et al 

2008, Fan et al 2010.

4. Which data to collect?

Depends on the research user’s questions.

Generally – Collect data to describe: 

the contents of the intervention evaluated (the before/after change-difference which was or 

is to be implemented), 

the implementation actions (and structure or systems for implementation), 

the surrounding elements which may affect implementation at different levels (eg group, 

facility, organisation, area, national).

About context: 

For controlled experimental trials: guidence for which aspects of context to collect data 

about and report is given in Consort 2012 reference below).

For observational studies (eg case studies, process evaluations) formulate your theory of 

which 5-10 context elements are most critical to effective implementation and will help and 

hinder implementation. Formulate this by finding previous research into a similar type of 

intervention or programmes, and look for which aspects of context this research suggests 

may be influential on implementation. If there is no such research, then use one of the

generic context frameworks or assessment models listed below in the appendix. 

Which context features help and hinder implementation will be different for, eg a patient 

fall prevention programme to those for implementing a computer decision support system

for physicians, to those for implementing a clinical bundle in an ICU to reduce central line 

infections. Some may be similar (higher level leadership support).

Generally the relevant context elements are features of the local facility, health system, 

funding and regulation that will affect implementation, and availability of investment 

resources, and payments for operations (eg reimbursement).

Data will be needed about whether or how these context influences changed over time, if 

the data is to be used to explain outcomes and implementation success that was observed.

For action research or quality improvement reports: If others are to use the study or 

reports, then some description of context will be necessary, following some of the general 

priciples above.

5. How to collect the data

Depending on your type of study and questions to be answered, the data can be collected by 
using 

o measures of aspects of context (eg see Shekelle et al 2011 for measuring 

instruments),

o interviews or surveys of informed observers,

o both of the above.

If money and time is short, then select a reasonable cross section of informed observers and 

ask them,

o which aspects of context helped and hindered implementation?

o from this list which was the most influential? (try to get them to rank order the list). 

Why do you say this (more evidence please)?

o over the period of implementation, did any of these influences change in strength, 

in a way in which their influence on implementation was significantly more, or less? 

6. How do I analyse date to find how much the context inlfuenced implementation?

For quantitative measurement data: when comparing different implementation cases, 

relate context variables to variables summarising success in implementation, or to outcome 

variables.

For qualitative data (eg from the informants, as suggested above):

o See if a pattern of similar views begings to show, 

o or whether there are strongly divergent views,

o and if these are associated with the interviewee’s role position or not. 

7. How do I report context data?

In a scientific publication: RCT and controlled trials: see Consort 2012 references below.

Observational: see SQUIRE reporting guidance (not much on context), or other papers 

(appendix) to see how they presented context data.

In an evaluation report: probably the best way is a model diagram summary with the details 

of what is in the boxes of the diagram in a separate table.

8. Checklist

 Who is the primary research user and their questions, and by when should the research 

deliver answers?

 Which are the potential research designs and why the chosen one?

 Which data are needed for a) others to replicate the intervention, b) others to assess how 

similar and different their context is, c) to be later able to explain the degree of 

implementation observed and what accounts for this and the outcome.

 Which data sources and types of data might provide these data?

 What is the data gathering plan, and methods and the times for gathering the data?

 How will you analyse data to find how much the context inlfuenced implementation?

 How will you present the context data?

9. Terms used in this guidance

Intervention: what ”comes between” what would otherwise have happened to the subject 

intervened-in, be it patient, provider, group or organisation. 

Implementation: sometimes it is useful to separate the actions, steps and structures 

established to deliver or carry out the intervention.

Context: everything which is not the intervention. 

Intervention/context boundary: a dividing line between actions taken to change a person, 

group or organisation (subject), and the surrounding influences which may also affect the 

subject but which are not considered part of the intervention.

Simple intervention or implementation: one item or action, constant or one time.
Complex intervention or implementation: multiple actions, possibly at different or 

overlapping times, possibly with mutually reinforcing.

Multiple level intervention or implementation: different and coordinated actions taken to 

change subjects at more than one level, such as actions to change individuals, and one or 

more groups, and organisations and to change influences eminating from higher-levels such 

as regulations or financing.

Cost of intervention and implementation: the resources used and quantified in money terms 

to a) operate the new way of working after the intervention (ie sustaining the intervention) , 

and b) used to implement the intervention (this may be short term implementation actions).

Three types of intervention and three types of research 

Depending on which type, you will need to need to collect and report slightly different data about 

intervention and context.

Three types of intervention

Prescribed: the intervention is carefully specified and the implementation ensures 

fidelity to the specification (eg drug trial tries to ensure the correct patients take the 

medication in correct doses at correct times)

Part-prescribed: a precribed ”core”, but adaption is allowed or encouraged of other 

aspects of the intervention (eg the training materials and process of the sessions are 

prescribed, but some latitude allowed about frequency (within the prscribed frequency 

of once every two to four weeks).

Principle intervention: the intervention is a ”change concept” such as ” Minimize hand 

offs” or ”Reduce multiple brands of same item” (see Langley et al 1996, pp 295-359). 

These require considerable interpretation by implementers, although examples may be 

given to implementers.

Three types of research

Experimental and quasi-experimental: eg RCT or other controlled trial.

Observational (”hands-off”): eg descriptive or evaluative study of an intervention 

programme or one without controls, possibly retrospective.

Action research or quality improvement project: where the research is intended to 

contribute to a change to the intervention to improve it in some way.

10. Appendix: frameworks for deciding which data to gather about intervention implementation

and context

Clinicians implementing clinical research:

o PARHIS Guidance (Stetler et al 2011) and related: 

o Context Assessment Index (CAI) (McCormack et al 2008) – the best validated 

o Alberta Context Tool (ACT) (Estabrooks et al 2008).

o CFIR Damschroder et al 2009, and French et al 2009

Quality Improvement Projects: 

o MUSIQ (Kaplan et al 2012) or French et al 2009)

o See also ORCA readiness for change assessment (Helfrich et al 2009).

o EMR implementation (Øvretveit et al 2007) 

o Generic change/innovation French 2009 Greenhalgh 2004

See also measures of aspects of context – for safety practice implementation in Shekelle et 

al 2011 which include:

o Patient safety culture,

 AHRQ surveys; 
 Patient Safety Climate (Pronovost et al 2003, Kho, et al 2005); 

 Safety Climate Survey (Kho et al 2005) 

o Teamwork, 

 ICU Nurse-Physician Questionnaire (Shortell et al 1991).

o Leadership, 

 ICU Nurse- Physician Questionnaire (Shortell et al 1991)

 Leadership Practice Inventory (Tourangeau & McGilton 2004)

 Practice Environment Scale (Lake 2002).

Stetler C Ritchie J Rycroft-Malone J Schultz A Charns M 2009 Institutionalizing evidence-based 

practice: an organizational case study using a model of strategic change Implementation Science 

2009, 4:78 doi:10.1186/1748-5908-4-78 (Pettigrew strategic change model 

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of 

health services research findings into practice: a consolidated framework for advancing 

implementation science. Implement Sci 2009, 4:50. 

Grol RPTM, Bosch MC, Hulscher MEJL, Eccles MP, Wensing M: Planning and Studying Improvement 

in Patient Care: The Use of Theoretical Perspectives. The Milbank Quarterly 2007, 85(1):93-138. 

Helfrich C, Damschroder L, Hagedorn H, Daggett G, Sahay A, Ritchie M, Damush T, Guihan M, Ullrich 

P, Stetler C: A critical synthesis of literature on the Promoting Action on Research Implementation in 

Health Services (PARIHS) framework. Implementation Science 2010, 5(1):82. 

Kaplan HC, Brady P, Dritz M, et al. The influence of context on quality improvement success in 

healthcare: a systematic review of the literature. Milbank Q 2010;88:500e59.

Kaplan, H Provost, L Froehle, C Margolis, P 2011The Model for Understanding Success in Quality 

(MUSIQ): building a theory of context in healthcare quality improvement, BMJ Qual Saf (2011). 

doi:10.1136/bmjqs-2011-000010

Kitson A, Harvey G, McCormack B: Enabling the implementation of evidence based practice: a 

conceptual framework. Quality in Health Care 1998, 7(3):149-158. 

Kitson A, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A: Evaluating the successful 

implementation of evidence into practice using the PARiHS framework: theoretical and practical 

challenges. Implementation Science 2008, 3(1):1. 

McCormack B, Kitson A, Harvey G, Rycroft-Malone J, Titchen A, Seers K: Getting evidence into 

practice: the meaning of ‘context’. J Adv Nurs 2002, 38(1):94-104. 

ORCA Helfrich C Li Y Sharp N Sales, A 2009 Organizational readiness to change assessment (ORCA): 

Development of an instrument based on the Promoting Action on Research in Health Services 

(PARIHS) framework Implementation Science 2009, 4:38 doi:10.1186/1748-5908-4-38:

Shekelle PG, Pronovost PJ, Wachter RM, Taylor SL, Dy SM, Foy R, Hempel S, McDonald KM, Ovretveit 

J, Rubenstein LV, Adams AS, Angood PB, Bates DW, Bickman L, Carayon P, Donaldson L, Duan N, 

Farley DO, Greenhalgh T, Haughom J, Lake ET, Lilford R, Lohr KN, Meyer GS, Miller MR, Neuhauser 

DV, Ryan G, Saint S, Shojania KG, Shortell SM, Stevens DP, Walshe K. Advancing the science of 

patient safety. Ann Intern Med. 2011 May 17;154(10):693-6. 

http://highwire.stanford.edu/cgi/medline/pmid;21576538
Van dusen Lukas et al Strengthening organizations to implement evidence-based clinical practices, 

Health Care Manage Rev, 2010, 35(3), 235-245.

Øvretveit J. Understanding the conditions for improvement: research to discover which context 

influences affect improvement success. BMJ Qual Saf 2011;20(Suppl 1):i18e23. 

Øvretveit JC, Shekelle PG, Dy SM, et al. How does context affect interventions to improve patient 

safety? An assessment of evidence from studies of five patient safety practices and proposals for 

research. BMJ Qual Saf 2011;20:604e10. 

11. Other references and resources

Craig, P et al (2008) Developing and evaluating complex interventions: new guidance, MRC, London: 

Medical Research Council www.mrc.ac.uk/complexinterventionsguidance 

Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S, 2008 The SQUIRE development group. 

Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual 

Saf Health Care 2008;17:i3–i9.

Fan, E Laupacis A Pronovost P et al 2010 How to Use an Article About Quality Improvement. JAMA. 

2010;304(20):2279-2287 (doi:10.1001/jama.2010.1692)

Grol R, Baker R, Moss F, editors. Quality improvement research: understanding the science of 

change in health care. London: BMJ Books, 2003.

Kernick, D (ed) (2003) Getting health economics into practice, Radcliffe Medical presss, Oxford.

Langley A. (1999) ‘Strategies for theorizing from process data’, Academy of Management Review, 24: 

691-704.

Langley GJ, Nolan KM, Nolan TW, et al. The improvement guide: a practical approach to enhancing 

organizational performance San Francisco: Jossey-Bass, 1996.

Luck J, Parkerton P, Hagigi F. What is the business case for improving care for patients with complex 

conditions? Journal of General Internal Medicine 2007;22(Suppl 3):396-402.

Mauskopf J, Sullivan SD, Annemans L, et al. 2007 Principles of Good Practice for Budget Impact 

Analysis: Report of the ISPOR Task Force on Good Research Practices – Budget Impact Analysis. 

Value in Health 2007;10(5):336-347.

Pettigrew AM 1992 The Character and Significance of Strategy Process Research. Strategic 

Management Journal 1992, 13(8):5-16.

Reiter, K. L., Kilpatrick, K. E., Greene, S. B., Lohr, K. N., & Leatherman, S. (2007). How to develop a 

business case for quality. International Journal for Quality in Health Care, 19, 50–55.

Van de Ven, AH 2007 Engaged Scholarship: A guide for organizational and social research. Oxford 

University Press, 2007.

Øvretveit, J 2012, Do changes to patient-provider relationships improve quality and save money? 

Volume 1 summary. The Health Foundation, London. www.health.org.uk Or web site 
http://public.me.com/johnovr 

CONSORT 2012 www.consort-statement.org

For RCTs: statement for reporting (and later revisions) www.consort-statement.org

http://www.consort-statement.org/consort-statement/3-12---methods/item5_interventions/

For ”intervention”, the above states: The description should allow a clinician wanting to use 

the intervention to know exactly how to administer the intervention that was evaluated in 

the trial.(102) For a drug intervention, information would include the drug name, dose, 

method of administration (such as oral, intravenous), timing and duration of administration, 

conditions under which interventions are withheld, and titration regimen if applicable. If the 

control group is to receive “usual care” it is important to describe thoroughly what that 

constitutes. If the control group or intervention group is to receive a combination of 

interventions the authors should provide a thorough description of each intervention, an 

explanation of the order in which the combination of interventions are introduced or 

withdrawn, and the triggers for their introduction if applicable.

Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of 

randomized controlled trials. The CONSORT statement. JAMA 1996;276:637-9.

Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. CONSORT for reporting 

randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS 

Med 2008;5:e20.

[PMID: 18215107]

SQUIRE standards for reporting quality improvement projects 

http://squire-statement.org/guidelines/explanation_and_elaboration/9_planning_the_intervention/

Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S, the SQUIRE development group. Publication 

guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health 

Care 2008;17:i3–i9.

http://conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2012/index.html NIH