Realist Evaluation: 

What Works for Whom What Works, for Whom

in What Circumstancein What Circumstance

Shirley M. Moore, RN, PhD, FAAN

Edward J. and Louise Mellon Professor of Nursin

and Associate Dean for Research and Associate Dean for Research

Frances Payne Bolton School of Nursing

Case Western Reserve University

Cleveland, Ohio
Poll Question #1 Poll Question #1

Have you ever used realist evaluation 

research or program evaluation? 

- Yes- No
Realist Evaluation Realist Evaluation

What works, for whom, in what 

circumstances 

A good fit for Implementation Researc
Close examination of h

variables are connect

An intervention is onl

yas strongas the w

link in its implementation chain. 

Explanatory models are needed to provid

evidence to strengthen every link in its im

plementation chain. 
Realist Evaluation Realist Evaluation

 Draws from the natural sciences using 

Draws from the natural sciences using 

comparisons 

 Makes use of qualitative and quantitat Result is a middle Result is a middle 

-range theory of how range theory of how 

something works 
Key Assumptions Key Assumptions

Interventions/Programs are “theories inc

-

Th i h bl i di 

hey a

im to c

hange problemat

ic condi

t

patterned behaviors patterned behaviors 

- They hope that the approaches introduc

achieve the aim 

- They incorporate ideas about how they 

They incorporate ideas about how they 

achieve their aims 
Assumptions (cont) Assumptions (cont)

 The incarnate theories (mechanisms) ar

unrecognized or poorly articulated 

 The incarnate theories are apt to change 

The incarnate theories are apt to change 

time as interventions/programs unfold 



Thi i / e 

intervent

ion

/program wor

k diff 

s differe

amongst different subgroups 

 Interventions are devised and work thro

ideas and intentions of those implicated ideas and intentions of those implicated 
Focus of Realist Evalua

Focus of Realist Evalua

Identify the mechanism of action that 

g enerates chan

ge in the intervention

Study the contexts that are most condu

positive outcomes 

(contextual factors are not “confounders

controlled) controlled) 
Complex Interventions Complex Interventions

 Wide range and variability of possible ou Diffi lt t d di i th d li d 

Difficulty s

tan

dardi

z

ing the 

d

elivery an

d 

 Variabilit

y in the tar

get 

p

o

pulation 

 Sensitivity to features of local context 

Degrees of flexibili ty or tailor

ing o

f t

h e 

i

permitted 

 Long causal chains linking the interventi

outcome(s) outcome(s) 
Assessment of 

Complex Interventions 

Research applications  self -management program for heart failure p

management program for heart failure p

 internet intervention for home recovery follo

replacement replacement 

 sleep management program for persons in nu

homes 

Pro

gram evaluation app lications

 professional education training programs  interprofessional education programs interprofessional education programs  institution of new clinical guidelines 
-

Implementaion EvaluatImplementaion Evaluat

Implementation Evaluation is typically co Multi-objective  Multi -agency  Multi-site  Frequently seek Frequently seek “wholesale transformat

wholesale transformat

 Shift institutional philosophy as well as 
Realist Evaluation Realist Evaluation

 Interventions/Programs are inspected fo

theories of change ( o r propositions abtheories of change 

(

y

o

ur propositions ab

they work) 

 The evaluation then tests the adequacy o

theory in different contexts theory in different contexts 

 The end product of inquiry being a bette

und di f hi h id k f 

derstanding o

f whi

ch ideas work 

for w

what circumstances, in what respects, a
Key Terms Key Terms

 Mechanism

3⁄4 What is it about programs or interventions that brin

effects “active ingredients” 

 Context 

3⁄4 Those features of the conditions in which programs 

interventions are introduced that are relevant to the the mechanism 

 Outcome Pattern 

3⁄4 The intended or unintended consequences of progra

interventions resulting from the activation of differemech i an sms in diff t t erent contexts
CMO Matrix CMO Matrix

Context Mechanism Outc

C1 

M1 

O

C

2 

M

2 

O

C3 

M3 

O
CMO Matrix CMO Matrix

Context Mechanism Outc

C1 

M2 

O

C

2 

M

2 

O

C3 

M2 

O
CMO Matrix CMO Matrix

Context Mechanism Outc

C1 

M1 

O

C

1 

M

2 

O

C1 

M3 

O
VA Quality Scholar CMO 

VA Quality Scholar CMO 

Context Mechanism Ou

C1 

A it 

Increased systems thinkin

g of 

QI team. 

Better iin 

patie

Across sites g Q p resultin

projects

C2 QI project team collect more focused Better i

in patie

Type of Learner (predoc or postdoc) 

data. resultin

projects

C3 

Bk d fF l 

QI projects are done more quickly and 

Better iin patie

Bk

gd 

of 

Facu

lty more quickly and thus change happens 

faster. 

in patie

resultinprojects
Stages in Realist Evalu

Stages in Realist Evalu

1) Elicit a

4) Revise understanding of CMO configurations as a theory on prelude to a further round programm

programm

1) H th i whom in w

circumsta

prelude to a further round 

of theory refinement 1) Hypothesis 

4) Theory Testing 2) Data Collection

3

) Data Analy

sis 

3) Analyse outcome 2) Collect dapatterns to see which can appropriate and which cannot be contexts and explained by initial theory 
Methods of Realist Eval

Methods of Realist Eval

1F l . 

Formu

late context-mechian

ism-outcome patte

configuration (CMO) hypotheses 

2. Devise methods of testin

gCMO hyp otheses

3 Test CMO hypotheses 3. Test CMO hypotheses 
Series of Pilot Studies Series of Pilot Studies

A goal is to draw lasting lessons about X 

process, intervention, program), across on the basis of a series of “relativel

y” l

pilot studies 

Focus is on the transferability, generaliza

and external validity of findings from c

studies studies 
Case Study Comparison

Case Study Comparison

C1 

M1 

O1 
Methods for Multiple 

Study Comparison

Data sources: Obser ation Obser

vation 

Interviews Document analysis Data Analysis: Pattern making Pattern making Scatter plots Data Matrix Data Matrix 

Tables 
Generalizing from the 

Particular 

The aim is to draw big conclusions fr

small number of studies 

Analytic generalization 

Can consist of countless Abstraction

or 

Particular instances
Based on fields where case study analysis 

such as comparative historical methodsuch as comparative historical method

 Small n analysis Comparative case study

Purpos

ive sampli ng

 Process tracin

g

 Typological models  C fi i l l i Configurat

iona

l ana

lys

is (Skocpol, 198

& B
Study Design and 

Intervention Comple

Predictive Knowledge 

RCT Realist Evaluation 

“Simple” 

Intervention 

Case QI Series Project 

Exploratory Knowledge Exploratory Knowledge 
Ground Clearing Issues Ground Clearing Issues

Usually no statistical or probability samplInsider perspective is important Insider perspective is important Systems studied are often changing as we 

them (provisional nature of findings) them (provisional nature of findings) 
1 Developing the Theori

1. Developing the Theori

Surfacing, articulating, prioritizing and 

formalizing program theories 

A set of theoretical statements to be tes

developed developed 
Example of a Theory Chart for Institu

Change to Institute Self-Management Ca

Theories of Change – seven key domains 

1. Procedures - all procedures must be made “user-friendly” 

2 Setting 2. Setting – patients will have more control over form and function 

patients will have more control over form and function 

3. Role (patients) – become more involved and proactive 4. Role (staff) – adjustment of role toward education and support 5 Routine 5. Routine – distortion in current distortion in current routines must occur to make this 

routines must occur to make this 

6. Authority – transfer more power to patients from staff 7. Efficiency – transfer power to cheapest provider should reduce co
2 Choosing the Cases 2. Choosing the Cases

 Sizing up the small n  Multiple case studies to explore the relative suc

Multiple case studies to explore the relative suc

different mechanisms as they are played out in 

contexts contexts 

 Similarities and differences in different context

exp

lored

 Discrepancies in the theory are observed  Choice of each successive case organized unde

sam

pling – what comes next in the case stud

y pg c
3 Assembling the Evide

3. Assembling the Evide

Process tracking – to trace the operation 

causal mechanism(s) at work in a givecausal mechanism(s) at work in a give

situation 

M1,2,3,etc 

A B 

Actual processes (mechanisms) are tracked Qualitative as well as quantitative data – mapped ont
4 Analysis 4. Analysis

Comparing further contexts in the search 

general models 

Contexts C1

M1,2,3,4,etc 

O1,2,3,4,etc 

Theory – new model is developed new model is developed 
New CMO Matrix Cre

Containing New Mecha

Context Mechanism Outc

C1 

M1 

O

C

2 

M

2 

O

C3 

M3 

O
VA Quality Scholars 

Evaluation: Questio

Wh t i th i t f ddi i t f What is the impact of adding interprofes

learning to the VAQS Program? 

What are the mechanisms through whic

i t n erprofessional program elements p

their impact? 

What characteristics of the local environ

cont ib t d t b tt i t f i 

tributed to better interprofessiona

learning? 
o ec s.

Program Outcomes 

1. Better improvement in patient outcomes resul

Q p j I projects.

2. Faster cycles of improvement on QI projects 3. Increased research publications re: QI. 4. Increased cross-di li scip nary QIcit ati ons.5. Increased number of p yh sicians as active mem

teams.

6. I d ti f ti f h lth f Increased satisfaction of healthcare profession

workforce.
Potential Mechanisms Potential Mechanisms

1. Increased sense of and application of 

kno ledge held b different members kno

wledge held 

by different members 

team. 

2. Use of a larger number of QI methods3. Increased systems thinking of QI team Increased systems thinking of QI team4. Increased number of contextual variab

into account when QI projects designe
Contexts

1. Does interprofessional (IP) learning differ 2 Do learning activities of pre and postdoc p2. Do learning activities of pre and postdoc p

differ? 

3. Does the impact of IP learning differ by pre 

postdocs? 

4. Does the impact of IP learning differ by me

background? background? 

5. Does the impact of IP learning differ by se

background? 
Ctt Mh i O 

VA Quality ScholarCMO Matrix 

Context Mechanism Ou

C I d t B tt i 

1 

Across sites 

Increased systems 

thinking of QI team. 

Better im

in patien

resulting resulting 

projects. 

CIncreased systems Better im

2 

Type of Learner 

Increased systems 

thinking of QI team. 

Better im

in patien

Type resulting 

of Learner 

(predoc or postdoc) 

g 

projects. 

C3 

Bkgd of Faculty 

Increased systems thinking of QI team. 

Better imin patien

resulting 
Challenges in Using Realist E

Challenges in Using Realist E

1. Distinguishing between context and 

mechanism 

2. Is there an endpoint? When to Is there an endpoint? When to 

“close 

”

3. Time-consuming 4. Can realist evaluation studies be replic5. How does Realist Evaluation differ fro

How does Realist Evaluation differ fro

traditional mediator/moderator analysi
Realist Evaluation Realist Evaluation

A revised gold standard appropriaA revised gold standard appropria

the evaluation of complex interve
Questions