Strengthening Organizations to Imp

Evidence

-Based

Clinical Practices

Carol VanDeusen Lukas, Carol VanDeusen Lukas, EdD

Center for Organization, Leadership & Management Res

VA Boston Healthcare System VA Boston Healthcare System

June 2011
COLMR stud

y team

• Ryann L. Engle, MPH• S ll K H l MBA S

ally 

K. 

H

olmes, MBA

• Victoria Parker, D.B.A.• Marjorie Nealon Seibert MBA Marjorie Nealon Seibert, MBA• Michael Shwartz, PhD• Jennifer L. Sullivan, PhD Jennifer L. Sullivan, PhD

+

VISN leaders
Study aim

• To implement & evaluate an organizational 

model hyp g 

othesized to strengthen the abilit

healthcare organizations to bring evidencebased clinical practices in routine operation
O i ti l d l t t d Organizational mo

d

el tes

t

e

d

Active top leadership 

commitment

Improved evidence-base

clinical practices

Links to senior 

management structures and management structures and clinical practices

processes

M lti u -di i li id disciplinary evidencebased

clinical process redesign 
Examples of operational elements 

of model

• Senior leadership commitment Senior leadership commitment

– Set high expectations for improvement– Invest own time on improvement-related activ

• Linkages to senior leadership

– Appoint a leadership champion – Identify clear path for team reporting to senio

Identify clear path for team reporting to senio

leadership for accountability & support

• Multi-disciplinary evidence-based redesign team

– Appoint members from affected disciplines & – Use systematic methods to analyze processe

performance
Study questions

• Is the organizational model implemented with hig

fidelity to the model design?

•

Are medical centers that implement the model w

fidelity more successful in improving performanc

targeted evidence targeted evidence

-based clinical practice than m

based clinical practice than m

centers that implement fewer elements?

• Why is the organizational model implementation 

successful or not successful?
St d d i Study design

• Designed in collaboration with directors and chie

medical officers of 3 participating VISNs, or NetwDept of Veterans Affairs (VA) Dept of Veterans Affairs (VA)

• Original study design: Mixed-methods pre-post 

comparison group intervention in 16 medical cencomparison group intervention in 16 medical cen

networks 

 1 Network randomly selected to implement th

organi i l dl izational model

 Other 2 Networks served as comparison grou
Clinical focus is hand-hygiene

• Clinical redesign process component require

clinical focus to engage staff

• Compliance with evidence Compliance with evidence

-based hand based hand

-hygie

hygie

guidelines evidenced-based and high priority

 Fundamental as

pect of infection control

• One of the simplest yet most effective 

shown to reduce nosocomial infections

 R i t f Th J i t C i i Requiremen

t o

f The 

J

oin

t

Commission

 New high priority for improvement in VA a

study designstudy design
Intervention in 7 medical cente

• Initial site visit for introduction of the project and 

assessment of baseline state of the model com

• Follow up to work with site to complete impleme

plan

• Repeat visits/phone calls every 4 Repeat visits/phone calls every

4

-6 months ove

6 months ove

years

• VISN-wide support

– Shared learning groups monthly– Leadership consortium quarterly
Th d t ree data sources

• Local observations of hand Local observations of hand-hygiene compliance hygiene compliance

– compliance rates

• Semi-structured interviews by 

 research team du

visits

– fidelity ratings– factors affecting implementation fidelity

• Site-visit impressions journals kept by research 

– factors affecting implementation fidelity factors affecting implementation fidelity
Fidelity of implementation Fidelity of implementation

Data source: Notes from semi-structured interview

as basis for ratings and as basis for ratings and narrative evidence of fi

narrative evidence of fi

each model element, completed by site-visit reteam at end of each visit

Measures:• Ratings on a 0-4 scale (0= element not present

element in place and consistently used as inten

• Component scores created by aggregatin

Component scores created by aggregatin

elements and calculating an unweighted 

• Overall site fidelity ratings calculated mea

component scores component scores

• Narrative evidence analyzed qualitatively by cr

comparisons structured by fidelity instrument
Example of fidelity rating tool

II. Management Structures and Processes 

Rating Narrative: Examples

1. Alignment and accountabilitya. Create incentive and reward 

t t t fh d

Successes are recogncelebrated. Where haperformance varies fro

structures to encourage use of handhygiene guidelines and, where needed, fundamental redesign of clinic processes 

performance varies froindividual(s) are chargaction and reporting b

b. Establish structures to link the hand-hygiene improvement efforts to senior manage

ment such that senior 

Structures and procesan oversight committe

se o a age e t suc t at se o reporting paths to senmanagement gives the initiative high priority and holds the design team accountable. • Reporting relationship to senior 

leadership team 

• Member of senior management 

d hi f bl i

epo t g pat s to se 

monitor hand-hygiene progress regularly to himprovement team acto provide support. A member of facility lecharged as formal chaproject and liaison to s

and chief or comparable service line leader in infectious disease as formal champions to advocate for redesign and help the design team solve problems

p jleadership. S/he assuleadership reviews mecorrective action is takProject champion assuredesign team has neresources and helps thproblems. 

2. Integration and resource support a. Link improvement efforts to senior management such that senior management facilitates cooperation across organizational boundaries and provides other resources to design team. 

Project champion and leadership are actively support the project as obtaining resources acorganization. For exa Protected time to 

initiative 

 Staff eq Staff, equipment a

uipment a

needed 

 IT support as need
Q tit ti fid lit i Quantit

ative fid

elity scoring

ks 

gn 

l 

ll 

er 

Facility 

Fidelity: 

eadership 

Fidelity: 

agement link

elity: redesig

efforts 

elity: overal

elity: overal

hange from 

baseline 

lity rank ord

le

mana

Fide

Fid

Fide

ch

Fidel

A 4.00 4.00 3.85 3.95 2.82 1 B 4.00 3.20 2.95 3.38 2.11

2

C 3.75 2.60 3.35 3.23 1.99 3 D 3.00 3.00 3.50 3.17 1.84 4 

E 2.00 2.20 2.25 2.15 1.21

5

F 2.50 1.80 1.65 1.98 0.41 6 

G 2.50 1.25 0.50 1.42 -0.05 7 

Note: Facilities A-D are high-fidelity sites, E-G are lower fidelity sites. 
Hand-hygiene compliance hygiene compliance

Data source: Observations of hand-hygiene comp

measured th h t t d b ti b 

d through structured observations by m

center staff

Measures: 

• Percent compliance for each observation period 

level. 

• Effect size of improvement in compliance calcula

comparing the baseline 3-month periods to the lmonth periods of the study

• Statistical significance tested through a weighted 

squares regressi d l ith ion model with:

• time (i.e., month) as independent variable• compliance percent as dependent variable • sample size in each data collection period as weight sample size in each data collection period as weight.
Hand-hyg p 

iene compliance ordered 

fidelity ranking

Facility erence pre- period 

rence post- period fect Size: 

ths pre-post fect Size: 

ths pre-post ssion model: 

coefficient ssion model: coefficient p 

value sion model: 

r-

squared F

Adhep

Adhe

p

Eff

3 mon

Eff

6 mon

Regres

time 

Regres

time c

Regresss

A 67.6 92.9 .67 .69 1.29 0.00 0.72 B 74 2. 91 5. .48 .40 0 98 . 0 00 . 0 57 .C 37.4 80.9 .92 .22 1.41 0.01 0.36 D 81.7 96.8 .52 .53 0.97 0.00 0.53 

E 69.1 75.2 .14 .07 0.20 0.11 0.62

F 61.5 68.3 .14 -.27 -0.40 0.47 0.03

G 80.1 70.8 -0.22 -0.29 -0.47 0.17 0.08 

Note: Facilities A-D are high-fidelity , sites E-G are lower fidelity sites.
F t ff ti fid lit Fac

tors affecting fid

elit

y

Data source: Notes from semi-structured interview

impressions journals completed by research teasite visits 

Measurement: Notes coded by members of team 

not visit the particular site being coded

• Thematic analyses beginning with individual site 

Thematic analyses beginning with individual site 

• Data organized into matrices for cross-site comp
Limitations

• Implementation in one Network in VA• Hand-hygiene observations done locally hygiene observations done locally• Different team members interacted with each sit

the intervention team actions might have differed unmeasured ways

• Weak commitment in comparison Networks
St d i li ti Study implications

• C fi i h i l i f EB Confirms expectations that implementation of EB

practices that cut across multiple processes of ccomplex process with many possibilities for failu

• I l t ti i t th d b f 

Implementation is strengthened by presence of 

components that interact and are mutually reinfo

– Active leadership commitment to improving the targete

practices, 

– Robust clinical process redesign to engage staff and in

evidence-based practices in routine operations

– Links to management structures and processes to sup

andi d i ntegrate redesign

• Implementation is strongly influenced by organiz

elements and context
• Lukas, , C.V. Eng, , , , , 

le, R.L., Holmes, S.K., Parker, V

Nealon, Seibert, M., Petzel, R.A., Shwartz, M., SJ.L. (2010). Strengthening organizations to impleevidence-based clinical practices based clinical practices. Healthcare Healthcare

Management Review, 35(3), 235-245.
Guidance for the Design of Implementat

Brian S. Mittman Mittman, PhD

Director, VA Center for Implementa

P ti rac ce and R h esearch S t uppor

VA Greater Greater Los Angeles Angeles Healthcare Healthcare S

June 9, 2011
Implementation research in he

Implementation research is the scientific study of mpromote the systematic uptake of research findingevidence‐based practices into routine practice, and

improve the quality and effectiveness of health se

It includes the study of influences on healthcare pand organizational behavior.
Studying implementation program program effe

effe

Two very different questions1. Does it work? Is it “effective”?

Should I use it?

2. How, why, when and where does it work?

How should I use it?
Evaluating implementation program effect

Debates regarding research approach

• Most large‐scale implementation studies are trial

(experimental or quasi‐experimental evaluations

i l t ti implementation programs – QUERI Steps 4/5/6

• Implementation Implementation trials are the focus of considerab

considerab

and ongoing development regarding research ap

desig , ns methods and rep g ortin
Guidance for implementation program ev

QUERI S i erv ce‐Di d recte P j ro ect (SDP) T 

e

Motivation• implementation projects are hybrid research/practic

involving complex social/b h l e avioral phenomena

• implementation projects require a unique set of des

meth do s, skills and compet i enc es

SDP Template goal

• provide guidance in designing, conducting and docu

implementation trials
Poll question: question: Audienc Audience Composi

Composi

1. Have you served as PI or key investigator for an

implementation trial submitted for VA funding

– Yes, and it was funded
Poll question: question: Audienc Audience Composi

Composi

2. Have you served as PI or key investigator for an

implementation trial submitted for non‐VA fun

– Yes, and it was funded

– Yes, but it was not funded

– No
Disclaimer; Disclaimer; no warranties warranties expressed expressed or 

The views and recommendations contained in thpresentation presentation are based on interactions interactions with unto

numbers of VA and non‐VA implementation rese

and research research stakeholders stakeholders, but do not necessarily necessarily

represent their opinions, nor those of any VA or n

funding agency or official official.
QUERI Service Service‐Directed Directed Project Project Tem

Tem

A. Specific aims

– Implementation and science aims (short‐, long‐

– Hypotheses: intervention impacts and processe

conceptual model

B. Background Background and significance significance

– Clinical issue (morbidity, mortality, burden)– Effective Effective practice practice (clinical (clinical evidence): evidence): strength strength,

acceptance, implementation gaps (magnitude, 

to close) (QUERI Steps 2, C, E, 3A)

– Implementation processes: evidence, insights
QUERI SDP Template

C. Previous studies studies

– Current practices: determinants, barriers and fa

to change (QUERI Step 3B, D)

– Implementation strategy: appropriateness (QU

3C), theory, empii l r ca evidence base and st t a us 

4‐phase framework)
QUERI SDP Template

D. Design and methods

– Theoretical/conceptual framework and its basis

• Results of (or plans for) diagnostic analysis (QU

3B,C,D): practice determinants; barriers and fa

change

• Discussion Discussion of key conditions conditions, requirements requirements for 

– Legitimacy of clinical evidence– Motivation for change (p , ex ectations, p 

ress

– Norms (organizational, professional, consu

– Clinician, staff, consumer education and sk

– Financial, administrative, logistical and tec

and facilitators
QUERI SDP Template

D. Design and methods (continued)

– Theoretical/conceptual framework (continued)

• Applicable theory(ies) or framework(s) based o

practice determinants and barriers/facilitators change, e.g.,– organizational organizational delivery delivery requires requires organizatio

organizatio

individual clinician practices require theori

psychology

– knowledge gap requires education

– gaps in practice norms, attitudes, beliefs re

influence
QUERI SDP Template

D. Design and methods (continued)

– Implementation program details and theoretica

• Overview; table of key elements relative to the

framework

• Details of operationalization of each element

– education program design, delivery– social influence strategy

– delivery system redesign; distinguish care 

change st t ra egy (as appli bl ) ca e

• Formative evaluation, tailoring/adaptation, mo

refinement

– Implementation generally requires multi‐level, mu

programs rather than interventions
QUERI SDP Template

D. Design and methods (continued)

– U l sua care ( i) comparison) condition

– Evaluation details

• Experimental design (RCT, ITS, pre/post)

• Sites, sampling, recruitment, randomization

• Balance Balance between between internal internal and external external validity validity (a

real‐world conditions): decisions regarding sites aimplementation program delivery (e.g., staffing), 

part, by Phase 1 vs. 2 vs. 3 status of trial
QUERI SDP Template

D. Design and methods (continued)

– E l ti va uation d t il e a s: i t mpac eval ti ua on

• Outcomes (patient, system outcomes)

• Contextual factors

– Evaluation details: formative/process evaluation

• Identify mechanisms of impact and measures: va

measures, data collection collection protocols protocols

• Influences on mechanisms: variables, measures, d

• Analysis plans and methods

– Evaluation details: other

• Sustainability, spread/scale‐up potential and path• E i conom cs

• Clinical intervention effectiveness (if hybrid E/I st
QUERI SDP Template

D. Design and methods (continued)

– Management plan

• Intervention management plan

• Evaluation management plan

• Staff qualifications: intervention, evaluation
Studying implementation program program effec

effec

Two very different questions1. Does it work? Is it “eff ti ec ve”?

Should I use it?

2. How, why, when and where does it work?

How should I use it?
Implementation research challeng

• What is the nature of the desired evidence, knowled

i i ht ns g s, und t di erstanding (and the associtd a e l i earn ng

abilities)?

• How is this und t di erstanding generally prod d? uce M

e

vs. impact‐oriented approaches

• Is our current current portfolio portfolio producing producing this understanding understanding 

generating this learning? (Are prevailing research ap

designs and methods – and p j ro ects – app p ) ro riate?)

• Must we rebalance our portfolio? How?