Are you affiliated affiliated with the 

• Yes• No
What is your primary primary rol

• Researcher• Clinician• Manager or Policy Maker• Student, Trainee, or Fellow

• Other
What is your level of particip

in i l t ti implementation resear

• Have been a PI on an implementation • Have been part of a study team for an

implementation study.

• Currently developing an implementat

p p ro osal.

• No hands on experience, just getting s
The Rewarding Early Abstinenc

Treatment Participation (REA

study

Hildi Hagedorn, Siamak Noorbalooc

Carl Rimmele Rimmele, and Daniel Kivlahan Kivlahan
REAP Study Objectives

1. T t es the eff ti ec veness of an i ti ncen ve program 

large sample of Veterans with alcohol and/or 

dependence. Comparing:

y Rates of negative alcohol and drug screens du

intervention

y Rates of attendance during the interventiony Percent Percent days abstinent abstinent out of the past 30 day

12 month follow‐ups.

2. Assess the costs of the intervention intervention.3. Complete a process evaluation to inform futu

implementation efforts.
Relation to QUERI Pipelin

Pipelin

• Mainstream HSR&D Effectiveness • Hybrid Type I design

– Includes elements of a Pre‐Impleme

Study (Step 3)

• E g. ., identify identify barriers barriers and facilitators facilitators t

implementation.
Study Conditions

y 330 Veterans seeking treatment for alcohol or s

dependence dependence at two VA SUD clinics clinics randomly randomly as

as

y Usual Care: Standard care provided at the clinic 

and urine testing 2x/week for 8 weeks.

y Incentive Program: Usual care + draw for incent

canteen vouchers) when negative samples are s
Costs of the Interventio

Interventio

Vouchers M = $103 (SD=

R id U i T t C Rapid Urine Test Cups M $61 (SD 2

($5.25/cup X 11.6 visits)

M = $61 (SD=2

Alco Sensor mouthpieces Alco Sensor mouthpieces M = $3 (SD=1

($0.24/piece X 11.6 visits)

M $3 (SD 1

Staff Costs $103

($41,000 base + 30% fringe) (16 apts. of 15 min. 

Mean $ per patient M=$269 (SD=9

* Highest Cost Patient = $462
Evaluation Framework Guidi

Guidi

y RE‐AIM

y Reach: What percentage of patients approached 

participate? participate? Did participants participants differ from those th

y Effectiveness: Tests of main study hypotheses.y Adoption: Adoption: What will be the greatest greatest barriers barriers to o

adopting this intervention? How can they be ove

y Implementation: What tools will programs need 

the intervention consistently?

y Maintenance: What resources would be require

changes changes, if any, will be needed to integrate integrate the i

i

into regular practice?
Theoretical Framework Guid

Guid

y PARIHS

y Evidence: What are the staff’s perceptions o

evidence supporting this i t ti ? ntervention? Does 

intervention fit with their current clinical pra

perceived needs of their patients? patients?

y Context: What are the characteristics of the 

and leadership in the clinics? What resource

available to the clinics?

y Facilitation: What types of resources, trainin

tools would be of greatest assistance to mai

the intervention?
Linking Data Collection Collection t

Frameworks

RE‐AIM Constructs Data Source

Reach Recruitment rates.

Demographic characteristics of those agreerefusing.

Effectiveness Effectiveness Main study outcomes outcomes comparing comparing control control to 

patients: Rates of negative urine screens, ra

retention.

Adoption Adoption Observations Observations of intervention intervention in clinic.

Perceptions of staff and leadership.

Implementation Perceptions of staff.

Maintenance Perceptions of leadership.
Linking Data Collection Collection t

Frameworks

PARIHS Construct Data SourceEvidence Perceptions of staff and leadership.Context Organizational readiness measure collected 

leadership.

Facilitation Observation of intervention in the clinic.

Perceptions of staff and leadership.
Process Evaluation Tool

Organizational Readiness to Change

Assessment ( ) staff :

 Knowledge of evidence base, attitudes t

intervention, organizational context context (lea

(lea

culture, resources, etc.)

Research Team Observation Observation Log:

 Record details of interactions with staff

particularly those focusing on reactions 

the intervention, barriers to implement

recommendations for improvements.
Process Evaluation Tool

Staff Post‐Intervention Interviews:

 Reactions to the intervention, perceptio

impact of the intervention on the clinic, 

and facilitators to implementation,

recommend ti a ons for changes to the

intervention.

P t os ‐I t ti n ervention L d hi ea ership It i n erv e

 Are they going to attempt to continue t

intervention? What lead to that decisio

what modifications will they make?

Patient Post‐Intervention Interviews

 Likes, dislikes, value, improvements.
Insights from Process Process Evalu

Evalu

• REACH: Patients were enthusiastic ab

entering the p g ro ram and enj yo ed it.

• EVIDENCE: Staff were not enthusiastic 

the intervention intervention at baseline baseline. Enthusia

Enthusia

increased dramatically during interven

period.

• MAINTENANCE MAINTENANCE: Staff suggested suggested a gro

intervention would be more feasible, 

patients patients were not interested interested in this.

• FACILITATION: Change in staff attitude
Uses of Process/Formati

Evaluation

• Effectiveness Trials:

– Collect information to inform future

implementation.

• Implementation Trials:

– Assess stakeholders’ perceptions of feas

value of il i mp ementation strategy.

– Understand and adapt implementation p

real time.

– Assess impact of theoretical constructs o

implementation outcome, e.g., the value 

theory.
EQUIP:

Implementation ResearImplementation Resear

in Specialty Mental Heain Specialty Mental Hea

Supported by HSR&D QUERI

Alexander S. Young, M D. ., MSHS . . . .

Amy N. Cohen, Ph.D.

Alison Hamilton, Ph.D.

EQUIP Investigators: VISN 3, 16, 17, and 22

VA Desert Pacific Mental Illness, Research, Education, and Clinical Ce

UCLA Department of Psychiatry
EQUIP:

Research – Operations Partne

• Improve care for schizophrenia schizophrenia

– Evidence‐Based Quality Improvement

– implementation methods & evaluation

• Cli i n c‐l l eve , 15‐month cont ll d ro e ti l r a (VA QUERI)

– partnership with 4 VA regional networks

– each with 1 intervention and 1 control site (8 med

• QUERI Step 4, Phase 2‐3

– evaluation of both implementation and effectivene

Northp

Long Beach

Los Angeles Bronx

Long Beach

Waco Temple Houston Shreveport
EQUIP: Specific Aims

• Assist VA VISNs to implement evidenc

based care for schizophrenia

• Evaluate the effect (relative to usual ca

care model implementation

– on provider competency, treatment

appropriateness appropriateness, patient patient outcomes outcomes, and se

se

• Evaluate processes of and variations in 

model implementation and effectiven
Formative & Process Process Evalua

Evalua

• Using mixed methods, evaluate

processes of and variations in care

mod l e i l t ti implementation and eff ti ec 

to strengthen implementation and 

– assess accept bilit tability of the care mod 

e 

barriers and facilitators to its

implementation

– understand how the project’s strateg

tools affect care model implementat

– analyze the impact of individual care 

components on treatment appropria
At Baseline

• Strategic planning

– choice of 2 evidence evidence‐based practices practices 

implementation

– care targets: targets: weight & work outcome

outcome

• Diagnostic evaluation

– structure of care for patients with

schizophrenia varied across sites

– availability & quality of these care ta

varied across sites
Implementation Tools & Strategie

E id v ence‐B d ase Q lit ua y I t mprovement 

Evidence base:• TMAP• EQUIP 1

Leadership support

Clinical champion

-1

EBQI Quality manager

QI Informatics suppor

“infrastructure”

Provider/patient educati

“priority-setting”

Performance feedbac
Conceptual Framework:

Simpson T f rans er Mdl o e

• Stages of organi ti l za onal

change

• Validated survey measures

for each stage

• 4 Action steps:

– Exposure: Introduction and training– Adoption: Intention to try the care model throu

program leadership decision and subsequent su

– Implementation: Exploratory use of the care mo

– Practice: Routine use of the care model
Intervention Strategies and Form

E l i A i i i b STM S

STM Stages Intervention Strategies and Tools Formative E

Eva

luat

ion Act

i

v

i

t

ies by STM S

Exposure

Ð

•Secure commitment•Training and Observation of care model by Regional PIs and Project Managers•Review evidence

Add l

•Program Train•Organizational •Provider Burno

•Address values

•Identify and prioritize needs•Begin tailoring intervention

Adoption

Ð

Predisposing activities:

S

•Field notes

•VI N Implementation Teams

•Opinion leaders•Continue tailoring•Continue to secure commitment, address values

Implementation

Ð

Enabling activities:

•Patient Assessment System•Assertive care•Discuss and start using provider supports & incentives•Social marketin

g

•Project documImplementation Managers’ field Coordinators’ lo

g •Provider & Clin

(pre- & mid-imp

Practice Reinforcing activities (performance monitoring & 

feedback):•Monthly Quality Meeting/Quality Reports

•Provider & Clin(post-implemen•Organizational 

•Implementation Team Meetings•Continue tailoring with provider input•Quality Reports

•Provider Burno
Data for Formative Evalua

P 

Impl

(STM

Pre

-

Implementation

(STM: Exposure 

& Ad ti )

Implementation

(STM: 

Implementation)

Developmental Implementation In

& Adoption) Implementation)

Developmental

• Field notes• Documents 

Implementation

-

Focused

• Field notes

In

• Field 

• Ke

y 

(minutes, etc.)• ORC & Burnout Inventor

y

• Quality Coordinator logs• Documents

y

interv• ORC 

y Inven

• Key stakeholder interviews

• Key stakeholder interviews

Pro

gress-Focused

• QI tools
Evaluation

• Summative

– 801 patients

– 201 providers (clinicians and managers)

– evaluate effect on provider competency, treatment ap

patient outcomes, service utilization

– patient patient interviews interviews at 0 and 12 months– VistA data on treatment use

• Process

– characterize provider competencies, organizational rea

barriers, facilitators

– interview interview providers providers & managers managers at 0, 6, and 12 month– survey providers and administration at 0 and 12 mont

– monitor use of informatics

– logs and minutes of implementation team meetings

– field notes from local QI teams
Institutional and Personal

Readiness for Change

TCU Organizational Readiness for Change (ORC) scale
Tailoring of Implementation

Based on Readiness

• Sites A and B: more ready to change

– no specific tailoring

• Site C: less ready to change

– needs (l ) ow : h i ht e g en awareness of gaps 

use clinical champions and educational p

– mi i ss on (dt mo erate b t u l t owes of all cli i 

n c

staffing kept consistent; consistency of m

– autonomy (dt mo erate b t u l t owes of all cli 

n 

clinicians help determine how to implem

care targets
Results: Implementatio

• Organization

– strong support– collaboration between services was difficu

(nutrition, primary care wellness program

specialty mental health)

• Clinician competencies

– improved through through education education and practice practice

• Managers used data to reorganize care

– sca es l p aced l in each c c lini– routine weighing of patients established– clinical staff trained to provide services
Results: Summative

• At baseline

– 45% of patients obese, mean BMI = 30

– 70% on medications that cause substantial wei

– 22% used services, mean sessions used = 2

• As a result of the intervention, patients were 

more likely to use services (χ2 more likely to use services (χ =14, p< 01) .

– mean sessions used increased to 11– no changes at control sites

• Control site patients: 13 pounds heavier at e

(±7.6 pounds, F=4.8, p=.03)

– controlling controlling for: pre‐baseline baseline weight, baseline baseline w

psychotic symptoms, negative symptoms

Reference Slides
Multiple Data Sources:

Measuring & Documenting Implemen

EQUIP Examples

Semi-structured it i 9 participation, level ointerviews: i l t ti

leaders, clinicians, managers

implementation

Organi i li zational site 9 li i

surveys: administrators & staff

9 clinic structure, processes, change

Field journals 9 group-level dynami

implementation deta

Administrative data 9 visits, prescriptions

patient surveys 9 kiosk self-assessmeActivity logs 9 time spent on aspec

study
Stages of Formative Evalua

Impl

Pre- Implementation Implementation

Developmental

“Diagnostic” of the existing context (b li

ImplementationFocused

“Actuality

” of

In

“Uses 

oth

(baseline assessment)

• Organizational readiness for 

h

Actuality of implementation

• Barriers to change• Adjustments to

oth

• Key 

exp

• Co

c

hange di 

• Expectations of project

• Existing services

Adjustments to interventions dia

con

Progress-Focused

• Existing services and structure of 

care

“Monitoring impacts & indicators of progress toward goals”• Dose & intensit

y of intervention
References

• Young AS, Forquer SL, Tran A, Starzynski M, Shatkin J: Identifying cl

competencies that support rehabilitation and empowerment in ind

severe mental illness. Journal of Behavioral Health Services & Rese

321‐333.

• Chinman M, Young AS, Schell T, Hassell J, Mintz J: Computer‐assiste

assessment in persons with severe mental illness. Journal of Clinica

2004; 65: 1343‐1351.

• Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS: EQUIP: Im

Chronic Care Principles and Applying Formative Evaluation Method

Care for Schizophrenia. Implementation Science. 2008; 3: 9.

• Young AS, Niv N, Cohen AN, Kessler Kessler C, McNagny McNagny K: The appropriate

appropriate

medication treatment for schizophrenia. Schizophrenia Bulletin. 20

access.

• Hamilton AB, Cohen AN, Young AS. Organizational Readiness in Spe

H lth ea Care. J l ourna of G l enera It l n ernal M di i e c ne. 2010; 25 S1: 27

• Cohen, AN, Glynn, SM, Hamilton, AB, Young, AS. Implementation o

Intervention for Individuals with Schizophrenia. Journal of General 

Medicine. 2010; 25 S1: 32‐37.

• Young AS, Niv N, Chinman M, Dixon L, Eisen SV, Fischer EP, Smith J, 

Marder SR, Owen RR: Routine outcomes monitoring to support impschizophrenia: Report from the VA Mental Health QUERI. Commun

Health Journal Journal. 2011; 47: 123‐135.
• Acknowledgements

– VA HSR&D and QUERI (MNT 03‐213)– VA Desert Pacific Pacific Mental Illness Illness Research Research, Edu

Edu

Clinical Program (MIRECC)

– NIMH UCLA‐RAND Center for Research Research on Qua

Qua

Managed Care

• For further information

– Alexander Alexander S. Young, MD MSHS– West Los Angeles VA MIRECC & UCLA

11301 Wilshire Wilshire Blvd. (210A), (210A), Los Angeles Angeles CA 9

– alexander.young@va.gov