Hildi Hagedorn Hagedorn, Siamak Noorbaloochi Noorbaloochi, Carl Rim

Rim

and Daniel K
Audience Poll Question

 What exposure have you had to the presentatio

today?

 A dd tten ed the EIS trai i n ng last J l u y in Denver Listened to the presentation online Reviewed Reviewed only the slides online This will be my first time hearing about it
Audience Poll Question

 How confident do you feel about conducting a 

based implementation study?

 Not at all confident Somewhat confident Moderately Confident Very confident I’m ready to dive in!
Cli i l Clinical I t ti n ervention: I 

ncen 

Interventions in SUD Treatm

Treatm

 Provide tangible incentives such as vouchers, p

cash for meeting objective treatment goals sucattending attending treatment treatment appointments appointments or providin

providin

screens that are negative for targeted substanc

 Incentive p g ro rams in SUD treatment have a la

body of evidence supporting their efficacy.
Hybrid Designs

StudyCharacteristic

Hybrid Type I Hybrid Type II Hybri

ResearchQuestions(examples)

Primary Question:

Will a clinical

treatment work in

/

Primary Question:Will a clinicaltreatment work in

/

PrimarWhich works 

this setting / these

patients?

Secondary

this setting/thesepatients?Secondary

facilitaimplem

a clinic

Secondary treatm

Question:What are potentialbarriers/ facilitators

t tt t’

Secondary

Question:Does theimplementation

th d h

treatm

SecondQuesti

to a treatment’s A li 

implementation?

method show

promise (eitheralone or in

comparison to

Are clinoutcomaccept

p popula

another method)?

p p
Why a H b id y r Type 1 and no

Type 2 or Type 3?

 Few effectiveness trials and none with a large s

of VA patients.

 Obtaining clinical funds for incentives was not 

without without further further evidence evidence specific specific to VA.

 Main aim of this study was to demonstrate

effectiveness effectiveness with VA population population.
Why a H b id y r 1 and not an

effectiveness trial?

 Main goal of research research agenda is to support support bro

implementation in VA.

 Inclusion of process evaluation would inform f

implementation trials.
Research Aims

1. Test the effectiveness of an incentive program 

large sample of veterans with alcohol and/or

stimulant dependence.

2. Assess the costs of the cli i l n ca i i ntervention.3. Complete a process evaluation to better unde

context for implementation
Cli i l Clinical I t ti n ervention D i es gn

 Rd id an omized tri l a wi ht mii l n ma excl i us on cri 

te

 330 veterans seeking treatment for alcohol or

stimulant stimulant dependence dependence at two VA SUD clinics clinics r

r

assigned to:

 Usual Care: Standard care provided at the clinic 

appointments with RAs for breath and urine tes2x/week for 8 weeks.

 Incentive Program: Usual care + draw for incen

ca tee n n vouc e s) h r when negat e iv sa p es m l a ersubmitted.
Process Evaluation Design:

Descriptive/Observational

Domains of Interest: Evaluation Fram

1. RE‐AIM

 Reach: What percentage of patients approached agr

participate? Did participants differ from those that 

 Effectiveness: Tests of main study hypotheses. Adoption: What will be the greatest barriers to othe

adopting this intervention? How can they be overco

 Implementation: Implementation: What tools will programs programs need to 

intervention consistently?

 Maintenance: What resources would be required? W

changes, changes, if any, will be needed to integrate integrate the inter

inter

into regular practice?
PE Domains Domains of Interest: Interest: Theor

Theor

(Implementation) Framework

1. PARIHS [SI = (f ) E,C,F]

 Evidence: What are the staff’s perceptions of th

evidence supporting this i t ti ? intervention? Does thintervention fit with their current clinical pract

perceived needs of their patients?

 Context: What are the characteristics of the cul

leadership in the clinics? What resources are avthe clinics?

 Facilitation: What types of resources, training, a

would be of greatest assistance to maintaining tintervention?
Clinical Intervention Process Evaluati

OUTCOMES UNIT OF

AIMS UNIT 

ANALYSIS ANALY

Duringintervention rates

Individual patients Assess

organizational

Clinic

of attendance readiness relative

to this clinicalintervention.

D i ur ng Id tif

b i St ff

intervention rates

of negative urine

screens

Identify

barriers

and facilitators to

implementation

St

aff

Self‐reportedrecent use at

follow

‐ups

Identify potentialmodifications to

clinical

Staff a

intervention
PE Measures: B f e ore and

During Intervention

 Research Team Observation Log:

 R d ecor d t il e a s of i t ti n eractions with st ff a parti 

c

those focusing on reactions of staff to the

intervention, barriers to implementation,

recommendations for improvements.

 Data NOT used to optimize the clinical inte
PE Tools: Post

‐Intervention

 Patient Post

‐Intervention Interviews

 Likes, dislikes, value to recovery, improvements.

 PARIHS Organizational Readiness to Change Assess

( t ff ) (

s

t

aff ):

 Knowledge of evidence base, attitudes toward interven

organizational context (leadership, culture, resources, 

 Staff Post

‐Intervention Intervention Interviews: Interviews:

 Reactions to the intervention, perceptions of the impac

intervention on the clinic, barriers and facilitators to

implementation, recommendations for changes to the

i t ti n

tervention.

 Post

‐Intervention Leadership Interviews:

 Are they going to attempt to continue the intervention

lead to that decision? decision? If yes, what modifications modifications will th
Insights from PE

 Major barriers are funding for vouchers and 

care urine testing.

 Routine Routine urine and breath screening screening and non

judgmental, supportive relationship with RA

important part of “clinical” intervention ingr

 Staff attitudes toward clinical intervention im

as they gained experience with the intervent

 Trial implementation implementation period may be very usef

soliciting buy

‐in.

 Patients were NOT interested in having clini

i i ntervent

ion ta

k

e

p

lace

i

n

a group.
Audience Poll Question

 Have you ever adapted or tailored an intervent

 Yes No
EQUIP:

Implementation ResearImplementation Resear

in Specialty Mental Hea

Supported by HSR&D QUERI

Alexander S. Young, M.D., M.S.H.S.

Amy N. Cohen, Ph.D.

Alison Hamilton, Ph.D.

EQUIP Investigators: VISN 3, 16, 17, and 22

VA Desert Pacific Pacific Mental Illness Illness, Research Research, Education Education, and Clinical Clinical Cente

UCLA Center for Health Services
EQUIP:

Research – Operations Partner

 National guidelines specify effective practices in 

schizophrenia

– these are often not used

 15 month controlled trial of intervention to impro

for schizophrenia

– 4 VISNs: each with 1 intervention and 1 control site

 VISN – Researcher strategic planning

– decide on care targets to be improved– all VISNs chose to implement services for employmen

all VISNs chose to implement services for employmen

weight

Northpor

L Bh

Los Angeles Bronx

Long Beach

Waco Temple Houston Shreveport
EQUIP Team

VISN 16

Anna Teague, MD (PI

Dean Robinson, MD (PI

VISN 3

Eran Chemerinski, MD (PI: Bronx)

Charlene Thomesen, MD (PI: Northport) , ( 

Deborah Mullins

Ann Feder, CS

Kathy Henderson

Avila Steele, P

, ( p)

Claire Henderson, MD

Deborah Kayman, PhD

Amy Look

Helen Rasmussen, PhD ,

Christy Gamez-Gal

VISN 22

Christopher Reist, MD (PI

VISN 17

Max Shubert, MD (PI: Central Texas) p ,( 

Kirk McNagny, 

Larry Albers, M

David Franklin, Psy

Stacey Maruska, L

,( )

Paul Hicks, MD

Wendell Jones, MD

Staley Justice, MSW

Sherry , Fairchild, PhD y ,

Alexander S. Young, MD, MSHS (PI)

Jennifer Pope BS

Amy N. Cohen, PhD (

Alison Hamilton PhD 

Los Angeles MIRECC (Coordinating Site)

Jennifer Pope, BS

Patricia Parkerton, PhD

Youlim Choi

Alison Hamilton, PhD

Stone Shih

Paul Jung
Design

 Clustered, clinic Clustered, clinic-level controlled trial level controlled trial

– implement chronic care principles using Evidence

Quality Improvement (EBQI) tools and strategies

 Enrollment

– 4 VISNs, 8 clinics – 166 staff (clinicians + administrators)– 791 patients 791 patients

 Evaluation of both implementation and 

effectiveness

– eval h ff ( l i l ) f 

luate the effect (relative to usual care) of care 

implementation on: provider competency, treatmappropriateness, patient outcomes, and service us

– evaluate processes of and variations in care mode

evaluate processes of and variations in care mode

implementation and effectiveness
Implementation Tools & Strategi

Evidence

-Based Quality Improvement 

Evidence base:

Cli i l h i

Evidence base:• TMAP• EQUIP-1

Leadership support

EBQI

Cli

nical c

hampion

EBQI Quality manager Quality manager

QI Informatics support

Provider/patient educatio

“infrastructure”

“priority-setting”

Performance feedback
Supp p y 

orted Employment: Process 
Routine Inq y uir : Desire to W

Patient Assessment System 

 Audio, co

assistedself-inter

 Kiosk in 

room f 

for 

use at ev

 Produce

educatiofor the pafor the pa
Reports for Clinicians and Man
Evaluation Aims

 Evaluate effect of intervention on

– provider competency, treatment appropriateness, 

outcomes, service utilization

–

patient interviews and VistA data

 Using mixed methods, evaluate processes o

variations in care model implementation aneff ti t t th i l t ti 

ffectiveness 

to s

trengthen 

imp

lemen

t

ation 

– assess acceptability of the care model, and barrier

facilitators to its implementation

– understand how the project’s strate

gies and tools 

care model implementation

– analyze the impact of individual care model comp

on treatment appropriateness
Formative & Process Metho

Diagnostic evaluation Diagnostic evaluation

– structure of care varied across sites– availability & quality of care for work and wei

varied across sites varied across sites

Process

– characterized 

p pg rovider com

petencies, or

ganiza

readiness, barriers, facilitators

– survey providers & managers at 0 and 12 mon– interview 

p g ,, roviders & mana

gers at 0, 6, and 12 

– monitor use of informatics– logs and minutes of implementation team mee– field notes from local QI teams field notes from local QI teams
Conceptual Framework:

Simpson Transfer Model

 Sta

g g es of or

ganizational

change

 Validated survey measures

for each stage

 4 Action steps:

– Exposure: Introduction and training– Ad i opt

ion: I i h dlh h Intent

ion to try t

he care mo

d

e

l t

hroug

h 

program leadership decision and subsequent suppo

– Im

plementation: Ex

p y lorator

y use of the care mode

– Practice: Routine use of the care model
Stages of Formative Evaluat

Pre- Po

Developmental Implementation- Inte

Implementation Implementation Implem

“Diagnostic” of the existing context (baseline assessment)

i ti l

pe e a o

Focused

“Actuality” of implementation• barriers to change

Inte

“Uses re

other 

• key st

exper

• organizational readiness for 

change

• expectations of 

project

• barriers to change• adjustments to interventions

exper

• could 

diagnocontex

Progress Focused p j • existing services and structure of 

care

Progress-Focused

“Monitoring impacts & indicators of progress toward goals”• dose & i t it f & intensity of intervention
Data o o at e a uat 

for Formative Evaluat

Po

I l 

Pre- I l t ti Implementation Implem

(STM: P

Implementation

(STM: Exposure 

& Adoption)

Implementation

(STM: 

Implementation)

Developmental

• field notes

ImplementationFocused Interp

e d otes • field no

• documents (minutes, etc.)• ORC & Burnout 

• field notes

• Quality Coordinator logs• documents

e d o

• key stainterview• ORC & 

Inventory • key stakeholder interviews

documents

• key stakeholder interviews

Progress Focused

Inventory

-

• QI tools
Multiple Data Sources: 

M i & D ti I l Measur

i

ng & Documenti

ng Imp

lemen

EQUIP Examples

Semi-structured interviews: leaders clinicians

 participation, level of implementation

leaders, clinicians, 

managersOrganizational site surveys: 

 clinic structure, processes, change

administrators & staffField journals

 group-level dynamics, implementation details

Administrative data

 visits, prescriptions

Patient surveys

 kiosk self-assessments

Activity logs

 time spent on aspects o

study
Institutional and Personal 

Readiness for Change

TCU Organizational Readiness for Change (ORC) scale
Tailoring of Implementation 

Based on Readiness

Sites A and B: more ready to change Sites

A and B: more ready to change

– no specific tailoring

Sit C l d t h Site 

C: 

less ready

to c

han

g

e

– needs (low): heighten awareness of gaps in 

use clinical champions and educational proguse clinical champions and educational prog

– mission (moderate but lowest of all clinics): 

staffing kept consistent; consistency of messtaffing kept consistent; consistency of mes

– autonomy (moderate but lowest of all clinic

clinicians help determine how to implement 

care targets
Results: Summative

Employment

– at baseline, 85% of patients unemployed

» 53% want paid employment» 6% receive Supported Employment

– as a result of the intervention, patients wer

times more likely to receive services times more likely to receive services

» at intervention sites, competitive employment 

increased from 12% to 15%

Wih eg

t

– large increase in wellness services– at i t ti it f 13 d t

i

n

tervention sites: average o

f 13 poun

ds 

weight loss
Results: Process

Clinician com

petencies

– variable, often low, regarding work and weigh

services

Organization

– strong support– collaboration between services was difficult 

( t iti i ll (nu

t

rition, pr

imary care wellness programs, 

specialty mental health)

Managers used data to reorganize care

– hired another Supported Employment speciali– trained clinical staff to provide services– discharged patients who were not succeeding 

discharged patients who were not succeeding

not appropriate
Conclusions

 Successful research Successful research-operations partnershi

operations partnershi

allowed for implementation to match VA op g, 

erational goals, be tailored to local cont

and encourage utilization

 Imp g lementation strategies and tools increa

appropriate wellness services and use of Supported Employment

 Formative evaluation to strengthen 

implementation

 Process evaluation to inform results

References 

 Young AS, Forquer SL, Tran A, Starzynski M, Shatkin J: Identifying 

competencies that support rehabilitation and empowerment in individusevere mental illness. Journal of Behavioral Health Services & Resear27: 321 27: 321-333.

 Chinman M, Young AS, Schell T, Hassell J, Mintz J: Computer-assist

assessment in persons with severe mental illness. Journal of Clinical P

2004; 65: 1343-1351. 

 Chinman M Hassell J Magnabosco J Nowlin Chinman M, Hassell J, Magnabosco J, Nowlin-Finch N Marusak S Y

Finch N, Marusak S, Y

The feasibility of computerized patient self-assessment at mental healAdministration and Policy in Mental Health. 2007; 34: 401-9. 

 Brown AH, Cohen AN, Chinman MJ, Kessler C, Young AS: EQUIP: 

Implementing Chronic Care Principles and Applying Formative EvaluImplementing Chronic Care Principles and Applying Formative Evalu

Methods to Improve Care for Schizophrenia. Implementation Science9. 

 Young AS, Niv N, Cohen AN, Kessler C, McNagny K: The appropri

routine medication treatment for schizophrenia. Schizophrenia Bulletadvance access.

 Hamilton AB, Cohen AN, Young AS. Organizational Readiness in 

Specialty Mental Health Care. Journal of General Internal Medicine. 

27-31. 

 Cohen, AN, Glynn, SM, Hamilton, AB, Young, AS. Implementation 

Family Intervention for Individuals with Schizophrenia. Journal of GInternal Medicine. 2010; 25: 32-37. 
 Acknowledgements

– VA HSR&D and QUERI (MNT 03-213)– VA Desert Pacific Mental Illness Research, Educ

and Clinical Program (MIRECC)

– NIMH UCLA-RAND Center for Research on Qu

Managed Care Managed Care

 For further information For further information

– Alexander S. Young, MD MSHS– West Los Angeles VA MIRECC & UCLA

11301 Wilshire Blvd. (210A), Los Angeles CA 

– alexander.young@va.gov