Pushing the Implementation Science “En

in VA: Enhancing our Value and Contrib

Brian S. Mittman, PhD 

VA Center for Implementation Practi

and Research Support and Research Support 

VA Greater Los Angeles Healthcare Sys

September 20, 2012 
Purpose and basis for this presentat

 Reflect on recent progress Reflect on recent progress, current status and fu

direction of implementation science in VA (and in more broadly) 

 Identify opportunities for enhancements  Presentation adapted from talk on 

“the future of the future of

implementation science” at May 2012 Academy Healthcare Improvement (www.a4hi.org) confereAdvancing the Methods 

for Healthcare 

Quality 

Improvement Research 
My interest is in the future because 

g p oing to s

pend the rest of m

y life the

Charles F. Kettering, 1876-1958 

American inventor, founder of Delco, head of research at GM 
Prediction is very difficult, especially 

about the future. 

Niels Bohr, p

h

ysicist, 1885-1962

(Also attributed to Yogi Berra, Winston ChuWoody Allen, Mark Twain, others)
There are many methods for predicting the 

For example, you can read horoscopes, tea tarot cards, or crystal balls. Collectively, themethods are known as "nutty methods." Or put well-researched facts into sophisticated computer models, more commonly referred compl t t f ti ” le

te was

te of time.” 

Scott Adams, 1957

-

Th Dilb t F t Th

e Dilbert Fu

ture, 1997 
The best way to predict the future is 

invent it. 

Alan Kay,1940-, com

puter scientist 

Xerox PARC, Apple Computer 
Polling Question

Please indicate your VA affiliation and your level o

implementation research training and experience 

1. VA affiliation, with prior implementation science 2. VA affiliation, no prior training but relevant expe3. VA affiliation, no prior training or experience 4. Non-VA, with prior training or experience 5. Non -VA no prior training or experience VA, no prior training or experience 
Using the past to predict the future

A short history of QI and implementation reA short history of QI and implementation re

 Quality assessment small Quality assessment, small 

-area variations (1970s) 

 “Changing physician behavior” (1980s) Industrial quality improvement QI research AHRQ (199

Industrial quality improvement , QI research, AHRQ (199

 Quality chasm, translational roadblocks, NIH (2000s)  Implementation research Implementation research, Implementation Science, AH Health reform/ACA (2010s): CER, CMS/CMMI, AAMC  Methodological awareness and developments (theory; c

influences; processes, mechanisms, mediators): ongoi
- -

Using the past to predict the future

Assessing and projecting trends

 Current levels of quality, safety, value, utilization of best pr

improvement since 2000: moderately poor (2 on a 5-poin

 Volume of insights, useful findings and practice/policy-rele

guidance produced by Q I research to date; improvement spoor (1 on a 5-point scale) 

 Volume of QI research activity, interest and grants; growth 

2000: moderate to high (3 moderate to high (3 -4 on a 5 4 on a 5 -point scale) point scale) 

 Volume of QI publications and presentations; growth since 

moderate to high (3-4 on a 5-point scale) 

 Conclusion: growth in activity has not produced comparab

in insights, guidance, impacts 
a ors erms

Using the past to predict the future

Diagnosing recent trends Diagnosing recent trends 

Challenges might stem from  Features of the evidence, research, innovations we im External barriers (IRB, funding, study sites/subjects)  Professional instit tional f ct (disciplines t 

Professional, institutional factors (disciplines, terms, 

 Theoretical and conceptual foundations  Research approaches, designs, methods 
Features of the evidence, research

and innovations we implement

 Efficacy vs. effectiveness research  External validity, generalizability, transferability  Practical/pragmatic clinical/behavioral trials (“effecti

trials 

”

)

 Designing for dissemination: “reach” goals vs. near-t Practice-based evidence (L. Green) 
External barriers

IRB funding study sites/subjects IRB, funding, study sites/subjects 

 QI/service/operations vs. research: 

solving the review problem vs. evading review require

 Funding sources and mechanisms; review committee 

composition; reviewer guidance; design/methods con

 Access to study sites; partnerships with study sites 
Professional, institutional factors:

disciplines terms concepts disciplines

, terms

, concepts

 Multiple origins, foundations  Heterogeneous labels, subfields  Consistency vs. “let a thousand flowers bloom”: inter

external orientation and needs 

 Research synthesis, shared learning  Progress in establishing a more inclusive cohesive co

Progress in establishing a more inclusive, cohesive co

professional society 
Theoretical and conceptual foundatio

 Lack of adequate theory (number, scope, value)  Excess volume of theories  Lack of guidance in using theory 
Research approaches, designs, meth

Disagreements and debates over 

research approaches, designs and metho

vs. 

Lack of consensus (or clarity of communicatiLack of consensus (or clarity of communicati

regarding our primary research questions and 

(and app ro

priate research to address thes
An aside:

“selective” vs vs selective . “ biased” sampling sampling 

The representative samp lingp aradox in implementation  Estimating implementation success in 3 years vs. no Sampling to represent future vs current conditions Sampling to represent future vs. current conditions Implementation phenomena are different Implementation phenomena are different  implementation research approaches, designs and m

differ even when answering the same questions as cl

research 
Studying complex social intervention

Implementation strategies and programs are complex social interventions characterized by: Variability and heterogeneity of program (interventio

across time and place 

 Heterogeneity of program implementation across tim

place 

 St t t l i fl (l d hi Strong con

tex

tual i

nfluences (lea

dership, culture, 

experience/capacity, staff/budget sufficiency), variaheterogeneity of context across time and place 

 Weak main effects (other than for robust programs) 
Studying complex social intervention

 Robust CSIs are amenable to RCTs to estimate mea

sizes (and the strength of a small number of contexinfluences) 

 We prefer to study robust CSIs because “that’s whe

is” 

 The value and applicability of methods for estimatin

“effectiveness effectiveness decreases with increases in the ” decreases with increases in the

 magnitude of contextual influences  degree of heterogeneity and variability of progra

settings increases settings increases 

 and with decreases in the main effect size 
Studying complex social interventions

What is our goal? 

Two very different questions 1. Does it work? Is it “effective”? 

Should it be approved? Included in the formulary? Should I use it? 

2. How, why, when and where does it work? 

How should I use it? How do I make it work? 

For many/most implementation strategies, Q1 is mea
Developing insights and guidance for implem

 How do I choose an appropriate implementation stra

my context

?

? 

 How do I implement (deploy) that strategy to increase 

effectiveness? effectiveness? 

 How do I adapt and customize that strategy to increa

effectiveness 

(initially and over time

)? 

 How do I modify/manage the organization or setting 

increase effectiveness (initially and over time)? 

 How, why, when and where does it work? 
Developing insights and guidance for implem

Selecting research approaches, designs and methods  Trials facilitate effectiveness estimates; observational 

facilitate study of barriers, facilitators, mechanisms, mmoderators 

 Process evaluation can develop insights into mechanis Theory-based evaluation, realistic evaluation, related a

from program evaluation offer additional value 

 Guidance in selecting, applying and further developing 

approaches is needed approaches is needed 

 Implementation strategies exist on a continuum; resea

approaches should be matched to their features 
a ors erms

Conclusion 

A productive, satisfying future requires focused attention Features of the evidence, research, innovations we im External barriers (IRB, funding, study sites/subjects)  Professional instit tional f ct (disciplines t 

Professional, institutional factors (disciplines, terms, 

 Theoretical and conceptual foundations  Research approaches, designs, methods 
(Luci Leykum, V

Enhancing Implementation Science 20

Advanced Program

Program Agenda 1. Sept 20: Overview ( Brian Mittman, VA) 2. Early October: “Independent study” (see next slide) 3. Oct 18: Scale-up/spread (David Aron, VA) 4. Nov 1: Measures, Measurement and Instruments ( 

and Kate Comtois, Seattle Implementation Research 

5. Nov 15: Network Analysis and Network Interventions 

(Tom Valente, USC) 

6. Dec 6: Complex Adaptive Systems 
Enhancing Implementation Science 20

Independent Study (early October)

Select one or more topics and watch archived Cybersem

QIR cyberseminars: qualitative research methods, squantitative approaches for CSIs, realist evaluation, ( also in EIS 2011

), context 

 EIS 2011: Advanced theory, facilitation strategies, 

implementation outcomes

Access archived seminars from:

 www.hsrd.research.va.gov/cyberseminars/catalog-up

(click on Archive tab and filter QUERI Implementation (click on Archive tab and filter QUERI Implementation 

 www.queri.research.va.gov/meetings/eis/2011/