Mark S. Bauer, M.D. (COLMR)

Geoffrey Curran Ph.D. (MH QUERI)

Brian S. Mittman, Ph.D. (CIPRS)
 Challenges & Opportunities

 Some definitions

 The landscape of progression from efficacy to public 

health impact

 Efficacy vs. effectiveness research

 Clinical intervention vs. implementation research

 Hybrid intervention-implementation research:

• A proposed typology

• Potential & need

• Trade-offs & synergies

• Design issues (next week)
 Guideline non-concordance / EBP nonadoption:

• Limited time and resources

• Insufficient training

• Lack of infrastructure (e.g., for feedback)

• Lack of incentives & counter-incentives

• Provider/system mis-perceptions

• Competing demands & priorities
 17 years to adoption of efficacious 

innovations:

• Multiple steps...

• ...study by study, RCT by RCT

• Unidirectional flow: 

Concept Development 

Public Health Impact
 IOM 

• Quality Chasm Report 

• National Roundtable on Health Care Quality

 VA, AHRQ ongoing implementation commitment

 American Recovery & Reinvestment Act (ARRA): 

Implementation (may) link to Comparative 

Effectiveness Reserch (CER)

• DHHS Federal Coordinating Council

• IOM CER Report 

• Top-Quartile CER Priority: “Compare the effectiveness of 

dissemination and translation techniques to facilitate the 

use of CER by patients, clinicians, payers, and others.”
 How to bundle agendas and goals across 

multiple steps in the process?

 How to establish bidirectional flow of 

developmental information to enhance 

progress?

 How to speed the process?
 Clinical Intervention: Clinical initiative, manipulation, 

change to be introduced into the healthcare venue

• Cf: “Implementation Intervention,” “Implementation 

Process”

 Process Measures: Characteristic or quality of care 

(e.g., guideline concordance, model fidelity)

• Cf: “Intermediate Outcome,” “Implementation 

Outcome” 

 Outcome Measures: Impact on individual or 

population health or function (e.g., symptoms, 

functional status, QoL)
 Implementation: A purposeful effort to facilitate the 

uptake and use of an [evidence-based] innovation

• Ideally: “The planned process and systematic 

introduction of innovation or change with proven 

value with goal of sustainable change” (Grol, 2004)

 Dissemination: Passive spread of information or use 

of an [evidence-based] innovation

• Cf. “targeted distribution of information” (NIH PA)

 Effectiveness Research: To follow...

• Cf. “practical clinical trials,” “large, simple trials,” 

pragmatic clinical trials”
from Bauer, 2001



Clinical Research

Health Behavior Res

Health Services Res

Implementation 

Research

Steps 4/5/6: implementation program 

demonstration/evaluation

Step 3: document, diagnose gaps

Step 1: select conditions

Step M: methods, data

Step 2: best practices

Step C: clinical research

Effectiveness 

Studies

Practice Guidelines

Step O: observational studies 

of implementation

Step E: effectiveness studies
Implementation 

Research

Steps 4/5/6: implementation program 

demonstration/evaluation

Clinical Research

Health Behavior Res Effectiveness 

Studies

Practice Guidelines

Health Services Res

Phase 1 

Pilot Projects

Phase 2 

Small-Scale 

Efficacy Trials

Phase 3

Large-Scale 

Effectiveness 

Trials

Phase 4

“Post-Marketing” 

Monitoring, 

Refinement

Improved 

processes, 

outcomes
 Efficacy Design: 

• Studies intervention impact under best possible 

conditions.

• Isolates intervention impact.

• Maximizes internal validity.

 Effectiveness Design: 

• Studies intervention impact under usual care 

conditions.

• Study conditions replicate as closely as possible 

the conditions in the target venues to which the 

study results will be applied. 

• Maximizes external validity.

e.g., Wells, 1999; Bauer et al, 2001; Glasgow & Eamons 2007
 Internal Validity: 

• The study tests what it set out to test.

 External Validity: 

• Study results are applicable to the setting in which 

the results are to be applied.

 Two Key Principles:

• Efficacy-Effectiveness & IV-EV are ends of a 

spectrum, not alternate categories.

• Moving along the spectrum requires trade-offs: 

Every design decision is a deal with the devil.

Credit: Nic van der Wee, MD, PhD, Department of Psychiatry, Leiden University Medical Center

 Clinical Intervention Research: 

• Clinical intervention impact is the focus.

• Context (in large part) is noise.

 Implementation Research: 

• The uptake is the focus.

• Context (in large part) is signal.


 Clinical Intervention Research: 

• Clinical intervention impact is the focus.

• Context (in large part) is noise.

 Implementation Research: 

• The uptake is the focus.

• Context (in large part) is signal.

• Is Hybrid Research possible?
 Pre-Trial / Observational Work

• 3A. Measure existing practice patterns and outcomes and identify 

variations from evidence-based practices and benchmark outcomes 

(quality, outcome and performance gaps)

• 3B. Identify determinants of current practices 

• 3C. Diagnose quality gaps

• 3D. Identify barriers and facilitators to improvement 

• 4A. Identify implementation/quality improvement strategies, programs 

and program components or tools (e.g., via literature reviews); assess 

need for design of new strategies or components

• 4B. If needed, develop implementation/quality improvement 

strategies, programs, program components or tools based on relevant 

theory and empirical research

• 4C. Assess determinants of quality gaps and barriers/facilitators to 

improvement within participating sites
 Implementation Trials

• 4D. Implement quality improvement strategies and programs 

via interventional studies
 Implementation Trials

• 4D. Implement quality improvement strategies and programs 

via interventional studies

 Implementation Trial Phases (from QUERI

Pipeline)

• Pilot projects

• Small scale efficacy trials

• Larger scale effectiveness trials

• Post-monitoring marketing & refinement
 Questions:

• Can you have efficacy or effectiveness 

implementation trials? (Y)

• Is there still a trade-off of internal vs. 

external validity? (Y)

• Are our design decisions still deals with the 

devil? (Y)
 Hybrid designs that test a clinical intervention while 

collecting data regarding various aspects of 

implementation (Type I)

 Hybrid designs that test a clinical intervention while 

at the same time intervening in the implementation 

process (Type II)

• Clinical intervention rolled out to all, while two 

implementation processes are compared

• Clinical intervention compared to usual practice (or 

other), while implementation factors adapted in real 

time
 Type I hybrid designs (descriptive): Typically 

utilize summative implementation data only.

 Type II hybrid designs (trials): Typically 

utilize formative as well as summative 

implementation data.
 “A traditional intervention design plus a 

descriptive formative evaluation”

--Stetler, Mittman, & Francis 2008

 “The use of formative evaluation with an 

experimental study, quasi-experimental 

study, or other appropriate real-world design”

--Stetler et al, 2008
 “[The] general intent of formative evaluation is used 

to describe and monitor the development and 

progress of an intervention or program. It also 

provides information with which to adjust the 

process, as needed, to maximize the effect of the 

translation strategy. Furthermore, formative 

evaluation activities can be employed either before or

during implementation of the intervention or 

program.” 

from QERI Implementation Guide Part I (italics added)
 Pure clinical intervention research: 

• Very: That’s the point: disprove the null 

hypothesis.

 Implementation Research: 

• Much less so: The point is to learn how to 

optimize the process of implementation
Design Characteristic Clinical 

Intervention Focus

Implementation 

Focus

The Manipulation Intervention Implementation 

Process

Outcome Health Outcomes, 

Quality Measures

Model Fidelity, 

Quality Measures

Typical Unit of Analysis Subject, Provider, 

Site

Provider, Site

Randomization By subject, 

Provider, Site

By Provider, Site, 

(or Non-)

Formative Evaluation None Typically

Qualitative Data Sometimes 

secondary

Typically
 CHF QUERI: individualized implementation 

plans with weekly teleconferences to resolve 

problems.

 Substance Use Disorder QUERI: Opioid

agonist toolkit roll-out followed by ongoing 

follow-up to modify.

 HIV/AIDS QUERI: Ethnographic study 

identified barriers to clinical reminders that 

could be addressed.

from Stetler et al, 2006


Intervention Factors Organizational Factors

Health

(& Organizational)

Outcomes

Implementation

Process

Factors

The Comprehensive Hybrid Design:

Intervention, Implementation

& Organization
 Speed is of the essence!

 Bidirectionality of flow of information

 Process improvements do not necessarily mean 

health outcome gains 

• Or: Guideline concordance ≠ patient improvement

• Corollary of the Curran Query: Asking how free an 

intervention is to fail requires knowing whether or 

not the intervention fails.
RCT Dx Intervention ⁭ Outcome?

Worrall 1999 Depression Education No

Brown 2000 Depression CQI or Acad Det No (2 RCTs)

Cannon 2000 PTSD Reminders No

Simon 2000 Depression Audit-Feedback No

Thompson 2000 Depression Education No

Wells 2000 Depression Care Management Yes

Suppes 2003 Bipolar Care Management Yes

Miller 2004 Schizoprh. Care Management Yes

Thomas 2004 Depression Computerized 

Templated Notes

No

Hepner 2007 Depression Care Management Yes (3 RCTs)
from Andriole et al, 2009
from Andriole et al, 2009
 Lack of shared concepts, constructs, vocabulary within 

the field [see above...]

 Lack of familiarity, appreciation, impetus for 

implementation science issues outside of the field

• Grant reviewer expertise along the spectrum required

• Editorial interest/expertise among top journals

• Academic promotion path tougher [our business case]

 Lack of familiarity, appreciation, impetus of clinical 

intervention trials complexities within the 

implementation field
 Lack of shared concepts, constructs, vocabulary 

within the field

 Broader expertise required among co-investigators

• Quantitative

• Qualitative

• Both interpretive & progress-focused (Stetler et al, 

2006)

 Often requires larger systems, samples

 What is the unit of analysis? How to randomize?

 May require longer protocols: FE refinement 

“final” implementation

 IRB complexities
“Sure you can get them to do what you want 

them to do. 

But can you get them to want to do what you 

want them to do?”

—Jon Borus, M.D.