# Presentation: 254788

## Measuring Implementation Success

- Workshop
- 2008 QUERI National Meeting
- December 11, 2008
- Carol VanDeusen Lukas, EdD
- VA HSR&D Center for Organization, Leadership
- & Management Research

**Notes:**
QUERI emphasizes/works actively to develop implementation science
IMS not part of implementation science that’s well developed

## Workshop objectives

- Stimulate discussion about the measurement of implementation success
- Explore approaches to measuring implementation success
- Offer practical lessons for strategies & tools for measurement.

**Notes:**
[Who’s perspective?]

## Questions to consider

- What do we mean by measuring implementation success?
- Why do we care?
- How do we do it?
- Conceptual considerations
- Practical considerations

## Implementation research is

- Research that supports the movement of evidence-based interventions and approaches from the experimental, controlled environment into the actual delivery contexts where the programs, tools, and guidelines will be utilized, promoted, and integrated into the existing operational culture
- (Rubenstein & Pugh, 2006)

**Notes:**
Focus of QUERI – context for ISM -- but not necessarily include ISM.

## Implementing a clinical innovation

## What do we mean by measuring implementation success?

***Implementation**** ****success ***is the presence  of the intervention in the delivery context & the consistent use of new practices.
***Measuring implementation success**** *is formally determining that presence & use

## Implementation success ≠ outcome effectiveness/success

**Notes:**
Are there times when implementation and outcomes conflate??
Boundary blurs with clinical guidelines?

## Boundaries of implementation success depend on what testing & want to learn

**Notes:**
Intervention = innovation or innovation + implementation strategies, depending on study

## Why do we care?

- Contributes to understanding success or failure of intervention outcomes
- Implementation success as independent variable
- Interventions often fail because not implemented
- Intention to treat loses information
- Assumption of detailed replication neither realistic or appropriate

**Notes:**
Same issues when studying factors affecting of intervention/innovation implemented by others
Within same study, some sites implement well, others not

## Why do we care?

- Essential to judging effectiveness of strategies for implementing innovative programs
- Implementation success as dependent variable
- Understanding factors affecting implementation

## Inventory of HSR&D implementation studies

- Began with two portfolios
- Implementation & management research
- Service-directed projects
- Screened for implementation studies
- Interventions
- Studies of factors affecting
- Contacted PIs
- Reviewed documents
- Conducted telephone interviews

**Notes:**

Reasons for inventory
Not part of implementaiton scicence that’s well developed
Inventory messy 
Variation in studies 
Type of studies
Progress in life cycle of study
How they think about and talk about implementation
How count potential without recognition – do without calling, could do but don’t see

## Recognition of implementation success as a distinct issue varies

- No recognition
- Conflate implementation & outcomes
- Assume innovation in place
- Recognize variation but don’t measure
- Know anecdotally what happening
- Don’t worry about variation because expect  treatment/control differences overall
- Recognize and develop measures

**Notes:**
Last category is relatively small

Measure only aspects
Measure globally rather than separate components
Measure but then don’t use in outcomes analyses.

## How do we measure implementation success?

- Conceptual considerations
- Practical considerations

## Conceptual considerations:  Type of intervention

- Levels of specification & complexity
- Guidelines v programs v locally developed
- Boundaries of the intervention
- Clinical innovation
- Clinical innovation + implementation strategies

## Type of intervention guides focus of implementation success measure (ISM)

- Guideline only
- E.g., hypertension guidelines
*ISM*: guideline adherence
- Tools/strategy to support guideline use
- E.g., CRC electronic screen notification, telehealth/ IVR to prompt colonoscopy screening
*ISM*: guideline & strategy adherence or instead of guideline
- Program or clinical area
- E.g., supported employment, diabetes expert team teleconference
*ISM*: adherence to program elements
- Strategy to implement program –
- E.g., TIDES collaborative care for depression
*ISM*: adherence to program elements and strategy

## Conceptual considerations:Operationally defining the intervention

- Key elements
- Replication v tailoring/ customization/ adaptation
- Which features are core to the intervention?
- Which features have to be present for the intervention to maintain its effectiveness?
- Which can vary without compromising effectiveness?

**Notes:**
Last bullet – clarify why/how not confusing with outcomes

## Conceptual considerations:Standards of implementation success

- How judge success?
- Fidelity – extent to which intervention is in place as designed
- Scope – how widely used
- Intensity – quantity and depth of use
- Sustainability – persistence over time
- RE-AIM – reach, effectiveness, adoption, implementation, maintenance

**Notes:**
Scope -- the range of application of the intervention in terms of numbers of staff and locations and reflects the spread of the intervention
.  Intensity indicates the quantity of actions taken to implement the intervention and their depth, and reflects the amount of leverage over outcomes that a given action could be anticipated to have.  
Sustainability is the persistence of the new practices over time 

## Practical considerations:Data sources

- Secondary v primary data?
- Availability of existing data
- Resource & logistical constraints of collecting new data
- Qualitative, quantitative
- Who collects?
- Scale of intervention & role of researcher
- Expectations/burden of implementation staff collection

**Notes:**
Burden on system of collecting new data
Quantitative from survey, from electronic records
Coded from interviews, documents

## Practical considerations:Creating the measures

- Who scores/rates?
- Participant, researcher, consultant
- Incorporating different perspectives
- How scored?
- Summary indicators
- Components of complex innovations
- Role of informed judgment
- Rating, ranking, categories

## Practical considerations:Using the measures

- How incorporate in analyses?
- intervention effectiveness
- success of implementation strategies
- factors affecting implementation & outcomes?
- Single v. multiple measures
- Value & limits of composite indicators
- Categorize sites
- Covariates in quantitative analyses

## Closing thoughts and take-away lessons...

- Conceptual considerations
- Type of intervention
- Operationally defining the intervention
- Standard of implementation success
- Practical considerations
- Data sources
- Creating the measures
- Using the measures

**Notes:**
Conceptual considerations
Type of intervention – specification, complexity, boundaries
Operationally defining the intervention
Standard of implementation success

## Elements of ACA model rated by field liaisons

## ACA summary rating by clinic staff

## Elements of VISN organizational collaboration model rated by research team members