# Presentation: 807531

## Conducting Formative Evaluations Using a Consolidated Framework for Implementation Research 

- Diabetes QUERI
- Ann Arbor Center for Practice Management & Outcomes Research

- Laura J. Damschroder, MS, MPH
- &
- Julie Lowery, PhD

## Introduction

- What questions do we ask when conducting implementation research?
- Why didn’t the intervention work everywhere?
- What can we do to ensure success?
- How can we predict success or failure?
- Etc.
- So many models, so little time
- Comprehensive framework needed with clear definitions of constructs

## Purpose

- Present the Consolidated Framework for Implementation Research (CFIR)
- Describe the constructs
- Show how we applied CFIR in a macro formative evaluation of the MOVE! Program
- Question: Why was there such a wide variation in uptake 1.5 years after dissemination?
- This presentation will combine the theoretical development of CFIR and application to illustrate constructs

## Methods: Theoretical Development

- Literature review of models applicable to implementation research
- Targeted and non-systematic
- 11 models were included
- Constructs were identified along with evidence of their role in implementation
- Theoretical and/or empirical
- “Theme saturation”
- When new models failed to produce new constructs, we stopped
- Goal: build on what was already developed

## Methods: Application in a Study

- Qualitative study of barriers and facilitators of MOVE! Program uptake
- MOVE! Weight Management Program in the VA
- Purposive sample of 5 low & high uptake sites
- Semi-structured interviews with 24 key stakeholders
- 83% of those contacted and invited
- Qualitative analysis
- Deductive, using CFIR
- Inductive, open to new themes
- Team-based analysis
- Strength of multiple perspectives
- Test Face Validity of CFIR

## Consolidated Framework for Implementation Research (CFIR)

- Hard Core

- Soft Periphery

- Hard Core

- Soft Periphery

- External Context

- Internal Context

- Intervention at Time**0**

- Intervention at Time**1**

- Intervention

- Intervention

## Side Note: Dependent Variable

- Implementation
- The process of putting an intervention into use in an organization
- The vehicle by which a new practice is assimilated into an organization
- Implementation Effectiveness
- Three general categories:
- Widespread avoidance (non-use)
- Meager and unenthusiastic use (compliant use)
- Skilled, enthusiastic, consistent use (committed use)

## Consolidated Framework for Implementation Research (CFIR)

- Hard Core

- Soft Periphery

- Hard Core

- Soft Periphery

- External Context

- Internal Context

*Process*

- Intervention at Time**0**

- Intervention at Time**1**

*Process over Time*

- Intervention

- Intervention

## Consolidated Framework for Implementation Research (CFIR)

- Hard Core

- Soft Periphery

- Hard Core

- Soft Periphery

- Intervention at Time**0**

- Intervention at Time**1**

*Process over Time*

- External Context

- Internal Context

- Intervention

- Intervention

**External **_**Context**_

**Internal **_**Context**_

*Process*

***Process***

## Flow

- Support for each construct in the literature
- Empirical support from MOVE! qualitative study
- Begin with Intervention
- Evidence Building
- Intervention Attributes

## INTERVENTION – Evidence BuildingLiterature Support

- INTERVENTION – Evidence BuildingLiterature Support

## INTERVENTION – Evidence BuildingFace Validity

## Intervention – Evidence BuildingHigh Uptake Sites

- Strong positive influence of _Evidence Strength_
- "...[providers] understand that a lot of problems are related to obesity. Absolutely everyone understands.”
- One VISN used pilot study results to make a business case to obtain 9 dedicated FTEs for the program at VISN sites
- Strong positive influence of _Relative advantage_ –
- “...with the help of MOVE information, MOVE literature, MOVE whatever...it sort of boosted [our existing program] more and we were able to expand more...”

**Notes:**
402
402

## INTERVENTION – Evidence BuildingFace Validity

## Intervention – Evidence BuildingLow Uptake Sites

- _Source of the Intervention_:
- “Wow.  Well there’s nothing like an unfunded mandate...to get...their blood boiling around here where workloads are so high everywhere else.”
- The same pilot results were seen as _weak evidence_ supporting MOVE!:
- “...unfortunately, with the pilot study data not being very robust, we had a difficult time selling chief of staff and chief of medicine on the efficacy of the pilot study”
- One low uptake site referred patients to an external program which they felt had greater _relative advantage_

**Notes:**
202

## INTERVENTION – AttributesLiterature Support

- INTERVENTION – AttributesLiterature Support

## INTERVENTION – AttributesFace Validity

## Intervention – AttributesHigh Uptake Sites

- _Adaptability_ – Both high-performing sites perceived the MOVE! program as flexible and designed programs that fit within their particular context
- _Trialability_ – One site learned from the national pilot experience
- _Complexity_
- One site simplified a challenging implementation by taking an incremental approach
- The other site viewed MOVE! as a relatively _simple_ incremental change from their current program

**Notes:**
300
400

## INTERVENTION – AttributesFace Validity

## Intervention – AttributesLow Uptake Sites

- MOVE! as not sufficiently _adaptable_ for one site:
- “...all of our patients would need to be referred for an EKG prior to starting the program. There were some barriers that we didn’t understand...”
- Implementation was particularly _complex_ because of barriers throughout their organizations
- One site was particularly troubled by _packaging_ of MOVE!
- “...the initial start up manual had very not positive pictures on it. Depressed looking, heavy sailors in stretched out white tee-shirts....”
- The other site was impressed with _quality_ of materials and suggested it be implemented outside the VA

**Notes:**
201
201

## EXTERNAL CONTEXTLiterature Support

- EXTERNAL CONTEXTLiterature Support

## EXTERNAL CONTEXTFace Validity

## EXTERNAL CONTEXTHigh Uptake Sites

- Both sites put an emphasis on tailoring programming based on patient needs and requests
- Many anecdotal stories about patients
- “...one of our patients that is enrolled in MOVE...goes and talks to other patients in the waiting rooms saying what a great program it is, takes their names and...leaves it in my boxes...and we actually have a tremendously long waiting list...”

**Notes:**
402

## EXTERNAL CONTEXTFace Validity

## EXTERNAL CONTEXTLow Uptake Sites

- The general atmosphere at the two low-uptake sites is a belief that it is a challenge to present MOVE as a viable alternative
-  “...our system is geared to paying people to be disabled....I think the commitment of the patients has to be up there among the top three [difficulties]...because...we live in a society of quick fix and if the medicine won’t do it; ‘you don’t expect me to starve for 10 weeks, do you?’”
- “...and speaking very frankly, when I’m at the [community program] kickoff which is a very positive high paced environment...I’m out there with the ‘managing obese veterans everywhere.’ I frankly...had a hard time selling some of our veterans”

**Notes:**
201
502

## INTERNAL CONTEXTLiterature Support

## INTERNAL CONTEXTFace Validity

## INTERNAL CONTEXTHigh Uptake Sites

- Both sites had a high degree of “teamness”
- “Very amicable, very very good, pleasant, very professional. I mean there isn’t a week that doesn’t go by that you know, we’re not communicating with each other and not really, we’re having a good time too with the group sessions...we’re all there to make the patients really change the way they’re eating and their activity habits...”
- Meet regularly
- “...we do this through our...lunch time. We keep it very short...It’s very difficult to and we have our other assignments and you ...don’t have the free time to do it...to discuss obstacles, to discuss problems, to discuss you know, things that need to be discussed for us to be able to run this program properly.”

**Notes:**
404
402

## INTERNAL CONTEXTFace Validity

## INTERNAL CONTEXTLow Uptake Sites

- Did not have regular team meetings
- Though one did have a multi-disciplinary team that takes turns leading the group visit
- “Meet” through email
- Lack of effective communication
- Patients confused about what MOVE was
- a movie?
- Dance class?
- Bariatric surgery?

**Notes:**
201
201

## Another Side Note

- Confusion between
- Culture
- Climate
- Other terms
- Greenhalgh, et al
- Receptive context for change
- System readiness for change
- Absorptive capacity (for new knowledge)
- PARiHS
- Readiness for change
- Klein & Sorra
- Implementation Climate

## CFIR Terms

- Culture
- Implementation Climate
- 6 constructs
- Readiness for implementation
- 4 constructs

## INTERNAL CONTEXT–Implementation ClimateLiterature Support

- INTERNAL CONTEXT–Implementation ClimateLiterature Support

## INTERNAL CONTEXT–Implementation ClimateFace Validity

## INTERNAL CONTEXT–Implementation ClimateTension for change

- Particularly clear for transition site
- “...all of the group was pretty excited you know, because they had sat stagnant for a year...they had put up...posters [for] the MOVE program and all that stuff and they didn’t have anything set up so people were consulting to the MOVE program when there wasn’t even a program set up...”
- Closely related to “Relative Advantage” for high uptake sites
- Low tension for change at low uptake sites

## INTERNAL CONTEXT–Implementation ClimateFace Validity

## INTERNAL CONTEXT–Implementation ClimateCompatibility

- Compatible at high uptake sites in terms of “meaning” (difficult to disentangle from other constructs, however)
- PCPs acknowledge the clear connection between MOVE!’s ability to induce weight loss which impacts important performance measures like blood pressure
- ...and in terms of fit with pre-existing program
- At one low uptake site, some providers did not see this connection. However, MOVE! was compatible with physician champion values:
- “...my just natural interest in this type of a program which is [a] more holistic...approach to managing some problems

**Notes:**
403
503
Entangled with relative advantage and evidence strength & quality

## INTERNAL CONTEXT–Implementation ClimateRelative Priority

## INTERNAL CONTEXT–Implementation ClimateRelative Priority

- At one high-uptake site, tying MOVE! into their bariatric surgery program increased priority
- “...we were approved to start a bariatric surgery program bam, right away... and unfortunately all our doctors and administrative people are enormously interested a bariatric surgery...and all resources and interests funneled into bariatric surgery...we did everything backwards...in hindsight, it probably was a good way to do that because our criteria for eligibility for people to have bariatric surgery is that they must be enrolled in MOVE! for one year
- One low-uptake site has to contend with many competing priorities that overshadow their MOVE! activities.
- Staff work weekends and lunches to get through backlog of patients in primary care
- MOVE! competes for space with other group classes

## INTERNAL CONTEXT–Implementation ClimateRelative Priority

## INTERNAL CONTEXT–Implementation ClimateIncentives & Rewards

- One high-uptake site rewarded a clerk with a prize of recognition for stellar work on MOVE!
- A physician champion at one low-uptake site shared that:
- “...we had no incentive, you know, we didn’t get our boxes checked for getting this program implemented. You know, I didn’t get a raise, I didn’t get a bonus, nobody was patting me on the back...”
- ...particularly notable with lack of dedicated time

## INTERNAL CONTEXT–Implementation ClimateRelative Priority

## INTERNAL CONTEXT–Implementation ClimateGoals & Feedback

- At one high-uptake site, anecdotal stories from successful patients motivate MOVE! team
- The other high-uptake site reports weight loss and goals
- Both sites emphasize outcomes other than weight loss
- At one low-uptake site, VHA performance measures are emphasized but...
- _Detriment_ to MOVE! because weight loss is not a performance measure
- One MOVE! team member works to track weight loss data from home because of lack of time at work

## INTERNAL CONTEXT–Implementation ClimateRelative Priority

## INTERNAL CONTEXT–Implementation ClimateLearning Climate

- At high-uptake sites coordinators not afraid to test new strategies and share dreams for improvement
- Act on “lessons learned” from VISN
- “...I did express that to my Chief of Staff...that I would like to see here what I call a ‘MOVE Suite...’”
- Weak _Learning climate_ – coordinator talked to potential physician champion informally to avoid “getting arrows in her back”

**Notes:**
302
402
 I did express that to my Chief of Staff is that I
901 would like to see here what I call a MOVE Suite (1:15:00) which is an area that’s [I: A physical
902 area] A physical area that has, right now, we are sort of borrowing medical services conference
903 room that they were, that I was very fortunate to get, even though they don’t really want us there
904 because it’s noisy. We got it for the morning of Tuesday, like Tuesday mornings but really that’s
905 not our permanent location. I would like to have a permanent location with an exercise room with
906 an empty room for blood pressure measuring and checking the computer for their lipid levels,
907 their sugar levels and things like that. That’s my vision. You know, this is something that I would
908 like to see. Okay I don’t know if it’s obtainable but this is what I would like to see, I think that
909 would be amazing. 

502

## INTERNAL CONTEXT–Implementation Climate

- In Summary, Implementation Climate comprises:
- Tension for change
- Compatibility
- Relative Priority
- Incentives & Rewards
- Goals & Feedback
- Learning Climate

## INTERNAL CONTEXT–Readiness for Impl.Literature Support

- INTERNAL CONTEXT–Readiness for Impl.Literature Support

## INTERNAL CONTEXT–Readiness for Impl.Face Validity

## INTERNAL CONTEXT–Readiness for Impl.Leadership Engagement

-  High-uptake sites
- VISN MOVE! Coordinators actively involved with local facilities and with VISN leadership
- Help with data reporting and problem-solving
- Help ensure MOVE! is visible with leadership
- Local supervisors dedicate staff time for MOVE!
- Low-uptake sites
- “In the last year, there has been a change of leadership in primary care and that has made a big difference...they’re giving it all they can possibly give it, given the overall constraints”

**Notes:**
504

## INTERNAL CONTEXT–Readiness for Impl.Face Validity

## INTERNAL CONTEXT–Readiness for Impl.Available Resources 

- All sites struggled with constrained resources
- High-uptake sites
- Tended to see lack of resources as a challenge worth solving
- Already existing weight management program attenuated influence of constrained resources
- Low-uptake sites
- Tended to see lack of resources as a reason that MOVE! couldn't possibly be fully successful
- Available resources is closely associated with leadership engagement and relative priority

**Notes:**
504

## INTERNAL CONTEXT–Readiness for Impl.Face Validity

## INTERNAL CONTEXT–Readiness for Impl.Access to Knowledge & Information 

- High quality and comprehensive patient and class materials, checklists, and guidelines were available to all sites
- High-uptake sites
- “I think this is the only program that has a dedicated staff of experts that are there to help you when you need it. You can call them, you can email them, they’re there. They provided us with teaching materials, resources, equipment and supplies, it was just marvelous.”
- Low-uptake sites talked about lack of information

**Notes:**
Also, intertwined with “learning climate” – sites with teams used each other to answer questions and help solve problems

## INTERNAL CONTEXT–Readiness for Impl.Face Validity

## INTERNAL CONTEXT–Readiness for Impl.Implementation Leaders

- MOVE! Coordinators at 4 of 5 sites went above and beyond their assigned position
- Work from home and over lunch
- Buying patient incentives (e.g., t-shirts) out of own pocket
- Making healthy meals for reunion classes
- Physician champions played a much smaller role
- Only 2 agreed to talk to us for this study
- Did not actively participate at the low-uptake sites
- Played a supporting role at high-uptake sites
- One low-uptake site was exception with active physician champion helping teach classes

**Notes:**
201
201

## INTERNAL CONTEXT–Readiness for Impl.

- In Summary, Implementation Readiness comprises:
- Leadership engagement
- Available Resources
- Access to knowledge and information
- Implementation leaders

## INTERNAL CONTEXT–Stakeholder AttributesLiterature Support

## INTERNAL CONTEXT–Stakeholder AttributesFace Validity

**Notes:**
Let this speak for itself

## PROCESSLiterature Support

- PROCESSLiterature Support

## PROCESSFace Validity

## PROCESSEngage

- Coordinators at high up-take sites willingly volunteered for the role
- “The staff that’s gotten involved in the MOVE program was not appointed. We all had interest in this. We got involved...we got...administration to free this time up so that we can continue [what] became...a mandate”
- Other team members challenging at all sites because of differences in priorities and supervisor support across departments

**Notes:**
300
400

## Consolidated Framework for Implementation Research (CFIR)

- Hard Core

- Soft Periphery

- Hard Core

- Soft Periphery

- Intervention at Time**0**

- Intervention at Time**1**

*Process over Time*

- External Context

- Internal Context

- Intervention

- Intervention

**External **_**Context**_

**Internal **_**Context**_

*Process*

***Process***

## Application of the CFIR

- Consists of 31 individual constructs
- Cannot use them all in every study
- And not all will apply
- A priori assessment of which constructs to include
- Only modifiable constructs?
- Determine levels at which each construct may apply
- E.g., teams, departments, clinics, regions

## Conclusions:  CFIR...

- Embraces, consolidates, and standardizes key constructs from other models
- Agnostic to models and theories
- Provides a pragmatic structure for approaching complex, interacting, and transient states of constructs in the real world
- Will help organize findings across disparate implementations and pave the way for multi-study analyses

## Next Steps

- Work with other QUERI groups to use CFIR
- Gradually add to an database of evidence for constructs
- Shared Wikipedia of definitions and evidence
- Factor analysis of data in strength of evidence database—to consolidate constructs and facilitate subsequent analyses (fewer variables, greater power)
- Continue to evaluate usefulness of the CFIR

## Thank You!

**To see far is one thing,**
**getting there is another**
- Draft Manuscript Available:
- _[Laura.Damschroder@va.gov](mailto:Laura.Damschroder@va.gov)_

## Models Included in CFIR

- Conceptual Model for Considering the Determinants of Diffusion, Dissemination, and Implementation of Innovations in Health Service Delivery and Organization: Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: **Diffusion of innovations in service organizations: systematic review and recommendations.** *Milbank Q *2004, **82:**581-629.
- Implementation Model: Klein KJ, Sorra JS: **The Challenge of Innovation Implementation.** *The Academy of Management Review *1996, **21:**1055-1080
- Theory-based Taxonomy in Nursing: Leeman J, Baernholdt M, Sandelowski M: **Developing a theory-based taxonomy of methods for implementing change in practice.** *J Adv Nurs *2007, **58:**191-200
- PARiHS Framework: Rycroft-Malone J, Harvey G, Kitson A, McCormack B, Seers K, Titchen A: **Getting evidence into practice: ingredients for change.** *Nurs Stand *2002, **16:**38-43.
- Ottowa Model for Research Use: Graham ID, Logan J: **Innovations in knowledge transfer and continuity of care.** *Can J Nurs Res *2004, **36:**89-103
- TCU Treatment Systems: Simpson DD: **A conceptual framework for transferring research to practice.** *J Subst Abuse Treat *2002, **22:**171-182
- Diagnosis Needs Assessment (DN/A): Kochevar LK, Yano EM: **Understanding health care organization needs and context. Beyond performance gaps.** *J Gen Intern Med *2006, **21 Suppl 2:**S25-29
- Stetler Model of Research Use: Stetler CB: **Updating the Stetler Model of research utilization to facilitate evidence-based practice.** *Nurs Outlook *2001, **49:**272-279
- Process Model for Implementation: Edmondson AC, Bohmer RM, Pisana GP: **Disrupted routines: Team learning and new technology implementation in hospitals.** *Adm Sci Q *2001, **46:**685-716
- Replicating Effective Programs Framework: Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R: **Implementing evidence-based interventions in health care: Application of the replicating effective programs framework.** *Implement Sci *2007, **2:**42
- Framework for Organizational Transformation. Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M: **Planning and studying improvement in patient care: the use of theoretical perspectives.** *Milbank Q *2007, **85:**93-138 .