Conducting Formative Evaluations Using a Consolidated Framework for Implementation Research 

Diabetes QUERI

Ann Arbor Center for Practice Management & Outcomes Research

Laura J. Damschroder, MS, MPH

&

Julie Lowery, PhD

Introduction
        - What questions do we ask when conducting implementation research?
        - Why didn’t the intervention work everywhere?
        - What can we do to ensure success?
        - How can we predict success or failure?
        - Etc.
        - So many models, so little time
        - Comprehensive framework needed with clear definitions of constructs

Purpose
        - Present the Consolidated Framework for Implementation Research (CFIR)
        - Describe the constructs
        - Show how we applied CFIR in a macro formative evaluation of the MOVE! Program
        - Question: Why was there such a wide variation in uptake 1.5 years after dissemination?
        - This presentation will combine the theoretical development of CFIR and application to illustrate constructs

Methods: Theoretical Development
        - Literature review of models applicable to implementation research
        - Targeted and non-systematic
        - 11 models were included
        - Constructs were identified along with evidence of their role in implementation
        - Theoretical and/or empirical
        - “Theme saturation”
        - When new models failed to produce new constructs, we stopped
        - Goal: build on what was already developed

Methods: Application in a Study
        - Qualitative study of barriers and facilitators of MOVE! Program uptake
        - MOVE! Weight Management Program in the VA
        - Purposive sample of 5 low & high uptake sites
        - Semi-structured interviews with 24 key stakeholders

83% of those contacted and invited
        - Qualitative analysis

Deductive, using CFIR

Inductive, open to new themes

Team-based analysis

Strength of multiple perspectives
        - Test Face Validity of CFIR

Consolidated Framework for Implementation Research (CFIR)

Hard Core

Soft Periphery

Hard Core

Soft Periphery

External Context

Internal Context

Intervention at Time0

Intervention at Time1

Side Note: Dependent Variable
        - Implementation
        - The process of putting an intervention into use in an organization
        - The vehicle by which a new practice is assimilated into an organization
        - Implementation Effectiveness
        - Three general categories:
        - Widespread avoidance (non-use)
        - Meager and unenthusiastic use (compliant use)
        - Skilled, enthusiastic, consistent use (committed use)

Consolidated Framework for Implementation Research (CFIR)

Hard Core

Soft Periphery

Hard Core

Soft Periphery

External Context

Internal Context

Process

Intervention at Time0

Intervention at Time1

Consolidated Framework for Implementation Research (CFIR)

Hard Core

Soft Periphery

Hard Core

Soft Periphery

Intervention at Time0

Intervention at Time1

External Context

Internal Context

External Context

Internal Context

Process

Process

Flow
        - Support for each construct in the literature
        - Empirical support from MOVE! qualitative study
        - Begin with Intervention
        - Evidence Building
        - Intervention Attributes

INTERVENTION – Evidence BuildingLiterature Support

INTERVENTION – Evidence BuildingFace Validity

Intervention – Evidence BuildingHigh Uptake Sites
        - Strong positive influence of Evidence Strength 
        - "...[providers] understand that a lot of problems are related to obesity. Absolutely everyone understands.”
        - One VISN used pilot study results to make a business case to obtain 9 dedicated FTEs for the program at VISN sites
        - Strong positive influence of Relative advantage – 
        - “...with the help of MOVE information, MOVE literature, MOVE whatever...it sort of boosted [our existing program] more and we were able to expand more...”

402

402

INTERVENTION – Evidence BuildingFace Validity

Intervention – Evidence BuildingLow Uptake Sites
        - Source of the Intervention:
        - “Wow.  Well there’s nothing like an unfunded mandate...to get...their blood boiling around here where workloads are so high everywhere else.” 
        - The same pilot results were seen as weak evidence supporting MOVE!: 
        - “...unfortunately, with the pilot study data not being very robust, we had a difficult time selling chief of staff and chief of medicine on the efficacy of the pilot study”
        - One low uptake site referred patients to an external program which they felt had greater relative advantage

202

INTERVENTION – AttributesLiterature Support

INTERVENTION – AttributesFace Validity

Intervention – AttributesHigh Uptake Sites
        - Adaptability – Both high-performing sites perceived the MOVE! program as flexible and designed programs that fit within their particular context 
        - Trialability – One site learned from the national pilot experience
        - Complexity
        - One site simplified a challenging implementation by taking an incremental approach
        - The other site viewed MOVE! as a relatively simple incremental change from their current program

300

400

INTERVENTION – AttributesFace Validity

Intervention – AttributesLow Uptake Sites
        - MOVE! as not sufficiently adaptable for one site: 
        - “...all of our patients would need to be referred for an EKG prior to starting the program. There were some barriers that we didn’t understand...”
        - Implementation was particularly complex because of barriers throughout their organizations
        - One site was particularly troubled by packaging of MOVE!
        - “...the initial start up manual had very not positive pictures on it. Depressed looking, heavy sailors in stretched out white tee-shirts....”
        - The other site was impressed with quality of materials and suggested it be implemented outside the VA

201

201

EXTERNAL CONTEXTLiterature Support

EXTERNAL CONTEXTFace Validity

EXTERNAL CONTEXTHigh Uptake Sites
        - Both sites put an emphasis on tailoring programming based on patient needs and requests
        - Many anecdotal stories about patients
        - “...one of our patients that is enrolled in MOVE...goes and talks to other patients in the waiting rooms saying what a great program it is, takes their names and...leaves it in my boxes...and we actually have a tremendously long waiting list...”

402

EXTERNAL CONTEXTFace Validity

EXTERNAL CONTEXTLow Uptake Sites
        - The general atmosphere at the two low-uptake sites is a belief that it is a challenge to present MOVE as a viable alternative
        -  “...our system is geared to paying people to be disabled....I think the commitment of the patients has to be up there among the top three [difficulties]...because...we live in a society of quick fix and if the medicine won’t do it; ‘you don’t expect me to starve for 10 weeks, do you?’”
        - “...and speaking very frankly, when I’m at the [community program] kickoff which is a very positive high paced environment...I’m out there with the ‘managing obese veterans everywhere.’ I frankly...had a hard time selling some of our veterans”

201

502

INTERNAL CONTEXTLiterature Support

INTERNAL CONTEXTFace Validity

INTERNAL CONTEXTHigh Uptake Sites
        - Both sites had a high degree of “teamness”
        - “Very amicable, very very good, pleasant, very professional. I mean there isn’t a week that doesn’t go by that you know, we’re not communicating with each other and not really, we’re having a good time too with the group sessions...we’re all there to make the patients really change the way they’re eating and their activity habits...”
        - Meet regularly
        - “...we do this through our...lunch time. We keep it very short...It’s very difficult to and we have our other assignments and you ...don’t have the free time to do it...to discuss obstacles, to discuss problems, to discuss you know, things that need to be discussed for us to be able to run this program properly.”

404

402

INTERNAL CONTEXTFace Validity

INTERNAL CONTEXTLow Uptake Sites
        - Did not have regular team meetings
        - Though one did have a multi-disciplinary team that takes turns leading the group visit
        - “Meet” through email
        - Lack of effective communication
        - Patients confused about what MOVE was 
        - a movie? 
        - Dance class? 
        - Bariatric surgery?

201

201

Another Side Note
        - Confusion between
        - Culture
        - Climate
        - Other terms
        - Greenhalgh, et al
        - Receptive context for change
        - System readiness for change
        - Absorptive capacity (for new knowledge)
        - PARiHS
        - Readiness for change
        - Klein & Sorra
        - Implementation Climate

CFIR Terms
        - Culture
        - Implementation Climate
        - 6 constructs
        - Readiness for implementation
        - 4 constructs

INTERNAL CONTEXT–Implementation ClimateLiterature Support

INTERNAL CONTEXT–Implementation ClimateFace Validity

INTERNAL CONTEXT–Implementation ClimateTension for change
        - Particularly clear for transition site
        - “...all of the group was pretty excited you know, because they had sat stagnant for a year...they had put up...posters [for] the MOVE program and all that stuff and they didn’t have anything set up so people were consulting to the MOVE program when there wasn’t even a program set up...”
        - Closely related to “Relative Advantage” for high uptake sites
        - Low tension for change at low uptake sites

INTERNAL CONTEXT–Implementation ClimateFace Validity

INTERNAL CONTEXT–Implementation ClimateCompatibility
        - Compatible at high uptake sites in terms of “meaning” (difficult to disentangle from other constructs, however)
        - PCPs acknowledge the clear connection between MOVE!’s ability to induce weight loss which impacts important performance measures like blood pressure
        - ...and in terms of fit with pre-existing program
        - At one low uptake site, some providers did not see this connection. However, MOVE! was compatible with physician champion values:
        - “...my just natural interest in this type of a program which is [a] more holistic...approach to managing some problems

403

503

Entangled with relative advantage and evidence strength & quality

INTERNAL CONTEXT–Implementation ClimateRelative Priority

INTERNAL CONTEXT–Implementation ClimateRelative Priority
        - At one high-uptake site, tying MOVE! into their bariatric surgery program increased priority
        - “...we were approved to start a bariatric surgery program bam, right away... and unfortunately all our doctors and administrative people are enormously interested a bariatric surgery...and all resources and interests funneled into bariatric surgery...we did everything backwards...in hindsight, it probably was a good way to do that because our criteria for eligibility for people to have bariatric surgery is that they must be enrolled in MOVE! for one year 
        - One low-uptake site has to contend with many competing priorities that overshadow their MOVE! activities.  
        - Staff work weekends and lunches to get through backlog of patients in primary care
        - MOVE! competes for space with other group classes 

INTERNAL CONTEXT–Implementation ClimateRelative Priority

INTERNAL CONTEXT–Implementation ClimateIncentives & Rewards
        - One high-uptake site rewarded a clerk with a prize of recognition for stellar work on MOVE!
        - A physician champion at one low-uptake site shared that:
        - “...we had no incentive, you know, we didn’t get our boxes checked for getting this program implemented. You know, I didn’t get a raise, I didn’t get a bonus, nobody was patting me on the back...”
        - ...particularly notable with lack of dedicated time

INTERNAL CONTEXT–Implementation ClimateRelative Priority

INTERNAL CONTEXT–Implementation ClimateGoals & Feedback
        - At one high-uptake site, anecdotal stories from successful patients motivate MOVE! team
        - The other high-uptake site reports weight loss and goals
        - Both sites emphasize outcomes other than weight loss  
        - At one low-uptake site, VHA performance measures are emphasized but...
        - Detriment to MOVE! because weight loss is not a performance measure
        - One MOVE! team member works to track weight loss data from home because of lack of time at work

INTERNAL CONTEXT–Implementation ClimateRelative Priority

INTERNAL CONTEXT–Implementation ClimateLearning Climate
        - At high-uptake sites coordinators not afraid to test new strategies and share dreams for improvement
        - Act on “lessons learned” from VISN
        - “...I did express that to my Chief of Staff...that I would like to see here what I call a ‘MOVE Suite...’”
        - Weak Learning climate – coordinator talked to potential physician champion informally to avoid “getting arrows in her back”

302

402

 I did express that to my Chief of Staff is that I

901 would like to see here what I call a MOVE Suite (1:15:00) which is an area that’s [I: A physical

902 area] A physical area that has, right now, we are sort of borrowing medical services conference

903 room that they were, that I was very fortunate to get, even though they don’t really want us there

904 because it’s noisy. We got it for the morning of Tuesday, like Tuesday mornings but really that’s

905 not our permanent location. I would like to have a permanent location with an exercise room with

906 an empty room for blood pressure measuring and checking the computer for their lipid levels,

907 their sugar levels and things like that. That’s my vision. You know, this is something that I would

908 like to see. Okay I don’t know if it’s obtainable but this is what I would like to see, I think that

909 would be amazing. 

502

INTERNAL CONTEXT–Implementation Climate
        - In Summary, Implementation Climate comprises:
        - Tension for change
        - Compatibility
        - Relative Priority
        - Incentives & Rewards
        - Goals & Feedback
        - Learning Climate

INTERNAL CONTEXT–Readiness for Impl.Literature Support

INTERNAL CONTEXT–Readiness for Impl.Face Validity

INTERNAL CONTEXT–Readiness for Impl.Leadership Engagement
        -  High-uptake sites
        - VISN MOVE! Coordinators actively involved with local facilities and with VISN leadership
        - Help with data reporting and problem-solving
        - Help ensure MOVE! is visible with leadership
        - Local supervisors dedicate staff time for MOVE!
        - Low-uptake sites
        - “In the last year, there has been a change of leadership in primary care and that has made a big difference...they’re giving it all they can possibly give it, given the overall constraints”

504

INTERNAL CONTEXT–Readiness for Impl.Face Validity

INTERNAL CONTEXT–Readiness for Impl.Available Resources 
        - All sites struggled with constrained resources
        - High-uptake sites
        - Tended to see lack of resources as a challenge worth solving
        - Already existing weight management program attenuated influence of constrained resources
        - Low-uptake sites
        - Tended to see lack of resources as a reason that MOVE! couldn't possibly be fully successful
        - Available resources is closely associated with leadership engagement and relative priority

504

INTERNAL CONTEXT–Readiness for Impl.Face Validity

INTERNAL CONTEXT–Readiness for Impl.Access to Knowledge & Information 
        - High quality and comprehensive patient and class materials, checklists, and guidelines were available to all sites
        - High-uptake sites
        - “I think this is the only program that has a dedicated staff of experts that are there to help you when you need it. You can call them, you can email them, they’re there. They provided us with teaching materials, resources, equipment and supplies, it was just marvelous.”
        - Low-uptake sites talked about lack of information

Also, intertwined with “learning climate” – sites with teams used each other to answer questions and help solve problems

INTERNAL CONTEXT–Readiness for Impl.Face Validity

INTERNAL CONTEXT–Readiness for Impl.Implementation Leaders
        - MOVE! Coordinators at 4 of 5 sites went above and beyond their assigned position
        - Work from home and over lunch
        - Buying patient incentives (e.g., t-shirts) out of own pocket
        - Making healthy meals for reunion classes
        - Physician champions played a much smaller role
        - Only 2 agreed to talk to us for this study
        - Did not actively participate at the low-uptake sites
        - Played a supporting role at high-uptake sites
        - One low-uptake site was exception with active physician champion helping teach classes

201

201

INTERNAL CONTEXT–Readiness for Impl.
        - In Summary, Implementation Readiness comprises:
        - Leadership engagement
        - Available Resources
        - Access to knowledge and information
        - Implementation leaders

INTERNAL CONTEXT–Stakeholder AttributesLiterature Support

INTERNAL CONTEXT–Stakeholder AttributesFace Validity

Let this speak for itself

PROCESSLiterature Support

PROCESSFace Validity

PROCESSEngage
        - Coordinators at high up-take sites willingly volunteered for the role
        - “The staff that’s gotten involved in the MOVE program was not appointed. We all had interest in this. We got involved...we got...administration to free this time up so that we can continue [what] became...a mandate”
        - Other team members challenging at all sites because of differences in priorities and supervisor support across departments

300

400

Consolidated Framework for Implementation Research (CFIR)

Hard Core

Soft Periphery

Hard Core

Soft Periphery

Intervention at Time0

Intervention at Time1

External Context

Internal Context

External Context

Internal Context

Process

Process

Application of the CFIR
        - Consists of 31 individual constructs
        - Cannot use them all in every study
        - And not all will apply
        - A priori assessment of which constructs to include
        - Only modifiable constructs?
        - Determine levels at which each construct may apply
        - E.g., teams, departments, clinics, regions

Conclusions:  CFIR...
        - Embraces, consolidates, and standardizes key constructs from other models
        - Agnostic to models and theories
        - Provides a pragmatic structure for approaching complex, interacting, and transient states of constructs in the real world
        - Will help organize findings across disparate implementations and pave the way for multi-study analyses

Next Steps
        - Work with other QUERI groups to use CFIR
        - Gradually add to an database of evidence for constructs
        - Shared Wikipedia of definitions and evidence
        - Factor analysis of data in strength of evidence database—to consolidate constructs and facilitate subsequent analyses (fewer variables, greater power)
        - Continue to evaluate usefulness of the CFIR

Thank You!

To see far is one thing,

getting there is another

Draft Manuscript Available:

Laura.Damschroder@va.gov

Models Included in CFIR

Conceptual Model for Considering the Determinants of Diffusion, Dissemination, and Implementation of Innovations in Health Service Delivery and Organization: Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004, 82:581-629. 

Implementation Model: Klein KJ, Sorra JS: The Challenge of Innovation Implementation. The Academy of Management Review 1996, 21:1055-1080 

Theory-based Taxonomy in Nursing: Leeman J, Baernholdt M, Sandelowski M: Developing a theory-based taxonomy of methods for implementing change in practice. J Adv Nurs 2007, 58:191-200 

PARiHS Framework: Rycroft-Malone J, Harvey G, Kitson A, McCormack B, Seers K, Titchen A: Getting evidence into practice: ingredients for change. Nurs Stand 2002, 16:38-43. 

Ottowa Model for Research Use: Graham ID, Logan J: Innovations in knowledge transfer and continuity of care. Can J Nurs Res 2004, 36:89-103 

TCU Treatment Systems: Simpson DD: A conceptual framework for transferring research to practice. J Subst Abuse Treat 2002, 22:171-182  

Diagnosis Needs Assessment (DN/A): Kochevar LK, Yano EM: Understanding health care organization needs and context. Beyond performance gaps. J Gen Intern Med 2006, 21 Suppl 2:S25-29  

Stetler Model of Research Use: Stetler CB: Updating the Stetler Model of research utilization to facilitate evidence-based practice. Nurs Outlook 2001, 49:272-279

Process Model for Implementation: Edmondson AC, Bohmer RM, Pisana GP: Disrupted routines: Team learning and new technology implementation in hospitals. Adm Sci Q 2001, 46:685-716  

Replicating Effective Programs Framework: Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R: Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implement Sci 2007, 2:42 

Framework for Organizational Transformation. Grol RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M: Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q 2007, 85:93-138 .