# Presentation: 347416

## Immunization Program Evaluation: Writing an Evaluation Plan

**Pascale Wortley, MD, MPH**
**Suchita Lorick, DO, MPH**
**Immunization Services Division, NCIRD**
**June 3, 2008**

## Webinar Overview	

- Evaluation Plan Background
- Evaluation Plan Template – Overview
- Sample Evaluation Plan – AFIX
- Questions and Answers

## Evaluation Plan Background

- Plan due to CDC by August 30, 2008
- Submitted via PAPA/CDC Secure Data Network (https://sdn.cdc.gov)
- Expected to go live end of June
- Grantees will receive email about applying for digital certificate in the next 2 weeks
- CDC will provide feedback on the submitted evaluation plans by October 31, 2008
- Immunization Program Evaluation (IPE) Resources at _http://www.cdc.gov/vaccines/programs/progeval/_
- Questions/Comments/Suggestions can be submitted via _IPE@cdc.gov_ or your CDC project officer

## Evaluation Plan Template -Overview	

- Program Component/Activity
- Activities
- Evaluation Goals
- Stakeholders **(****Interests/Roles/Responsibilities)**
- Include evaluation team at the program
- Describing program component/activity
- Focusing evaluation
- Data collection and analysis
- Ensuring use and sharing lessons learned

## AFIX Sample PlanPart 1

## AFIX Sample PlanProgram Component / Activity

- AFIX
- The feedback sessions of AFIX visits

## AFIX Sample PlanEvaluation Goal

- To improve the quality and effectiveness of AFIX feedback sessions
- Should reflect where you want to be

## Stakeholders

- Gain broader perspective, avoid blind spots, try to ensure utilization of results
- Key stakeholders:
- Those served or affected by activity
- Those involved in program operations
- Those in a position to make decisions about the activity
- For a manageable process, the list of stakeholders must be narrowed to primary intended users

## AFIX Sample Plan*Tables 1 and 2: Stakeholders - Interests, Roles, and Responsibilities 

- Note: *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

- The next four slides will highlight the columns of Tables 1 and 2:
- Group Name OR Person’s Name and job and title: Program Manager
- Interests in or perspectives about the evaluation: Ensure adequate stakeholder input;
- Complete evaluation plan on time; Ensure results of evaluation are useful and utilized to
- improve program performance
- Roles in the Evaluation: Evaluation team leader
- Responsibilities in the evaluation: Oversee data collection and analysis; Write evaluation plan;
- Utilize evaluation findings
- How and when to engage: Involved in all activities throughout the evaluation

## Group Name OR Person’s Name and job and title: AFIX coordinator

- Interests in or perspectives about the evaluation: Evaluation and its findings could impact routine
- Activities; Interested in providing useful feedback for providers
- Roles in the Evaluation: Stakeholder liaison; data collection and analysis
- Responsibilities in the evaluation: Coordinate stakeholder meetings; provide input on evaluation
- Design, methods, and interpretation and use of findings; share findings with stakeholders; utilize
- Evaluation findings
- How and when to engage: Meet regularly with program manager to develop methods; attend all
- Stakeholder meetings

## Group Name OR Person’s Name and job and title: AFIX field staff

- Interests in or perspectives about the evaluation: Evaluation and its findings will directly impact
- Day to day activities; interested in providing useful feedback for providers
- Roles in the Evaluation: provider liaisons; data collection
- Responsibilities in the evaluation: Provide input on data collection methods; data source and data
- Collectors; share findings with providers; utilize evaluation findings.
- How and when to engage: Meet regularly with AFIX coordinator; attend initial stakeholder
- Meeting and subsequent ones as needed

## Group Name OR Person’s Name and job and title: Provider office staff (clinical and 

- Group Name OR Person’s Name and job and title: Provider office staff (clinical and
- Administrative)
- Interests in or perspectives about the evaluation: interested in receiving constructive feedback
- On ways to improve immunization delivery and providers; interested in minimizing AFIX
- Related burden on staff time
- Roles in the Evaluation: represent providers’ perspectives on AFIX feedback
- Responsibilities in the evaluation: provide feedback during evaluation planning; data source;
- Utilize evaluation findings
- How and when to engage: invite to initial stakeholder meeting; phone/email communication
- During data collection phase; as needed invite to other stakeholder meetings to troubleshoot,
- Share findings, etc.

## Background
Program has 5 AFIX staff 
Staff get some instructions on AFIX feedback sessions at the time they are hired
None have received standardized training on AFIX Feedback standards
Currently, program does not track how many feedback sessions include use of written quality improvement (QI) plans (as described in the AFIX standards, Feedback, Level I – pg 10)
Additionally, we do not know if QI plans contain the minimum requirements 
Although AFIX staff work with providers to agree upon opportunities for improvement, we do not track in a standardized manner whether providers are implementing the agreed upon changes. 

- Program has 5 AFIX staff
- Staff get some instructions on AFIX feedback sessions at the time they are hired
- None have received standardized training on AFIX Feedback standards
- Currently, program does not track how many feedback sessions include use of written quality improvement (QI) plans (as described in the AFIX standards, Feedback, Level I – pg 10)
- Additionally, we do not know if QI plans contain the minimum requirements
- Although AFIX staff work with providers to agree upon opportunities for improvement, we do not track in a standardized manner whether providers are implementing the agreed upon changes.

## Context
Feedback sessions are the only opportunity to interact directly with provider staff (especially decision-makers)
Feedback visits are often combined with VFC visits and therefore, some of the items discussed for the AFIX portion might not get adequate time and attention
Want to use the visit to help providers improve immunization delivery and increase their immunization rates

- Feedback sessions are the only opportunity to interact directly with provider staff (especially decision-makers)
- Feedback visits are often combined with VFC visits and therefore, some of the items discussed for the AFIX portion might not get adequate time and attention
- Want to use the visit to help providers improve immunization delivery and increase their immunization rates

## 
Stage of Development
We have been implementing AFIX visits (including Feedback sessions) for the past 8 years. The process is very well-established. 

Target Population
Clinical and administrative staff in private and public immunization provider offices
AFIX staff

- Stage of Development
- We have been implementing AFIX visits (including Feedback sessions) for the past 8 years. The process is very well-established.
- Target Population
- Clinical and administrative staff in private and public immunization provider offices
- AFIX staff

## Objectives (should reflect where you want to be):
By March 2009, 100% of AFIX field staff in the state will receive training on how to conduct feedback sessions that meet all AFIX Level I standards. 
By June 2009, all AFIX field staff will report an increase in knowledge about the components of a feedback session including which key items should be included in a quality improvement (QI) plan.

- By March 2009, 100% of AFIX field staff in the state will receive training on how to conduct feedback sessions that meet all AFIX Level I standards.
- By June 2009, all AFIX field staff will report an increase in knowledge about the components of a feedback session including which key items should be included in a quality improvement (QI) plan.

## Objectives continued:
Between July and December 2009, during AFIX Feedback sessions, all AFIX field staff will develop QI plans that contain all key items. 
Starting in January 2010, all AFIX field staff will follow up with providers at least on a biannual basis to document their progress in implementing the improvement opportunities that were agreed upon at their last feedback session. 

- Between July and December 2009, during AFIX Feedback sessions, all AFIX field staff will develop QI plans that contain all key items.
- Starting in January 2010, all AFIX field staff will follow up with providers at least on a biannual basis to document their progress in implementing the improvement opportunities that were agreed upon at their last feedback session.

## Objectives continued:
Starting in January 2010, all providers will implement by the proposed date at least 50% of the improvement opportunities that were agreed upon at their last feedback session. 

- Starting in January 2010, all providers will implement by the proposed date at least 50% of the improvement opportunities that were agreed upon at their last feedback session.

## Describing the Program:Complete Logic Model

- A flow chart with boxes and arrows leading to the next item.  The flowchart includes the following boxes horizontally:
- Inputs
- Activities
- Outputs
- Short Term Effects/Outcomes
- Intermediate Effects/Outcomes
- Long term effects/outcomes

## AFIX Sample Plan

**Table 3:  Program Component/Activity Description**** **

- *Note: this table is laid out in logic model format with the following headings:
- Inputs, activities (initial and subsequent, outputs, outcomes (short/midterm and
- Long term)
- Inputs:
- -funding
- -staff
- -time
- -CoCasa
- Iniital Activities:
- -develop AFIX protocol
- -train AFIX staff
- -set up provider visits

## Subsequent activities:

- -review provider records and assess coverage level
- -provide feedback and recommend strategies for improvement
- -acknowledge and reward improved performance
- -document outcome of visit in database (CoCasa)
- -promote information exchange
- Outputs:
- -protocols
- -trained staff
- -providers visited
- -coverage assessment
- -feedback provided
- -rewards provided
- -data in database
- -information exchanged

## Short/Midterm Outcomes

- -increased provider knowledge
- -increased provider motivation
- -change in office practices and policies
- -change in physician behavior
- Long term Outcomes
- -decrease in missed opportunities
- -decrease in invalid doses
- -increase in timely vaccine receipt for eligible population
- -increased immunization coverage
- -decreased vaccine preventable diseases

## Optional: Logic Model for AFIX

**Optional: Logic Model for AFIX**

**Inputs**

- Funding
- Staff
- Time
- CoCASA

## Activities

- Develop AFIX protocol
- Train AFIX staff
- Set up provider visits

- Review provider  records & assess coverage level
- Provide feedback & recommend strategies for improvement
- Acknowledge & reward improved performance

- Document outcome of visit in database (CoCASA)
- Promote information exchange

**Note: The activities ‘train AFIX**
**staff’ and ‘provide feedback and **
**recommend strategies for improvement’**
**are circled in red because the sample**
**evaluation focuses on these areas**

## Outputs

- Protocols
- Trained staff
- Providers visited

- Coverage assessment
- Recommendations
- Rewards provided

- Data in database
- Information exchanged

## Outcomes

- Increased provider knowledge
- Increased provider motivation

- Change in office practices and policies
- Change in physician behavior

-    missed opportunities
-    invalid doses
-    timely vaccine receipt

-      Vaccinations
-      VPD

**Note: The outcome  ‘change in office**
**Practices and policies’ **
**is circled in red because the sample**
**evaluation focuses on this area**

## AFIX Sample PlanPart 2

## Focusing the Evaluation

- Formulate and prioritize questions
- Develop indicators and targets for each question
- Identify data sources to help you answer these questions
- Outline the who, when and how of data collection and analysis

## Prioritizing evaluation questions

- Is the question:
- Important to your program staff and stakeholders?
- Does it reflect key elements of your logic model?
- Will it provide information you can act upon to make program improvements?
- Can it be answered using available program resources?
- Are there any available data sources?

## AFIX Sample PlanPriority Evaluation Questions

- Process Questions
- How many providers receive a quality improvement (QI) plan containing the 4 key items* (Level 1, AFIX Feedback Standards)?
- Do AFIX staff know how to write a QI plan (including 4 key items)?
- Outcome Questions
- How many providers implement the interventions (AKA “recommendations”) that were agreed upon during their AFIX Feedback visit?

***4 key items: intervention, action steps, responsible party, date of implementation**

## AFIX Sample PlanTable 8: Data Collection and Analysis

- Evaluation Question #1: How many providers receive a quality improvement (QI) plan
- containing the 4 key items (Level 1, AFIX Feedback Standards)?

- Indicators
- Percent of providers that received a QI plan following an AFIX visit
- Percent of QI plans that include all four key items
- Targets
- 80% of providers received a QI plan following an AFIX visit
- 100% of QI plans contain all four key items
- Data Sources
- AFIX field staff; site visit notes
- QI plans

## AFIX Sample PlanTable 8: Data Collection and Analysis continued

- Data Collection
- Method: Review site visit notes from 2007-2008; if not available or incomplete, survey AFIX field staff
- Timeline: December 2008-January 2009
- Person Responsible: AFIX coordinator and AFIX field staff
- 2. Method: Review a sample of QI plans from 2007-2008 developed by each AFIX field staff
- Timeline: December 2008-January 2009
- Person Responsible: AFIX coordinator

## AFIX Sample Plan*Table 8: Data Collection and Analysis continued

- Evaluation Question #2: Do AFIX staff know how to write a QI plan
-  (including 4 key items)?
- Indicators:
- Percent of AFIX field staff trained on how to write a QI plan (including all four key
- Items
- 2. Percent of AFIX field staff who feel confident in their ability to write a QI plan
- Targets:
- 100% of AFIX field staff received training on QI plans before conducting AFIX visits
- 100% of AFIX field staff feel confident in their ability to write a QI plan
- Data Sources:
- Staff training records
- AFIX field staff

## AFIX Sample Plan*Table 8: Data Collection and Analysis continued

- Data Collection:
- Method: Review staff training records
- Timeline: December 2008
- Person Responsible: AFIX coordinator
- 2. Method: Survey AFIX field staff about barriers and facilitators of QI plan
- Development
- Timeline: December 2008-January 2009
- Person Responsible: AFIX coordinator

## AFIX Sample Plan*Table 8: Data Collection and Analysis continued

- Analysis:
- Method: Frequencies
- Timeline: December 2008
- Person Responsible: AFIX coordinator
- 2. Method: Frequencies; qualitative content
- Timeline: January 2009
- Person Responsible: AFIX coordinator

## AFIX Sample Plan*Table 8: Data Collection and Analysis continued

- Evaluation Question #3: How many providers implement the interventions
- (AKA “recommendations”) that were agreed upon during their AFIX Feedback visit?
- Indicators:
- Percent of providers that implemented one or more recommendations (by the
- Proposed date)
- 2. Percent of feedback visits that were conducted with provider staff who have the
- Authority to implement changes
- Targets:
- Providers implemented at least 50% of recommendations (by the proposed date)
- 100% of AFIX feedback sessions are conducted with provider staff who have the
- Authority to implement changes
- Data Sources:
- Providers; AFIX field staff; site visit notes and QI plans from 2007-2008
- Site visit notes from 2007-2008

## AFIX Sample Plan*Table 8: Data Collection and Analysis continued

- Data Collection:
- Method: Survey (a sample of) providers who received visits during 2007-2008 to
- Identify which recommendations were implemented; survey (a sample of)
- Providers and all AFIX field staff about barriers and facilitators of implementation
- of recommendations; Review site visit notes and QI plans from 2007-2008
- Timeline: December 2008-February 2009
- Person Responsible: AFIX coordinator and AFIX field staff
- 2. Method: Review site visit notes and QI plans from 2007-2008
- Timeline: December 2008 – January 2009
- Person Responsible: AFIX field staff
- Analysis:
- Method: Frequencies; qualitative content
- Timeline: February 2009
- Person Responsible: AFIX coordinator and AFIX field staff
- 2. Method: Frequencies
- Timeline: January 2009
- Person Responsible: AFIX field staff

## *Table 9: Disseminating Findings (“Ensuring Use”) 

***Table 9: Disseminating Findings (“Ensuring Use”)**** **

**Target Person or Groups Name**: Program Manager
**Target’s Evaluation Use**: Determine the current status of AFIX feedback sessions;
- Utilize findings to improve training provided to field staff and maximize feedback
- Sessions to ensure more providers are implementing agreed upon recommendations
**Dissemination Items and Methods**: provide a complete report of findings
- (electronic and hardcopy)
**Timeline for Dissemination: **at the completion of the evaluation (March 2009)
**Person Responsible: **AFIX coordinator
**Target Person or Groups Name: **AFIX field staff
**Target’s Evaluation Use: **Identify areas of improvement for AFIX feedback sessions;
- Adjust methodology for recommendations made during feedback sessions
**Dissemination Items and Methods: **present a summary of evaluation findings
- During monthly staff meetings
**Timeline for Dissemination: **at the completion of the evaluation (March 2009)
**Person Responsible: **AFIX coordinator

## *Table 9: Disseminating Findings (“Ensuring Use”) continued 

***Table 9: Disseminating Findings (“Ensuring Use”) continued**** **

**Target Person or Groups Name: **Immunization providers (All AFIX providers
- Including data sources)
**Target’s Evaluation Use: **Improve implementation of agree upon interventions;
- Use as a benchmark to see how they are performing compared to other providers
- with regard to implementing recommendations
**Dissemination Items and Methods: **provide a summary of relevant findings
- (electronic and hardcopy)
**Timeline for Dissemination: **at the completion of the evaluation (March 2009)
**Person Responsible: **AFIX field staff
**Target Person or Groups Name: **NIC 2009 conference
**Target’s Evaluation Use: **learn about methodology for evaluating AFIX feedback
- visits
**Dissemination Items and Methods: **presentation
**Timeline for Dissemination: **February 2009
**Person Responsible: **program manager

## Summary

- Evaluation plan will be submitted via PAPA/CDC Secure Data Network
- Due date is August 30, 2008
- Evaluation plan template (June 2008 version) includes all the items that you will need to submit to us

## Resources

- Immunization evaluation resources are available at the Immunization Program Evaluation (IPE) website at _www.cdc.gov/vaccines/programs/progeval_
- Questions / Comments / Suggestions
- You can Contact us either via the IPE email at _IPE@cdc.gov_ or via your CDC project officer
- THANK YOU!

## Extra Slides

## Indicators

- A measurable piece of information that helps you answer your evaluation question
- Relates to the objectives of your evaluation
- Can be related to processes or outcomes
- Also referred to as a performance measure in the IPOM
- Can use existing ones or develop ones tailored to a particular question

## Targets

- Reasonable expectations about what “success” means
- Should create one for each indicator
- Based on the current status of an activity
- Consider program requirements

## What are Data Sources?

- 	Where or from whom you will get data for each of your indicators to answer your evaluation questions.

- Data Source: Documents
- Examples: medical records, meeting minutes, surveillance reports, interview records
- Data Source: Individuals
- Examples: staff, providers, partnership members
- Data Source: Observations
- Examples: data obtained from observations of staff, environment (reception area),
- office flow, activities, etc.

## Advantages/Disadvantages of Data Sources

- Data Source: Documents
- Advantages: data available and accessible
- Disadvantages: value of data depends on how accurately it was recorded; may lack data
- needed for the evaluation
- Data Source: Individuals
- Advantages: can be collected directly from target population
- Disadvantages: may be unreliable due to social desirability and/or recall difficulty
- Data Source: Observations
- Advantages: can supplement self-report; provide information on behavior, skills, environment
- Disadvantages: value of data depends on training of observer and specificity of instrument

## Data Collection Methods

- Method: Survey
- Advantages: anonymity possible; can administer to groups; efficient and cost effective
- Disadvantages: forced choices limit response; wording may bias response; impersonal
- Method: Individual interviews
- Advantages: can build rapport; can probe for more information; can get breadth/depth of information
- Disadvantages: time consuming, expensive, interview style may bias
- Method: focus groups
- Advantages: can get breadth and depth of information in short time frame; can convey key information regarding the program
- Disadvantages: need trained facilitator; time consuming to analyze responses
- Method: Observation
- Advantages: Can assess fidelity as activities occur
- Disadvantages: interpretation of behavior difficult; expensive and time consuming
- Method: Document review
- Advantages: information already exists; does not disrupt program
- Disadvantages: depends on quality of information; time consuming

## This document

- This document can be found on the CDC website at:
- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_4th-508.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_4th-508.ppt)__ _