# Presentation: 478617

## Immunization Program Evaluation: Writing an Evaluation Plan

**Pascale Wortley, MD, MPH**
**Suchita Lorick, DO, MPH**
**Immunization Services Division, NCIRD**
**June 3, 2008**

***"The findings and conclusions in this report are those of the authors and ***
***do not necessarily represent the official position of the CDC."***

## Webinar Overview	

- Evaluation Plan Background
- Evaluation Plan Template – Overview
- Sample Evaluation Plan – AFIX
- Questions and Answers

## Evaluation Plan Background

- Plan due to CDC by August 30, 2008
- Submitted via PAPA/CDC Secure Data Network (https://sdn.cdc.gov)
- Expected to go live end of June
- Grantees will receive email about applying for digital certificate in the next 2 weeks
- CDC will provide feedback on the submitted evaluation plans by October 31, 2008
- Immunization Program Evaluation (IPE) Resources at _http://www.cdc.gov/vaccines/programs/progeval/_
- Questions/Comments/Suggestions can be submitted via _IPE@cdc.gov_ or your CDC project officer

## Evaluation Plan Template -Overview	

- Program Component/Activity
- Activity (ies)
- Evaluation Goal (s)
- Stakeholders (Interests/Roles/Responsibilities)
- Include evaluation team at the program
- Describing program component/activity
- Focusing evaluation
- Data collection and analysis
- Ensuring use and sharing lessons learned

## AFIX Sample PlanPart 1

## AFIX Sample PlanProgram Component / Activity

- AFIX
- The feedback sessions of AFIX visits

## AFIX Sample PlanEvaluation Goal

- To improve the quality and effectiveness of AFIX feedback sessions
- Should reflect where you want to be

## Stakeholders

- Gain broader perspective, avoid blind spots, try to ensure utilization of results
- Key stakeholders:
- Those served or affected by activity
- Those involved in program operations
- Those in a position to make decisions about the activity
- For a manageable process, the list of stakeholders must be narrowed to primary intended users

## AFIX Sample Plan

| Group Name OR Person’s Name and job title | Interests in or perspectives about the evaluation | Roles in the evaluation | Responsibilities in the evaluation | How and when to engage |
| --- | --- | --- | --- | --- |
| Program Manager | Ensure adequate stakeholder input; Complete evaluation plan on time; Ensure results of evaluation are useful and utilized to improve program performance | Evaluation team leader | Oversee data collection and analysis;  Write evaluation plan; Utilize evaluation findings | Involved in all activities throughout the evaluation  |
| AFIX Coordinator | Evaluation and its findings could impact routine activities; Interested in providing useful feedback for providers | Stakeholder liaison;  Data collection and analysis | Coordinate stakeholder meetings; Provide input on evaluation design, methods, and interpretation and use of findings; Share findings with stakeholders; Utilize evaluation findings | Meet regularly with PM to develop methods; Attend all SH meetings |

**AFIX Sample Plan**
***Tables 1 and 2: Stakeholders - ****Interests, Roles, and Responsibilities**** **

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

## AFIX Sample Plan

| Group Name OR Person’s Name and job title | Interests in or perspectives about the evaluation | Roles in the evaluation | Responsibilities in the evaluation | How and when to engage |
| --- | --- | --- | --- | --- |
| 2 AFIX field staff | Evaluation and its findings will directly impact day-to-day activities; Interested in providing useful feedback for providers | Provider liaisons;  Data collection | Provide input on data collection methods; Data source and data collectors; Share findings with providers; Utilize evaluation findings | Meet regularly with AFIX Coordinator; Attend initial SH meeting and subsequent ones as needed |
| Provider  office staff (clinical 
and admin.) | Interested in receiving constructive feedback on ways to improve immunization delivery and providers; Interested in minimizing AFIX-related burden on staff time  | Represent providers’ perspectives on AFIX Feedback;   | Provide feedback during evaluation planning; Data source; Utilize evaluation findings | Invite to initial stakeholder meeting; Phone/Email communication during data collection phase; As needed, invite to other SH meetings to troubleshoot, share findings, etc. |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Tables 1 and 2: Stakeholders (****Interests, Roles, and Responsibilities)**

## AFIX Sample PlanDescribing the program component/activity

**Background**
- Program has 5 AFIX staff
- Staff get some instructions on AFIX feedback sessions at the time they are hired
- None have received standardized training on AFIX Feedback standards
- Currently, program does not track how many feedback sessions include use of written quality improvement (QI) plans (as described in the AFIX standards, Feedback, Level I – pg 10)
- Additionally, we do not know if QI plans contain the minimum requirements
- Although AFIX staff work with providers to agree upon opportunities for improvement, we do not track in a standardized manner whether providers are implementing the agreed upon changes.

## AFIX Sample PlanDescribing the program component/activity (Cont’d)

- Context
- Feedback sessions are the only opportunity to interact directly with provider staff (especially decision-makers)
- Feedback visits are often combined with VFC visits and therefore, some of the items discussed for the AFIX portion might not get adequate time and attention
- Want to use the visit to help providers improve immunization delivery and increase their immunization rates

## AFIX Sample PlanDescribing the program component/activity (Cont’d)

- Stage of Development
- We have been implementing AFIX visits (including Feedback sessions) for the past 8 years. The process is very well-established.
- Target Population
- Clinical and administrative staff in private and public immunization provider offices
- AFIX staff

## AFIX Sample PlanDescribing the program component/activity (Cont’d)

**Objectives (should reflect where you want to be):**
- By March 2009, 100% of AFIX field staff in the state will receive training on how to conduct feedback sessions that meet all AFIX Level I standards.
- By June 2009, all AFIX field staff will report an increase in knowledge about the components of a feedback session including which key items should be included in a quality improvement (QI) plan.

## AFIX Sample PlanDescribing the program component/activity (Cont’d)

**Objectives (con’t):**
- Between July and December 2009, during AFIX Feedback sessions, all AFIX field staff will develop QI plans that contain all key items.
- Starting in January 2010, all AFIX field staff will follow up with providers at least on a biannual basis to document their progress in implementing the improvement opportunities that were agreed upon at their last feedback session.

## AFIX Sample PlanDescribing the program component/activity (Cont’d)

**Objectives (con’t):**
- Starting in January 2010, all providers will implement by the proposed date at least 50% of the improvement opportunities that were agreed upon at their last feedback session.

## Describing the Program:Complete Logic Model

- Context
- Assumptions
- Stage of  Development

## AFIX Sample Plan

| Inputs | Activities
Initial                Subsequent |  | Outputs | Outcomes
Short-/Mid-term   Long-term |  |
| --- | --- | --- | --- | --- | --- |
| Funding

Staff

Time

CoCASA | Develop AFIX protocol

Train AFIX staff

Set up provider visits | Review provider records & assess coverage level

Provide feedback & recommend strategies for improvement

Acknowledge and reward improved performance

Document outcome of visit in database (CoCASA) 

Promote information exchange | Protocols

Trained staff

Providers visited

Coverage assessment

Feedback provided

Rewards provided

Data in database

Information exchanged | Increased provider knowledge

Increased provider motivation

Change in office practices and policies

Change in physician behavior | Decrease in missed opportunities

Decrease in invalid doses

Increase in timely vaccine receipt for eligible population

Increased immunization coverage

Decreased VPD |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Table 3:  Program Component/Activity Description**** **

## Optional: Logic Model for AFIX

**Optional: Logic Model for AFIX**

- Funding
- Staff
- Time
- CoCASA

- Develop AFIX protocol
- Train AFIX staff
- Set up provider visits

- Review provider  records & assess coverage level
- Provide feedback & recommend strategies for improvement
- Acknowledge & reward improved performance

- Protocols
- Trained staff
- Providers visited

- Coverage assessment
- Recommendations
- Rewards provided

- Increased provider knowledge
- Increased provider motivation

- Document outcome of visit in database (CoCASA)
- Promote information exchange

- Data in database
- Information exchanged

- Change in office practices and policies
- Change in physician behavior

-      Vaccinations
-      VPD

-    missed opportunities
-    invalid doses
-    timely vaccine receipt

## AFIX Sample PlanPart 2

## Focusing the Evaluation

- Formulate and prioritize questions
- Develop indicators and targets for each question
- Identify data sources to help you answer these questions
- Outline the who, when and how of data collection and analysis

## Prioritizing evaluation questions

- Is the question:
- Important to your program staff and stakeholders?
- Does it reflect key elements of your logic model?
- Will it provide information you can act upon to make program improvements?
- Can it be answered using available program resources?
- Are there any available data sources?

## AFIX Sample PlanPriority Evaluation Questions

- Process Questions
- How many providers receive a quality improvement (QI) plan containing the 4 key items* (Level 1, AFIX Feedback Standards)?
- Do AFIX staff know how to write a QI plan (including 4 key items)?
- Outcome Questions
- How many providers implement the interventions (AKA “recommendations”) that were agreed upon during their AFIX Feedback visit?

***4 key items: intervention, action steps, responsible party, date of implementation**

## *Table 8: Data Collection and Analysis

| INDICATOR(S) | TARGET(S) | DATA SOURCE(S) | DATA COLLECTION | ANALYSIS |
| --- | --- | --- | --- | --- |
| Evaluation Question #1: How many providers receive a quality improvement (QI) plan containing the 4 key items (Level 1, AFIX Feedback Standards)? |  |  |  |  |
| 1. Percent of providers that received a QI plan following an AFIX visit | 1. 80% of providers received a QI plan following an AFIX visit | 1. AFIX field staff; site visit notes  | 1. Method:  Review site visit notes from 2007-2008; if not available or incomplete, survey AFIX field staff
Timeline: Dec 2008-Jan 2009 
Person Responsible:  AFIX coordinator and AFIX field staff | 1. Method:  Frequencies
Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator and AFIX field staff |
| 2. Percent of QI plans that include all four key items | 2. 100% of QI plans contain all four key items | 2. QI Plans  | 2. Method: Review a sample of QI plans from 2007-2008 developed by each AFIX field staff Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator | 2. Method: Frequencies Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Table 8: Data Collection and Analysis**

## *Table 8: Data Collection and Analysis (Cont’d) 

| INDICATOR(S) | TARGET(S) | DATA SOURCE(S) | DATA COLLECTION | ANALYSIS |
| --- | --- | --- | --- | --- |
| Evaluation Question #2: Do AFIX staff know how to write a QI plan (including 4 key items)? |  |  |  |  |
| 1. Percent of AFIX field staff trained on how to write a QI plan (including all four key items) | 1. 100% of AFIX field staff received training on QI plans before conducting AFIX visits | 1. Staff training records | 1. Method: Review staff training records Timeline: Dec 2008
Person Responsible: AFIX coordinator | 1. Method: Frequencies
Timeline: Dec 2008
Person Responsible: AFIX coordinator |
| 2. Percent of AFIX field staff who feel confident in their ability to write a QI plan | 2. 100% of AFIX field staff feel confident in their ability to write a QI plan
 | 2. AFIX field staff | 2. Method:  Survey AFIX field staff about barriers and facilitators of QI plan development
Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator | 2. Method: Frequencies; Qualitative content 
Timeline: Jan 2009
Person Responsible: AFIX coordinator |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Table 8: Data Collection and Analysis**** ****(****Cont’d)**** **

## *Table 8: Data Collection and Analysis (Cont’d)

| INDICATOR(S) | TARGET(S) |  | DATA SOURCE(S) | DATA COLLECTION | ANALYSIS |
| --- | --- | --- | --- | --- | --- |
| Evaluation Question #3: How many providers implement the interventions (AKA “recommendations”) that were agreed upon during their AFIX Feedback visit? |  |  |  |  |  |
| 1. Percent of providers that implemented one or more recommendations (by the proposed date) |  | 1. Providers implemented at least 50% of recommendations (by the proposed date) | 1. Providers; AFIX field staff;
Site visit notes and QI plans from 2007-2008 | 1. Method:  Survey (a sample of) providers who received visits during 2007-2008 to identify which recommendations were implemented; survey (a sample of) providers and all AFIX field staff about barriers and facilitators of implementation of recommendations; Review site visit notes and QI plans from 2007-2008
Timeline: Dec 2008-Feb 2009 
Person Responsible:  AFIX coordinator and AFIX field staff | 1. Method:  Frequencies; Qualitative content
Timeline: Feb 2009 
Person Responsible: AFIX coordinator and AFIX field staff |
| 2. Percent of feedback visits that were conducted with provider staff who have the authority to implement changes  |  | 2. 100% of AFIX feedback sessions are conducted with provider staff who have the authority to implement changes  | 2. Site visit notes from 2007-2008 | 2. Method:  Review site visit notes and QI plans from 2007-2008
Timeline: Dec 2008-Jan 2009 
Person Responsible:  AFIX field staff  | 2. Method:  Frequencies
Timeline: Jan 2009 
Person Responsible: AFIX field staff |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Table 8: Data Collection and Analysis (****Cont’d)**

## *Table 9: Disseminating Findings (“Ensuring Use”) 

| TARGET PERSON OR GROUP’S NAME | TARGET’S EVALUATION USE(S) | DISSEMINATION 
ITEMS AND METHODS  | TIMELINE FOR DISSEMINATION | PERSON RESPONSIBLE |
| --- | --- | --- | --- | --- |
| Program manager | Determine the current status of AFIX Feedback sessions; Utilize findings to improve training provided to field staff and maximize Feedback sessions to ensure more providers are implementing agreed upon recommendations | Provide a complete report of findings (electronic and hardcopy) | At the completion of the evaluation  (~  March 2009) | AFIX Coordinator |
| AFIX field staff | Identify areas of improvement for AFIX Feedback sessions; Adjust methodology for recommendations made during feedback sessions. | Present a summary of evaluation findings during monthly staff meetings | At the completion of the evaluation (~ March 2009) | AFIX Coordinator |
| Immunization providers
(All AFIX providers including data sources) | Improve implementation of agreed upon interventions; Use as benchmark to see how they are performing compared to other providers with regard to implementing  recommendations | Provide a summary of relevant findings (electronic and hardcopy) | At the completion of the evaluation (~ March 2009) | AFIX Field staff  |
| NIC 2009 Conference | Learn about methodology for evaluating AFIX Feedback visits | Presentation | February 2009 | Program Manager |

- *Table numbers are based on numbering of tables in the Guide to Immunization Program Evaluation

**AFIX Sample Plan**
***Table 9: Disseminating Findings (“Ensuring Use”)**** **

## Summary

- Evaluation plan will be submitted via PAPA/CDC Secure Data Network
- Due date is August 30, 2008
- Evaluation plan template (June 2008 version) includes all the items that you will need to submit to us

## Resources

- Immunization evaluation resources are available at the Immunization Program Evaluation (IPE) website at _www.cdc.gov/vaccines/programs/progeval_
- Questions / Comments / Suggestions
- You can Contact us either via the IPE email at _IPE@cdc.gov_ or via your CDC project officer
- THANK YOU!

## Extra Slides

## Indicators

- A measurable piece of information that helps you answer your evaluation question
- Relates to the objectives of your evaluation
- Can be related to processes or outcomes
- Also referred to as a performance measure in the IPOM
- Can use existing ones or develop ones tailored to a particular question

## Targets

- Reasonable expectations about what “success” means
- Should create one for each indicator
- Based on the current status of an activity
- Consider program requirements

## Data Sources

- 	Where or from whom you will get data for each of your indicators to answer your evaluation questions.

| Data Sources | Examples |
| --- | --- |
| Documents | medical records, meeting minutes, surveillance reports, interview records |
| Individuals | staff, providers, partnership members |
| Observations | data obtained from observations of staff, environment (reception area), office flow, activities, etc. |

## Data Sources: Advantages/Disadvantages

| Data Source | Advantages | Disadvantages |
| --- | --- | --- |
| Documents
 | data available and accessible | value of data depends on how accurately it was recorded 
may lack data needed for the evaluation. |
| Individuals | can be collected directly from target population | may be unreliable due to social desirability and/or recall difficulty |
| Observations | can supplement self-report
provide information on behavior, skills, environment | value of data depends on training of observer & specificity of instrument |

**Data Sources: Advantages/Disadvantages**

## Data Collection Methods

| Methods | Advantages | Disadvantages |
| --- | --- | --- |
| Surveys | Anonymity possible
Can administer to groups
Efficient & cost effective | Forced choices limit response 
Wording may bias response
Impersonal |
| Individual interviews | Can build rapport
Can probe for more info
Can get breadth/depth of info | Time consuming
Expensive
Interview style may bias |
| Focus groups | Can get breadth & depth of info in short time frame
Can convey key info re program | Need trained facilitator
Time consuming to analyze responses |
| Observation | Can assess fidelity as activities occur | Interpretation of behavior difficult
Expensive & time consuming |
| Document review | Info already exists
Doesn’t disrupt program | Depends on quality of info
Time consuming |

## This document

- This document can be found on the CDC website at:
- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_4th.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_4th.ppt)__ _