# Presentation: 347415

## Practical Evaluation for Public Health Programs

**2007 AIM / PROGRAM MANAGER MEETING**
**November 6, 2007**
**Betty Apt**
**Division of STD Prevention**
**Centers for Disease Control and Prevention**

## Background

- Two previous webinars
- How to choose what to evaluate
- CDC framework for evaluating public health programs
- Today- Address elements you will need to report on in your 2009 grant applications

## What is Program Evaluation?

- ...the **systematic** collection of information about the activities, characteristics, and outcomes of programs **to make judgments** about the program, **improve** **program effectiveness**, and/or **inform decisions** about future programming.
- 		*(Patton, Utilization Focused Evaluation, 1997)*

## Why Evaluate?

## Sharks Tracking Ketchup: How We Can Go Wrong

- County did extensive outreach to perform syphilis screening in the community in a variety of venues & settings.  Lots of resources devoted to this.
- After one year they had not looked at their data.  As part of evaluation project they did.
- Found
- No new cases of syphilis.
- Interventions that work in one setting might not work in another setting.
- Need to evaluate what you are doing.

## What Can Program Evaluation Do?

**Increase Program Knowledge **
**Direct Program Improvement**
**Maximize Resources**
**Provide Accountability**

## CDC’s Evaluation Framework

- Step 1: **Engage Stakeholders**
**Step 2: Describe the Program**
**Step 3: Focus the Evaluation Design**
**Step 4: Gather credible evidence**
**Step 5: Justify conclusions**
**Step 6: Ensure use and share lessons learned**

**Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11). **

## CDC’s Evaluation FrameworkStandards

- Utility
- Feasibility
- Propriety
- Accuracy

**Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR1999; 48 (No. RR-11). **

## Program Evaluation Grant Requirement...

- As part of the submission of the continuing grant application for 2009, in August 2008, grantees are required to submit an evaluation plan.

## Evaluation Plan

- An evaluation plan is the “blueprint” that lays out a plan for implementation of your evaluation. The plan describes what the evaluation will look like as well as what information will be collected and what will be done with the results.

## Why do you need an evaluation plan?

- Outline evaluation design, activities and procedures
- Define responsibilities, timeline
- Clarify expectations

## Getting started

- Choose an evaluation component or activity
- Form an evaluation team at your program
- Assess the team’s capacity and resources to conduct evaluation
- Identify relevant stakeholders

## Some Potential Components/Activities to Evaluate

- Immunization information systems
- Adolescent immunization
- Perinatal hepatitis B prevention
- AFIX
- VFC

## Some Potential Criteria...

- Where am I *spending the most?*
- Where am I *concerned the most?*
- Where are my *big successes*?
- Where are my *failures?*
- Other...
- Other...

## Step 1: Engaging Stakeholders

## Who are Stakeholders?

**Stakeholders are individuals or organizations who are interested in, or affected by, the program and/or evaluation.**

## Some Key Stakeholders

- Type: Decision Makers
- Definition: Decide and direct program operations, including how evaluation findings are used
- Examples: Immunization program director, program manager, HD director, health** **commissioner, legislators
- Type: Implementers
- Definition: Involved in program operations
- Examples: Immunization program manager, program staff, private physicians, staff in physician offices
- Type: participants
- Definition: Served by the program / activities
- Examples: Clients/patients, parents, health service providers
- Type: partners
- Definition: Support/invested in the immunization program or target population
- Examples: Funders, managed care plans, schools, medical organizations

## Reasons to Involve Stakeholders

- Find out what they are interested in.
- Increase likelihood that evaluation results will be acted upon.
- Establish what role they will play in the evaluation.
- Obtain “reality check” on utility and feasibility of evaluation plan and methods.

## Tips for Working with Stakeholders

- Communicate to them the benefits of their involvement in the process.
- Incorporate their input, opinions, and insights (or explain why not possible).
- Promise only what you can deliver.
- Maintain regular communications about progress.

## Step 2: Describe the component

## Describing the Program Through 

**Logic Models **

## Defining Logic Models

**A graphic representation, or a flow chart, that shows how a program is supposed to work, i.e.,  how the components of the program should produce desired outcomes. **

## Benefits of Logic Models

- Provides roadmap of expected program progress and results.
- Provides a framework and guide for program planning and evaluation.
- Builds clarity and understanding among stakeholders on how a program is supposed to work.
- Identifies resources needed.
- Identifies gaps.

## Describing the Program:Complete Logic Model

- A flow chart with boxes and arrows leading to the next item.  The flowchart includes the following boxes horizontally:
- Inputs
- Activities
- Outputs
- Short Term Effects/Outcomes
- Intermediate Effects/Outcomes
- Long term effects/outcomes

## Inputs = resources
Activities = what the program and its staff actually do
Outputs = tangible products of activities
Outcomes = results of activities (who or what will change)

- Activities = what the program and its staff actually do
- Outputs = tangible products of activities
- Outcomes = results of activities (who or what will change)

## Types of Logic Models

- Type: Global
- Description:" Big picture” of entire program
- Type: Nested
- Description: segment or component of a program

## Global Logic Model for State X STD Program

- Note: The following slides show the boxes for each aspect of the logic model

**INPUTS**
- Funds
- Assigned Staff
- Technical assistance and collaboration

## ACTIVITIES

- -Provide community and individual behavior change interventions on syphilis.
- -Provide med and lab services.
- Provide Ct screening of young women.
- -Ensure syphilis partner services.
- -Promote leadership and program management.
- -Conduct surveillance and data management.
- -Provide professional development.

## Female admitees in juv detention facilities tested for Ct.

_**Community/ Individual Behavior Change Interventions**_
- -Interventions on syphilis implemented among at risk MSM.
_**Medical and Lab Services**_
- -Lab/med facilities and providers report test results.
**Female admitees in juv detention facilities tested for Ct.**
_**Partner Services**_
- -Syphilis cases’ partners identified.
_**Leadership and Program Management**_
- -Strategic plan in place.
- -Program operation plan to monitor program activities.
- -Appropriate program policies in place.
_**Surveillance and Data Management**_
- -Reported cases of P&S syphilis and Ct sent to CDC in timely manner.
_**Training and Professional Development**_
- -Staff training needs regularly assessed.
- -Training opportunities on syphilis and Ct provided.
_**STD Outbreak Response Planning**_
- -Plan includes required elements.** **

## SHORT-TERM OUTCOMES

- Increased knowledge:
- -consequences;
- -Safe behaviors;
- -Self assessment of risk.
- Increased intention to use condoms.

**INTERMEDIATE OUTCOMES**
- Increased safer sex behaviors:
- -Abstinence
- -Mutual monogamy
- -Fewer concurrent partners

**LONG-TERM OUTCOMES**
- -Reduced syphilis incidence
- - Reduced Ct prevalence

## Nested Logic Model of Chlamydia Screening Program for 

**Adolescent Females in County Z Juvenile Detention Centers**

**Inputs**

_**Funds**_**   **
**                **
_**JDC Staff**_** **
_**Technical Direction, Assistance, and Collaboration**_
_**Materials**_

## Activities

**Provide professional development in Ct services**** **

**Provide JDC support for developing and implementing CT services**** **

**Conduct Ct services for female adolescent admittees****.**** **

## Outputs

**JDC clinical staff participated in professional development**** ****events. **

**JDC administration authorized time and staff for development and implementation of screening**** ****program. **

**Female adolescents counseled and screened**** **

## Short-term outcomes

_**JDC Providers**_**Improved knowledge and skills for Ct services **

_**JDCs**_
**Increased numbers of programs in JDCs that provide Ct services.**** **

## Intermediate Outcomes

_**JDC Providers**_
**Increased clinicians’ adherence to Ct clinical practice guidelines**** **

_**Adolescent Females**_
**Decreased number of partners. Increased health seeking behavior. **

## Long-term Outcomes

_**JDCs**_
**Reduction of Ct prevalence among adolescent females in JDCs. **

## Logic Model for AFIX

- Inputs

- Funding
- Staff
- Time
- CoCASA

## Activities

- Develop AFIX protocol
- Train AFIX staff
- Set up provider visits

- Review provider  records & assess coverage level
- Provide feedback & recommend strategies for improvement
- Acknowledge & reward improved performance

- Document outcome of visit in database (CoCASA)
- Promote information exchange

## Outputs

- Protocols
- Trained staff
- Providers visited

- Coverage assessment
- Recommendations
- Rewards provided

- Data in database
- Information exchanged

## Outcomes

- Increased provider knowledge
- Increased provider motivation

- Change in office practices and policies
- Change in physician behavior

-    decreased missed opportunities
-    decreased invalid doses
-    increased timely vaccine receipt

-     Increased  Vaccinations
-      Decreased VPDs

## Describing the Program Through... 

**Goals and**
**SMART Objectives**

## What is a goal?

- States what should happen as a result of the program/activity
- Describes the expected long-term effects of a program/activity.
- Serves as the foundation for development of objectives.

## Examples

- To improve the health status of Georgia children.
- To increase the proportion of children who are immunized.
- To decrease vaccine preventable diseases.

## What is an objective?

- Describes how you plan to achieve your goal.
- Reflects a _partial accomplishment_ of the goal.
- Serves as the _basis for monitoring progress_ towards achieving program goals.
*If you are meeting your objectives, you will hopefully meet your goal. *

## What’s the Connection between program goals and program objectives?

## Characteristics of a Well-Written Objective

_**It is SMART**_

_**S**_pecific
_**M**_easurable
_**A**_chievable
_**R**_elevant
_**T**_ime-Bound

## SMART Objectives and Evaluation

- Monitor progress
- Identify problems

## Specific

- “who”
- “what” –action
- one action verb per objective
- “where” – location of the action

## Specific

- “who” - target population
- Children ages 18-24 months
- “what” – action
- Conduct 1 workshop on AFIX protocol in every health region of the State

## Measurable

- “How much” change is expected
- Increase the number 18 -24 month old children who receive all vaccinations _**from 20% to 40%.**_

## Achievable

- Should be realistic – personnel, money, etc.
- At least 70% of private providers will implement one or more of the recommended interventions by the proposed date.

## Relevant

- Relates to the program goals.
- At least 50% of parents who attended the health fair and completed an evaluation will report increases in knowledge about VPD

_**T**_**ime-bound**

- “When” the objective will be measured/met
- By March 31, 2008, ...

##    Example 1 	

-   By Dec 31, 2009, 100% of AFIX feedback sessions conducted in 2009 will have included a quality improvement plan that includes all four key items.

***SMART?***
***Yes***

**S****pecific: **who, what, where

**M****easurable: **how much change

**Achievable: **realistic

**R****elevant: **relate to goals

**T****ime-bound: **when

##    Example 2	

-   During 2008, physician practices will receive appropriate feedback.

***SMART?***
***NO***

**S****pecific**
**M****easurable**
**Achievable**
**R****elevant**
**T****ime-bound**

##    Types of Objectives

**Process Objectives**
- Describe program activities
- Specify actions to be taken to measure program implementation

**Outcome Objectives**
- Relate to changes in the target population or end result of a program (changes in policies, regulations, sustainability)
- What the target population will know or will be able to do by the conclusion of the program

## Types of Outcome Objectives

- Type: Short Term
- Definition/Examples: Initial changes in target population/organization/group
- (awareness, knowledge, attitudes, skills)
- Type: Intermediate
- Definitions/Examples: Interim changes
- (norms, behavior, policy).
- Type: Long Term
- Definitions/Examples: Changes in health status, morbidity as result of sustained program implementation.

## SMART Objectives Group Exercise

- The State Immunization Program manager is concerned about the quality of feedback AFIX field staff are giving to physician practices.
- As a group, for the AFIX program, write:
- 		1 process
- 	1 short-term,
- 	1 intermediate, and
- 	1 long-term outcome objective based on the SMART criteria.

## Examples of Objectives

**Goal:  Improve vaccination coverage among children **
**and adolescents in the state through the AFIX program****.**

- Include the following:
- process
- short term outcome
- intermediate outcome
- long term outcome

## Steps 3 & 4: Evaluation Focus & Data Collection

## Plan Your Evaluation by...

**What questions you want the evaluation to answer about your program?****     **

## Examples of Evaluation Questions

- Type: Process
- Definition: Determines if program activities are implemented as intended;
- Provides feedback for program improvement; conducted throughout project life.
- Examples: Were recommendations of the reviewers implemented by the providers?
- Type: Outcome
- Definition: Measures program effects; changes in target population’s knowledge,
- attitudes, skills, intentions, behaviors; or organizational changes
- (e.g., policy adoption); linked to process evaluation; provides feedback for program
- improvement.
- Examples: Did the number of missed opportunities decrease?

## Purpose of Evaluation Questions

- Helps focus the evaluation.
- Guides the evaluation planning process.
- Facilitates decision-making about evaluation methods to use.
- Helps ensure use of the evaluation findings

## How Questions Direct the Data Collection and Analysis Plan

**EXAMPLE QUESTION: During feedback sessions in 2007,did all physician **
**practices receive a quality improvement (QI) plan containing all four key items? **

- 1. Opportunity for improvement on which to focus
- 2. Define action steps for implementing the intervention
- 3. Responsible party for implementation
- 4. Date to implement intervention

## Steps To Develop Evaluation Questions

**Involve stakeholders.**** **
**2.  Brainstorm on possible evaluation questions.**
**	*****What do you want the evaluation to answer about the program component or activity?***

## Steps to Develop Evaluation Questions (continued)

**3.****  **Prioritize questions with criteria of:
- Important to stakeholders;
- Will provide information that can be acted upon;
- Can be answered with available resources;
- Can be answered within available timeframe; and
- Will be supported by program decision-makers.
- 4. Develop a final list.

## Data Collection and Analysis

**EXAMPLE QUESTION: During feedback sessions in 2007, did all physician **
**practices receive a quality improvement (QI) plan containing all four key items? **

- Flow chart diagram with the following boxes from left to right:
- Indicators, target, data sources, data collection, analysis

## What is an Indicator?

- The piece of information that you need to give you the answer to your evaluation question
- A measure that shows the progress made
- The “how will I know” answer

## What are Data Sources?

- 	Where or from whom you will get data for each of your indicators to answer your evaluation questions.

- Data Source: Documents
- Examples: medical records, meeting minutes, surveillance reports, interview records
- Data Source: Individuals
- Examples: staff, providers, partnership members
- Data Source: Observations
- Examples: data obtained from observations of staff, environment (reception area),
- office flow, activities, etc.

## Advantages/Disadvantages of Data Sources

- Data Source: Documents
- Advantages: data available and accessible
- Disadvantages: value of data depends on how accurately it was recorded; may lack data needed for the
- evaluation
- Data Source: Individuals
- Advantages: can be collected from target population
- Disadvantages: may be unreliable due to social desirability and/or recall difficulty
- Data Source: Observations
- Advantages: can supplement self-report; provide information on behavior, skill environment
- Disadvantages: value of data depends on training of observer and specificity of instrument

## Data Collection Methods

**Surveys**
**Interviews**
**Focus Groups**
**Observation**
**Document Review**

## Data Collection Methods

- Method: Survey
- Advantages: anonymity possible; can administer to groups; efficient and cost effective
- Disadvantages: forced choices limit response; wording may bias response; impersonal
- Method: Individual interviews
- Advantages: can build rapport; can probe for more information; can get breadth/depth of information
- Disadvantages: time consuming, expensive, interview style may bias
- Method: focus groups
- Advantages: can get breadth and depth of information in short time frame; can convey key information regarding the program
- Disadvantages: need trained facilitator; time consuming to analyze responses
- Method: Observation
- Advantages: Can assess fidelity as activities occur
- Disadvantages: interpretation of behavior difficult; expensive and time consuming
- Method: Document review
- Advantages: information already exists; does not disrupt program
- Disadvantages: depends on quality of information; time consuming

## Data Collection Procedures

- Use existing data when feasible
- Understand agency policies and regulations that may affect data collection
- Identify who will be responsible
- Identify the timeline of when the data will be collected.
- Be clear about the data you want to collect and sensitive to the time and effort that will need to be expended by the data providers
- Design instruments as needed

## Using Mixed Data Sources/Methods

- Involves using more than one data source and/or data collection method.
- Advantages:
- Allow examination of different facets of the same phenomenon
- Obtain comprehensive information
- Increase validity of results

## Step 5: Justify Conclusions

## Now that you’ve collected the data, what do you do with it?  

**Analyzing data **
**Who**
**When**
**How**
**Interpretation of results and sharing findings**

## Things to Consider When Analyzing Data

- Quantitative Methods
- Develop a database for all fields in the data collection instrument
- Depending on type of analysis, specific quantitative skills may be needed
- Qualitative Methods
- Categorize similar findings (coding, subcoding)
- Look for patterns

## EXAMPLE QUESTION: During feedback sessions in 2007, did all physician 

**practices receive a quality improvement (QI) plan containing all four key items*? **

- Indicator:
- Percent of practices that received a QI plan following an AFIX visit during 2007;
- percent of QI plans that included all four key items
- Target:
- 100% of practices received a QI plan following an AFIX visit during 2007;
- 100% of QI plans contained all four key items
- Data Sources: AFIX field staff; site visit notes or records
- Data Collection:
- Method – Review site visit notes or records; survey field staff
- Timeline – twice a year
- Person Responsible – state health department AFIX coordinator
- Analysis:
- Method – frequency
- Timeline – twice a year
- Person responsible – State health department AFIX coordinator

- *Four key items:
- 1. Opportunity for improvement on which to focus
- 2. Define action steps for implementing the intervention
- 3. Responsible party for implementation
- 4. Date to implement intervention

## Data Collection and AnalysisGroup Exercise

- Evaluation Question:  During 2007, did physicians implement recommendations by the proposed date as outlined in the QI plan?
- As a group, using the exercise handout, complete
- the table by developing appropriate:
- indicators,
- targets
- data sources,
- data collection methods, and
- type of analysis.

## CDC Framework for Program Evaluation

## Share the results and lessons learned from the evaluation with stakeholders and others

Use your evaluation findings to modify, strengthen, and improve your program

**Use your evaluation findings to modify, strengthen, and improve your program**

**Step 6:  ****Ensure Use and ****Share Lessons Learned**

## Step 6:  Ensure Use and Share Lessons Learned

**Share the results and lessons learned from the evaluation with stakeholders and others**
**Use your evaluation findings to modify, strengthen, and improve your program**

## Interpreting Findings

**Always refer to your evaluation questions.**
**Identify findings that will help answer the evaluation questions.**
**Stakeholders may provide some insight about the findings**

## Developing Recommendations

- Your evaluation’s recommendations should be:
- Linked with the original purpose of your evaluation.
- Based on answers to your evaluation questions.
- Linked to findings from your evaluation
- Tailored to the users of the evaluation results to increase ownership and motivation to act.

## How to Share the Evaluation Results/Recommendations

- Consider information needs of the audience/stakeholders.
- Tailor message and format of dissemination to the users of the evaluation results
**Oral **
**Written**
**Full Report**
**Executive Summary**

## Type of Dissemination Methods

- _Evaluation Reports_
- Provide an executive summary.
- Use examples, graphics, quotes to highlight findings.
- Present data simply and concisely.
- Use active verbs to shorten sentences.
- Organize results by evaluation question.

## Type of Dissemination Methods

**Oral Presentations**
- Use slide show; provide handouts
- Involve audience in discussion of how to use findings to improve program, help set policy, etc.
**Mass Media**
- Use portions of executive summary.
- Follow your agency’s protocol on how to address the media.

## Acknowledgements

-   _ISD/NCIRD_			      _AIM_
- Nancy Fasano			Helen Fox Fields
- Suchita Lorick			Jesse Greene
- Greg Reed				Claire Hannan
- Pascale Wortley
- Evaluation Workgroup
- 		  _Division of STD Prevention_
- 	Health Services Research & Evaluation Branch
- 	Yamir Salabarría-Peña
- Tom Chapel

## This document

- This document can be found on the CDC website at:
- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/prog-man-meeting07-508.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/prog-man-meeting07-508.ppt)_** **