# Presentation: 476227

## Practical Program Evaluation—Follow Up: Working a Case Example

**By:**
**Thomas J. Chapel, Office of Workforce and Career Development**
**Betty Apt, Division of STD Prevention**
**Centers for Disease Control and Prevention**

## Today...

- Review main points from overview by applying to a case example
- Choosing a component to evaluate
- Identifying stakeholders that matter most
- Developing simple logic models
- Choosing a strong evaluation focus
- Discuss November workshop

## Why Bother? Because We Want Programs to Improve

- Continuous Quality Improvement (CQI) cycle.
**Planning**—*What* actions will best reach our goals and objectives.
**Performance measurement**— How are we doing?
**Evaluation**—W*hy* are we doing well or poorly?

- What do we do?

- Why are we doing well or poorly?

- How are we doing?

- How do we do it?

## What can Program Evaluation do?

- Increase Program Knowledge
- Direct Program Improvement
- Maximize Resources
- Provide Accountability

## Avoid Wasted Effort!

**Avoid Wasted Effort!**

## Two Pieces of Work

- Choose a program component to evaluate
- Apply evaluation steps and standards

## Choosing a Program Component to Evaluate

- Evaluation can focus on any/all parts of a large program like the immunization program

## Some Potential Criteria..

- Where am I ***spending the most?***
- Where am I ***concerned the most?***
- Where are my ***big opportunities***/new areas?
- Where are my ***big successes***?
- Other...
- Other...

## Some Potential Components to Evaluate

- Immunization information systems
- Adolescent immunization
- Perinatal hepatitis B prevention
- AFIX
- VFC

## Today’s Case Example: AFIX—Reasons for Selecting Component

- Substantial program effort
- Research shows it works...
- But implemented in various ways...
- Want to ensure “our” approach is working

## Next—Apply Evaluation Steps and Standards

## Evaluation Step—Stakeholders

- Identify and engage stakeholders
- Describe the program component
- Focus the evaluation design
- Gather credible evidence
- Justify conclusions
- Ensure use and share lessons learned

## Which Stakeholders Matter Most?

_*Who is*_
_*Affected*_ by the program?
_*Involved*_ in program operations?
- Intended _*users*_ of evaluation findings?

_***Of these, who do we most need to:***_
- Enhance _*credibility?*_
_*Implement*_ program changes?
_*Advocate*_ for changes?
_*Fund, authorize*_, expand program?

##        Some Key Stakeholders

| Types | Definitions | Examples |
| --- | --- | --- |
| Decision makers | Decide and direct program operations, including how evaluation findings are used | Immunization program director, program manager, HD director, health commissioner, legislators  
 |
| Implementers
 | Involved in program operations | Immunization program director, program manager, field staff
 |
| Participants | Served by the program | Physicians, parents, community members |
| Partners | Support/invested in immunization program or target population | Managed care plans, health systems, AAP and AAFP chapters, CBOs |

-        Some Key Stakeholders

## Payoffs for Involving Stakeholders

- Strong input on purpose and design of the evaluation
- “Reality check” on utility and feasibility of evaluation plan and methods.
- Increase credibility of evaluation findings.
- Increase likelihood that evaluation results will be acted upon.

## Involve Stakeholders from the Start!

- Potential stakeholder roles at all steps:
- Input on evaluation design
- Input on data collection methods
- Assist with data collection/analysis
- Outreach to the target population for the evaluation
- Interpretation and use of results

## Case Example: AFIX Stakeholders

- Program staff
- Clinical staff (clinic managers, nurses, physicians)

## Evaluation Step—Description

- Identify and engage stakeholders
- Describe the program component
- Focus the evaluation design
- Gather credible evidence
- Justify conclusions
- Ensure use and share lessons learned

## Seeking Consensus On...

- The activities and outcomes depicted
- The “roadmap”
- Which outcomes = program “success”
- This will facilitate future discussions of:
- How *much* progress on outcomes = program “success”
- Choices of data collection/analysis methods

## Describing the Program:Complete Logic Model

**Activities **

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

## What the program and its staff actually do

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*What the program and its staff actually do*

## Results of activities: Who/what will change?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

_*Results of activities: *_*Who/what will change?*

## Simple Logic Model—AFIX

**Simple Logic Model—AFIX**

**	****	****ACTIVITIES****	****  ****	****                OUTCOMES****	**
**            ****Early              Later****	****	****       Short-Term       Long-Term****	****	****	**

## Case Example: AFIX Logic Model

## Resource “platform” for the program

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Resource “platform” for the program*

*Tangible products of activities*

## AFIX Full Logic Model

## Moderators: Contextual factors that will facilitate or hinder getting our outcomes

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

_*Moderators: *_*Contextual factors that will facilitate or hinder getting our outcomes*

## Evaluation Step—Focusing the Evaluation

- Identify and engage stakeholders
- Describe the program component
- Focus the evaluation design
- Tailor evaluation to program and stakeholder’s needs
- Determine resources and personnel available
- Develop and prioritize evaluation questions
- Gather credible evidence
- Justify conclusions
- Ensure use and share lessons learned

## Evaluation Standards Help Choose Focus

_***“***__***Utility”***_*—* Which stakeholders/others want the evaluation findings? How/for what purpose  will they use the findings?
_***“***__***Feasibility”***_— How reasonable are these questions given: How long program has been in existence; program intensity; data collection logistics/constraints.

## Generating and Prioritizing Evaluation Questions

_***Utility: Questions which:***_
- Are important to stakeholders;
- Are consistent with program goals and objectives
- Will provide information that can be acted upon
_***Feasibility: Questions which:***_
- Can be answered with available resources and within available timeframe
- Are consistent with program maturity and intensity
- Will be supported by program decision-makers.

## Setting Focus: Process and Outcome Evaluation

- Process Evaluation
- Determines if program activities are implemented as intended.
- Tracks *who*, *what*, *when*, and *where *program information.
- Provides feedback loop for program improvement.
- Conducted throughout project life.
- Outcome Evaluation
- Measures program effects; changes in target population’s knowledge, attitudes, self-efficacy, skills, intentions, behaviors; or organizational changes (e.g., policy adoption).
- Linked to process evaluation.
- Provides feedback loop for program improvement

## Case Example: Setting AFIX Focus

**Utility**
_***Purpose/User:***_  Program staff
_***Use:***_ Determine if:
- AFIX feedback is actually implemented by providers.
- If not, determine barriers and challenges to use of feedback
- And, determine how to improve activities and followup to surmount identified barriers and challenges
**Feasibility**
_***Stage/Intensity:***_ Well-established and multi-faceted program.  Should be making these impacts if working
_***Resources:***_ [Assumption] Evaluation questions of interest have some ready data sources that would be credible for those interested in findings

## AFIX Logic Model—Focus

## AFIX—Sample Evaluation Questions

- Proposed focus: both process and outcomes
- Outcome:
- Are AFIX recommendations implemented by practices within 6 months of visit? (short term outcome)
- Process
- Factors that may be related to implementation/non-implementation of AFIX recommendations:
- Are AFIX staff appropriately trained?
- Are practice contacts the right people?
- What is the quality of the feedback? Are practices satisfied with feedback received?

## Where Next...

***Stakeholders***
***Throughout!***

**Where Next...**

**Data Collection Sources and Methods**

**Data Collection Procedures**

**Data Analysis**

**Data Interpretation**

**Shared Results and Lessons Learned**

**Results used to improve program**

**Indicators**

## But...

- These steps more likely to be done well because we did a good job on earlier steps!!!

## Practical Program Evaluation

- Coming! This Fall...

## Fall Training Session

- Reaffirm/reinforce today’s points
- Work through remaining steps:
- Define indicators
- Identify data collection sources and methods
- Consider data analysis and interpretation
- Maximizing use

## Pre-Work for November

- Consider criteria for choosing a program component
- Give some preliminary thought to:
- Potential program components
- Key stakeholders
- Purpose/use of evaluation

## Practical Program Evaluation

- Life Post-Session

## Helpful Publications @ www.cdc.gov/eval

## Helpful Resources

- NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: _[http://www.cdc.gov/eval/whatsnew.htm](http://www.cdc.gov/eval/whatsnew.htm)_
- Logic Model Sites
- Innovation Network:
- 	_[http://www.innonet.org/](http://www.innonet.org/)_
- Harvard Family Research Project: _[http://](http://www.gse.harvard.edu/hfrp/)__[www.gse.harvard.edu/hfrp](http://www.gse.harvard.edu/hfrp/)__[/](http://www.gse.harvard.edu/hfrp/)_
- University of Wisconsin-Extension: _[http://www.uwex.edu/ces/lmcourse/](http://www.uwex.edu/ces/lmcourse/)_
- CDC/DASH:_[http://www.cdc.gov/healthyyouth/evaluation/resources.htm#4](http://www.cdc.gov/healthyyouth/evaluation/resources.htm#What can Program Evaluation do?)_
- CDC/STD: _[http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm](http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm)_
- Texts
- Kellogg Foundation Logic Model Development Guide: _[www.wkkf.org](http://www.wkkf.org/)_
- W.K. Kellogg Foundation Evaluation Resources: _[http://](http://www.wkkf.org/programming/overview.aspx?CID=281)__[www.wkkf.org/programming/overview.aspx?CID](http://www.wkkf.org/programming/overview.aspx?CID=281)__[=281](http://www.wkkf.org/programming/overview.aspx?CID=281)_
- Rogers et al.  Program Theory in Evaluation.  New Directions Series: Jossey-Bass, Fall 2000
- Chen, H.  Theory-Driven Evaluations.  Sage.  1990

## Community Tool Boxhttp://ctb.ku.edu

## This document can be found on the CDC website at:

- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_2nd.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_2nd.ppt)_