# Presentation: 519055

## Practical Program Evaluation—Using CDC’s Evaluation Framework

**By:**
**Thomas J. Chapel, MA, MBA**
**Office of the Director**
**Office of Workforce and Career Development**
**Centers for Disease Control and Prevention**

## Today...

- Present CDC Evaluation Framework steps and standards
- Show central role of “program description” and “evaluation focus” steps
- Discuss simple logic model(s) for an immunization program
- Show how logic model helps with key evaluation tasks
- Set-up work for Session 2 in the Fall

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## Defining Evaluation	

_***Evaluation***_ is  the systematic investigation of the merit, worth, or significance of any *“object”*
*	**	**	**	**	**	**Michael Scriven*
_***Program***_** **is any organized public health action/activity implemented to achieve some result

## Integrating Processes to Achieve Continuous Quality Improvement

- Continuous Quality Improvement (CQI) cycle.
**Planning**—*What* actions will best reach our goals and objectives.
**Performance measurement**— How are we doing?
**Evaluation**—W*hy* are we doing well or poorly?

- What do we do?

- Why are we doing well or poorly?

- How are we doing?

- How do we do it?

## Framework forProgram Evaluation

**Notes:**
And so we have a complete picture of the CDC Eval Framework.

A series of six steps which cycle back.

Four clusters of standards that act as a template for each step to guide each step and measure result.

## Underlying Logic of Steps

_***No eval is good unless***_... results are _***used***_ to make a difference
_***No results are used unless***_... a _***market***_ has been created prior to creating the product
_***No market is created unless***_.... the eval is _***well-focused***_, including most relevant and useful questions
***And...***

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## Establishing the Best Focus Means...

_***Framework Step 1:***_  Identifying who cares about our program besides us?  Do they define program and “success” as we do?”
_***Framework Step 2:***_ What are milestones and markers on the roadmap to my main PH outcomes?

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## The Four Standards

- No one “right” evaluation. Instead, best choice at each step is options that maximize:
_***Utility***_:  Who needs the info from this evaluation and what info do they need?
_***Feasibility***_:   How much money, time, and effort can we put into this?
_***Propriety:***_  Who needs to be involved in the evaluation to be ethical?
_***Accuracy:***_  What design will lead to accurate information?

## Practical Program Evaluation

- Constructing Simple Logic Models

## You Don’t Ever Need a Logic Model, BUT, You Always Need a Program Description

- Don’t jump into planning or eval without clarity on:
- The big _***“need”***_ your program is to address
- The key _***target group(s)***_ who need to take action
- The kinds of actions they need to take (your intended _***outcomes***_ or objectives)
_***Activities***_ needed to meet those outcomes
- “Causal” _***relationships***_ between activities and outcomes

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Logic Models and Program Description

***Logic Models :  ****Graphic depictions of the *_***relationship***_* between your program’s activities and its *_***intended ***_*effects*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Linking Planning, Evaluation and Performance Measurement

- Goals

- Objectives

- Actions/
- Tactics

- Activities

- ST or MT
- Outcomes

- LT
- Outcomes
- or Impacts

- Process Measures
- Progress Measures
- Impl. Measures

- Outcome Measures
- Impact Measures
- Key Performance Indicators
- Success Factors

- Plan

- Eval

- PM

## Step 2: Describing the Program:Complete Logic Model

**Activities **

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## What the program and its staff actually do

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*What the program and its staff actually do*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Results of activities: Who/what will change?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

_*Results of activities: *_*Who/what will change?*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Resource “platform” for the program

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Resource “platform” for the program*

*Tangible products of activities*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Moderators: Contextual factors that will facilitate or hinder getting our outcomes

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

_*Moderators: *_*Contextual factors that will facilitate or hinder getting our outcomes*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Contextual Factors

_***P***_olitical
_***E***_conomic
_***S***_ocial
_***T***_echnological

**Notes:**
To reiterate key points about logic models:

They are graphic representations of INTENDED relationships among activities and outcomes of a program.  Also, of the interrelationships of these activities and the interrelationships of these outcomes.

They are a disciplined way to produce a “map” which can start the conversation, but will not in and of themselves provide answers.

They are a means and not an end.  The “end”–clarifying what a program is and what it intends to do can be accomplished without logic models, and conversely, use of logic models does not guarantee a good, or accurate result.

## Practical Program Evaluation

- Logic Model Case Illustration

## Childhood Lead Poisoning Prevention

- Lead poisoning is widespread environmental hazard facing young children, especially in older inner-city areas.
- Main sources of lead poisoning in children are paint and dust in older homes with lead-based paint.
- Effects ameliorated through a combination of medical and nutritional interventions.  But, ultimately, source must be contained/ eliminated through renovation or removal of the lead-based paint by professionals, although some reduction possible through intensive housekeeping practices.
- Programs receiving CDC money aim to: screen children, identify those with elevated blood lead levels (EBLL), assess  environments for lead sources, and case manage both their medical treatment and the correction of their environment.
- The grant money ***cannot*** directly pay for medical care or for renovation of homes.

**Notes:**
Logic modeling is helpful for 

Program description: What does my current program consist of and what am I trying to accomplish.

Program enhancement: Is this what I really want to achieve with my program and is there the proper mix of activities to achieve my outcomes?

Framing a new effort.  Here there is no program, only a problem to be addressed.  What are all the forces that need to be brought to bear to solve this problem.

## Constructing Logic Models:  List Activities and Outcomes by....

- Examining program descriptions, MISSIONS, VISIONS, PLANS, ETC and extracting these from the narrative, _***OR***_
- Starting with outcomes, ask “how to” in order to generate the activities which produce them, _***OR***_
- Starting with activities, ask “so what” in order to generate the outcomes that are expected to result

**Notes:**
Program description is the simplest use of logic models in that it accepts the program as given and seeks only to accurately depict it graphically.
The steps are simple:
List the activities and the effects (outcomes).  You can do either first.
Arrange each in a time sequence.  Which effects come first, and then what, and then what.  This is sometimes moot for the activities, all of which may happen concurrently, but not always. For example a prevention program may have a sequence of activities that include outreach, screening, referral, and treatment–all activities that serve as jumping off point to achieve the first sets of effects.
Draw arrows to show which activity(ies) are directed at which effects.
Review and refine as necessary. Often, students new to logic modeling get discouraged by how “messy” the model is.  This is actually a good thing and shows that the program is more complex than originally imagined.  The relation of activities and effects is not often clear, and many versions of the model may be necessary before you produce a clean and attractive presentation model.

## Then...Do Some Sequencing...

- Divide the ***activities*** into 2 or more columns based on their ***logical***_*** ***_sequence.  _*Which activities have to occur before other activities can occur?*_
- Do same with the ***outcomes***. _*Which outcomes have to occur before other outcomes can occur?*_

**Notes:**
Program description is the simplest use of logic models in that it accepts the program as given and seeks only to accurately depict it graphically.
The steps are simple:
List the activities and the effects (outcomes).  You can do either first.
Arrange each in a time sequence.  Which effects come first, and then what, and then what.  This is sometimes moot for the activities, all of which may happen concurrently, but not always. For example a prevention program may have a sequence of activities that include outreach, screening, referral, and treatment–all activities that serve as jumping off point to achieve the first sets of effects.
Draw arrows to show which activity(ies) are directed at which effects.
Review and refine as necessary. Often, students new to logic modeling get discouraged by how “messy” the model is.  This is actually a good thing and shows that the program is more complex than originally imagined.  The relation of activities and effects is not often clear, and many versions of the model may be necessary before you produce a clean and attractive presentation model.

## Listing Activities and Outcomes: Lead Poisoning

- Activities
- Outreach
- Screening
- Case management
***Referral*** for medical tx
- Identification of kids with elevated  lead (EBLL)
- Environmental assessment
***Referral*** for env clean-up
- Family training

- Effects/Outcomes
- Lead source identified
_***Families***_ adopt in-home techniques
_***Providers***_ treats EBLL kids
_***Housing Authority ***_eliminates lead source
*EBLL reduced*
*Developmental “slide” stopped*
*Q of L improved*

## Global Logic Model: Childhood Lead Poisoning Program

**Global Logic Model: Childhood Lead Poisoning Program**

**Early Outcomes**

**Later Outcomes**

**Later Activities**

**Early Activities**

***If we do...***
- Outreach
- Screening
- ID of elevated kids

***And we do...***
- Case mgmt of EBLL kids
- Refer EBLL kids for medical treatment
- Train family in in-home techniques
- Assess environment of EBLL child
- Refer  environment for clean-up

***Then....***
- EBLL kids get medical treatment
- Family performs in-home techniques
- Lead source identified
- Environment gets cleaned up
- Lead source removed

***And then...***
- EBLL reduced
- Develop’l slide stopped
- Quality of life improves

## Sometimes, Less is More...

- A simple table-format logic model ***may*** be all you need for many audiences
- BUT, for ***comprehensive description***, may need to add inputs and outputs

## Lead Poisoning: Sample Inputs and Outputs

- _Inputs Needed for Activities_
- Funds
- Trained staff
- Relationships with orgs for med tx and env clean-up
- Legal authority to screen

- _Outputs of Activities_
- Pool (#) of eligible kids
- Pool (#) of screened kids
- Referrals (#) to medical treatment
- Pool (#) of “leaded” homes
- Referrals (#) for clean-up

## Global Logic Model: Childhood Lead Poisoning Program

**Global Logic Model: Childhood Lead Poisoning Program**

**Early Outcomes—**

**Later Outcomes**

***Funds***
***Trained staff***
***R’ships with orgs for med tx and clean up***
***Legal authority***

**Outputs**

**Later Activities**

- Inputs

- Early Activities

- Outreach
- Screening
- ID of elevated kids

- Do  case mgmt
- Refer for medical treatment
- Train family in in-home techniques
- Assess environ’t
- Refer house for clean-up

***Pool (#) of eligible kids***
***Pool (#) of screened kids***
***Referrals (#) to medical treatment***
***Pool (#) of “leaded” homes***
***Referrals (#) for clean-up***

- EBLL kids get medical treatment
- Family performs in-home techniques
- Lead source identified
- Environ cleaned up
- Lead source removed

- EBLL reduced
- Develop’l slide stopped
- Quality of life improves

## For Planning and Evaluation “Causal” Arrows Can Help

_***Not***_  a different logic model, but same elements in different format
- Arrows can go from:
***Activities to other activities:  Which*** activities feed ***which*** other activities?
***Activities to outcomes***:  ***Which*** activities produce ***which*** intended outcomes?
***Early effects/outcomes to later ones***: ***Which*** early outcomes produce ***which*** later outcomes

## Lead Poisoning: “Causal” Roadmap

- Screening

- Do Environment
- Assessment

- ID Source and
- Refer for clean-up

- MedicalManagement

- Lead Source
-  Removed

- ReducingEBLLs

- ImprovedDevelopmentand Intelligence

- MoreProductiveand/or QualityLives

- Family performs
- in-home techniques

- ID kids with
- EBLL

- Outreach

- Train
- Families

- Refer for
- Medical Treatment

- Case
- Management

_*Activities*_

_***______________________Outcomes__________________________***_

## Applying Teaching Points to Immunization Example

## Immunization Example—Activities and Outcomes

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Example—Table Format

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Example—”Roadmap”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Can Emphasize Any Part of Program for Planning/Eval

- Where am I spending the most?
- Where am I concerned the most?
- Where are my big opportunities/new areas?
- Where are my big successes?

## Example—Perinatal Hep B “Zoom-In”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Note!

- Logic Models make the program theory _***clear***_, not _***true***_!

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Logic Models Take Time...So Be Sure to Use Them

- Not worth it as “ends in themselves”
- But can pay off big in evaluation:
- Clarity with stakeholders
- Setting evaluation focus

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Which S’holders Matter Most?

_*Who is*_
_*Affected*_ by the program?
_*Involved*_ in program operations?
- Intended _*users*_ of evaluation findings?

_***Of these, who do we most need to:***_
- Enhance _*credibility?*_
_*Implement*_ program changes?
_*Advocate*_ for changes?
_*Fund, authorize*_, expand program?

## Lead Poisoning: “Causal” Roadmap

- Screening

- Do Environment
- Assessment

- ID Source and
- Refer for clean-up

- MedicalManagement

- Lead Source
-  Removed

- ReducingEBLLs

- ImprovedDevelopmentand Intelligence

- MoreProductiveand/or QualityLives

- Family performs
- in-home techniques

- ID kids with
- EBLL

- Outreach

- Train
- Families

- Refer for
- Medical Treatment

- Case
- Management

_*Activities*_

_***______________________Outcomes__________________________***_

## Using the Logic Model with Stakeholders

- Do they agree/disagree with:
- The activities and outcomes depicted?
- The “roadmap”?
- Which outcomes = program “success”?
- How *much* progress on outcomes = program “success”?
- Choices of data collection/analysis methods?

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Applying Teaching Points to Immunization Example

## Example—”Roadmap”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Example—Perinatal Hep B “Zoom-In”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Evaluation Can Be About Anything

- Evaluation can focus on any/all parts of the logic model
- Evaluation questions can pertain to
- Boxes---did this component occur as expected
- Arrows---what was the relationship between components

**Notes:**
Need
Looks at the adequacy or relevance of a particular program or intervention
Identified the particular unmet needs of a target population
Identify the types of services that are likely to be most attractive and effective for the target population
What is the health problem and its consequences?
What is the overall size of problem and in various segments of population
What are the determinants of the problem?
Who are the target groups?
Process
Progress of a program/intervention
Effort and resources used
Examines overall implementation
Compares projected/anticipated activities with actual ones
Outcome
Impact or effect of a program/intervention
Did the program achieve it’s stated goals?
Attempts to establish a cause and effect relationship
Efficiency
Deals with the issue of cost
Are funds spent for the intended purpose?
Does the program achieve its success at a reasonable cost?; A criteria used to compare programs/interventions

## Did we get the inputs we needed/were promised?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Did we get the inputs we needed/were promised?*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Were activities and outputs implemented as intended? How much? Who received?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Were activities and outputs implemented as intended? How much? Who received?*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Which outcomes occurred? How much outcome occurred

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Which outcomes occurred? How much outcome occurred*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Did we account for moderators? Is there evidence of contextual barriers or facilitators?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Did we account for moderators? Is there evidence of contextual barriers or facilitators?*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Did outcomes occur because of our activities and outputs?

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Context
- Assumptions
- Stage of  Development

*Did outcomes occur *_***because ***_*of our activities and outputs?*

*(How) was implementation quality *_***related to ***_*inputs?*

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Step 3.  Some Typical Evaluation Emphases

- Implementation (Process)
- Is program in place as intended?
- Effectiveness (Outcome)
- Is program achieving its intended short-, mid, and/or long-term effects/outcomes?
- Efficiency
- How much “product” is produced for given level of inputs/resources?
- Causal Attribution
-  Is progress on outcomes due to your program?

**Notes:**
Need
Looks at the adequacy or relevance of a particular program or intervention
Identified the particular unmet needs of a target population
Identify the types of services that are likely to be most attractive and effective for the target population
What is the health problem and its consequences?
What is the overall size of problem and in various segments of population
What are the determinants of the problem?
Who are the target groups?
Process
Progress of a program/intervention
Effort and resources used
Examines overall implementation
Compares projected/anticipated activities with actual ones
Outcome
Impact or effect of a program/intervention
Did the program achieve it’s stated goals?
Attempts to establish a cause and effect relationship
Efficiency
Deals with the issue of cost
Are funds spent for the intended purpose?
Does the program achieve its success at a reasonable cost?; A criteria used to compare programs/interventions

## Setting Focus: Some Rules

- Based on “utility” standard:
_***Purpose:***_  Toward what end is the evaluation being conducted?
_***User:***_  Who wants the info and what are they interested in?
_***Use:***_ How will they use the info?

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## (Some) Potential Purposes

- Show accountability
- Test program implementation
- “Continuous” program improvement
- Increase the knowledge base
- Other...
- Other...

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## Deciding on the “Right” Focus: “Harvesting” Step 1...

_***Needs of Key S’holders from Step 1:***_
- What are key s’holders most interested in?
- Must I include this in my evaluation focus?

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

##  “Reality Checking” the Focus

- Based on “feasibility” standard:
_***Stage of Development:***_ How long has the program been in existence?
_***Program Intensity:***_** **How intense is the program?  How much impact is reasonable to expect?
_***Resources:***_  How much time, money, expertise are available?

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Some Evaluation Scenarios

_***Scenario I:***_  At Year 1, other communities want to adopt your model but want to know “what are they in for”

## Scenario 1:

_***Purpose:***_  Examine program implementation
_***User:***_  The “other community”
_***Use:***_  To make a determination, based on your experience, whether they want to adopt this project or not

## Lead Poisoning: “Causal” Roadmap

- Screening

- Do Environment
- Assessment

- ID Source and
- Refer for clean-up

- MedicalManagement

- Lead Source
-  Removed

- ReducingEBLLs

- ImprovedDevelopmentand Intelligence

- MoreProductiveand/or QualityLives

- Family performs
- in-home techniques

- ID kids with
- EBLL

- Outreach

- Train
- Families

- Refer for
- Medical Treatment

- Case
- Management

_*Activities*_

_***______________________Outcomes__________________________***_

## Some Evaluation Scenarios

_***Scenario II:***_  At Year 5, declining state revenues mean you need to justify to legislators the importance of your efforts so as to continue funds.

## Scenario 2:

_***Purpose:***_  Determine program impact
_***User:***_  Your org and/or the legislators
_***Use:***_
_*You*_ want to muster evidence to prove to legislators you are effective enough to warrant funding, or
_*Legislators *_want you to show evidence that proves sufficient effectiveness to warrant funding

## Lead Poisoning: “Causal” Roadmap

- Screening

- Do Environment
- Assessment

- ID Source and
- Refer for clean-up

- MedicalManagement

- Lead Source
-  Removed

- ReducingEBLLs

- ImprovedDevelopmentand Intelligence

- MoreProductiveand/or QualityLives

- Family performs
- in-home techniques

- ID kids with
- EBLL

- Outreach

- Train
- Families

- Refer for
- Medical Treatment

- Case
- Management

_*Activities*_

_***______________________Outcomes__________________________***_

## Immunization Example

- Think about some “pressures” to evaluate
- Where is the pressure coming from?
- Who will use the data?
- What will they use it for?
- Thinking about that: What part(s) of the model most need to be part of the evaluation?

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## Example—”Roadmap”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Example—Perinatal Hep B “Zoom-In”

**Notes:**
Describe program
Develop clear and succinct description of program
Clarify program’s purpose, activities, and capacity to meet goals
Should include:
Need for program
Expected effects
Proposed activities
Resources available to conduct program
Program’s stage of development
Social and political context in which program implemented
Necessary for two reasons:
Ensure that stakeholders share same level of understanding about program’s components, implementation and intended effects
Foster strategic thinking about program

## Practical Program Evaluation

- Coming! This Fall...

## Fall Training Session

- Reaffirm/reinforce today’s points
- Work some cases
- Help you:
- Use simple logic model
- Choose an appropriate focus
- Construct evaluation questions/indicators
- Give thought to data collection

**Notes:**
Self-explanatory
We’ll start with a quick definition
Read definition
Note that it is “systematic” which means disciplined and objective
Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation
Define PROGRAM broadly as well

## In Short...

## Upfront Small Investment...

- Clarified relationship of activities and outcomes
- Ensured clarity and consensus with stakeholders
- Helped define the right focus for my evaluation
- Clarified vision, mission, goals, objectives, and their interconnection
- Helped me clarify my “critical path”
- Help me cut to the “heart” of my program and...
- How best to get there

**Notes:**
Need
Looks at the adequacy or relevance of a particular program or intervention
Identified the particular unmet needs of a target population
Identify the types of services that are likely to be most attractive and effective for the target population
What is the health problem and its consequences?
What is the overall size of problem and in various segments of population
What are the determinants of the problem?
Who are the target groups?
Process
Progress of a program/intervention
Effort and resources used
Examines overall implementation
Compares projected/anticipated activities with actual ones
Outcome
Impact or effect of a program/intervention
Did the program achieve it’s stated goals?
Attempts to establish a cause and effect relationship
Efficiency
Deals with the issue of cost
Are funds spent for the intended purpose?
Does the program achieve its success at a reasonable cost?; A criteria used to compare programs/interventions

## Practical Program Evaluation

- Life Post-Session

## Helpful Publications @ www.cdc.gov/eval

## Helpful Resources

- NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: _[http://www.cdc.gov/eval/whatsnew.htm](http://www.cdc.gov/eval/whatsnew.htm)_
- Logic Model Sites
- Innovation Network:
- 	_[http://](http://www.innonet.org/)__[www.innonet.org](http://www.innonet.org/)__[/](http://www.innonet.org/)_
- W.K. Kellogg Foundation Evaluation Resources: _[http://www.wkkf.org/programming/overview.aspx?CID=281](http://www.wkkf.org/programming/overview.aspx?CID=281)_
- University of Wisconsin-Extension: _[http://www.uwex.edu/ces/lmcourse/](http://www.uwex.edu/ces/lmcourse/)_
- Texts
- Rogers et al.  Program Theory in Evaluation.  New Directions Series: Jossey-Bass, Fall 2000
- Chen, H.  Theory-Driven Evaluations.  Sage.  1990

## Community Tool Boxhttp://ctb.ku.edu

## This document can be found on the CDC website at:

- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course.ppt)__ _