# Presentation: 047913

## Process evaluation

- Pascale Wortley, MD, MPH
- Immunization Services Division, NCIRD
- April 17, 2008

## Definitions

- _Process evaluation_: examines whether program activities been implemented as intended
- _Outcome evaluation_: measures program effects in the target population

## In what situations can process evaluation be useful?

- When expected outcomes are not observed, process evaluation can suggest reasons.
- When outcomes are positive, process evaluation may help understand which components are especially important.
- When an activity is newly implemented and it is too early to expect changes in outcomes.
- When an outcome evaluation is not feasible due to resource constraints.

## Examples: Standing orders in Nursing Homes

- Goal: increase nursing home vaccination rates in 14 participating states
- Intervention: QIOs promote implementation of  standing orders in nursing homes
- Outcome: vaccination rates did not increase significantly
- Limited process evaluation conducted, degree of intensity of the intervention is not known
- Conclusion: we do not know what it takes for QIOs to work effectively with NH to promote standing orders.

## Examples: “Gimme 5”

- School-based intervention to increase consumption of fruits, 100% juice, and vegetables
- Curriculum based intervention
- Classroom teaching
- Point-of-purchase education
- Family participation
- Outcome evaluation: modest effect
- Process evaluation:
- Teacher satisfaction with training
- Teacher self-report
- Teacher observation
- Phone interviews of parents
- Observation at point-of-purchase education sessions

## Examples: “Gimme 5”, cont’d

- Teacher satisfaction with training: largely satisfied
- Teacher self-report: reported completing 90% of curriculum tasks
- Teacher observation: about 50% of tasks completed—selectively
- Phone interviews of parents: a substantial proportion did not do curriculum-related homework with their child, about 1/3 did not view any of the 3 videos sent home. Only 10% participated in point-of-purchase educations session
- Observation at point-of-purchase education sessions: corroborated low participation reported by parents
- Conclusion: sub-optimal implementation

## Context

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Assumptions
- Stage of  Development

- Basic Logic Model

**Notes:**
Lots of this will have been decided when you “staked the claim” but worth considering again.
How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions.  In that case, then a need to reconcile where to put the resources.
Key questions include:
Ripple effect or reasonableness:  This one should have been answered in “staking the claim”.  The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact.  Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
Accountability:  This is what the major stakeholders are expecting of you.  As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction.   Still, in designing an evaluation, the information of most importance to the stakeholder is a major consideration.
Measurability:  As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.

## Context

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Assumptions
- Stage of  Development

*What the program and its staff actually do*

*Results of activities: who, what will change*

## Context

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Assumptions
- Stage of  Development

- Sphere of control

- Sphere of influence

## Context

**Inputs**

**Outputs **

**Intermediate Effects/ Outcomes**

**Short-term Effects/ Outcomes **

**Long-term Effects/**
**Outcomes**

- Assumptions
- Stage of  Development

**Outcome evaluation**

**Process **
**evaluation**

## What are we trying to learn through process evaluation?

- Do we have the right mix of activities?
- Are we reaching the intended targets?
- Are the right people involved as partners, participants, and providers?
- Do the staff/volunteers have the necessary skills?
- How well do our activities meet with our priorities

## Example 1: Child immunization outreach campaign—pocket of need

- Peer-to peer counselors identified through church or CBO have been trained to approach mothers of young children, discuss immunization, provide materials, refer to clinics
- Media campaign: radio, newspaper, flyers, posters

## Example 1: Child immunization outreach campaign—pocket of need

- Outputs:
- Number of counselors trained
- Number of contacts made with target population
- Number of materials created, handed out
- Number of PSAs aired, and time
- Number of newspaper ads

## Example 1: Child immunization outreach campaign—pocket of need

- II. Process evaluation questions
- Were counselors well-trained, was training standardized?
- Did peer counselor knowledge increase with training?
- Do mothers find the message appropriate?
- What percent of target population approached?
- Did target population hear ads, read newspaper articles, see posters?
- Were there barriers to getting to the clinic?
- How well do counselors deliver message?

## Example 1: Child immunization outreach campaign—pocket of need

- III.  Outcome evaluation questions
- Did mothers knowledge or attitudes change?
- Did mothers bring children to clinics?
- Did vaccination rate increase?

## Example 2: AFIX

- Outputs
- Number of assessments conducted
- Number of feedback sessions performed
- Number of staff trainings, of staff trained

## Example 2: AFIX

- Process evaluation questions
- How well are AFIX staff trained?
- Is training standardized?
- Do AFIX staff carry out interactions as per training?
- Did AFIX staff make recommendations to improve vaccination practices?
- Are AFIX staff interacting with the right people?
- Are provider staff satisfied with interactions?
- What key messages did they take away from feedback session?

## Example 2: AFIX

- Outcome evaluation questions
- Were recommendations implemented?
- Did chart organization improve?
- Did missed opportunities decrease?
- Did vaccination rates increase?

## Example 3: influenza vaccination clinic

- A county health department is holding new influenza vaccination clinics in a predominantly African American  neighborhood. Information to advertise the clinics has been disseminated through a local paper, flyers and posters in community centers, and radio spots.

## Example 3: influenza vaccination clinic

- Outputs
- Number of clinics held
- Number of persons attending
- Number of radio announcements
- Number of flyers/posters posted

## Example 3: influenza vaccination clinic

- Process evaluation questions
- Percent of clinic participants 50+ or chronic disease?
- Percent of clinic participants who don’t usually get vaccinated?
- What percent of target population is aware of clinics?
- Reasons for not attending clinics?

## Example 3: influenza vaccination clinic

- Outcome evaluation questions
- Has vaccination rate increased in neighborhood
*(Note: may be difficult to measure, but can be extrapolated from process evaluation information on number of “new vaccinees”)*

## Some useful concepts

- Reach
- Quality of implementation
- Appropriateness
- Satisfaction
- Barriers

## Reach

- Degree to which intended audience participates in intervention
- Percent of target population that heard messages
- Percent of persons attending influenza vaccination clinic that do not usually get vaccinated
- Percent of mothers contacted by peer-to-peer counselors

## Quality of implementation

- Was activity implemented properly, according to standards or protocol
- Was AFIX feedback session conducted as per guidelines?
- Are AFIX staff interacting with right people?
- Did peer counselors interaction with mothers follow training or protocol?
- Is training of staff for a given activity standardized?
- Was training curriculum delivered in its entirety?

## Appropriateness

- Interventions or messages that are delivered may only be effective if judged appropriate by target population, or if designed in manner to achieve objective
- Did messages “speak” to target audience?
*   **Note: short term outcomes also related to appropriateness, e.g.*
*	**Did knowledge or skills increase as a result of training?*
*	**Was information provided in training subsequently used?*

## Satisfaction

- The extent to which participants are satisfied with training or interaction may influence subsequent behavior
- Provider satisfaction with AFIX session
- Peer counselor satisfaction with training

## Barriers

- This attempts to understand why something didn’t happen, and may identify key environmental variables
- Reasons mothers didn’t bring their children to the clinic (transportation, clinic schedule, other?)
- Reasons providers didn’t implement AFIX recommendations (resources?)
- Reasons seniors didn’t come for influenza vaccination clinic (unaware of clinic, concern about vaccine, no perceived need?)

## Prioritizing evaluation questions

- Is the question:
- Important to your program staff and stakeholders?
- Does it reflect key goals and objectives of your program?
- Does it reflect key elements of your logic model?
- Will it provide information you can act upon to make program improvements?
- Can it be answered using available program resources?
- Are there any available data sources?

## Process evaluation: Methods

- Quantitative
- Qualitative
- Combination

## Qualitative methods

- Case studies
- Structured or semi-structured interviews
- Focus groups
- Direct observation
- Reviews of program meeting minutes, progress reports

## Quantitative methods

- Surveys
- Information collected from program participants
- IIS

## Involving stakeholders

- Gain broader perspective, avoid blind spots, try to ensure utilization of results
- Key stakeholders:
- Those served or affected by activity
- Those involved in program operations
- Those in a position to make decisions about the activity
- For a manageable process, the list of stakeholders must be narrowed to primary intended users

## Working with an external consultant

- Scope of work can range from:
- Conducting specific tasks
- Designing and conducting an evaluation
- External evaluator _must_ work closely with you and stakeholders to ensure result is useful

## Resources

- _[www.cdc.gov/vaccines/programs/progeval](http://www.cdc.gov/vaccines/programs/progeval)_
- 	Immunization evaluation resources
- Contact us
- Email via website
- Via your public health advisor
**This document can be found on the CDC website at:**
- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-course-3rd.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-course-3rd.ppt)_** **

## This document

- This document can be found on the CDC website at:
- _[http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-course-3rd.ppt](http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-course-3rd.ppt)_