Program Evaluation Doesn’t Have to be Scary: 

Increase Buy-In with Message Framing

Thursday, March 22, 2018
HOSTED BY
PRESENTERS

Xan Young, MPH

Senior TA Consultant

American Institutes for Research

Michael Baran, PhD

Principal Researcher

American Institutes for Research
• Program evaluation basics

• Understanding culture’s role in strategic message framing

• Barriers to communicating about program evaluation

• Framing strategies to increase buy-in

AGENDA
PROGRAM EVALUATION BASICS
What first comes to mind when you think about program evaluation?

(Write your answer in the “Submit Questions Here” box.)

QUESTION
A systematic study using research methods to collect 

and analyze data to assess how well a program is 

working and why 

DEFINITION OF PROGRAM EVALUATION

Source: Government Accountability Office. (2012). Designing Evaluations: 2012 Revision. (GAO Publication No. 12-208G). 

Washington, D.C.: U.S. Government Printing Office.
Has the program attained its implementation objectives?

• Are you implementing the services or training that you initially planned to implement?

• Are you reaching the intended population?

• Are you reaching the intended number of participants?

• Are you developing the intended collaborative relationships?

Has the program attained its outcome objectives?

• Are participants exhibiting the expected changes in knowledge, attitudes, behaviors,

or awareness?

• Can these changes be attributed to the program?

TYPES OF QUESTIONS EVALUATION CAN ANSWER

Source: Administration for Children and Families. (2010). The Program Manager’s Guide to Evaluation: Second Edition. 

Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/program_managers_guide_to_eval2010.pdf
• Implementation

Were your program’s activities put into place as originally intended?

• Effectiveness

Is your program achieving the goals and objectives it was intended to accomplish?

• Attribution

Can progress on goals and objectives be shown to be related to your program, as opposed to

other things that are going on at the same time?

• Efficiency

Are your program’s activities being produced with an appropriate use of resources such as

budget and staff time?

• Cost-Effectiveness

Does the value or benefit of achieving your program’s goals and objectives exceed the cost of

producing them?

TYPES OF QUESTIONS EVALUATION CAN ANSWER

Source: Centers for Disease Control and Prevention. (2011). Introduction to Program Evaluation for Public Health Programs: 

A Self-Study Guide. Retrieved from https://www.cdc.gov/eval/guide/cdcevalmanual.pdf
LOGIC MODEL

• A logic model is a useful tool for program planning and evaluation.

• It visually describes how a program should work, presents the planned

activities for the program, describes how activities will be documented, and

focuses on anticipated outcomes.

• It often uses diagrams or pictures that illustrate the logical relationship

among key program elements through a sequence of "if-then" statements.

Source: Understanding Logic Models. Retrieved from https://www.ojjdp.gov/grantees/pm/logic_models_understanding.html
KEY ELEMENTS OF A LOGIC MODEL

Source: Retrieved from https://www.ojjdp.gov/grantees/pm/Logic_models_safeplay1.pdf
SAMPLE LOGIC MODEL

Source: Adapted from https://www.cdc.gov/std/Program/pupestd/Components%20of%20a%20Logic%20Model.pdf

Goal: To reduce the prevalence of STDs among men in City X

Process objective: By (month/year), staff will conduct three professional 

development workshops on STD screening recommendations for clinical 

providers caring for men in City X.

Inputs Activities Outputs Short-Term 

Outcome

Intermediate 

Outcome

Long-Term 

Outcome

Funding

Staff time

Screening 

guidelines

Workshop 

curriculum

Meeting space

Conduct 

workshop on 

STD guidelines 

for clinical 

providers caring 

for men

Workshop on 

STD guidelines 

for clinical 

providers caring 

for men

Increased 

knowledge of 

STD guidelines 

among clinical 

providers 

attending the 

workshop

Increased 

routine 

screening for 

STDs in men by 

clinical 

providers 

attending the 

workshop

Decreased 

prevalence of 

STDs among 

men
SAMPLE LOGIC MODEL

A Safe Place for Kids to Play

A logic model developed by Courtney and Bailey Peters when they were 9 and 11 years old

Source: Work Group for Community Health and Development. (2013). Community tool box. Retrieved from https://ctb.ku.edu/en/

table-of-contents/overview/models-for-community-health-and-development/logic-model-development/example
CULTURE’S ROLE IN STRATEGIC MESSAGE FRAMING
COMMON REACTIONS TO EVALUATION

Source: Centers for Disease Control and Prevention. EvaluACTION: Putting Evaluation to Work. Retrieved from 

https://vetoviolence.cdc.gov/apps/evaluaction
WE NEED MORE EFFECTIVE MESSAGES!

• We need to better understand what’s underlying the common reactions

(cultural models).

• We need to be strategic about triggering or avoiding those cultural models

to increase buy-in.

Source: Work Group for Community Health and Development. (2013). Community tool box. Retrieved from 

http://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development/logic-modeldevelopment/main 
MISSED MESSAGE EXAMPLE 1

• Expert Message:

“Prison only makes a youth offender more likely to commit future crime. 

Prison doesn’t do any good for youth who engaged in minor crimes.”

• General response:

“Yes, prison is too cushy, like a holiday camp! We need to take away 

prisoner activities and make prison more harsh so that they never want 

to come back.”

Source: Baran, M., et al. (2015). “Like a Holiday Camp” Mapping the gaps on criminal justice reform in England and Wales. 

Washington, DC: The FrameWorks Institute. 
MISSED MESSAGE EXAMPLE 2

• Expert Message: 

“Stress is harmful for young children’s cognitive development.”

• General response: 

“I experienced stress and it made me stronger.”

Source: Talking Early Child Development and Exploring the Consequences of Frame Choices: A FrameWorks Message Memo. 

Washington, DC: The FrameWorks Institute. 
EXPLANATIONS FOR MISSED MESSAGES

• Message was too complicated.

• Message was poorly distributed.

• Message was not memorable or attention grabbing.

• Message did not trigger strong emotion.

• Message did not contain facts (or contrasted facts with myths).

• People misunderstood the message.

Another possibility...

• Culture gets in the way!
CULTURAL MODELS IN MIND

Source: Shore, B. (1996). Culture in mind: cognition, culture, and the problem of meaning. London: Oxford University Press. 

• From 0-3, the brain develops 700 synaptic connections per second.

• Cultural models are implicit patterns of thinking widely shared in a given 

population. 

• They help us organize the vast, complicated stimuli of life into a shared 

common sense.
HOW DO CULTURAL MODELS AFFECT 

COMMUNICATION?

• The way that messages are framed (values, metaphors, key words, 

messenger, tone, etc.) cues cultural models.

• Those cultural models then determine how a person understands the 

message. 

• Unlike what a “code” model would predict, miscommunication is the norm.

Source: Shonkoff, J., & Bales, S. (2011). Science does not speak for itself: Translating child development research for the 

public and its policymakers. Child Development, 82(1), 17–32.
HOW DO WE GATHER EVIDENCE FOR A 

COMMUNICATIONS STRATEGY?

• Conducting in-depth 

interviews that ask the most 

basic questions

• Analyzing implicit “common 

sense” understandings 

• Mapping the cultural models 

in the space

Phase 1: Formative Research Phase 2: Message Testing

• Developing messages to 

target productive 

understandings and avoid 

problematic ones

• Testing hypotheses with a 

mix of qualitative and 

quantitative methods
REFRAMED MESSAGE EXAMPLE 1

• Expert Message: 

“Prison only makes a youth offender more likely to commit future crime. Prison 

doesn’t do any good for youth who engaged in minor crimes.”

• General response: 

“Yes, prison is too cushy, like a holiday camp! We need to take away prisoner 

activities and make prison more harsh so that they never want to come back.”

Source: O’Neil., et al. (2016). New Narratives: Changing the frame on crime and justice. Washington, DC: The FrameWorks Institute. 
REFRAMED MESSAGE EXAMPLE 1

• Cultural models: 

retribution, rational actor

• Reframe with: 

“Prisons sweep young people into a powerful stream of crime from which it is 

difficult to escape.”

Source: O’Neil., et al. (2016). New Narratives: Changing the frame on crime and justice. Washington, DC: The FrameWorks Institute. 

• Expert Message: 

“Prison only makes a youth offender more likely to commit future crime. Prison 

doesn’t do any good for youth who engaged in minor crimes.”

• General response: 

“Yes, prison is too cushy, like a holiday camp! We need to take away prisoner 

activities and make prison more harsh so that they never want to come back.”
REFRAMED MESSAGE EXAMPLE 2

• Expert Message: 

“Stress is harmful for young children’s cognitive development.”

• General response: 

“I experienced stress and it made me stronger.”

Source: Talking early child development and exploring the consequences of frame choices: A FrameWorks message memo. 

Washington, DC: The FrameWorks Institute. 
MISSED MESSAGES EXAMPLE 2

• Cultural models: 

individualism, resilience = willpower

• Reframe with:

A taxonomy of normal stress, tolerable stress, and toxic stress.

Source: Talking early child development and exploring the consequences of frame choices: A FrameWorks message memo. 

Washington, DC: The FrameWorks Institute. 

• Expert Message: 

“Stress is harmful for young children’s cognitive development.”

• General response: 

“I experienced stress and it made me stronger.”
BARRIERS TO COMMUNICATING ABOUT 

PROGRAM EVALUATION
CULTURAL MODEL: JUST DO IT

• Lack of understanding about implementation as a science or practice.

• Common sense tells you it will work, so why evaluate?

• If it worked somewhere, it will work here.

• Outcomes are straightforward.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic 

report. Washington, DC: The FrameWorks Institute. 
CULTURAL MODEL: EVALUATION STARTS AFTER 

IMPLEMENTATION IS COMPLETED

• Evaluation is just for the end of a project – to show if it “worked.”

• Obscures that evaluation throughout can show you WHY implementation is 

working or not, and allow you to adjust as you go.

• Makes it hard to understand planning – how implementation and 

evaluation planning have to happen simultaneously.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
CULTURAL MODEL: SHORT-TERM THINKING

• Evaluation is to measure what happened at the end of this single program.

• If the evaluation doesn’t capture why the program worked or didn’t work, it 

misses the chance to improve not only this program, but future programs 

and the field at large.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
CULTURAL MODEL: MORE = BETTER

• Delivering more services and reaching more people can be valued higher 

and can be more salient than the quality of the program or the outcomes.

• As such, people think that evaluation takes away staff, resources, and time 

from what really matters.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
CULTURAL MODEL: OVERWHELMING COMPLEXITY

• Implementers may think, “how do you evaluate a program when you 

understand the overwhelming complexity of implementing it in the real 

world?”

• Implementers struggle to balance fidelity with adaptability to context.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
CULTURAL MODEL: JUDGING THE PROGRAM 

AND THE IMPLEMENTERS

• Implementation in the real world is complex, and if the program does not 

achieve the ideal expected results, implementers are to blame.

• Makes people hesitant to evaluate because it will make them look bad and 

will make them look like they are wasting resources.

• Causes fear that the program will be deemed ineffective and defunded.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
FRAMING STRATEGIES TO INCREASE BUY-IN
REFRAMING THE CONVERSATION

1. Clarify fidelity versus adaptation to increase understanding of implementation.

Every situation is different. This program has core 

“effectiveness factors” that must be maintained, and 

then we will adapt to our local context.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
REFRAMING THE CONVERSATION

2. Use the values of Ingenuity and Responsible Management to avoid the 

overwhelming complexity cultural model.

Our organization has a long history of being creative 

and resourceful when it comes to solving problems and 

improving outcomes for young people. We can use 

those skills to carefully plan and manage evaluation so 

that we know what is working and why, and so that it 

doesn’t take too much staff time.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
REFRAMING THE CONVERSATION

3. Use an explanatory metaphor to show why evaluation shouldn’t just 

happen at the end.

Imagine a classroom where students were only given 

one test, at the end of the year. Sure, you’d be able to 

see what they learned, but you’d have missed the 

opportunity to see what was going on during the year 

so you could adjust the teaching. The continuous 

evaluation is key so that students learn more.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
REFRAMING THE CONVERSATION

4. Validate and empower implementers so they don’t feel like they are being 

judged.

We know that the real world is a messy place and 

that programs don’t always hit the outcomes they 

want right away. We want to continually evaluate so 

we can adjust the program as needed. And we need 

the input from people like you who are seeing what’s 

really going on!

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
REFRAMING THE CONVERSATION

5. Focus attention on quality rather than quantity. 

We all want to reach as many people as we can. But 

we won’t be doing any good if our program doesn’t 

work well for the people we are reaching, and 

evaluation is critical for that.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
REFRAMING THE CONVERSATION

6. Link the benefits of evaluation to your mission. 

Some benefits include: (a) adjusting the program so it 

works better, (b) learning what is working and why so 

that we can keep improving, and (c) telling 

compelling success stories, to get more buy-in, to get 

more resources, to do more good work.

Source: Lindland, E., et al. (2015). “Just Do It” Communicating implementation science and practice: A FrameWorks strategic report. 

Washington, DC: The FrameWorks Institute. 
QUESTIONS?
Xan Young, MPH

Senior TA Consultant

American Institutes for Research

202-403-5290

xyoung@air.org

Michael Baran, PhD

Principal Researcher

American Institutes for Research

301-592-3354

mbaran@air.org

CONTACTS