FINISHED FILE

AMERICAN INSTITUTES FOR RESEARCH

PROGRAM EVALUATION DOESN’T HAVE TO BE SCARY:

INCREASE BUY-IN WITH MESSAGE FRAMING DESCRIPTION

WEBINAR

MARCH 22, 2018

11:00 A.M. CDT

Services provided by: 

Caption First, Inc. 

P.O. Box 3066 

Monument, CO 80132 

1-877-825-5234

www.captionfirst.com 

 ***

This text, document, or file is based on live transcription. 

Communication Access Realtime Translation (CART), captioning, 

and/or live transcription are provided in order to facilitate 

communication accessibility and may not be a totally verbatim 

record of the proceedings. This text, document, or file is not 

to be distributed or used in any way that may violate copyright 

law. ***

 >> XAN YOUNG: Okay welcome everyone. We will go ahead and 

get going now. Welcome to today’s webinar, titled. This webinar 

is being recorded and will be available for later viewing on the 

youth.gov website. Thank you for being here. Hold on. Let's 

see if I can get this to advance. There we go. This webinar is 

hosted by the interagency working group on used programs. 

 On behalf of all the federal partner departments and 

agencies, which support the programs and services focusing on 

youth. In a moment my co-presenter and I will introduce 

ourselves but first to Okori Christopher, our technology 

specialist, will provide a quick overview of the technology 

we're using and logistics. Okori?

 >> SPEAKER: Thank you for joining. This session today is 

listen only, so I'm going to mute all lines calling in. To 

submit your question, we ask that you do so using the Q & A pod 

on the left-hand side on any pictures of our presenters today. 

Those questions go to us, and we'll make sure that the 
presenters address them as soon as the Q & A section starts. 

 For technical questions, please use that to submit them to 

me, and I will contact you directly and try to work out any 

issues you may be having on your side. With that, I'll pass it 

back over to you. 

 >> XAN YOUNG: Thanks Okori. Let's move on to 

introductions. My name is Xan Young. I’m a senior TA consultant 

here at AIR. I work with community members, professionals, and 

organizations to promote, policy, practice and system changes 

with the goal of improving the lives of youth as well as other 

people across the life span. Michael Baran will be 

co-presenting with me today. 

 Michael, would you like to introduce yourself? 

 >> MICHAEL BARAN: Thank you, Xan. So my name is Michael 

Baran. I'm a principal researcher at AIR. I'm not a program 

evaluation expert, per se, but I am a cultural anthropologist. 

And what I specialize in is doing deep ethnographic culture 

research on a variety of issues in a way to understands how 

people are thinking and acting in their daily lives and then to 

use that information to improve the way that we do 

communications, the way that we do messages, outreach, and to 

improve outcomes on the whole range of issues. 

 Before I started working at AIR, I worked at the Frameworks 

Institute, which really specializes on this idea of framing, and 

I'll be drawing on some of the research that I did there in this 

webinar. 

 >> XAN YOUNG: Great. Thanks, Michael. So we'll start out 

by discussing program evaluation in general; what it is and why 

it's worth doing. And I'll move through this part fairly 

quickly because this content may be familiar to many of you. 

Michael will cover the rest of the presentation starting 

with an explanation of culture's role in strategic message 

framing, followed by an overview of common barriers to 

communicating about program evaluation, and then ending with a

description of some framing strategies you can use to increase 

buy-in. So you might be asking yourself, by-in from whom? I'll 

address that now and we'll also revisit it later. 

We put this webinar together because we know that any 

successful program evaluation really requires the participation 

of many different people. It's not enough for a federal funder 

to require a programmer evaluation or for a project director to 

commit to engaging in evaluation. The reality is that many 

other people need to participate in the process. 

You might need people to document how they implemented the 

program, collect information on sign-in sheets, distribute 

feedback forms, conduct follow-up calls, provide access to 

datasets, et cetera, et cetera. So if you don't have buy-in 
from those people, you won't succeed. 

So by the end of this webinar, you will know how to engage 

key stakeholders in support of the evaluation work. So let's 

get started. First, what comes to mind when you think about 

program evaluation? Why don't you write your answer on the 

“submit questions here” box that Okori pointed out earlier? 

I'll keep an eye out for what you write there, and then 

I'll read a sample of your comments out loud to the rest of the 

group. 

So before we start talking to you about program evaluation, 

just what does that phrase evoke for you? Either an association 

with it, a definition? Whatever you feel led to share. Okay 

I’m seeing a lot of responses, thank you.

Here are words that come to mind when people hear that 

phrase are understanding, exploration, program outcomes, 

measures, surveys, fidelity, transparency, someone else said a 

report that will be very little read. The quantitative data. 

Let's see. That it's necessary. The process of data collection 

and analysis. The effectiveness of variables we try to 

influence, accountability. Wow, lots of different pieces of 

feedback here. Great. Systemic way of gathering information to 

answer questions. Uh-huh. 

It also evokes feelings like words like testing, boring, 

long. Nervousness. Okay, so I think that's a pretty good 

snapshot of the diversity of feelings and definitions that we 

come to this work with. Thank you so much. 

We may revisit some of these comments from you later 

because you started to touch upon some of the things that 

Michael will be covering in later slides. So one of the things 

I like to do is define terms. I found it's important not to 

assume that the words I use have the same meaning to me as my 

listener. Take the word surveillance. Public health 

professionals hearing that term might think of CDC's youth risk 

behavior survey, but someone else might imagine an FBI 

surveillance van and a bunch of wiretapped phones. It's very 

useful when you’re having a conversation with other people to 

make sure you use words the same way and this will be a 

recurring theme in the discussion today. 

So to demonstrate that, we're going to briefly describe 

program evaluation starting with the word "program." There's no 

standard government definition of a program. It can be defined 

any number of ways. Whether a program is defined as an 

activity, a project, a function or a policy, it must have an

identifiable purpose or set of objectives an evaluator assesses 

how well the purpose or objectives are met. 

On this slide I share a definition from GAO that says, 

program evaluation is a systemic study using research methods to 
collect and analyze data to assess how well a program is working 

and why. And actually it's important to note that evaluations

may also assess whether programs had unintended or undesirable 

outcomes. This can be a very important finding. 

Okay. So an evaluation can answer many different kinds of 

question. It can tell you whether a program attained the 

implementation objectives by answering questions like, are you 

implementing the services or training you initially planned to 

implement? Are you reaching the intended population? 

Are you reaching the intended number of participants? Are 

you developing the intended collaborative relationships? It can 

also help you determine if a program obtained the outcome 

objective answering questions like, are participants exhibiting 

the expected changes, knowledge, attitudes, behaviors or 

awareness, and can these changes be attributed to the program 

itself? 

So here's another way to think about the kinds of questions 

an evaluation answers, and you can answer implementation, 

effectiveness and attribution questions, which are some of what 

I covered on the previous slide. But here I’m also highlighting 

that it can measure efficiency. Are your program's activities 

produced with an appropriate use of resources such as budget and 

staff time, and it can help you determine cost-effectiveness. 

Does the value or benefit of achieving the goals or objectives 

exceed the costs of producing them? 

So you can design your evaluation to answer any number of 

these questions. To answer questions like these, you do first 

need to be clear about what you're trying to accomplish. A

logic model is a great tool for this, both for program planning 

and for evaluation. It visually describes how programs should 

work, presents the planned activities for the program, describes 

how activities will be documented, and focuses on the 

anticipated outcomes. It often uses diagrams or pictures that 

illustrate the logical relationship among key program elements 

through a sequence of if, then statements. 

Sometimes the image says “if, then,” and sometimes it's 

implied by arrows. In a sense it's a kind of roadmap. It's 

describing where you are, where you're going, how you're going 

to get there. It can help you identify program goals, 

objectives, activities, desired results, clarify underlying 

assumptions, communicate key elements of the program, and 

importantly, specify what to measure in your evaluation. 

So here's a simple graphic that shows the key elements of a 

logic model. You know what I'm talking about. You basically 

say if we use these resources for activities to produce these 

outputs, so these people can change their ways which leads to 

these outcomes, then we’ll have these fabulous results. So 
here's an example of a sample logic model slightly adapted from 

one you can find on the CDC website at the URL at the bottom of 

the slide. 

Here the goal is to reduce the prevalence of sexually 

transmitted diseases among men in City X, and your objective is 

by month/year, staff will conduct three professional development 

workshops on STD screening recommendations for clinical 

providers caring for men in City X. I won't read the whole 

thing out, but you can kind of see how the inputs like funding 

and staff time are followed by activities, which result in the 

completed activities of these outputs, the workshops and you 

move on to the workshops leading to an increased knowledge about 

key things like the STD guidelines among the clinical providers 

and knowing about those guidelines leads to them being used and 

increased routine screens for STDs in men by the clinical 

providers who attended the workshop. 

Then, presumably as a result of the screening, you actually 

have a decreased prevalence of STDs among men. Here's another 

one. This one is to illustrate you shouldn't be intimidated by 

logic models. This one was this was drawn by Courtney and Bailey

Peters when they were 9 and 11 years old, and they made it after 

their mother Dorian Peters told them what she did at work that 

day. 

She was on the staff of the Idaho Asthma Coalition and 

attended a presentation on logic modeling. When she came home 

and told her kids what she did that day, they decided to try it, 

and this is what they came up with. What I like about this is 

that it reminds you that it doesn't have to be fancy. You can 

hand-draw this thing. 

It's also really helpful to see what kind of assumptions 

get revealed by a logic model. In the interest of time I won’t 

have you type in what you see here, but take a look at it and 

see if you can identify some assumptions these kids had. One 

assumption seems to be that if they make $1,500 they will be 

able to build a playground, right? Or get new playground 

equipment for the city park. 

But looking at these images, you can find more assumptions. 

What do they think they will be able to accomplish? What's 

going to be the result of the new playground equipment? A safe 

place for kids to play. Okay. 

So one of the reasons I love making logic models is that it 

reveals those hidden assumptions, and it allows you to adjust 

your implementation plan or identify the kind of data that you 

would really need to make sure you collect to support your 

program evaluation. So I'll stop there and say, it's clear that 

I'm a big fan of logic models and program evaluation in general, 

but not everyone feels the same way I do. 
So I'm going to pass this over to Michael, and he's going 

to take us through the next section of the presentation. 

 >> MICHAEL BARAN: Great. Thank you, Xan. Who isn't a fan 

of logic models? In this section basically I'm going to answer 

why do we need to think about culture when we're talking about 

messaging, message framing? It's not something we often think 

that much about, so I'm going to get into that first in a 

general way before talking specifically about program 

evaluation. 

To start, let's look what happens when a message goes out 

there. When people hear the kinds of things that van talked 

about. There's research from the CDC showing that people have 

these kinds of reactions; say these kinds of things:

It's too complex, it’s too expensive, I can just tack it on 

at the end and it’s too time-consuming, my resources are better 

used providing activities, I don’t have enough staff, or it will 

make me look bad. As I went through the comments in that box

when Xan asked what evaluation means to you, I can see that many 

of you are much more on board already program evaluation because

there weren't too many responses like this, but there were some, 

you know. It's boring, it's long. Nervousness. Things like 

that. Right? 

So these are the things that people say. What's really 

critical now is that when you try to respond to that, you can't 

just come back with the same kind of thing. The same kind of 

thing that she was talking about and saying, no, it's not like 

that and repeat what you already said. To be effective we need 

to understand what's underlying what people say in a deep way. 

The deep patterns of thinking that underlie what people are 

saying, and then also to use that to be strategic and to be more 

effective in our messaging. So first I want to explain how this 

works with two examples. Not, like I said, from program 

evaluation, so don't get caught up too much in the project 

details, but just to show you how this works overall. 

So this first example I'm going to talk about was from a 

project that I did with the Frameworks Institute for a group 

called transform justice in the U.K. Okay? This is a group in 

the U.K., and they were working with a whole bunch of experts, 

scientists, and advocates, and one of the messages they wanted 

to get out was this. This is a composite, not a direct quote. 

Prison only makes a youth offender more likely to commit future 

crime. Prison doesn't do any good for youth who engaged in 

minor crimes. Okay? 

This is not just a value that they have, right? They're 

basing this on evidence, so there's a lot of evidence out there 

showing that to be the case. The task force for community 

preventative services, for example, reviewed a lot of that 
research and found that transferring juvenile offenders to the 

adult criminal system leads to an increase in the risk of use 

re-offending, so when an expert in the U.K. says something like 

this, what's the reaction? What do people take away? These 

experts are trying to convey some of the science, really. 

They're trying to convey that sending people to prison has 

an effect on people's identities, which is something that causes 

more crime in the future. It creates connections with other 

criminals and actually prunes connections sometimes with people 

outside prison, which also leads to an increase in crime. It 

can limit opportunities for people once they get out of prison, 

for example, making it harder to get a job. They're talking 

about things like that. 

Now, when people hear this message, though, the response is 

something like this: Yes. So it sounds like they agree. Yeah, 

prison is too cushy, like a holiday camp. Very British 

expression. We need to take away prisoner activities and make 

prison more harsh so they never want to come back. Right? 

They're basically agreeing, yeah, prison doesn't work, but not 

because of the things that the experts wanted to talk about, 

cutting ties and making it harder to get a job and systemic 

issue. 

No, because it's too cushy. You have this total mixed 

message between what the expert is trying to get out or what the 

general public hears or sometimes people in the field hear. 

Here’s a second example of a missed message. This one is 

coming from a project I worked on with the Harvard University 

Center on the Developing Child. So they were trying to get 

across this message that prolonged stress is harmful for a young 

child's cognitive development in a very physical way. It really 

impairs brain development. 

And when people hear this, the typical response is 

something like, oh, stress. I experience stress, and it made me 

stronger and who I am today. Stress isn't that bad for kids. 

They need it. Right? 

So again, total missed message. The expert is trying to 

convey one message, and the public or people in the field 

receiving a very different message. There are lots of 

explanations that people have for why messages get missed, 

especially people that work in communications, right? They 

often say things like, well, the message is too complicated, and

that’s especially true when it's a message from a scientist. 

The message was poorly distributed. No one saw it. It 

wasn't using popular channels. The message wasn't memorable. 

It didn't grab your attention. It wasn't exciting. That's why 

people didn't receive it in the right way. It didn't trigger 

strong emotion. You have to really trigger emotion for your 
message to get across. It didn't have facts in it, so why would 

people believe it? Or maybe on the flipside had facts but 

contrasted with myths. There's interesting research about the 

ineffectiveness of myth fact sheets, for example. The basic 

idea being that what happens is that in the short-term people 

misremember the myths as what's true, as the facts. And that it 

then gets worse over the long term when they test it. 

So that format is not very effective. 

There's this idea that people are just misunderstanding the 

message because they don't understand the issue in general. 

Right? But there's something else going on here, which is what 

I'm going to talk about, and that is that culture gets in the 

way, okay? So what is culture? This is the way that I approach 

culture. 

If you talk to 1,000 anthropologists, you’ll probably get a 

thousand different definitions of what culture is. We're very 

good at that. This is the way I approach it the way a lot of 

psychological cognitive anthropologists like me approach the 

issue as well. We think about culture as models in our minds, 

right? 

If you think about a developing child's brain, very young 

from zero to 3 years old, for example, they're forming about 700 

synaptic connections per second, which is pretty amazing. 

They're literally building the cognitive architecture of their 

mind in this cultural environment. They're forming the 

connections that are going to create their understandings of 

what everything means and how everything works. 

This helps us organize the way we think about the world, 

right? We couldn't think about all the stimuli we’re getting on 

a second by second basis anew every time. We have to have

patterns to make sense of it, and it basically builds up a 

common sense about the world. 

Then these cultural models that are there, they get triggered by 

the framing of any message. 

When I talk about a message frame, what I'm talking about 

is not necessarily the content but the way it's presented. 

There are many different framing elements. I'm going to talk 

about a couple of these later. There are things like values, 

metaphors, keywords, the messenger, the tone of the message, 

these sorts of things. All of those frames end up having the 

effect of queueing, activating, triggering the cultural models 

that are there in our minds. 

We used to have this idea that communication, you know, you 

thought of the brain as the general processing mechanism, and we 

have this idea that if you have an idea you want to convey, you 

encode it, right? This is the code model. You encode it into 

words or written words or visuals and send it out to a receiver. 
That receiver unpacks that code and gets basically the message 

you are trying to get across. With the code model, you’re 

surprised by miscommunication. That's not the way it's supposed 

to work. 

With this model, instead--with the idea that you send out 

messages that trigger cultural models and the cultural models 

then determine how a person understands the message-- when 

you’re thinking with that model, you understand that 

miscommunication is what we should expect. That's the norm. We 

plan for that, okay? That helps us be more effective. 

How do we plan for it? If we have the time and we're 

starting a new project, we gather evidence in two phases. So we 

first have a formative research phase to find out what's getting

in the way, what's causing the mixed messages, and then we have 

a second phase in which we develop messages that we think are 

going to trigger productive models and avoid problematic ones. 

Then we test those messages to make sure that they actually do 

work in the way that we expect them to. So what does that look 

like in the previous examples that I gave? 

So here's the first example of criminal justice reform in 

the U.K. All right? Now, when we actually did a whole mapping 

of all the cultural models going on here, we can see what's 

underlying this general response. So in this case there are two 

cultural models that I’ll highlight. Basically this idea of 

retribution is going on, the idea that you have to pay for your 

crime with equal punishment rather than, for example, looking at 

what's going to work to reduce future crime, right? That's not 

the model at play here. 

It’s about, there was a crime, it has to be paid for with 

an equal punishment. Then, there's an idea of a rational actor 

model. That the way people make decisions is by logically 

weighing pros and cons. If you made prison more harsh, people 

would think twice and maybe decide, no, it's not quite worth it 

to commit this crime. We know that's not the way people make 

decisions, but that's the way people think that people make 

decisions. Right? 

So what do we do about that is when we understand that 

these models are getting in the way, what do we do about that? 

We develop a metaphorical way of talking about the same thing 

that allows people to be more receptive to those ideas. Ideas of 

pruning connections and making criminal connections and limiting 

opportunities and creating identities. Something like this: 

Prison sweeps young people into a powerful stream of crime from 

which it's difficult to escape. 

It's subtle and metaphorical, and you can visualize it and 

it channels people's thinking to be receptive, to be able to 

access the kinds of things that experts are talking about. 
Usually those things are there. 

They're just a little bit more excessive, but something 

like this with the way it frames the issue actually allows 

people to understand what the expert is trying to say better. 

There's both qualitative and quantitative data suggesting that 

something like this is effective for that. 

It channels people's thinking, okay. Kind of metaphorical

in itself. Second message; second example. This is the one 

about stress. What's going on here? There's a very strong 

sense of individualism thinking, individualist thinking here, 

and there's also this idea that all you really need to overcome 

hardship is to pull yourself up, to try harder. If you use your 

willpower, you can do it. 

While a standard communication recommendation might be 

something like, well, that scientist's message is too 

complicated, and you have to dumb it down and simplify it. What 

we end up doing is actually making it more complicated, so we 

started testing this idea of a taxonomy of stress. There's 

normal stress, things like going to school for the first day and 

stuff like that that is actually good for you. There's 

tolerable stress which is very serious life events happen, but 

if you have the right supports, you can get past that. Then 

there's toxic stress. Things like abuse and neglect that really 

inhibits the brain developing in the proper way. 

When you then take that same content that the expert was 

trying to get out, but instead of calling it stress you call it 

toxic stress, that actually works. The message is not missed 

anymore, and the reaction is like, oh, yeah, toxic stress. 

That's bad. We definitely should limit that. 

Then the message works in the way that you are intending 

it. Okay. So now let's turn specifically to program 

evaluation. What are the cultural models that get in the way? 

That's what we want to understand first. 

The first thing I want to point out here is we don't have 

formative research to draw on for this topic that's specifically 

focused on program evaluation. So what we do have is research 

that the Frameworks Institute has done on program implementation 

that also addresses program evaluation. So that's what I'm 

going to draw on here. 

The second thing to note is that the cultural models that 

I'm going to be talking about that get in the way of effective 

messaging, you personally might not have these problematic 

patterns of thinking that get in the way. I think from some of 

the reactions that we saw from Xan’s first question, a lot of 

you probably don't have these problematic culture models. You 

might very much understand program evaluation in the way that an 

expert does. You might be an expert on program evaluation. 
As Xan mentioned in the beginning, some people you work 

with have these problematic cultural models, so it's helpful for 

you to know what you're up against. Okay. So the first culture 

model is a “just do it” cultural model, right? You've got a 

plan, common sense tells you it will work. It worked somewhere 

else. It's going to be great. So just do it. What do you need 

evaluation for? You might have heard reactions like this if you 

have tried to get people on board with program evaluation in the 

past. 

A second culture model, if you do have to have program 

evaluation, people often think of it as something that starts 

after implementation is done. It's at the end to see if it 

works. This way of thinking, of course, completely obscures the 

idea that continuous evaluation is critical to show what is 

working or not and why that is so that you can adjust as you go. 

And that's something that has to be planned right from the 

start along with implementation. This cultural model makes it 

hard to see that. Like I said, I know some of you do see that. 

I think in the comments someone even specifically said, you 

know, planning right from the very beginning. So that's great 

to see. 

Third, we have a cultural model we might call short term or 

small picture thinking. The idea here is that evaluation is 

about this single program, right? Not really thinking about the 

long-term for this program or the wider picture of people trying 

to make a difference on this issue and how data you collect

could contribute to that, but really focus on this one 

particular program and this one particular iteration. 

A fourth cultural model that often gets in the way is a 

focus on quantity. More is better. Delivering more services, 

reaching more people. That is what it's all about. That is 

impact, right? The bigger the numbers, the bigger the impact. 

It's all about the numbers. If it's all about the numbers, then 

that continuous program evaluation designed to improve quality 

is kind of at odds with that, because it's going to take away 

staff, going to take away resources, going to take away time 

from what really matters, and that you will definitely see folks 

that you are trying to get on board with this thinking what 

really matters is reaching more people and feeling frustrated 

that the evaluation is going to take away from that. 

A fifth cultural model sometimes gets in the way. 

Sometimes this paralyzing way of thinking about the overwhelming 

complexity of the world. How can you evaluate a program that 

keeps fidelity to the original idea, when the real world is so 

messy. We have to adapt it for here for where we are, and that's 

going to screw it all up basically. Okay? That kind of 

thinking can get in the way because it really can be 
almost -- people almost have a fatalistic way of thinking about 

it, that it's just too complicated. 

Then, finally because it's adapting to the messy, real 

world, there's this idea that some people have that the -- that 

it's not going to achieve the desired results, and that's going 

to make us look bad. It's going to look like a waste of 

resources. It's going to make it so we don't get more resources 

in the future. That definitely makes the idea of evaluation 

pretty scary, right? 

Okay. Now how can we take that where we know about what's 

going on and what we know about framing and make it less scary? 

In this lass section I have some suggestions for more effective 

communication, for getting more buy-in. 

One big caveat is not these suggestions have not been 

tested in that phase two that I mentioned earlier where 

you -- after you developed hypotheses for what's likely to work, 

you actually go out and test them. These have not been tested, 

but they are very good, educated guessing for knowing what we 

know about communications in general and what we know from that 

first formative phase of research that was done. 

So first recommendation is to start off being very up-front 

about fidelity versus adaptation to context with something like 

this. Every situation is different. This program has core 

effectiveness factors that must be maintained, and then we'll 

adapt to our local context. 

So this helps to avoid some of that worry that adapting to 

local context is going to mess everything up. Right? 

Second is this idea to use values to frame the message. I 

want to pause for a minute to talk about values. Almost every 

message, even if you don't realize it, has a value built in. 

That's a subtle cue telling the person what's at stake, why it 

matters, how to approach it, something like that. 

You can take the same content and switch the subtle framed 

value in there and have dramatically different results to give 

you a concrete example, there was a project I was working on in 

Canada which was looking to increase support for people who had 

substance abuse issues. So we were working with some 

organizations there. We did an analysis of the frames that were

already used by that group, and it turns out they were framed 

with a value of empathy. 

When they talked about substance abuse issues, they were 

sort of taking this that put yourself in the shoes of the person 

addicted to substances, you know? Don't you feel like they're a 

person, too and they're human and need support and stuff like 

that? It seemed very logical and obvious to them that this 

would be a good way to talk about the issue. 

When we went out then and took the same content and tested 
it against other values, they were framing that same content and 

even no value, which strips all of the values out of this 

particular phrase, we found something that even to us is pretty 

shocking. Not only was the value of empathy not helpful for 

increasing support, and this was a quantitative online survey 

with thousands of people testing this value-framed sentence 

against the whole series of dependent variables. 

So not only does the value of empathy in this particular 

case not increase support, it actually decreased support in a 

significant way compared to that same context that had no value. 

Right? 

So by then talking about what they were talking about, 

using the value of empathy, they were getting worse outcomes 

than if they had not had any value in there at all, which was 

pretty shocking to us. The good news is there were other values 

that were more successful. 

I was just listening to your talk. I wish I can remember 

the name of the guy. He talked about testing values to frame 

issues related to conservation and the environment. 

He found that conservatives had values that resonated with 

them and liberals had values that resonated with them. If you 

took the liberal content and framed it with conservative values, 

the same message, it actually had a much better success rate to 

getting people to support issues related to the environment. 

So really fascinating stuff. 

Okay. Tangent aside, in this particular case, FrameWorks

recommends these two values. The value of ingenuity and 

responsible management to avoid this overwhelming complexity 

cultural model. It would look something like this. 

Our organization has a long history of being creative and 

resourceful when it comes to solving problems and improving 

outcomes for young people. We can use those skills to carefully 

plan and manage evaluations so we know what's working and why 

and so it doesn't take too much staff time, okay? Just 

embedding those values in there, and giving people not feeling 

the stress of the overwhelming complexity cultural model. 

Next, you can use a different framing tool. Explanatory 

metaphors, and I want to explain what an explanatory metaphor 

is. It's taking something -- we use metaphors all the time, 

right? They’re in poetry everywhere, but we can use them for 

communication.

The idea is taking something very complicated and talking 

about it in terms of something much more concrete and simple 

that people can understand. You can't just pick any metaphor to 

explain your issue even though a lot of times experts and 

advocates would like to and often they do actually frame their 

issues using explanatory metaphors. 
Sometimes they go very wrong. I’ll give you my favorite 

example, which was, again, working with the Center on the 

Developing Child but this time on the issue of resilience. 

There were some experts there that thought that the perfect 

way to talk about resilience in children was to talk about 

orchid children and dandelion children and how what we really 

meant is dandelion children that can grow in any environment and 

didn't need a lot of care. They were just going to thrive no 

matter what, right? Whereas, of course, the opposite is the 

case for orchids. 

When we went out and tested that message with people, you 

know, we could do this test right now, but I won't take time to 

do it. What comes to mind when you first hear that if you have 

kids especially? When we talked to people, no one wanted a 

dandelion kid. Everyone wanted an orchid child. They're 

beautiful, special, precious just like your children, right? It 

completely backfired which is why you have to test these things, 

okay? 

This is one we suggest might be effective to talk about 

program evaluation because of some of the formative research 

that was found when the Frameworks Institute did that research. 

It's something like this. Using people's familiarity with tests 

and quizzes. We all went to school to say something like this. 

Imagine a classroom where students are given one test at the end 

of the year. 

Sure, you'd see what they learned, but you'd miss the 

opportunity to see what was going on during the year, so you can 

adjust the teaching. The continuous evaluation is key so that 

students learn more. So a metaphor for something like if people 

have the trouble with the idea of continuous evaluation. 

Fourth, focus on implementers and address their fears 

directly. So something like this. We know that the real world 

is a messy place and programs don't always hit the outcomes they 

want right away. We want to continually evaluate so we can 

adjust the program as needed, and we need input from people like 

you who are seeing what's really going on. 

So empowering implementers in the process. That's 

critical. Focus attention on quality rather than quantity. 

Something like this: We all want to reach as many people as we 

can, but we won't be doing any good if our program doesn't work 

well for the people we're reaching, and evaluation is critical 

for that. 

They’re going to think about quantity no matter what, but 

focus on quality as well and how evaluations can increase the 

quality. 

Then finally always linking the benefits directly to your 

mission. Focusing on the positive, something like that some 
benefits include adjusting the program so it works better, 

learning from what's working and why so we can keep improving, 

telling compelling stories to get more buy-in and more resources 

and to do more good work. As you can see, these framing 

suggestions, these framing words are not magic-wired. That's 

not the idea. 

They’re not things you should memorize even. Just 

hopefully some helpful ways you can talk about program 

evaluation in your own words by using these framing tips and 

have slightly easier time getting people on board. 

They work because you know what's getting in the way of 

people understanding these issues like you do. You know better 

what's making it scary in the first place, and you can use these 

framing tools to overcome that. All right? So that brings us 

to 15 minutes that we wanted to leave for questions. 

So if you would like to type them as Okori mentioned, we 

have the lines muted. If you want to type them in the submit 

questions here box, we will try to address as many as we can 

now, and if we don't get to them all, we'll also able to respond 

in writing and get those answers out to all of you. 

 >> XAN YOUNG: Yeah, thanks everyone. As a reminder, you 

can post comments. Maybe you managed to obtain buy-in using one 

of these techniques we mentioned or another one that worked. 

Don’t hesitate to let us know about your story as well. While 

we wait for questions to be typed, I wanted to share one comment 

already posted. Someone typed in, I think there's also 

oftentimes a confusion between process/implementation type 

evaluations and outcome evaluations. 

Too often decision-makers only care about outcomes and not 

enough about metrics related to process and delivery. That 

comment was written before Michael shared some slides about ways 

to frame this very issue, but you're absolutely right that, you 

know, a key question to answer is where are program activities 

put in place as originally intended? Some people call this 

measuring fidelity. We also hear Michael talk about them as key 

elements.

If we don't know what happened, we really don't know how to 

implement the outcomes, you know? If we say everybody go -- all 

the teachers implement this program with six sessions and two 

hours each session and look at the outcomes and it didn't seem 

to work, but we don't know anything about what happened after we 

told people to do it, we're missing a piece of the puzzle. 

What if we find out teachers on average implemented the 

first two-hour session and decided it was a waste of time and 

didn’t do the next five sessions. That would have been a great 

thing to know early on in the process. So process evaluation is 

one way that we can kind of monitor how things are going and 
making mid-course adjustments and make sense of what we see in 

the outcomes. 

 >> MICHAEL BARAN: I wanted to address one question I see 

here, Xan. This is a question of how to address this in tribal 

communities, which is a great question to think about when we 

say cultural models, what culture exactly are we talking about? 

Those are decisions that are really key when we design that 

first phase of research, and although I didn't do this first 

phase of research that I was talking about with the Frameworks 

Institute on program implementation and evaluation, my guess is 

that it did not include a lot of Native-Americans in that 

research. 

They made some decisions about demographic variables that 

were going to be included in the sample and then looked for 

shared assumptions and shared understandings among that group. 

Then I'm almost positive that's a very diverse group in terms of 

race, class, age, having children or not, all sorts of 

demographic variables. But my guess is they did not include 

people from tribal communities, and so what you want to do,

then, is you'd want to do -- you could do a scaled-down version 

of a phase 1 research. If that was key for you, you could go 

and do research specifically with tribal communities and see if 

these same cultural models were at play or different ones to 

take into account. 

 >> XAN YOUNG: Great. Thanks. So just flipping through 

some of these messages as they pop up. Okay. So here's a 

comment that just came in. I agree with mixed messages and 

framing, it's a big problem. One example is when people say 

that African-Americans are disproportionately represented in the 

criminal justice system. You know, the person saying that might 

have meant to show that there was a systemic issue there, and 

it's a problem, but the listener might interpret it as a sign

that black people are more prone to criminal activity. 

 So that's a good example of a mixed message where one 

person is sharing data thinking that the interpretation they 

have is the same as interpretation of the listener. And 

oftentimes people don't realize that that message was missed in 

the conversation. 

 >> MICHAEL BARAN: Yeah. That's a great observation. I 

saw that, and I actually on that very issue saw that exact thing 

happening where this very strong individualistic way of 

thinking. So those facts that the experts or advocates often 

think speak for themselves. 

 They tell such a convincing story to the experts themselves 

that they think all I would have to do is show these facts or 

these graphs to my audience, and they're going to take away the 

same thing that I do because they’re so convincing. As this 
listener pointed out sometimes the cultural models really 

determine how we interpret even those basic facts or statistics 

and often in completely opposite ways. 

 It's fascinating to watch, and sometimes it's frustrating 

and sad to watch as well, but it's very true. 

 >> XAN YOUNG: Here's one person that is saying that in her 

experience she has found that it’s useful to try and connect 

things you're trying to achieve in the evaluation to experiences 

that people might have already had in their personal lives. For 

example, if you're trying to talk to them about how you're 

trying to achieve an increased interest in something. 

The example she shared was it's like having a bunch of 

friends giving you feedback on which picture to use in your 

Tinder profile. So trying to have a conversation with people 

about what does it mean to see increased interest? How do we do 

that? Let's compare it to something you might have had a 

conversation about before. 

Or nutrition labels being connected to figuring out what kind of 

bread to buy was another example she shared. 

 >> MICHAEL BARAN: Yeah. There's dual strategies going on 

there, right? Sort of an explanatory metaphor and connecting to 

your personal life, so it's great. 

 >> XAN YOUNG: We're pausing for a second to flip through. 

We love all the feedback we're getting here. 

 So we have this question coming in about how to gain buy-in 

for evaluating competences for implementers or facilitators. 

Issues like they might understand the mechanics of the 

implementing the intervention but have gaps in competences like 

culture humility, sensitivity that could actually cause harm. 

So I think what you're getting at, certainly culture 

linguistics is critical for any implementation, but also 

consider ways of measuring implementation that are going to get 

at these concerns, right? Like, having observations, for 

instance, to see -- to go out and see how people are 

implementing it included in the training and expectation about 

the way the training is delivered and our things like that. 

Obviously, cultural and linguistic competency is more 

complicated than these two simple things, but to make sure that 

how our program is implemented, the decision about how 

implementing it is made, the way those most affected are engage 

in the implementation and the way the people who implement the 

program understand how to go about it, those are all important 

parts of making your implementation plan and considering what is 

important to evaluate. 

 >> MICHAEL BARAN: Yeah. A great question in here as well 

about whether there are any strategies for getting people on the 

same page within an organization on the message about evaluation 
or really that question about everything, whatever you try too 

message about. That's a great question. It takes a lot of 

work, you know? Sometimes it's getting people on board with the 

right message takes practice. 

It doesn't always come naturally to use these frames we're 

suggesting are more effective. In fact, I remember running a 

workshop once and at the end a participant said, so basically 

what you’re saying is if it feels good to say it, don't say it. 

And that's not exactly the right rule, but there's 

something to that that we often have habitual ways of talking 

about an issue because they resonate with us, but if our goal is 

to get people on board who might not already be, then we have to 

practice using those strategies to take a normal communication, 

a normal message that someone would want to say about evaluation 

and pick it apart to find what frames are embedded in there that 

you might not realize. 

Then practice rewriting that or resaying it again and 

again. It takes some work. You can do it. 

 >> XAN YOUNG: While we see some comments coming in, I'm 

just reminded of another online learning event I was at recently 

where some colleagues in Los Angeles talked about the way they 

went about implementing a program to increase positive activity 

in local parks that were not at the time considered family, you 

know, places to spend time and feel safe and commune with your 

neighbors. Instead, they were kind of viewed as places where a 

lot of crime happened, gang activity, et cetera. 

They really went to community members who were using the 

park, which included gang members, and talked about, you know, 

what would -- how can we come to a common understanding about 

the parks being a neutral space? Some of the language that they 

ended up adopting and kind of using across the board with 

community members who had had a lot of conflict with each other 

in the past in the park was, especially young people, was to 

refer to the common frame of reference around visiting areas in 

prison and those being a safe space where people wouldn't have 

fights with each other. 

People who previously were incarcerated it turned out had a 

lot of, you know, understanding that you might have conflict in 

any number of locations in a prison, but when it comes to 

spending time with your loved ones who are coming to visit you, 

that's a neutral zone. Everyone should be able to go in there 

at the right time to be with their friends and loved ones and no 

one will mess with anyone else. 

When they talked about the language to describe places like that 

and could we agree the parks are that kind of space in our 

broader community, they obtained buy-in and they were able to 

implement a program and then really keep checking in, like, is 
it working? People really started to think about this space in 

a different way using that kind of understanding and language. 

So that became part of both their implementation plan for 

building buy-in in the community for the program, but also part 

of the evaluation. Did we see a significant change in the 

number of people that think of parks this way, especially among 

those who had been previously incarcerated. They were able to 

measure there was that shift. 

So I think that ties together some of what, you know, we 

were hearing about -- and then they were also able to measure 

some outcomes, and in the spirt of outcome evaluation, and show 

there was a decrease in violence in the community and there was 

a decrease in obesity because people were actually being more 

active and there was a real active commitment to fitness 

programs in this use of the parks to try and engage people 

around fitness and physical activity as well. 

So it's an interesting thing where we see the real needs to 

tie in cultural understanding, messaging framing to the whole 

process so that you can evaluate and plan well for your 

implementation and have meaningful measures for your evaluation 

to understand what worked. 

 >> MICHAEL BARAN: Yeah. That's a great one to leave on, 

Xan. Before you all go, we hope you really enjoyed this. We've 

definitely enjoyed presenting this for you. Before you go, we'd 

love to hear any feedback you have in the chatbox. If you could 

just maybe say something that you learned or a way you could 

envision using this in your work, we'd love to hear from you. 

Again, thank you so much for participating. 

 >> XAN YOUNG: We're always here for a reason, so we want 

you to apply this in your real-world context. When you write 

your final comments in the question box, let us know kind of 

where you do your work and, you know, whether you got something 

out of this webinar you'll be able to use, because we're going 

to look through all that to learn more about what's useful to 

others in the future. If you have any suggestions also about 

additional information you want or other kinds of webinars we 

should deliver, or content we should put up on the youth.gov 

website, this is your chance to let us know that too. 

It’s great how this can be applied to do your work. Thank 

you for your time. We're really grateful that you spent this 

hour with us. I see the comments come in. Thank you so much 

for staying on to type those in. We are going to -- there's 

some logistical things coming up. I want make sure to let you 

know if people want to receive the slides, there is a download 

pod on the screen. 

If you haven't downloaded a copy of the slides on handouts 

click on the one that says evaluation webinar notes and you'll 
have it right away. This webinar will also be -- it has been 

recorded and it will be posted on youth.gov along with copies of 

these slides. So you can always find it there as well, and send 

other people to come watch it or get the resources. Thank you 

so much. One person said, really loved the thought process on 

messages, so critical. Thanks so much. 

We really appreciate -- and you're from the U.S. department 

of treasury. It's nice to see people letting us know where they 

work. I won't keep anyone here by reading these out, but we'll 

read through them carefully. Thank you for your time. You can 

stay on as long as you want typing the messages, and we'll end 

the audio now thanking you and wishing you a great day. 

 (Webinar concluded at 12:02 p.m. CDT)

***

This text, document, or file is based on live transcription. 

Communication Access Realtime Translation (CART), captioning, 

and/or live transcription are provided in order to facilitate 

communication accessibility and may not be a totally verbatim 

record of the proceedings. This text, document, or file is not 

to be distributed or used in any way that may violate copyright 

law. ***