Office of Research & Evaluation Webinar 

Using Evidence for Scaling Community-Based Interventions 

That Work 

1

MARY: Hi, everyone. Thanks so much for joining us 

today, and welcome to our June, 2018 webinar. My name 

is Mary Hyde, and I am the Director of the Office of 

Research and Evaluation at Corporations for National 

and Community Service. Our office’s objectives are to 

support our agency’s mission by building knowledge on 

civic engagement, volunteering, and national service. 

We conduct in house research, but also fund research 

through competitive grants to researchers, scholars, 

and dissertators at institutions of higher education, 

and support research and evaluation of our programs 

and our grantees with other sources of support, like 

those you are going to hear today. Our webinar series 

is one way we share our ongoing research and 

findings. Today we are very excited to host another 

research and evidence webinar, titled Using Evidence 

for Scaling Community-Based Interventions That Work. 

And I would like to acknowledge and thank all of the 

national service programs who are hard at work 

building that evidence base so that we can help 

improve lives and strengthen community conditions.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

2

We are also very excited to welcome members from the 

Federal Evidence and Evaluation Community of 

Practice, who are both in the audience here on site 

with us and online today, and the webinar will serve 

as part of their workshop series. Because this is 

part of the series, we will keep the lines open until 

3:30 instead of ending at 3:00. For those of you who 

have joined us in the past, we tend to not go that 

long, but today’s a special occasion.

Here with us today is Dr. Diana Epstein, who leads 

the evidence team at the Office of Management and 

Budget, and she will say a bit more about the 

workshop series. Diana.

DIANA: Thanks, Mary. [unintelligible] to be here with 

you today and I want to thank everyone at CNCS for 

allowing us to jump onto this webinar and cobrand it 

as part of our broader Evidence and Evaluation 

Community in Practice series. So the Evidence and 

Evaluation Community Practice is an informal group 

that we run, our team at OMB coordinates. The goal is 

to really engage folks from across the federal 

government from evaluation offices, research offices, 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

3

analytics offices, pretty much anyone who’s 

interested in efforts to build and use evidence for 

decision making. Particular focus on program 

evaluation. So we do a couple things. We run a MAX 

site that has a bunch of resources, both in formals 

or OMB directives, as well as resources that other 

agencies have produced, and also serves as a 

collaborative workspace for some of the interagency 

groups that have evaluation as a central component. 

We also are running trainings like this, workshops, 

about once a month. So the way we’ve been doing this 

over the past year or so is we rotate through; we 

have a different agency host every month, both 

onsite, and then also hopefully having a webinar 

component for those who are not able to attend in 

person. So I’d really encourage you all to check out 

the MAX site on Federal Evidence and Evaluation 

Community of Practice, and hopefully you will join 

some of our events in the future. And just a plug for 

our team as well. We’re here as a support unit, so 

we’re here to support you. If you or others at your 

agency would like to reach out to us, we are more 

than happy to offer support.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

4

MARY: Great. Thank you, Diana. And welcome to all of 

you on the line and in the room. Before I move into 

introducing our speakers, we’d like to cover a few 

housekeeping items. As this is the second webinar we 

are hosting using our new platform, Adobe Connect. So 

with that, I will turn it over to Emily.

EMILY: All right. Thank you for everyone joining us 

online [unintelligible] Adobe Connect. I want to let 

you know this webinar will be recorded and posted 

online following the presentation. Unlike previous 

webinars, there is no dial-in phone line. All audio 

will be broadcast over the internet using your 

computer’s speakers. All participants online will be 

in listen-only mode until the Q and A session 

following the presentation, at which time you can ask 

a question using your computer’s microphone. We ask 

that you ask the question directly into your 

computer’s microphone as that will give us the best 

sound quality. You can also ask questions at any time 

during the presentation, using the Q and A in the 

chat boxes below. As I mentioned earlier, this 

webinar is recorded and if you have any questions or 

experience technical difficulties, please let us know 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

5

in the chat box. I believe that takes care of all of 

our housekeeping items.

MARY: Thank you. So let me introduce our speakers 

today. We’re going to being with a few introductory 

comments from Dr. Lily Zandniapour, who is a research 

and evaluation manager here at CNCS, and who has been 

developing and championing the evidence base at CNCS. 

We are also going to be hearing from Anthony Nerino, 

who is a research analyst here at CNCS, and he 

oversees the analysis and reporting of the national 

volunteering and civic engagement data, as well as 

leading the scaling project here at the corporation. 

In addition, we have Nan Maxwell and Scott Richmond, 

joining us from Mathematica Policy Research. Nan 

Maxwell is a senior researcher with more than 30 

years of experience in conducting basic and applied 

research. She has extensive experience in survey 

research and has served as a principal investigator 

for numerous randomized controlled trial and quasiexperimental design impact studies of programs 

designed to prepare youth for successful careers and 

post-secondary education. Also to help at-risk 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

6

groups, such as the homeless, parolees, those with 

mental illness, and opportunity youth gain employment 

and life stability. She has worked with an array of 

clients including federal, state, and local 

governments; nonprofit organizations with missions to 

improve employment or education outcomes, and forprofit organizations scaling promising interventions 

in a community. Scott Richmond is a survey researcher 

with diverse experience in several key areas, 

including education, family support, and early 

childhood, and has extensive experience in directing

data collection activities for evaluation studies. He 

is also an expert in developing systematic review 

processes and synthesizing the results for our broad 

audience. Having served as a project director for 

systematic review projects funded by foundations and 

nonprofit organizations, we were interested in 

improving youth and community well-being. We will 

conclude after these speakers share with us their 

knowledge with a few concluding remarks. After the 

remarks we will open up the discussion for Q and A. 

During the webinar, please feel free to ask any 

questions or provide comments in the chat box. At 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

7

this point, I will hand it over to Dr. Lily 

Zandniapour.

LILY: Thank you, Mary. Good afternoon everyone, and 

thank you for joining us today. Before our guests 

begin their presentations, Anthony and I are going to 

give you some context for today’s discussion and some 

background about the journey that brought us to this 

point. 

Just a little bit about the agency. The Corporation

for National and Community Service, as many of you 

know, is a key funder of the nonprofit sector, and 

its mission is to improve lives, strengthen 

communities, and foster civic engagement through 

service and volunteering. The agency’s flagship 

programs include AmeriCorps, Volunteers in Service to 

America or VISTA, Senior Corps, and National Civilian 

Community Corps or N triple C. These programs provide 

an opportunity for thousands of Americans of all ages 

and backgrounds to serve their country each year. The 

Edward M. Kennedy Serve America Act reauthorized the 

agency in 2009 and expanded national service 

programs. The Act stipulated that CNCS and its 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

8

grantees support programs in six focus areas, as you 

see on the slide. In addition to these six focus 

areas, the agency has focused efforts on 

organizational capacity building as well. At CNCS 

programs we work in a variety of areas, diverse 

areas, covering rural and urban areas. 

I just wanted to say a few words about sort of 

measurement, particularly output and outcomes. CNCS 

programs have used performance measure since the 

inception of the agency. These performance measures 

have been periodically updated, most recently in 

2008, and part of the then-current strategic planning 

process. With regard to impact, however, there were 

few if any reverse evaluations done prior to that 

point. Over the past decade, and in particular 

following the signing of the Serve America Act, CNCS 

began to more intentionally organize and develop an 

evidence base for a national service program and 

increase its focus on evaluation, evidence-building, 

and use.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

9

Both the agency and the grantees invest in 

significant resources in evaluation of their 

interventions and programs during this period. One of 

the things that we [unintelligible] as a catalyst for 

this evidence, focus on evidence, was that the 

legislation called for the implementation of the 

social innovation fund, which is a tiered evidence 

based initiative part of several of these across the 

federal agency. And this particular program really 

had evidence effectiveness baked in its DNA. The aim 

and the mission of that program was to scale what 

works. And so that program really served as a 

catalyzer in the agency. Sort of as, I want to say, 

as an evidence that the agency itself is a learning

organization. AmeriCorps, the agency’s largest 

program, has since become much more evidence-driven.

For example, it’s adopted a singular tiered-evidence 

framework. It’s implemented evaluation requirements 

based on funding levels, and it’s increased its 

weight on evidence for review process. It’s developed 

and rolled out evidence-based planning grants. So a 

whole lot has been happening with our largest program 

around evidence. 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

10

Senior Corps, another one of our key programs, has 

also started offering augmentation grants so that 

grantees can implement evidence-based programs. In 

addition to all of that, most of our programs, and in 

specific I want to all out CIF and Senior Corps as 

well as AmeriCorps have undergone impact evaluations 

at the funding stream level. And they’ve been put 

through rigorous impact evaluations. As for this 

agency’s use of evidence and focus on third-part 

evaluations and systematic evidence reviews, it’s 

sort of expanded. A sort of body of evidence has 

started emerging. And in order to better understand 

it, we started commissioning meta-analysis types of 

studies to really grasp what’s been going on and what 

we are learning about the evidence face of the 

programs.

At the same time, we tried to share a lot of what 

we’ve learned on our website. Evidence Exchange is a 

repository where a lot of our evaluation and research

reports are housed. We have expanded our evaluation 

resources and all of those can be tracked on the 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

11

website. We’re moving in the direction of trying to 

make a lot of these more self-paced and more 

interactive over time. And finally, we could use this 

2017 state of evidence report where we try to really 

write a summary of what we’ve learned so far around 

our evidence. 

With sort of that as a backdrop, our vision going 

forward is really to leverage all these investments

we’ve done to date. Our hope is to take what works 

and bring it to more people across different 

communities nationwide and our hope is to sort of use 

national service to support that expansion and 

growth. And to do that, we wanted to sort of develop 

further our knowledge base about what works and how 

that can be scaled up. So with that, I’m going to 

pass it on to Anthony to give you more of an overview 

of this particular project.

ANTHONY: Thank you, Lily. In 2016, the Office of 

Research and Evaluation contracted with Mathematica 

Policy Research to conduct a multi-year, multicomponent learning project that could move us closer 

to our goal. The project is designed to deepen our 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

12

understanding of agencies’ most effective evidencebased innovations and to expand the agencies and the 

its grantees knowledge base on how to successfully 

scale them. The project aims to answer a number of 

critical questions. 

Which CNCS-supported interventions and components

demonstrate the strongest evidence of effectiveness?

How do CNCS-funded organizations define and 

operationalize scaling?

How do they ensure they’re scaling their programs 

with fidelity, and the extent and reasons why they 

may adapt or modify the program?

What facilitates or hinders the scaling process?

And finally, what does scaling readiness of the 

organizational program and intervention level look 

like? As part of the project, we want to develop 

policies and practice guidelines and technical 

assistance for materials for CNCS and its grantees. 

This is knowledge that can inform the path forward to 

those thinking of scaling their interventions. To 

date, the project has generated a synthesis and 

analysis of the existing and emerging evidence 

gleaned from evaluation reports; and the next phase 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

13

will include in-depth case studies or process 

evaluations of a select group of grantees to capture 

their experiences around scaling using the scaling 

right in this framework. In addition to these indepth studies, we also hope to draw a broader set of 

experts to gather insight on the topic. Consequently, 

we plan to put together a field working group, 

potentially comprising project directors, 

researchers, federal grant makers, and foundation 

staff. And it’s our hope that we are able to share 

these learnings beyond the agency and its grantees 

with the larger grant making and nonprofit community. 

With that, I want to introduce and turn over our 

presentation to Scott Richmond and Nan Maxwell from 

Mathematica, and thank you for being here.

SCOTT: So as Anthony mentioned, Mathematica has been 

working with CNCS for the past year and a half on the 

Scaling Evidence-Based Models Project, in an effort 

to generate practical knowledge on how CNCS might 

foster the successful scaling of effective 

interventions. Anthony and Lily provided a nice 

overview of the project components and how they align 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

14

to the project’s overall goals, but our presentation 

today will focus on the evidence and scaling activity 

review components and the processes we developed to 

execute these tasks. 

The evidence and scaling plan review components can 

best be thought of as different sections of a larger 

pipeline, which are represented by the two different 

shades of blue in the graphic. The first part of the 

pipeline is designed to identify the CNCS-funded 

interventions with evidence of effectiveness. That 

is, interventions that have been found through a 

rigorous research to improve the outcomes for 

individuals receiving the interventions’ services. 

The second part of the pipeline is designed to learn

the extent to which grantee organizations demonstrate 

a readiness to scale these interventions. So by 

working to identify the interventions with evidence 

of effectiveness, and a readiness for scaling, this 

project will help support CNCS’ efforts to identify 

which interventions work and how to make them work 

for more people.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

15

So I’ll be discussing the first part of the pipeline, 

which focuses on the evidence review while Nan will 

be discussing the second part, which is based on the 

review of the grantee’s scaling plans. I’ll note that 

while the project is also designed to help CNCS and 

its grantees to strengthen their capacity to build 

evidence and a readiness to scale, we won’t be 

discussing that piece during the presentation today. 

Thanks a lot.

So the first section of the pipeline for identifying 

interventions with evidence of effectiveness is to 

collect and categorize the current evidence that 

exists for interventions that are of potential 

interest. This evidence can be identified through 

various avenues, such as evaluation sponsored by 

federal agencies, funding opportunity requirements 

where applicants must submit evidence as part of 

their application process, and then evidence review 

clearing houses that compile evidence and then review 

them against evidence quality standards. 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

16

For this project, we used the evidence sets CNCS 

compiled through its notice of funding opportunity 

evidence requirements and also the evaluations that 

CNCS grantees must perform as part of their funding 

requirements from the agency. Because the quality of 

evidence can vary, it’s important to then categorize 

that evidence, such as how CNCS uses its evidencetiered framework to rate the quality of the evidence 

for the interventions it supports. CNCS contracts 

with third-party evaluators to review grantee 

evidence documents based on the research methods that 

are used and then provide an evidence rating that can 

range from pre-preliminary to strong. An example of 

one of the categories, preliminary, would be a study 

that looks at the credit scores of individuals before 

and after they receive a financial literacy program, 

but the study would not have also included a 

comparison group to compare the treatment group to.

The moderate and strong ratings are evidence that is 

generated from studies that use more rigorous 

research design, such that if done well, they would 

provide sense of competence that the intervention 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

17

caused any differences between the treatment and 

comparison groups. So an example of these sorts of 

studies would be a study that examines children’s 

healthy eating habits among those who are assigned to 

receive a school lunch initiative program and 

comparing them to a similar group of children who are 

not assigned to receive that program. So for this 

project, the interventions we initially considered

for our review, were those that evidence with a 

moderate or strong rating. 

So the next step of the pipeline is to identify the 

evidence that aligns the project’s research 

questions. Because our project was interested in how 

CNCS’ grantees planned to scale effective 

interventions, we developed screening criteria to 

identify which evidence used more rigorous research 

designs, evidence that showed consistently favorable 

results, and aligned the scaling activities that 

grantees had for their planned interventions. The 

evidence that met these criteria would then move on 

to the next phase of the pipeline where we would take 

a more in-depth look at the quality of the evidence 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

18

itself. So the first set of criteria we developed

pertained to the design, the reporting, and the 

objectivity of the evidence submitted by the CNCS 

grantees. So for example, we wanted evidence that 

used a comparison group in its study design to get a 

better understanding of what would have happened to 

the treatment group had they not received the 

intervention. We also needed details on the study 

design and methods to get a better sense of whether 

the study was using adequate procedures when 

implementing the design that they had in place. 

The next criteria we developed is the need for the 

evidence documents to have consistently favorable 

impact findings for the program participants. And 

this is done as a prerequisite to successfully 

scaling an intervention; there needs to be some 

underlying evidence that that intervention was 

successful to begin with.

And then the last criterion we developed at this 

stage was that grantees needed to have submitted 

evidence for the specific intervention that they 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

19

sought to scale. So for example, a grantee planning 

to scale a specific parenting education program 

needed to have submitted evidence for that specific 

program itself, and not just any parent education 

program, as a general proof of concept that this 

approach could work.

So for the last step of this part of the pipeline, we 

assess the quality of the evidence based on the 

internal validity criteria we establish, which were 

informed by standards developed by existing evidence 

review clearing houses. Using internal validity 

criteria provide greater confidence that any 

differences between a group of individuals receiving 

interventions and a group that did not could be 

attributed to the intervention itself and not to 

other factors. In order to have confidence that the 

intervention caused any differences between the 

treatment and comparison groups at the end of the 

intervention, you first have to know whether these 

groups were similar to each other before the 

intervention began. You want groups that are equal to 

either other at the start of the intervention so that
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

20

you know you are comparing apples to apples, and that 

the only difference between the groups at the start 

of the study is that one of them received the 

intervention. If you don’t have this, then you can’t 

be certain that any outcome differences assessed at 

the end of the intervention were due to the 

intervention itself or maybe some preexisting 

differences that the groups had before the 

intervention even began. 

So to help make sure that the studies we reviewed 

were not comparing apples to oranges, we developed 

civic criteria to address issues that can arise and 

threaten a study’s internal validity. These issues 

could include losing a large number of sample 

participants over the course of the study, switching

participants from one group to another after the 

intervention has already begun, or not showing that 

the final treatment in comparison groups were similar 

to each other before the start of the intervention.

So the fourth criterion we developed is about 

identifying whether there were other factors embedded 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

21

within the study design, also known as study design 

confounds, that could otherwise explain the outcome 

differences between the intervention between 

comparison groups. 

So at this stage of our broader pipeline, 

interventions that had evidence meeting the project’s 

quality standards were then considered having 

evidence of their effectiveness. We then reviewed the 

plan’s scaling activities that CNCS grantees had for 

these specific interventions, which Nan will speak to 

next.

NAN: Thanks, Scott. Before discussing the scaling 

portion of the pipeline, we really need to define 

scaling. And as Lily pointed out, we are discussing 

three different types of scaling. We’re looking at 

expansion, that is surveying more people in the same 

location. We’re looking at replication, so that 

you’re serving the same population, but at a 

different location. And then we’re looking at 

adaption, that is serving a different population or 

modifying the intervention for the same population.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

22

Now we note that, when scaling an intervention 

differences often emerge between the intervention as 

it was designed and the intervention that’s actually 

planned to be scaled. Furthermore, there’s 

differences that likely arise between the 

intervention that was implemented and the evaluation 

of it or the plan’s scaling of it. Now these are 

really important differences, and don’t want to 

minimize these differences with my next statement, 

but we’re going to ignore them right now, for the 

sake of brevity and when you talk about the 

intervention and scaling the intervention, we’re just 

going to talk about the intervention as it was 

designed, or the intervention model, so to speak.

So in order to look at the scaling readiness for the 

interventions that Scott identified as being, having 

evidence, we really need to develop a framework, so 

Mathematica developed a framework for looking at 

interventions and organizations that implement them, 

and determining whether they’re ready or not to be 

scaled. So the framework, as you can see on the 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

23

slide, identifies five different conditions of 

readiness to scale. The first three of wellspecified intervention, well-defined target 

population, and implementation supports, allow us to 

assess whether the intervention is ready to scale. 

The last two conditions, enabling context and 

implementation infrastructure, allow us to assess 

whether the organization is ready to scale the 

intervention. And we’ll go through each of these 

conditions, [unintelligible].

So the first condition is the intervention being 

well-specified. And having a well-specified 

intervention means that the intervention is described 

in sufficient detail to make clear what activities 

are crucial, and then describes details, actually 

behavioral descriptions of those activities, to 

ensure that they can be implemented as they were 

intended. So I like to think about this as like a 

recipe. Someone can enjoy a cake and ask for the 

recipe, and someone can say, “Oh, it’s made with 

butter, eggs, flour, sugar, milk.” That’s great. But 

without the details about those ingredients and how 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

24

do you put them together, you’re unlikely to be able 

to replicate that cake. So just like the cake, CNCS 

grantees will not be ensured of successfully scaling 

an intervention without it being well-specified. And 

this means a grantee should have a clear sense of not 

only what those key activities are, but also each one 

of these elements about those components or key 

activities. How is the activity to be delivered? Mode 

of delivery. Must it be in person? Can technology 

substitute for in person? Can it be a combination of 

the two? How intense? How much of the component must 

a participant receive? How long should the activity 

last? How often should it be undertaken, and then how 

much of it should a participant get? What staff are 

needed to implement the activity effectively? And 

this is really critical for CNCS because knowing what 

kind of staff are needed helps them match volunteers 

appropriately to the service opportunities out there. 

You have to know what setting the activity will work 

best in, and setting includes both location like 

rural or urban area kind of thing, and it also 

includes venue. Is the implementation designed to 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

25

unfold; is the activity designed to unfold in a park, 

at home, in a school, in a classroom, whatever? So 

these all need to be defined for each activity. 

And then finally, the intervention needs to define 

the requirements for being considered a participant 

and a completer. And although each of those 

activities are core components, might improve 

outcomes for some individuals, it’s the combination 

of them. It’s the taking it together that really, the 

designer had in mind when developing the 

intervention. It’s that combination that will make 

the outcomes happen.

The second component of scaling readiness is that the 

intervention defines the target population. Now, each 

intervention is designed to work for a certain 

population. Say, the low income; say, kids in grades 

2 and 3. It might not work on other populations, 

which means that the target population has to be 

clearly identified.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

26

Now, of note in that description of the target 

population, it also has to include any criteria for 

including that people must have in order to enroll in 

the intervention or criteria that would exclude them. 

You know, for example, a program might be designed to 

include only youth that were at least age 18, but 

younger than age 25. 

The third component is that the intervention has to 

have supports for its implementation. And these 

support ensure that the intervention can be 

implemented in the way it was designed. And there’s 

six different types of support that are needed. The 

first is that the intervention needs to have a team, 

or at least an individual, who is responsible for 

monitoring it, to ensure that it’s implemented 

successfully, and that problems are identified as 

they arise. The second type of support is that the 

intervention needs to specify performance procedures 

for staff that provide benchmarks for them to meet 

when delivering the intervention. The third is that 

the intervention needs to have continuous quality 

improvement to ensure that quality is maintained 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

27

while it’s being implemented or scaled. The fourth is 

that the intervention needs to clearly identify the 

training that staff will need to implement it with 

quality. The fifth, the intervention needs to have 

clearly identified lines of communication, so that 

both staff and partners have knowledge about 

operations in order to effectively implement. And 

then finally the intervention needs to be supported 

by data systems that provide information about 

participants and the intervention per se, so that 

decision-making can be effective. 

Now let’s switch over to what the organization needs 

to have in place in order to be ready to scale an 

intervention. First, and maybe not foremost but 

certainly first, is that leaders and stakeholders 

must be on board with the scaling, and partners are 

one of those stakeholders there. These individuals 

have to provide to support. They have to promote the 

intervention and its scaling efforts, and certainly 

not put up roadblocks to it. 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

28

Second, the organization must have a culture that 

supports both the intervention and its scaling. And 

what this means is that the organization values 

innovation, learning, moving forward. It also means 

that improvements are made in responses to challenges

encountered. And there will be challenges 

encountered, that’s a given. And the organization has 

to have an attitude of ‘we will conquer these 

challenges; we will overcome them. We will improve 

and move forward and not give up’. 

And finally, the organization has to have a 

supporting infrastructure. Now what does this mean? 

It means that in scaling, it has to have sufficient 

financial resources, it has to have sufficient staff, 

materials, physical space, so that it can scale. It 

also has to have a human resource system that 

supports staff selection, training, and ongoing 

supervision.

And finally, let’s just close the loop that Anthony 

started, 15 minutes ago, 20 minutes ago. The 

project’s been going on for about a year and a half, 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

29

and we’ve really made some tremendous progress, 

although on any given day... Specifically, we reviewed 

and evaluated research evidence from CNCS grantees, 

and that has allowed us to identify intervention that 

CNCS funds that have demonstrated effectiveness. Then 

we’ve also, as you’ve heard, developed a framework by 

which to identify interventions and organizations 

that are ready to be scaled. We’ve applied the 

conditions for scaling to say which effective 

interventions and organizations might be ready to 

scale, and then we’ve selected organizations that are 

currently scaling effective interventions for indepth study, the case studies that Anthony talked 

about. 

So what’s coming up? Preview here. In the next few 

years, we’re going to continue the learning and 

knowledge development here. We’re going to conduct an 

in-depth study of the selected organizations to 

understand their scaling. We’re going to use that 

information that we gather from these key studies to 

identify the factors that challenge and facilitate 

scaling. And then we’re going to disseminate findings 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

30

widely to a wide variety of audiences. We really, 

really, really want CNCS and its grantees to make 

practical use of the research. You know, the goal is 

to foster successful scaling of effective 

interventions, and with that comes improved lives, 

strengthened communities, and a better civic 

engagement.

And finally, I’m giving some prompts here for Q and 

A. You’ve heard our thinking about evidence and 

scaling and we’re real curious about your thoughts 

about what’s going on out there. So during the Q and 

A period, we’re hoping that you can think about and 

provide discussion on what role does evidence play in 

your organizations. And what indicators does your 

organization use to determine if it’s ready to scale? 

So with those thoughts, I’ll turn it back to Mary. 

Thank you.

Mary: Thank you very much. Do Anthony or Lily have 

any additional comments?

ANTHONY: I just want to say thank you for the 

presentation and for the work you’ve done on this 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

31

project, and I want to highlight that this project 

and a number of other projects, this one in 

particular, has really been the result of a whole 

group of people coming together and thinking about 

it, both in the moment and over time. And Diana 

Epstein was a factor in bringing this project to 

light. I’ve had the pleasure of working with Scott 

and Nan on getting this implemented and Lily’s been 

tremendously valuable to me in terms of insight or 

scaling from her experience with CIF. So I just want 

to point out that that collective effort really kind 

of shines a light on the notion that as we move 

forward, we’re really developing a very visible 

notion as a learning agency, that we’re taking the 

information that we get and we’re looking at it in 

terms of where it applies and what more information 

is needed for us to make very conscious decisions 

about where we go and what we do, both with national 

service and generally, with a lot of our grant 

making. So, thank you.

MARY: Great. Thank you. Before we open this up to Q 

and A, I would also like to introduce Amy 

[unintelligible], our program examiner within the 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

32

Education, Income Maintenance, and Labor Division at 

the Office of Management and Budget. We really 

appreciate you being here today and I would like to 

give you the floor for any comments you’d like.

AMY: Thank you, first and foremost, to our 

excellent panel. [unintelligible] the corporation, 

CNCS, for allowing me to be here. You know, something

you said, Nan, about CNCS being a learning agency; 

that really speaks to the president’s management 

agenda that was released in March of this year. 

Specifically, the objective of federal government in 

the 21st century will be to provide high-quality and 

timely information to inform evidence-based decision 

making in learning. So that is the objective of being 

administration writ large. But then also just as 

great stories of taxpayer dollars going into the 

future. So that said, we totally support CNCS and are 

looking forward to working with you and supporting 

your efforts.

MARY: Great. Thank you so much. With that, I’m going 

to turn it over to Q and A, and I think Andrea will 

sort of help us walk through how that works with the 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

33

technology as well as folks in the room, because we 

want your thoughts.

ANDREA: We’re going to try it. So for our in-person 

audience, just please state your name before asking 

your question. And for our online audience, you can 

ask a question by typing it in to the Q and A box, or 

you could also use the microphone on your computer by 

selecting the raised hand feature from the menu 

above, and we will grant you microphone rights in the 

order the questions come in. So, anything else, 

Emily?

EMILY: No, I think that covers it all, and if anyone 

online wants to ask a question, just please make sure 

you speak directly into your computer’s microphone.

ANDREA: So could we go back to the two questions you 

two had posted and just as prompts? But of course, 

any questions are welcome. And we do have one here 

from the chat that I’ll just read to get us started. 

So this is from Allison Augustin. “Hello, would love 

more detail of what is considered a confounding 

factor. We know that isolating the impact of the 

AmeriCorps member within a complex environment, such 

as a school, can be challenging.” So any thoughts on...
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

34

MALE: Yeah. Great question. An example of 

confounding factors would be where you’re unable to 

disentangle what the intervention might be doing 

versus just the context in which it may be 

implemented and could otherwise explain outcomes. So 

I think one example to illustrate this is you are 

trying to assess an education program that teachers 

were putting a new curriculum in the classroom, but 

your treatment condition; it was just novice teachers 

and then in the control condition, it was just more 

experienced teachers. Because of those differences 

and who was implementing the intervention, the one 

group that then the control group, would have been 

qualitatively different from the other and you don’t 

know if any differences you would see is because of 

the intervention, or just because of the differences 

in the teaching experiences that these two groups 

have. So that would be an example of a confound, 

where you can’t really attribute any outcome 

differences to the intervention versus those other 

factors that were inherent in the study design.

MALE: [unintelligible], the Congressional Research 

Service. So I don’t have any special expertise in 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

35

grants, but I understand that many of them are 

structured to last three years or five years, and one 

question that comes up occasionally is that when a 

grantee takes up an intervention, it takes a while 

for the intervention to mature, raising the question, 

do we have a valid construct here of what the 

intervention is as practiced, as they use an impact

evaluation to look at it, they also use CQI and 

probably a variety of evaluation and research methods 

to kind of think about their process. So can you talk 

a little bit about just maturity of the intervention 

and kind of the glide path that occurs?

MALE: Yeah, absolutely. And I think your question 

really touches on a lot of different things. One, it 

[unintelligible] the timing of the impact evaluation 

itself, knowing that some interventions might need a 

longer runway to really say, “We’re finally here with 

the model we’re trying to implement.” And if you do 

that impact evaluation too soon, then you’re going to 

prematurely get impact that you would otherwise had 

gotten had you waited. But I think, touching on what 

Nan spoke about too, it also speaks to the need for 

those sorts of implementation supports, in terms of 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

36

the intervention specification and readiness, so that 

knowing that you may need a longer runway to get that 

model and put it into practice, that’s why you need 

the CQI processes. That’s why you need an 

implementation monitoring team to identify that those 

sort of issues are occurring and how do you address 

them over time, because without that sort of 

implementation supports, you might think, “I am doing 

the model as it was intended,” but you may not 

actually know that because you don’t have the 

supports in place. You don’t have the data systems 

that have that feedback for those. So it sort of 

speaks to the need of, knowing that in advance, those 

are sort of challenges you might experience, so to be 

ready to scale, you want to do as much as you can to 

have those in place before you embark on that 

journey.

MALE: So can I ask you a quick follow up? How does 

this technical assistance play into that? Because in 

medicine, you have doctors kind of gliding on the 

same path with you as a patient, doing 

[unintelligible] time series. You know, how does this 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

37

feel? How is this working? And so, kind of like, for 

CNCS or for other helpers there, financially?

FEMALE: I just wanted to say that I think that 

what you say is quite valid. It’s something that CNCS 

would be grappling with. Right now, we’re trying to 

sort of get more feel for how these interventions are 

being scaled up and how grantees are approaching 

them. Because there’s going to be some change that 

does happen as you try to replicate and expand. And 

sort of getting a good grasp of where those 

interventions have veered off of what was supposed to 

be to the extent that they’re not change; are they 

really being implemented with [unintelligible]? All 

of those sorts of aspects come into play. I think 

what we’re trying to do now at this stage is gain 

enough experience and tap what’s happening on the 

ground so we have to better position to do the 

support that you were talking about. So I think we’re 

still a bit away from that, but we know that a lot of 

organizations and programs have started on this 

journey, so they have some experience that we want to 

tap into, and then all of those things come into 

play.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

38

FEMALE: [unintelligible] My question is more in terms 

of the frame, specifically the framework for the 

scaling. The first thing that comes to my mind, with

experience of community work is that of adaptation or 

modification of evidence-based program. But at least 

scaling, right now, to me, sounds more like size of 

this number [unintelligible], this number. But 

usually, and you actually addressed this also, the 

scaling is not so simple as just numbers because 

different times, it’s different kinds of individuals, 

[unintelligible] they are the range of modifications 

can be so wide, really. So I don’t know if it’s 

scaling any more [unintelligible]. Maybe what I’m 

trying to get at, why is scaling as important as 

probably all the factors that may have to do with 

different kinds of modifications? Is scaling in terms 

of size, the most important factor when you modify or 

try to implement an evidence-based program or is 

[unintelligible] other aspects, even with the 

framework that you mentioned, I am thinking, “Okay, 

it’s staff.” Staff can be numbers. Three is staff, 

five is staff. But staff can also mean the experience 

the kind of background that they have. And then, if 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

39

you think about grantees, usually they implement 

something based on their own experience 

[unintelligible], but they’re not really detailed of 

evidence-based programs necessarily, about you need 

three staff per the ratio, you know. Five per 

[unintelligible] and they need to be experienced in 

these. So how [unintelligible] the need 

[unintelligible] is so much that I’m thinking about 

what framework were you talking about. The kind that 

meets [unintelligible]. 

FEMALE: You know, you’re absolutely right. It’s 

not; the framework’s simple, real world is not 

simple. And you touched on a lot of the complexity in 

your question. I think the point, or one of the 

points, is that you really have to be mindful of what 

you’re doing. You know, you really need an 

intervention that you know what it’s supposed to look 

like; you know where you’re starting from. It’s welldefined, the population is well-defined. Yes, you can 

adapt it. Most developers of interventions allow for 

some adaptation, you know, of it. That’s without a 

doubt. But you have to be mindful where it is, and 

then when you’re going out from that, you have to 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

40

really, again, be mindful. Is there evidence? What 

have you got? What evidence do you have to suggest 

that you’re going to be successful on this new type 

of population? What evidence is out there? And yes, 

it’s not going to be evidence that’s exactly on your 

intervention, but maybe it’s a reading literacy 

program that similar programs have been effective on 

this new population. Or similar programs have been 

effective in this new area. You just have to be 

mindful and think through how it’s going to be 

adapted and what evidence is out there and what 

evidence is out there that there are going to be 

challenges and can you overcome them? It’s not as 

simple as the diagrams, absolutely not. But 

purposeful and mindful are words that come to my 

mind.

FEMALE: There’s a question down here as well.

FEMALE: Yeah, I’m sorry. I’m going to take a few 

from the chat. So Eric Cruz says, okay, sorry. She 

wants to know if CNCS is moving into evidence-based 

research, will all programmatic grants as a senior 

primary core be required to move into higher 

evidence-based tiers. And if so, are federal and 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

41

direct [unintelligible] be considered research and 

will it be of a higher level needing less funds for 

operating a program? How will that work? Will we be 

able to apply for additional funding to cover that 

cost?

Finally, moving programs into research means that 

grant will be needed to be submitted to the IRB which 

can take two months. Will there be enough time for 

that? So just questions around how this is funded.

MARY: So I would say I’m not a program person. This 

is Mary Hyde. I wouldn’t venture to speak to 

programmatic policy issues but I would say that I 

would hope that some of this information would be 

used to inform the ways in which we structure our 

grants, the ways in which we guide the investments, 

and certainly the AmeriCorps [unintelligible] 

national grants have three years to conduct an 

evaluation, so certainly building in time for the IRB 

is something that could be accommodated by that time 

frame. But from an agency perspective, I think that 

is the space where we start to speak more

intentionally with the programmatic side of the house 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

42

and say, “Okay, what are the implications of this?”, 

or how we fund or how dollars are allocated.

FEMALE: So we’re going between the live audience 

and the chat. So now the live audience.

FEMALE: Hi, this is Laurie Frank [unintelligible] 

Policy Fellow, and I’m interested in the relationship 

between interventional readiness and organizational 

readiness and your plans for refining your framework. 

So I was very interested to see that some of your 

next steps are to look at facilitators and challenges 

to scaling, which is terrific. But wanted to assess 

your level of optimism that you’ll find generalizable 

principles; you know, how much is idiosyncratic to 

given conditions and how much really can be 

generalized, and then what your plans are for feeding 

that back into your framework.

FEMALE: Great question. And since we haven’t 

started yet, I can have all the optimism in the 

world. And I do, actually. The framework wasn’t 

developed in isolation. It’s grounded; very, very 

heavily grounded actually, in implementation science. 

And so it comes from a strong literature base. That 

said, like I said in my discussion points, we’re 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

43

going to face challenges and we will have to feed 

those challenges back to the framework, absolutely. 

So I don’t know. At this point we’re not heading to 

the field until the fall. I think we’ve got good 

instruments in heading into the field that will 

capture, that have been designed based on the 

framework, as well as the nuances of the 

organizations and interventions selected. So I think 

we’ll get the information that will allow us to go 

back and reconcile everything. I don’t know. 

Personally, I hope there’s not a lot of 

reconciliation, but I can say that because it’s only 

June. Check back. Check back next year this time. I 

may have a completely different answer.

FEMALE: We’ll do a follow-up in a year.

Mary: Yes.

FEMALE: So why don’t you go ahead and then I’ll 

take a few from the chat.

FEMALE: Okay, great. Hi, I’m [unintelligible]. 

I’m from the Office of Evaluation Sciences at GSA. 

Thank you very much for your presentations. My 

question is, what we do is we evaluate different 

program changes and my question is when to continue 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

44

building evidence versus when to suggest implementing 

or replicating or scaling. So are there criteria you 

would use around determining recommendations for 

those different types of scaling?

MALE: So just in terms of continuing to fund or 

taking what’s to scale? Can you clarify that?

FEMALE: Well, continuing to study something. So in 

different contexts. Like, how do you decide when it’s 

appropriate to replicate something in a different 

context, with different populations versus it’s 

appropriate to go ahead and implement it or decide 

whether you want to replicate the study or scale it

MALE: Gotcha. Thank you. So I mean, I would say, if 

your starting point, what you’re doing doesn’t have 

evidence of effectiveness, then you shouldn’t be 

trying to scale until you have that underlying 

success. Because you’re trying to either adapt to a 

population or replicate/expand. You don’t want to do 

that if what you’re initially doing is not working. 

So I think from there you would go back to your logic 

model or revisit what is it that you are doing, and 

kind of do some adaptations at that sense to really 

go back to the lab, and see if I give more dosage, if 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

45

I add these components or less of that. So I think, 

as Nan was saying, a lot of our model is based on 

implementation science and may just be revisiting

some of these same exact parameters where maybe I 

need to adjust my core elements. Or I could think 

about who the staff was. How many of the staff? Am I 

doing the right implementation support? So I think 

you can think about if you’re not having that 

underlying evidence to begin with, kind of revisit 

what is my model and then keep working to develop 

more evidence. But we would say you wouldn’t want to

try to bring it on the road until you know what 

you’re doing in your local context or what you’re 

trying to do [unintelligible].

FEMALE: I would say, assume that there is underlying 

evidence for something. But there might be less 

evidence for specific population or a specific 

context. At what point would you feel comfortable or 

have defined criteria around when would make sense to 

continue studying something versus...

FEMALE: This is Lily Zandniapour. I just wanted to say 

that we see the evidence-building piece as an ongoing 

process. And don’t really see this. It’s an iterative 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

46

process at each stage it might be using a different 

frame, a different approach, different methodology 

based on the question you are trying to answer. But 

really, this process builds on each other. I don’t 

see sort of a particular end point to this. So I 

think evidence building and evaluation sort of is a 

kind of that river that runs through it. So you want 

to do that even post establishing this strong 

evidence-based program. You might be looking more at 

the fidelity of implementation. Your questions might

be pointed at that at the first phase of replication 

or expansion or adaptation. And then continue to 

build that evidence of effectiveness, because it is 

always, at the end of the day you want to sort of 

strengthen that body of evidence. So I just don’t see 

that as an end game to this.

FEMALE: I would add, also, because I think I hear a 

little something different in your question. Part of 

what the next phase of this project is, by going into 

the three case study sites, is to learn intentionally

from folks who’ve had varying experiences and success 

with scaling. And not only from the evidence 

perspective, but what was it that made people decide 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

47

to scale? What happened along the way? So people have 

decided to intentionally scale back, and why is that? 

It’s both a programmatic decision as well as an 

evidence decision, which is just a little something I 

heard also in your question. So we hope to learn from 

them what those things are.

FEMALE: Okay, so a few questions online. This is from 

Jennifer [unintelligible]. I’ll just read it to you 

then because a number of [unintelligible]. Are the 

conditions for scaling shown in the scaling framework 

based on the theoretical model or based on findings 

from the review of grantee-generated evidence?

FEMALE: That’s easy. They’re based on, I don’t know 

that you’d call it a theoretical model, but it’s a 

based on scaling that has been done in other areas, 

the principles that have been derived from those 

other areas that the framework was actually 

developed, mathematically developed, the framework 

before this project, so...

FEMALE: So this is from Tina [unintelligible]. Can you 

provide some examples of the programs supported by 

CNCS grants that were part of the research conducted 

to date?
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

48

MALE: What was the question?

FEMALE: Can you provide some examples of the programs 

supported by CNCS grants that were part of the 

research conducted to date?

MALE: Well, when we did the meta-synthesis metaanalysis, we were careful to look at what focus areas 

were addressed by the program. So we recognized that, 

while we run six [unintelligible], we have activity 

in those six areas. We really were producing evidence 

in three of them to an appreciable degree, and one of 

them to a very appreciable degree, and that was 

evidence. So many of the programs that we looked at 

had to do with education interventions. Secondary to 

that were economic opportunities and those were backto-work programs or budgeting programs, fiscal or 

financial learning programs, and that sort. And in 

the evidence area, the things we looked at spanned 

pre-K all the way up through college readiness and 

college completion. So we had evidence in all of 

those areas. But the strongest ones emerged, the ones 

that we looked at hard for selecting the process 

evaluations, were primarily in health, I mean 

education, and then healthy futures. And they had to 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

49

do with mostly pre-K and early childhood learning and 

then healthy futures, again around early childhood 

development, child abuse prevention. Those are the 

programs that emerged with the strongest evidence and 

also provided us with a great deal of variety with 

regard to their scaling experiences. We didn’t want 

to pick three successful scaling projects; we wanted 

to pick projects that had varying degrees of success 

and had to retrench and rethink going forward what 

they were going to do so that they could teach us 

something about that whole process. But in terms of 

the specific grantees, no, we’re not at a point where 

we want to release those just yet, because we have to 

announce who we selected for the process evaluations 

and then also who we’re going to approach to help 

serve on our field working group. And that’s not 

public knowledge yet.

FEMALE: Thank you. I’ll take one more from here and 

then I’ll open it up here. So Diane Manheim, this 

model seems to give preference to research-based 

interventions rather than field-based success. Will 

this limit interventions with promise that are 

considered?
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

50

FEMALE: Well, I guess almost by definition, the answer 

is yes. It’s yes. There’s, we’re only looking at 

those with evidence. I don’t know what to say. Yes. 

But what I wanted to say was, actually, the mechanism

that we’ve created with this field work is to draw on 

a broader set of experiences. So we have the ones

that are promising also feeding into this. But you 

want to start with a foundation of what has had some 

evidence that it works and has been sort of tested in 

some application and expansion processes. 

MALE: I would also like to add that while 

interventions are not equivalent across all of our 

grantees, neither are evaluations. Grantees choose to 

do an evaluation and have a limited budget; they have 

a limited time frame and they approach it from 

different perspectives, and that produces the types

of evaluations that we have to look at and determine 

the strength of that. And those evaluations may find 

very positive impacts, and that may very well be 

true, but they may be missing components in the 

explanation of that finding that limit our really 

embracing that as strong evidence. So part of this 

project is designed to identify what are those key 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

51

components that need to be looked at in an evaluation 

regardless of who’s doing it, they need to consider 

these components when reporting that, so that when we 

look at it, we can make a fair judgement across all 

of them. And that’s part of the learning process 

that’s going on now.

FEMALE: And can I just add, taking it, if I can have a 

little latitude here, taking it out of the CNCS 

framework and into the broader evaluation and scaling 

footnote, but just for a sec; you have to start 

somewhere with this process. You know, CNCS has to 

start somewhere; the evaluation world has to start 

somewhere. And I don’t think anyone, I’ll let you 

guys speak for yourself, is saying that evidencebased and then you go into this scaling and apply the 

framework for readiness for scaling is the be-all and 

end-all right now, but it’s a good, solid place to 

start. And that doesn’t mean that once this pipeline, 

it’s no longer on the screen, is followed, that you 

can’t start burying the pipeline. And it’s sort of 

just like building evidence and scaling; this process 

is much like it. You start somewhere and then you see 

if you can go in a different direction, just like 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

52

scaling takes you in a different direction. And I do 

think this is a solid place to start this process and 

this thinking.

MALE: So this is Clint again from the Congressional 

Research Service, and by the way, I’m speaking only 

for myself now, not for my agency bosses. But maybe 

building on that observation and question; so the 

tiered evidence model sets up kind of a troika, 

promising to proven – I don’t like the term proven –

but I wonder with your case studies, if you might 

find if the model creates perverse incentives for 

grantees to go for the money, rather than going to 

get stuff in the strong category instead promising. 

Because it’s almost like they’re self-diagnosing and 

they might pick an intervention off the shelf that 

may not be fully suitable for their circumstances and 

I wonder if your case study approaches might be 

looking at that, to the extent that that’s a 

financial phenomenon out there. 

FEMALE: Hopefully we’ll get at that. Hopefully the 

case studies will provide some insight. Again, it’s 

case studies. It’s not like we’re doing millions of 

them. But hopefully; what you raise is something that 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

53

I do think about a lot, perverse incentives. And it’s

a great question. We’re on the lookout for it, yeah.

FEMALE: Another thing though, I want to say is it also 

offered opportunity, because if you really have 

programs that solve challenges in the communities and 

you have a lot of evidence that they do, then you 

want to be in a position to bring that to communities

that don’t have access to those solutions. So I think 

part of this is moving us in the direction where 

you’re attempting to; because at the end of the day 

the goal is not the evidence, the goal is that...the 

goal is to solve challenges on the ground, and if we 

have solutions, we want to be able to bring them 

where they’re needed. And then that would allow for 

us to also have that ability to innovate and support 

that innovation and promise while you’re really 

trying to bring solutions where you know that there 

is merit to those solutions. So I think it’s 

straddling these lines and always being aware that 

you have to be intentional and look at it and make 

sure you’re not creating those perverse incentives 

and you’re sort of being stewards of taxpayer money, 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

54

I think as far as we’re concerned, being in this 

position.

FEMALE: Hi, [unintelligible] OMB. Yeah, this tension 

of funding the evidence, this is more a general 

comment, but funding the evidence across several 

billion dollars, you know, 1800 different federal 

programs, I don’t think anyone including the 

Corporation is at the exclusion of funding innovation 

grants. Considering the spectrum that the tiered 

model, it can’t stand to reason that everyone’s going 

to put the money into what works and not perhaps fund 

the grantees that are still learning and tweaking and 

experimenting. So to perhaps counsel the grantees on 

the phone, what have you, yeah. This is a spectrum 

and there’s [unintelligible] innovation grants, and 

then those that are higher up the tier. 

FEMALE: Oh, two more questions. 

FEMALE: [unintelligible] with [unintelligible]. I was 

wondering; what exactly is the purpose of this study? 

Is it to test this model of scaling? [unintelligible] 

something that was true, you have to start somewhere 

and that program in mentioning that there were case 

studies that will be considered. If we have to start 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

55

somewhere, because I also heard the description of 

this study, so the interventions that have been 

considered and they are pretty different from each 

other [unintelligible] mental health and that. So if 

we need to start somewhere, a framework that we want 

to test, assuming that this is really the purpose of 

this study, wouldn’t it be wiser...again, it’s probably 

a question of [unintelligible] methodology...to focus 

more on, for example, case studies that are similar 

somewhere so that you have preliminary information on 

a framework that is focused on some kind of specific 

content or population, rather than just going across 

very different interventions to see a framework with 

information about its effectiveness based on three 

interventions that are so different? You see where 

I’m going in terms of focusing the study to really, 

what are we going to learn about it if we have to 

[unintelligible] if we’re all so spread out?

FEMALE: I just want to say, your point is well taken. 

But I think you have to kind of think about the 

context a little bit. So CNCS works across various 

focus areas. In that sense we are unique. Like, we’re 

not Department of Education. We have national service 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

56

so our base is a bit different. So they have a broad 

spectrum of programs that we fund across the 

nonprofit sector and I think that’s partly what has 

made us sort of venture into different directions. If 

we were the Department of Ed, probably started with a 

much more narrower thing. But this is not to negate 

what you’re saying. We are sort of starting, as you 

said, somewhere. And I think as we learn more, we’ll 

decide what is the best strategy to go forward, 

because at this point, we have looked at what we have 

in terms of an evidence-based, we’ve looked at the 

strongest solutions that we’ve funded. And we’re 

looking at them as a test case to see what we’re 

learning about getting them, again, to 

[unintelligible] across the country. It doesn’t mean 

we will not want to look back and we think this as we 

learn more.

FEMALE: Can I also add that I don’t see this as a 

testing of the framework? I see the framework as sort 

of setting parameters for what we’re looking at. I 

totally agree if we were testing a framework that it 

would be better to have interventions that are pretty 

similar, but maybe vary the context in which they’re 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

57

unfolding, or something like that. I don’t see it as 

testing the framework. I see the framework as 

providing structure to what we’re doing and then 

feeding off of what really says is really trying to 

provide practical, very practical, I hope, 

information to CNCS and the grantees about the 

challenges and what are the challenges you’re going 

to face when you scale and what can help you overcome 

them. And given the fact it’s practical knowledge, 

and the array of programs to CNCS funds, we really 

different want to get a broad view.

FEMALE: Okay, so I’m just going to take a couple more 

of...go ahead.

FEMALE: Hi. This is Laurie Frank, [unintelligible]. 

Part of what I think some of us are wondering about 

is your sampling frame, more the selection for this, 

and I’m interested in your orientation toward or your 

tolerance for modifications. The butterfly question. 

At what point does scaling become innovation because

things have changed so much in the course of scaling?

MALE: Yeah, and I think the way we set up our 

project is in hopes to tackle that, and I think one 

of the things Nan talked about in the framework is, 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

58

it’s not just what are the elements, but what are the 

core elements of the program. So something of the 

intervention. So at what point are you deviating so 

far from that that you’re really scaling something 

entirely? But then, if you are making modifications

or adaptations, which of course are necessary when 

you’re bringing new context. You might have to do 

that. But are you staying true to maybe some of the 

core essential elements of the intervention to stay 

grounded in the underlying evidence for it, or are 

you deviating so much from it? And if you happen to 

deviate from it, are you doing it in a way that, as 

Nan said, that’s purposeful and that’s mindful? Are 

you bringing in evidence or other underlying 

justifications to say, we are changing the dosage 

which existed for the original model but we feel 

confident because we’re bringing in this evidence to 

support [unintelligible]? So the way we reviewed some 

of these documents is what is the adaptation. But we 

also wanted to collect information of did the grantee 

justify or did they provide a rationale for why 

they’re doing it, too. So we were very mindful of 

that, that adaptation is sometimes necessary and it’s 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

59

also, it happens, but are you doing it in a way 

that’s at least informed by evidence or for some 

other rationale, too, and staying hopefully true to 

the original model but knowing maybe you’re tweaking 

around the edges?

FEMALE: Does that answer? 

FEMALE: Okay.

FEMALE: So just a few from online. Vicki Jones. She 

wants to know about the best practices for 

identifying quote/unquote organizational readiness 

for scaling and she points to slide three and she 

said it seems fuzzy; qualitative rather than 

quantitative. So it’s more of a remark. 

[unintelligible] move to that slide, thirteen.

FEMALE: Slide three or slide thirteen?

FEMALE: Thirteen, I believe. 

FEMALE: It’s more of a comment.

FEMALE: It is more of a comment, and I agree. I’m 

trying to think of a succinct way of answering the 

comment or addressing the comment. I guess the 

succinct way of saying it is underlying each of these 

bullets are sub-bullets, and then sub-bullets to the 

sub-bullets. And so when we say leaders and key 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

60

stakeholders and partners support the interventions, 

we’ve got under that operational definitions of what 

that means, and then sometimes under those 

operational definitions, it gets more specific. So 

what I presented is sort of the first line, you know, 

very high level and very astutely you’ve noticed that 

there’s a lot that goes under this. 

FEMALE: So just moving on to two more. Sandy 

Schiffers. If the intervention keeps changing, isn’t 

it impossible to conduct an RCT or even a QED? I 

think back to adaptation.

FEMALE: I do think that this level RCT experimental 

designs that is as many of you know, come from 

national sciences where the situation is much more

controlled, and I do that that if a program is making 

a lot of adaptations and tweaking and refinement, it 

is not at point where an RCT or a quasi-experimental 

design should be done. It typically has to, the 

program has to be mature and stable enough with a 

number of other qualifiers like many things need to 

be well-defined to make that a worthwhile endeavor. 

And as you all know, some of those have some 

[unintelligible] are fairly expensive, so you don’t 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

61

want to just do it willy-nilly and you know you want

to do it at the right time. So I want to acknowledge 

that and say that that is not an appropriate time if 

the program is not ready for an impact evaluation 

that’s fairly rigorous, I don’t think that it should 

be done. Also, there are certain programs that are 

just not constructed for this type of design, and you 

have to acknowledge that and move forward and sort of 

use the best methodology for studying a program based 

on its sort of stage of development, if that makes 

sense.

MALE: I can add on to that. It also mentioned this 

nuance, too. It brings up the nuance of the program 

model and then that for which was actually evaluated. 

And you want to make sure that you’re specifying, 

that you might have started [unintelligible] be over 

here with the model that we’re testing, but if you 

made so many modifications, then you’re not actually 

testing that anymore; you’re testing the adaptations 

and having clear specifications that, you know, we 

started with this teacher practice program but 

because of these modifications, the evidence is now 

for those modifications. But then, has the study done 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

62

well because of all the changes along the way, too. 

So it also speaks to you really need to be really 

clear about what you’re actually studying both in 

terms of the original conception but then actually 

what happened in practice, because what happened in 

practice, that’s the evidence.

FEMALE: So, in terms of models, I’ll read this and 

I’ll go to the live audience. This is 

[unintelligible]. Hello, I’m looking for examples of 

high-quality evidence models and evaluation models, 

methods, and scaling projects specifically for 

environmentally-based interventions. There are great 

examples that involve social services and educational 

interventions, and how populations of people are 

affected, but few related to the environment.

MALE: I’m sorry, I was...

FEMALE: But there’s you in terms of environmentallybased interventions, are there any examples that 

mostly they’re in education and social services.

MALE: It’s true that we looked at the strongest 

programs we have with regard to both their evidence 

and also their activity in terms of the, you know, 

when we look at the grants, they’re scaling up or 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

63

they’re scaling out or they’re scaling deep. And they 

provided us with an opportunity to look at some 

variation and to pull some knowledge from that. With 

regard to a lot of our programs, particularly in the 

conversation area, yes, we want to look at different 

programs and scale them. But the first question you 

might ask is, scale them in what direction? Or how do 

you want to scale this? Is it scaling up? Is it 

scaling in one particular focus area or is it scaling 

in another focus area? Such as, or using that focus 

area rather to measure scaling so it’s just capacity 

building. And we’ve spoken about this. There’s a lot 

of ways to do evaluations and a lot of ways to look 

at what those outcomes are, and then study those 

outcomes in a rigorous way. And I think a lot of 

programs are just learning that. And we’re hoping 

that this project will identify those key elements 

that all programs can use. In response to some of the 

other statements that were made about whether or not 

an intervention changes so much it’s hard to measure 

it, yeah. Certainly that would be a concern, but one 

of the things we certainly know is that if you scale, 

you need certain components to be present to do that. 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

64

The organization wanted the intervention well-defined 

and all of that stuff. And I think in some areas some 

of our grantees are going to have to think hard about 

how they define their intervention and look at that; 

what does that mean? And then, what are ways of 

measuring that out from there? So we’re hoping that 

this applies to all of our grantees across all of our 

focus areas, that key elements will be relevant to 

them as well. And hopefully, we can provide some 

guidance with regard to where they might go.

FEMALE: Okay, so Leticia Braga. Have efforts been made 

to compare the evidence standards across agencies? 

For example, the Department of Education uses the 

terms moderate and strong evidence, but the 

definitions include different components. What do you 

see as the balance between adapting definitions to 

the needs of specific agency area of focus and having 

enough commonality to develop a common vocabulary?

FEMALE: I can take that one. So yes, I think this is 

something that we are doing and have done. We had 

actually tiered evidence work group that was meeting 

for a while on specific [unintelligible]. This is one 

of the questions that came up, and something that 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

65

members of the group discuss on a regular basis. I 

think the group’s in a little bit of a hiatus, but 

you know, we’re thinking about potentially some 

additional meetings particularly around this very 

question and I think it is a balance. So there’s 

certainly some desire to standardize and if you look 

at some of the recent legislation that is included to 

your components, in one way or another, you’ll see 

that those definitions are starting to converge and 

that new programs are sort of referred back to 

definitions that are already in statute. So we 

definitely are seeing that. I think at the same time 

there’s recognition that the evidence base is not the 

same in all areas, that there are certainly some 

policy areas in which we have very little evidence 

and the priority should be on building that evidence 

base; whereas in other areas, the evidence base is 

much more well-developed, and that might be more 

amenable to [unintelligible] tier structure with that 

higher level. So I think it is a balance. On some 

level, we want to set the aspiration high, and in all 

areas we would hope that the evidence base is 

developed sufficiently to eventually get to that 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

66

highest level. And we’re also recognizing that it 

does vary quite widely, but appreciate the question 

and I think it’s something that is on quite a few 

people’s mind these days.

FEMALE: [unintelligible] NIH. Maybe I’m going further 

back in the pipeline; I’m not completely sure how 

CNCS’ grant mechanism works, exactly, but I was 

curious specifically about the health focus programs 

and grants. Does CNCS consume information from NIH, 

for like aging and early childhood type of work? Is 

there a dialogue already going on?

FEMALE: I wouldn’t say it’s ongoing. We have, 

particularly in the space of again we have had 

conversations and enjoyed some of their sort of 

retreats, if you will, with some of their researchers 

with the overlap with Senior Corps, so there are 

occasions where we try to sync up and certainly, in 

the example of the Senior Corps, augmentation grants, 

and I think we’re referenced early on. There is a 

list of NIA evidence-based programs that they select 

from. So wherever we can find those natural 

synergies, we try to definitely do so. In early 

education, I would say there’s been more tenuous 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

67

conversations. We’ve had stronger conversations at 

IES and the Department of Ed in the prior 

administration than we’ve had in the current, but 

it’s always an aspiration.

FEMALE: Okay. I was at NIA before and I remember 

meeting with [unintelligible].

FEMALE: Yes, exactly.

FEMALE: ...so I was curious how it is in the other 

areas.

FEMALE: Alright, so Mary Newton says, I’m not sure I 

understand what you mean by [unintelligible]. It

sounds like a fancy name for expand. Can you clarify 

this term a little bit more?

FEMALE: Okay, where was that slide? Seven? Bringing 

that up. Eight. So expansion is certainly one form, 

but we have put definitions to scaling and your 

question is spot-on in that the first thing you have 

to do is define scaling and people define it in very 

different ways. So expansion is one part of it, but 

we set the definition of expansion to more of the 

same chart of population in the same area, in the 

same location. So you’re just sort of growing your 

business, so to speak, where you are. But then we do 
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

68

go beyond that and talk about replication, where 

you’ve got the same program in a different location, 

maybe a different organization, wherever, with the 

same population. And then the adaption which is the 

one I think that we’re all struggling with because it 

is modifying the intervention or modifying it for a 

new population. That really is the heart of the 

discussion, and when does it become a new 

intervention? So we do have these three definitions. 

They’re not quite standard, you know. Across the 

field, people do define it differently, but this is 

what we invoked.

FEMALE: And then Allison Augustin actually has a 

question that is a bit of a follow up. What are the 

indicators City Year uses to assess whether we can 

scale our program as marketplace demand? By that I 

mean the extent to which our school district partners 

are requesting our services and willing to invest to 

support them. Being able to respond to district 

needs, share a strategy, and show the effectiveness 

of our program allows us to do that, but scaling 

won’t happen without the market quote/unquote demand.
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

69

MALE: Yeah, absolutely. I mean, I think the 

[unintelligible] local context itself. You don’t want 

to bring an intervention where there may not be a 

need for it. If you say you want to expand but 

there’s no more population to expand to, then that 

wouldn’t be a successful effort. So in addition to 

not just the demand, it’s also the support that might 

exist, too. [unintelligible] specifically on the 

intervention readiness and the organization 

readiness, but that’s not to say that the local 

context and reading where you’re trying to expand or 

replicate or adapt, that also is very important to 

the success of the scaling, too.

FEMALE: And I also add that I think the local context 

and demand for the intervention does affect whether 

you scale up or scale down. You know, we’re using the 

word scaling, but scaling down is captured in local 

demand.

FEMALE: Okay, so, we’re near the end, but I just want 

to give everyone – I know Mary’s going to close – but 

are there any last comments from the live audience or 

Anthony, Lily, any of the speakers?
Office of Research & Evaluation Webinar

Using Evidence for Scaling Community-Based Interventions 

That Work

70

MARY: Well, thank you again to our speakers and to 

those who provided comments and as well to the 

audience who’s joined us in this discussion today. 

Special thank you to our ICF colleagues as well as 

our other [unintelligible] colleagues and our friends 

at OMB who have really tried to pull all this 

together and coordinate this series. It’s been a nice 

collaboration. We will be sending out a post-webinar 

survey, so please let us know your thoughts on this 

webinar or ideas for future webinars, and we will 

post this recording in about a month. So thank you 

for joining and thank you for your time today.

[APPLAUSE]

END OF FILE