Using Evidence for Scaling Community-Based Interventions 

 That Work – CNCS Webinar Transcript 

1

ANDREA: Hi, everyone. Thanks so much for joining us today. My 

name is Andrea Robles [phonetic] and I’m from the 

Office of Research and Evaluation at the Corporation 

for Research and Community Service. Our office’s 

objective is to support our agency’s mission by 

building knowledge on civic engagement, volunteering, 

and national service. We conduct in-house research but 

also fund research through competitive grants to 

researchers, scholars, and dissertators at 

institutions of higher education and support research 

and evaluation of our programs and grantees.

Our webinar series is one way we share our ongoing 

research and findings. Today we are very excited to 

host another research and evidence webinar titled 

“Using Evidence for Scaling Community-Based 

Interventions That Work: Perspectives from the Field.” 

As we did in June of last year, we are very excited to 

welcome members from the Federal Evidence and 

Evaluation Community of Practice, who are both in the 

audience and online today. So, thanks for coming and 

listening. This webinar is part of a larger series of 

workshops organized by federal agencies and 
Using Evidence for Scaling Community-Based Interventions 

 That Work – CNCS Webinar Transcript 

2

partnerships with the OMB Evidence Team as part of the 

Evidence and Evaluation Community of Practice. They 

aim to have one event per month and always post slides 

afterwards on the Community Max page.

And if you work for a federal agency and are 

interested in hosting an event or would like to learn 

more about the Community of Practice, please reach out 

to Erica Zielewski. And her email is on one of the 

slides right now. So, again, thanks for coming and I’m 

glad everyone’s back to work. 

Before we begin, we’d like to cover a few housekeeping 

items for Adobe Connect and I'll pass it to Emily.

EMILY: All right, thank you everyone for your patience with 

Adobe Connect. We want to let you know that this 

webinar will be recorded and posted online following 

the presentation. Unlike previous webinars, there is 

no dial in phone line. All audio is broadcast over the 

internet using your computer speakers. All 

participants will also be in “Listen Only” mode 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

3

throughout the presentation. But you can ask questions 

at any time by using the chat box below.

As we mentioned earlier, this webinar is being 

recorded and if you have any questions or if you

experience technical difficulties, please let us know 

in the chat box and we will send you a personal 

message to [unintelligible]. I believe that takes care 

of all the housekeeping items.

ANDREA: Okay, thanks, Emily. So, our program today will be as 

follows: I will now pass this to Dr. Mary High who’s 

the Director of Research and Evaluation; Dr. Lee 

Sangapore [phonetic], who’s a Research and Evaluation 

Manager; and Anthony Areno [phonetic], Research 

Analyst, all at CNCS, who will provide some context 

around scaling and then introduce the speakers. We 

will hear presentations from five CNCS grantees about 

their experiences scaling program models and have a 

brief concluding remark from Dr. High. Then we will 

open the discussion for Q&A.

During the webinar, please feel free to ask questions 

or provide comments in the chat box and we will get to 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

4

them during the Q&A. So, I’m now going to hand this 

over to Dr. Mary High.

MARY: Thanks, Andrea, and welcome to today’s webinar. I just 

wanted to put this in a bit of a larger total context 

for today. I think, hopefully, everyone knows that 

using data and evidence for decision-making continues 

to be a bipartisan priority for the federal 

government. The most recent example of this priority 

can be found in the recently signed Foundations for 

Evidence-Making Policy Act of 2018. Included in this 

foundation act is a set of evaluation and evidencebuilding activities in Title 1 that ask federal 

agencies to engage in a number of activities, 

including scaling evidence-based interventions. 

Today we’re going to focus on one of the important 

ways we can use evidence to inform our decisions 

within the federal context. And we’re going to hear 

about folks who take evidence-based programs and 

practices and expand them across communities so that 

more people can benefit from what works. Specifically, 

here at CNCS, we have a number of examples of how 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

5

AmeriCorps grantees have been taking evidence-based 

programs and bringing them to scale in their 

communities. We’re going to hear directly today as 

part of our third webinar in our scaling series, 

directly from folks who have actively been engaged in 

implementing this notion of scaling evidence-based 

practice.

So, we’re going to hear a little bit about their 

firsthand experience about the evidence interventions 

that they have, but also, some of the challenges and 

opportunities that are presented by taking these sorts 

of things to scale. So, those that are interested or 

may have already joined us for the two prior webinars, 

again, this is the third in a series on this topic. 

The first one sort of framed this notion and how we at 

CNCS are sort of examining and learning about scaling 

evidence-based interventions. The second one really 

talked about a program that has been scaled based on 

continuing evidence that it’s built around its 

effectiveness. And today we’re going to hear a variety 

of perspectives. So, it’s going to be, I think, a 

really informative conversation. And with that I’m 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

6

going to pass it onto Anthony Narena [phonetic] to 

introduce today’s speakers.

ANTHONY: Thank you very much, Mary. One of the things that we 

have been thinking about as we engage in this project 

and the project is something that we’ve been working 

with our contractor on, our contractor Mathematica has 

helped to unpack what goes into scaling. What are the 

key components? What are the variants? What are the 

ways that grantees can look at their program? And 

scaling naturally follows from the demonstration of 

some evidence-based for impact.

So, in thinking about that and in thinking about how 

we can bring effective solutions to different 

communities, we thought it would be interesting to 

look at or informative to look at how the scaling 

takes place? Where might it take place with ease? 

Where might there be challenges? What are the things 

that are going on that would facilitate or impede this 

process? In the contract that we have now, some of the 

deliverables that we’ve received and there will be at 

some point six documents that lay documents out 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

7

guidance for scaling for grantees. These will also be 

important for funders, foundations, and different 

groups that are interested in strengthening their 

impact. But before we release those, before they go 

through our clearance process, we thought it would be 

interesting to have a conversation with our programs 

that have demonstrated strong evidence. In the process 

of selecting these, we found five that met these 

criteria who have been evaluated and have been looking 

at and enacting scaling in their work.

So, today’s webinar is really going to be a 

presentation followed by what we hope to be a very 

vigorous conversation, opening up these individuals

for questions so that they may guide what you do. 

These are folks that are very skilled in this and we 

hope that they will bring you some information and 

some insight. We think as we move through this process 

that experiential learning and the development of 

guidance and development of technical assistance is 

going to be very helpful for all of our grantees. This 

webinar is a step in that process. 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

8

So, as we go forward -- okay, oh, I need to do that. 

Okay. We’d like to ask the panel if they would address 

three basic questions in their brief presentation and 

these would include: what did you do to get evidence 

and what drove your decision to scale knowing that the 

on-the-ground circumstances are very important in that 

decision? What are the critical preparatory steps you 

have to take for your organization to support scaling? 

And what were the key challenges and what have you 

learned from effective scaling?

Today’s panel is going to include LaVal Brewer former 

Executive Director of Playworks; Lara Dreier, Senior 

Manager of Federal Initiatives for College Possible; 

Sadie O’Connor, Managing Director Reading and Math; 

Adam Mauer, HR Consultant for Citizens Schools; and 

Dean Elson, Chief Knowledge Officer for Reading 

Partners. With that, I would like to pass it onto 

LaVal and we can begin the conversation.

LAVAL: Good morning. My name is LaVal Brewer and I’m the 

Former Executive Director for Playworks in Southern 

California. Playworks is a national non-profit that 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

9

works at elementary schools around the country to 

bring safe and healthy play to every child. They do 

that but through bringing play through recess. And 

that recess time actually ends up changing the system 

and the impacts that are happening on school campuses 

on a daily basis. We do that through three very 

formalized programs. One is our coach program, who’s 

hired by Playworks and often is an AmeriCorps member, 

is placed at a school every day, all day for the 

entire school year. And they help to transform the 

recess time and also to bring the value of games and 

play to children of that school. 

The second opportunity is where we partner with a 

school and we have a staff person who’s actually 

working with four schools. And that staff person works 

with that school every day for a week throughout the 

entire year. And they have four schools that they’re 

working with.

And then our final way that we make this work now is 

by training schools to do this work to create systems 

change so that they can have safe play at their 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

10

school, thus changing their school climate and 

increasing social and emotional learning during the 

playtime.

So, what did we learn? How did we come to this 

decision? We also started at Stanford Mathematica a 

control study back in 2013. In that, we were able to 

confirm that our theory of change gave us the ability 

to scale outside of our own footprint and our own 

ability to do this work ourselves. So, 20 years ago

when we started doing this work, we thought, well, 

we’ll place an AmeriCorps member or a Caring 

Consistent adult, which is part of our theory of 

change, at a school campus and they will change the 

school climate through recess and children will have 

greater play. What we began to realize was that we 

wanted every child to have safe and healthy play in 

the United States and that our current theory of 

change and our current evidence said that we can do 

this by actually helping schools to have that safe and 

healthy place process. To take our theory of change 

and not say that we can deliver this ourselves, but 

that schools who have these Caring Consistent adults 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

11

and who have a system for making this work happen, can 

actually do it themselves.

So, what we did was we tested it out. So, we do a lot 

of the valuation of Playworks. We evaluate every year. 

We ask principals what they think. We talk to 

students. We talk to schools. We talk to school 

partners. And we use a lot of the valuation to help us 

make decisions.

So, as we were moving forward in this process, we 

thought, are we actually achieving our vision? Are we 

actually helping every child in America have safe and 

healthy play or are we just having safe and healthy 

play at the schools that we currently have 

partnerships with and we’re happy with our current 

impact? We realized we weren’t happy with our current 

impact and that what we really wanted was to achieve 

the actual vision and that we would use our theory of 

change and the evidence that we’ve had on how to 

create safe and healthy plays at schools and we would 

bring that to schools around the country. And that we 

would actually unleash schools to have this ability to 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

12

do this work themselves. So, if we were doing it 

ourselves and we were keeping it ourselves and we were 

the ones that were making this happen, we would have 

incremental change. We would have impact. But we 

wouldn’t change the system and our evidence told us 

that the system would change even if we weren’t the 

person doing it.

So, how do we do that? So, we had to say, “Well, how 

do we go about changing our system? How do we go about 

growing and scaling, not just having impact? In other 

words, not going from 10 to 20, but how do we go from 

one location to five locations? How do we actually 

scale up in different places? So, we began to say, 

“How do we test this?” So, the first thing we needed 

to do was we needed to figure out if our AmeriCorps 

funding in our local locations could expand to new 

locations outside of where we currently were? And in 

that case, it was in our metro. Can we get outside the 

metro? Can we get in new spaces?

So, I’m in California. We were thinking are there two 

places in California that we can grow that made sense 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

13

for us to expand and scale so that we would have 

greater conversion within those spaces? We chose 

Riverside in San Bernardino and we chose San Diego. 

So, my role and responsibility was to go out and study 

those geographic locations to determine if the theory 

of change that we had and the school partners that 

existed in those spaces would be a good fit for us to 

do a test and to do a pilot to see if this would 

actually work by expanding our reach and expanding 

outside of our actual reach?

The second thing we did was we said, “What are our 

actual outcomes? What do we want to desire to have 

happen in these locations?” So that we could keep real 

clear what our actual goal was and that that goal 

would stay connected to the theory of change. That we 

wouldn’t just scale and grow in order for scale and 

growth, but we would scale and grow in order for us to 

continue to implement the theory of change. And that 

we would stay within the evidence we know was working 

well. Thank you for moving that forward.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

14

So, we did that work and we started to go out and 

actually test that. So, did a small test in San Diego 

and we did a small test in Riverside and we found that 

the Riverside test was more productive than the San 

Diego test and that threw us off because San Diego is 

the second largest school district in California and 

Los Angeles has the largest school district in 

California. We thought it would be similar. And what 

we determined was that by doing the preparatory work, 

we would see that there was a different space in the 

scaling and we had to work through that process.

So, the final thing -- what were the challenges? So, 

the challenges we found and what we determined in 

scaling is scaling is not easy work. You have to plan. 

You have to create your outcomes. You have to test 

small. And then you have to evaluate that test against 

your outcomes, not against how you feel but what are 

the outcomes? Actually get the outcomes you were 

deciding. And then you would go to a small pilot. And 

then you could take this pilot and you can expand that 

pilot across the country. That’s what’s currently 

happening right now.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

15

So, in scaling you really need to take your time. You 

need to determine what you can do. And you can also 

have to determine if you have the team members -- in 

our case, the AmeriCorps members -- who could actually 

do the work with the leadership team we had in place. 

Thank you.

LARA: Thanks, LaVal. My name is Lara Dreier and I am the 

Director of Federal Partnership at College Possible. 

College Possible supports students from low income 

backgrounds who are pursuing a college degree. And in 

our flagship model, students start with us doing 

programming as juniors in high school. And they’re 

paired with a near peer AmeriCorps coach every year 

until they graduate from college. And so, that is our 

flagship model. We operate that in six metro areas 

around the country. 

And a couple of years ago, we launched a new program 

called Catalyze. Catalyze serves students who are 

first generation and from low income backgrounds to 

our already in college. Many of these students may 

have participated in a college access program while 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

16

they were in high school. But many of those programs 

don’t offer continued support through college degree 

attainment, even though this is a group of students 

that has dramatically lower graduation rates than 

their higher income peers. 

So, our Catalyze coaches are embedded on college 

campuses, supervised by college staff, and they’re 

implementing the College Possible curriculum. And the 

idea is that this model has the potential to allow us 

to serve exponentially more students more quickly than 

if we were continuing to just expand our flagship 

program into more space and regions. That process, 

extending the flagship program, is extremely resource 

intensive since it requires us to build a whole staff 

to support the operation. 

So, we are working to scale our Catalyze program. So, 

we’ll talk a little bit about how we came to that 

conclusion. The first thing we looked at was our own 

internal data. That initially drove us to the 

realization that support through college graduation 

was critical. Though we actually started as a college 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

17

access only program, so we worked with students only 

until they graduated from high school. And what we 

found when we started tracking student progress beyond 

high school was that our students weren’t seeing the 

success we wanted once they got to college. So, that 

led us to developing the college success component of 

our flagship program. 

We also turned to national data in helping us decide 

how or whether to scale. So, there’s a lot of data out 

there. I’m sure many of you are familiar with it. 

Georgetown reports that there are 240,000 low-income 

students that graduate from high school every year who 

are college ready but who don’t enroll. In addition to 

that, high-income students are significantly more 

likely to earn a college degree than students in the 

lowest income quartile. And there is a lot of economic 

data that demonstrates the increasing importance of 

having post-secondary credentials to sustain our 

workforce needs.

So, in our flagship program, we’re serving about 3,000 

high students every year, which is, we think, a 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

18

critical service to those students, but it’s a drop in 

the bucket in terms of the scope of the problem 

nationally. And so, we wanted to figure out a way to 

serve a lot more students really quickly. So, that is 

part of what drove us to explore how our services 

might integrate into existing infrastructures that 

already exist on college campuses rather than 

continuing to build something from scratch.

And then, another part of our decision hinged on 

sustainability. Our Catalyze model requires campuses 

to make an investment in the program, which reduces 

our reliance on grants to ensure that we can continue 

the program and sustain it financially. 

So, ultimately, we found that our college success 

program was the component of our program that was the 

most scalable in terms of resource investment and it 

was also the greatest need in terms of solving the 

problem that we refer to as the degree divide -- the 

difference in degree attainment between high-income 

and low-income students.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

19

So, in terms of preparation for scaling, we learned 

some valuable lessons when we were expanding our 

flagship program into new cities and high schools 

around building relationships and buy-in before 

entering a new market. So, there was a lot of energy 

spent in listening sessions with various colleges and 

universities to ensure that the program that we were 

designing was the program that they would value. And 

while all colleges and universities, of course, want 

their students to graduate, there’s a risk of folks 

feeling defensive or territorial if our approach 

appears to duplicate services they already provide. 

So, we wanted to be really careful about that. 

The other thing that we’ve done is really build out a 

robust staff to support it, which includes bringing in 

people with corporate and marketing backgrounds as 

well as nonprofit expertise. And this has really 

allowed us to help articulate the value proposition to 

the institutions, which, I will admit, is a different 

mindset and skillset than many of us nonprofit lifers 

might have in our toolbelt. So, that has been really 

valuable. You may not be able to see the details of 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

20

the visual on this slide, but it’s really just a 

visual representation of the business case that we’re 

making for the program.

So, in terms of challenges that we face, I would say 

they fall into two main categories. Because our 

Catalyze program is more decentralized than our 

flagship programming and the day-to-day management and 

supervision is done by campus staff instead of college 

possible team members, there are just some really 

basic challenges around building the infrastructure to 

ensure that we continue to have really strong grant 

compliance. So, those of you on the call who have had 

host sites forever, if that is nothing new to you, 

you’ve been doing that probably since day one. But for 

us, it’s a relatively new model to embrace and so it’s 

been a learning curve. As we figure out how not to 

support members directly, but how do we support their 

supervisors?

And then I would say more fundamentally, that 

decentralized model also means, for better or worse, 

that the campus staff feel more ownership of the 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

21

program. And so, in some cases, that means that they 

want to change things about the model and we have to 

decide what elements of our program are critical and 

require fidelity and where we think there’s room for 

campus-specific modifications?

So, a really simple example of this is one of our 

partners who wanted to provide coaching only to firstyear students, whereas our model at its core continues 

to coach students through graduation. So, we have a 

lot of discussions about whether that was an area we 

felt we could flex in and, ultimately, we worked with 

the campus so that our coaches could still support 

students through graduation but we focused our 

recruitment efforts heavily on first-year students so 

we could help the campus ensure that those students 

weren’t falling through the cracks.

So, that’s just a little bit about our experience and 

I will then turn it over to the next presenter.

SADIE: Good afternoon. My name is Sadie O’Connor. I am the 

Managing Director at Reading and Math Inc. and Reading 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

22

Corps is one of the AmeriCorps programs that we 

administer. So, Reading Corps is a tutoring program 

that places AmeriCorps members into either preschools 

or elementary schools with the goal to help students 

become proficient readers by the end of third grade. 

In our preschool program, our tutors are placed into a 

classroom and they’re working with all of the students 

in the classroom throughout the day, providing some 

one-to-one support, small group support, and even 

classroom-wide support to help those students acquire 

the literacy skills that they need to be successful 

for kindergarten.

In our Take 3 model in our elementary schools, a tutor 

is placed into the elementary school and they’re 

really providing one-on-one, 20-minute intervention 

sessions with students throughout the day. So, they 

have a caseload of students that have been identified 

as being at risk that they’re working with every day 

of the week. 

We have what I articulated on this slide of sort of 

what we consider to be three pillars of our program. 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

23

Evidence-based literacy interventions that are used 

consistently by all of our tutors. Database decisionmaking is really integral in looking at assessment and 

making changes to the interventions or the instruction 

that’s happening to make sure we’re meeting individual 

student needs. And then, providing onsite coaching for 

all of the tutors so that they get immediate feedback 

about their tutoring and we’re ensuring fidelity to 

the model.

In our 16-year history that Reading Corps has existed, 

we’ve scaled from being 24 AmeriCorps members in a few 

locations to now having more than 1500 in 600-plus 

locations throughout the State of Minnesota and is 

replicated in 12 states.

So, the first question is what did we do to get our 

evidence and what really drove our decision to scale? 

And what you’re seeing right now is a visual of how we 

think about how we build evidence within Reading 

Corps. And it starts with establishing promise and 

then moving into refining and evaluating, moving into 

scaling and improving, and then moving into continuous 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

24

improvement. And when I started reflecting and 

thinking about these questions about building evidence 

and reflecting back to our earlier years as we were 

just getting started, I thought about establishing 

promise and how important those two key questions 

really were to help give us a really solid foundation 

that helped us build our evidence and then, 

ultimately, scale. 

So, those two key questions: Is the program model 

stable? One of the things that we did early on was 

engage literacy experts in helping us design and 

develop what the model looks like. Right at the 

beginning, we looked to the research base to help us 

determine what interventions we would really us. And 

so, we defined interventions and assessment, a 

training program that could really be consistently 

applied across the whole program with all of our 

tutors.

And then is it promising? We had a legislator that 

initially gave us some seed money to get started and 

one of the things that she said was right at the get 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

25

go this has to produce results. If it produces 

results, we’ll continue to consider funding it. So, 

that was really baked into what we did right at the 

outset, was conducting evaluations, not largescale 

evaluations but small analysis that enabled us to show 

that we were making an impact and had some promise.

And then, what drove our decision to scale? I think 

it’s pretty simple. For us, it was really demand. 

Demand from our school partners. We did offer it to 

any school in Minnesota that was interested in it. And 

as the principals started to talk to each other and 

they could see the results the program was having with 

their own students, we were starting to get more 

demand from school districts and schools. We’re also 

getting more demand from legislators because we were 

starting to secure more state funding. And also demand 

from the philanthropic world.

So, the next question is what was critical trackwork 

that we did and how we are really supporting scale-up? 

I think on the outside, like I described earlier, the 

essential elements of our model were really clear. We 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

26

got really clear about that right in the beginning. We 

wanted to make sure that the same model was used in 

all of our schools. So, if you went to a school that 

was in Northern Minnesota, close to Canada, if you 

went to school in the Twin Cities and you maybe went 

to one in Southern Minnesota, you’d see our AmeriCorps 

members delivering the same things -- the same 

interventions, the same assessments. They’re all 

trained in the same way. So, that really allowed us to 

have a really stable theory of change.

The second thing is that these early results helped to 

prove that our model was effective. We did some really 

pretty low-cost match analysis, match sample analysis, 

for five consecutive years as we were just getting 

started. And that really was helpful to show that the 

impact that we were having was strong and, like I 

said, it was a low-cost thing to provide some early 

results.

The third thing is then finding people who influence 

resources, people who care about funding and 

implementing programs that work and who are also 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

27

interested in this issue of third grade reading. 

Obviously, resources are essential to scale. So, 

finding those people, whether it was in the 

philanthropic community, education system, school 

district leaders, our state funding, our state 

legislators. People who care about scaling that works 

and nurturing those relationships with them. Getting 

them out to see our program in action. Talking to our 

tutors, talking to our principals, talking to the 

teachers so that they could become early adopters. 

And then the fourth thing is just a constant 

refinement of our operational systems, which has just 

been 16 years in the making. So, as our program has 

grown, always rethinking how we’re doing things to be 

efficient and effective and providing a good 

experience for our AmeriCorps members. 

This is a really small example, but think about signin process that you have for a training. If you have a 

group of 30 members or 50 members, it’s pretty easy to 

have paper sign-in sheets. If you have 1500 members 

coming to a training, paper sign-in sheets are no 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

28

longer effective or efficient. So, something like that 

moving to scanned barcodes with name badges was just a 

small example of the types of operational systems that 

just have to change as we’ve grown and scaled.

And then finally, key challenges and what we’ve 

learned about effective scaling. It’s hard to isolate 

these to just a few. We’ve learned so much over the 

years about what it takes to scale. But a few things 

that really came to mind. The first being thinking 

about a statewide infrastructure. We started Reading 

Corps and all of our staff were housed in the Twin 

Cities, in our office in the Twin Cities. But we were 

really serious about reaching all parts of Minnesota 

and being able to provide a good experience to those 

schools. And so, at a certain point in our history, we 

decided to open up regional offices so that we could 

have staff located throughout the state that could 

really directly support the AmeriCorps members in the 

schools. And so, now today we have seven regional 

offices throughout Minnesota. And now as we’re scaling 

nationally, we also have offices in our other 

locations.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

29

The second thing is just thinking about a system for 

student data management. When we first started, we 

were using sort of an off-the-shelf system for our 

student data that produced graphs and had some really 

nice capabilities. But as our program grew and our 

needs became more unique and we had more desire to be 

able to export the data files and manipulate and think 

about how we’re using that data, we were just finding 

the off-the-shelf product really wasn’t meeting our 

needs anymore. So, we did make a decision to invest in 

building our own student data management system that 

was completely customized for our program, which just 

allows us to have better access to our data, which is 

useful for us at the evaluation level but also really 

at the ground level for our tutors and coaches.

And the third, of course, as all of you are aware, 

scaling and operating programs obviously require 

financial resources. So, just that constant attention 

and dedication to raising money to support the 

program. And I will say that the continuum of daily 

evidence is really important. And for us to not really 

be satisfied that outcomes are strong, but always 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

30

challenging to think about how can we strengthen and 

improve and refine. And it helps funders continue to 

see us as leading in this field. 

So, yeah, I’m happy to take questions in the Q&A, as 

all of the presenters above.

ADAM: Hi, my name is Adam Auer and I’ve had the opportunity 

to work with Citizen Schools in a number of capacities 

over the last six years. First, my role was as Chief 

Talent Officer and today I lead HR and many components 

related to AmeriCorps as a provider of outsource 

services through my company, Positively Partners. 

Positively Partners is a social enterprise that works 

with nonprofits in Boston and DC and across the 

country to deliver mission-aligned HR services at 

lower costs.

Today I’m speaking on behalf of Citizen Schools with a 

focus on its scale journey, which has significantly 

evolved over its past 20 years of operation. And I 

think it’s a different journey than the stories that 

are being shared by other panelists because Citizen 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

31

Schools achieved significant scale, then contracted, 

and is now thinking about scale in some different 

ways. 

To look at our mission, as an organization Citizen 

Schools seeks to ensure that every adolescent has the 

skills, mindset, and networks needed to inspire them 

as lifelong learners in ways that position them to 

thrive in the 21st century. It’s a mission that has 

remained the cornerstone of our work for more than two 

decades. But how we live it and execute this work 

continues to evolve and it’s shaped the way we think 

about scale and opportunities to work both inside and 

outside of the existing education system.

Our goal is to be a disruptive innovator in support of 

middle school students and the broader ecosystem in 

which students exist. Our vision statement, the next 

slide, if I can move it along. Our vision statement, 

the longer-term aim of our work is now to serve one 

million adolescents by 2025. We’re providing hands-on 

learning experiences for students that are led by 

experts from the community. And that’s always been the 

core of our work. We link students with experienced 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

32

experts across the professional discipline that help 

to engage middle school students and prepare them for 

college and careers. 

And while hands-on learning experience is the core, 

how we make these experiences available to students, 

the approach to reaching more students and assuring 

the quality of those interactions has really evolved. 

And I think the best way to think about that is 

through the lens of Clay Christiansen’s work on 

disruptive innovation. Like many organizations, we 

launched into this journey to effect systems change 

from a position outside the system. Thinking how we 

could build infrastructure to support that change and 

then kind of integrate that later into the system. And 

we did it through our after-school programs, which 

leveraged AmeriCorps teaching fellows. And we created 

an extended day for middle school students that 

connected them with these hands-on learning 

opportunities. And also, just help middle schools 

generally kind of think about how to serve the needs 

of students in the 21st century.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

33

Our journey to scale brought us to seven states at our 

largest and we had a cohort of almost 130 AmeriCorps 

members serving in our program. It also was a 

tremendous opportunity to learn about what worked 

better and to understand the challenges that middle 

schools face growing the skills and mindsets needed 

for success in the 21st century. It also relies on a 

really robust infrastructure to deliver a program. It 

often required us to build infrastructure that 

mirrored what was already in place across many 

schools. And that added cost. That meant that many 

schools that needed us, communities that could benefit 

most from our program weren’t able to afford us. And 

that led us down a journey to think about scale 

differently, to consider ways we could better partner 

with schools to leverage what was already working, 

what could be buttressed, and when we could work as a 

member of the school community in different ways to 

support common goals.

So, the journey to think about a differently about how 

we could spark and support innovation from within the 

system caused us to evolved our program models in ways 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

34

that continues to envision a kind of direct service, 

this provision of after-school programming, as the 

core of what we do. Yet disaggregated two components 

of our work in ways that allow us to scale in concert 

with school partners and rely on their infrastructure 

so that we can complement them, rather than sort of 

build structures around them. In essence, we evolved 

in a way that allows us to partner with schools as 

part of their existing school day as an alternative to 

our after-school program offering.

So, if you look at the slide, it starts with sort of 

expanding the school day at the core, our direct 

service component. And that provides the opportunity 

for us to continue to allow AmeriCorps members to 

serve and to do great work within schools and to 

establish proof points that demonstrate the larger 

impact on hands-on learning programming. But then 

we’ve evolved the first icon there, Catalyst, which 

provides the opportunity for us to teach and support 

others in the delivery of the hands-on learning 

experiences. And then the third component to the 

right, US2020, creates the infrastructure and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

35

scaffolding needed to introduce the experts, those 

skilled professionals into the school day. So, in 

concert, these three program elements provide a 

revised pathway to scale, which will continue to 

expand the organization’s reach beyond what it was 

able to accomplish through direct service alone.

The next slide shows kind of our journey to impact. 

And this begins again with direct service at our core. 

But then in this megaphone sort of amplifies that 

impact through a new pathway to scale, which calls for 

us to nurture the supply chain for expert mentors who 

lead the hands-on learning, to grow the relationships 

and demand for our partnership with schools, to grow 

capacity for this program within communities, and then 

achieve broader scale in the form of a movement. And 

to bring that full circle, back to what I mentioned 

around Clay Christiansen, this is just achieving the 

goal of transformative education in a new way that 

sort of is more embedded in the system and is less 

reliant on the robust operational infrastructure for 

program delivery that Citizen Schools was kind of 

building and tweaking to ensure quality delivery.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

36

My last comments just kind of relate to how AmeriCorps 

continues to fit into this. It’s been interesting as 

we kind of have shrunk a little bit our direct service 

offerings, AmeriCorps has continued to be such a

prominent component of our work. Both through kind of 

traditional service opportunities, but has also given 

us a lot of opportunity to think about vistas for 

capacity building. And then just this larger movement 

building has helped us to envision career pathways 

differently for those AmeriCorps members to think 

about ways that they can participate in this movement 

after their service as community liaisons, but also 

embedded within schools [unintelligible].

Happy to answer questions at the end, but thanks for 

hearing a little bit more about our journey.

DEAN: Hi, everyone. I’m Dean Elson [phonetic]. I’m the Chief 

Knowledge Officer at Reading Partners. Reading 

Partners is proud to be celebrating our 20th 

Anniversary year of providing services to students and 

schools throughout the country. Our vision at Reading 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

37

Partners is a future where all children in the United 

States have the reading skills necessary to reach 

their full potential. And our mission is to help 

children become lifelong readers by empowering

communities to provide individualized instruction with 

measurable results. 

Our program model is an evidence-based tutoring 

program. We work in public schools, including charter 

schools and as a public-school literacy intervention 

that is absolutely powered by AmeriCorps members. 

We’re a proud national service program. Our AmeriCorps 

members and our community volunteers work in schools 

with students. The majority of our schools are Title 1 

schools in under resourced communities and they work 

with them twice weekly for about 45 minutes. These are 

kindergarten through fourth grade students primarily 

who are struggling with reading and learning the basic 

foundational literacy skills they need to succeed in 

school in their life.

It looks like some of our slide content got cut off a 

little bit here, so I'll just help you as we move 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

38

along. So, this first slide is talking about the 

question related to evidence building and our decision 

to scale-up. So, like many of the other presenters 

today, I'll share with you that we also have internal 

student assessment data for several years that showed 

we were looking and seeing strong results for our 

students and seeing strong improvement in their 

literacy skills and their growth during the school 

year. They were able to reduce grade level reading 

gaps with them. And so, before I joined Reading 

Partners back in 2011, the organization had begun a 

small-scale match comparison study with the design and 

the oversight from a university professor and some 

data analysis by a doctoral student. And this allowed 

us to engage in what I would call a lower stakes 

evaluation or study and it allowed us to learn a 

little bit more and engage in a more rigorous 

evaluation of our impact than the organization had 

ever done previously. 

The early results from that study, along with high 

satisfaction from principals we were working with, a 

long waiting list for new school partners, and also an 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

39

aligned board and leadership team really pushed us to 

look for additional funding and opportunities to

scale. We really believed that we had a successful 

program model and we wanted to bring it to more under 

resourced communities. And so, several foundations, 

individual donors, board members and others made 

introductions and that really allowed us to explore 

partnership opportunities in these states and 

districts. They also had become an AmeriCorps program 

back in 2010. And as a proud national service program, 

we work with state AmeriCorps commissions as well. And 

that has provided funds and support to our 

organization to be able to grow and bring the program 

to more states. We are currently serving approximately 

11,000 students a year in 14 regions. And so, those 

regions are in nine states plus Washington, DC.

A turning point, a real catalyzing moment for Reading 

Partners happened back in 2010 and 2011 when we 

competed for and won a federal Social Innovation Fund 

or SIF grant through the Corporation for National 

Community Service. And that grant provided funding to 

us and opportunity to request matching funds from 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

40

other donors and a way to share our work and get 

others excited about what we were doing. And really 

allowed us a new way to pay for our first time ever 

actually a large, independent, highly-rigorous program 

evaluation. And through this SIF grant and the support 

of the Edna McCullough Park Foundation and our 

partnership with MDRC, our research firm, we were able 

to launch and go through the process of a randomized 

control design study or evaluation. And so, that was 

carried out in 2012 to 2015. It was a big process or a 

large process for us as an organization to learn and 

grow as we went along. We learned a lot from that 

study and the outcomes were really helpful for us as 

we talked about the benefits that Reading Partners 

brought.

And I want to just highlight here that both that SIF 

study as well as the second SIF grant that brought us 

to the State of Colorado and a partnership with CNCS 

again in Mile High United Way allowed us to grow to 

Colorado. Both the SIF studies allowed us to look at 

implementation data as well and I find that really 

important. I want to underscore the need to do that. 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

41

How do you learn? What are the lessons you can learn 

about your implementation fidelity and program quality 

that maybe you didn’t know before? And I believe other 

presenters today had mentioned the same thing. You’re 

not focused only on the impacts.

And then, lastly, just in terms of our decision to 

scale, I just want to highlight that our early 

evidence of success and student impact, those early 

studies along with the SIF studies as well and the 

valued asset, that school and district partners felt 

that we were and have really allowed us to continue to 

scale and bring our services to more schools and more 

districts over time.

This next slide focuses on the question on the 

critical preparatory work and how the organization 

supports scaling up. So, as with many other presenters 

today, we really focus on having a clear program model 

and theory of change and evolving that over time. We 

believe that ensuring the model has certain 

standardized elements that have been tested, I think 

you want to do that at least internally initially and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

42

then work toward external tests and evaluations as 

well.

I think it’s important to ask, “What do you know about 

your program’s success and how do you know it?” and 

“What are the first things first that you know you 

have to get right from the model to achieve success in 

a new location?” Some of our other presenters today 

talked about partnerships and that’s certainly 

incredibly important here at Reading Partners. Fee for 

service and our philanthropic investment, local 

leadership, key staff roles, etc.

We also feel that alignment from the board, the 

leadership team, and also our Regional Executive 

Directors and boards around expansion and scaling up 

to address the challenge while focusing on continual 

improvement. All of that is important too.

Our key functions, many key functions need to have 

both good leaders with good talent, strong talent and 

functional expertise. And over time we’re able to 

build that expertise and add more talent to the 

organization. We’re very pleased that we’ve been able 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

43

to do that. There is a program, data evaluation, 

development, finance, HRIT operation, etc.

We’ve also talked about the importance of having an 

expansion checklist and a clear timeline for go and no 

-go decisions. When in the year is the right time for 

you as an organization to make those decisions? And in 

our experience, the time that I’ve been here, we have 

looked at bringing the program to additional locations 

and decided not to, in fact, follow through. Sometimes 

during a rapid expansion cycle, some places it just 

wasn’t feasible. We looked at our internal capacity 

and operational capacity to be able to so and, in some 

cases, we had to back off. I would recommend that you 

consider that carefully. 

I think it’s important to achieve clarity and being 

transparent about where we were going to scale and why 

and what the opportunities that existed around 

sustainable funding. We felt it was very important to 

not just think about the first year you go to a city, 

but more what’s the three-year timeframe around 

expansion? And the discussions with school and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

44

district partners about the value, ahead of our model 

and how it aligned to their local vision or their 

blueprint to see success.

Obviously, as everyone talked about, it’s important to 

continue to raise more funding for expansion and 

improvement and so that’s really key. Making sure 

we’re aligning annually as an organization on our key 

priorities and projects. We openly debate the musthave investments versus the nice-to-have investments 

and projects that might need to be pushed out farther.

And, finally, anyone that knows me knows my data 

matter a lot to me and the teams that I work with. And 

so, having data on learning and the ability to tell an 

impact story are important as well. And I underscore 

here again that qualitative data can be very important 

as well quantitative data. 

I know I’m running short on time, so I want to go 

quickly through the next slide. I’d be happy to answer 

more questions as we get into the next section. But a 

few key bullets here about what are key challenges at 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

45

Reading Partners and what we’ve learned about 

effective scaling? One is to document success. Have 

written programs, fidelity guidelines, and quality 

standards. Provide training on those guidelines. And 

use quality assurance processes that are tied to 

program operational and financial data. 

It’s important to balance scaling, I think, with core 

program improvement. So, while you’re trying to 

continue to improve and reflect on and change certain 

elements potentially of your core model as you learn 

and you grow. Sometimes you only have so much capacity 

and so many resources to do. You can’t do everything. 

So, trying to find the right balance for your 

organization with scaling and that core program 

improvement is important.

Encouraging innovation, fidelity to a model that’s 

proven successful is really important to continued 

success. But we also believe an organization needs to 

have a certain degree of innovation and regional or 

site flexibility. So, within that, I think expanding 

the sources of innovation. Not all great ideas come 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

46

from national headquarters. Think about the process 

you need to use to encourage and support innovative 

ideas and sharing across your network as you scale up.

The last few things: secure multi-funding commitments. 

I talked about that earlier. That’s really important. 

You should expect everyone in the organization to be a 

fundraiser. I really like what was said earlier about 

the importance of adding talent sometimes outside the 

nonprofit space and thinking about corporate and 

marketing folks to join your organization as well. 

And then lastly, operational efficiency certainly 

matters. Use cost and other financial metrics as well 

as operational metrics to drive your core business 

functions. And figure out when and how scaling 

provides same or similar levels of program success 

with fewer resources and which resources can get you 

to more with less.

Finally, prepare the talent bench. I think it’s 

important to recognize that not everyone will be at 

the organization for 5-10 years. You will have 

important leaders and other folks leave the 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

47

organization from time to time and so it’s important 

to be aware of and plan for talent needs. Even before 

there are signs that someone might leave, you want to 

mitigate the potential loss of expertise and 

knowledge. And, again, that goes back to the 

importance of documenting your success in processes as 

well.

MARY: Okay, thank you. I think we heard a lot of great 

information. I just wanted to point out a couple of 

things that struck me as each of you were sharing your 

scaling experiences and journeys. I just thought it 

would be interesting to point them out and then 

certainly turn it over to questions that I’m sure 

folks have both in the room and on the phone. 

So, one thing that really struck me as you were all 

talking was that you all seem to be nicely embedded 

inside of organizations that appreciate a culture of 

learning and really support a culture of learning. And 

by that, I mean, for whatever reasons, you all sort of 

have this story of having this openness to inquiry, to 

the pathway of continuous learning, not being afraid 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

48

to realize that maybe you were on the wrong path and 

pivoting and taking another path. So, I think that’s 

an important part of this scaling conversation, that 

you are able to operate in an environment that allows 

you to fail, to regroup, and to try again.

I think what was also striking about each one of your 

journeys is that you also had a very systematic 

approach not only to your evidence-building because 

you all did a very thoughtful, rigorous sort of 

approach to building your evidence in various ways, 

but also to your decisions around scaling. You 

realized that in addition to having to incrementally 

build your data and demonstrate the results and to 

prove your outcomes, you also had to make your 

decisions around whether or not you wanted to expand 

or not. And in some cases, expanding and then deciding 

to retreat a bit for whatever reason. So, I think 

that’s also reflective of a willingness to embrace 

sort of this continuous learning and improvement 

cycle. 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

49

Another thing that struck me as you all were speaking 

was the sort of tension between fidelity and 

flexibility. You all started your stories with, “We 

had to really figure out what our stable model was. We 

had to make sure we kind of knew what our core 

components were. That’s what we could bring to our 

outcomes.” But then you started, you know, bringing to 

to different campuses, to different communities, to 

different states and you realized that you needed some 

flexibility in there. And then what is that balance? 

What are the pieces that you absolutely have to keep 

in order to achieve your outcomes versus what pieces 

actually need to be flexible? Because I think the 

other piece that you all spoke to was the importance 

of your relationships and that this is not only an 

evidence-building process, but it’s a relationshipbuilding process. And that whether it’s a partner in a 

school, whether it’s a partner who’s going to bring 

resources to your effort, or whether it’s a 

partnership with folks inside of your own 

organization, this is a process that requires 

extensive relationship nurturing as well.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

50

And then, lastly, I think I would point out I think 

each of you spoke to the role that national service 

can play in expanding these solutions in communities. 

I think you spoke to different ways. They can be the 

actual service delivery providers. They can be someone 

that’s helping you with your operational 

infrastructure, which also has the scale that you’re 

scaling. And so, I think that was interesting to think 

about how national service can be a part of this sort 

of expanding solutions that have been proven to work. 

So, for those on the phone who perhaps are new to the 

national service world, thinking about ways in which 

streams of service might help you take some evidencebased practices or programs to scale. I think that’s 

an interesting resource of folks to consider.

So, with that, thank you again. I'll hand it over to 

Andrea to facilitate some of the Q&A.

ANDREA: Okay. Let’s see. We’re going to open it up to Q&A. I’d 

like to just mention a few ways we can do this. First 

of all, we’ve had an incredibly active audience. Thank 

you so much for all on the chat. So, I have copied and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

51

pasted all your questions. So, I think we will take 

some turns, doing some on the chat box, opening the 

line, and then people in the room. So, for our inperson audience, please state your name before asking 

your question. For our online audience, can you ask a 

question by typing it in the Q&A box? You can also ask 

a question using the microphone on your computer by 

selecting the raised hand feature from the menu above 

and we will grant you microphone rights in the order 

the questions come in.

So, let’s see. I'll just take a few from the chat and 

then turn to you. So, let’s stop with Robert Rader. 

Can Playworks share a typical aligned output and 

outcome performance measure? I'll read a couple of 

these and then our speakers can take it. Carol Cutler 

White: How is evidence quantified? Alicia Sanders 

asks: How did you disseminate findings so that 

critical decision makers were aware of the results of 

reading core? So, I’ll just start right there and then 

we’ll take some from in here. So, any of the speakers 

want to answer?
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

52

FEMALE: I can start with responding to the question about how 

we disseminated findings to critical decision makers. 

You know, what we found to work the best was having 

in-person meetings and usually sort of smaller 

meetings where we could really sit down face-to-face 

with that person to talk about what the results look 

like and meant. What we found is that if it’s just an 

email or something that’s sent to somebody, they 

really don’t get the full context or maybe don’t take 

the time to read sort of the meat of what’s in there. 

So, we found that in-person, be able to walk through 

and present it usually on an individual basis was

really critical.

We also found that actually bringing them out on a 

site visit if they had time and were able to, really 

helped them visualize and see what that model was to 

connect it to the results.

ANDREA: Anyone else?

LARA: This is Lara. I can respond to the quantifiable 

evidence question. I would say that we look at a 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

53

number of different types of evidence to support and 

inform our programming. The first and sort of most 

obvious are student outcomes. Are students persisting 

in graduating? But, obviously, we can’t wait around 

for four to six years to find that out. And so, we 

look for sort of predictive factors on a more 

intermittent basis. So, we look at things like have 

students registered for classes for the following 

semester? Have they filed their financial aid 

applications?

And then we also look at coaching metrics. So, our 

model is centered on coaching sessions between our 

AmeriCorps members and the students that they support. 

And so, we’re tracking those activities in our 

database and taking a look at duration. We’re taking a 

look at modality. Are they having a voice-to-voice 

conversation or is over text or Facebook? And then 

also what are the topics that they’re discussing? What 

types of questions and concerns are the students 

raising? 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

54

And then finally, we also look at students’ survey 

results. So, we survey our students. We occasionally 

do focus groups and things like that to try to get 

more qualitative information around the student 

experience and how that intersects with what we’re 

trying to accomplish.

ANDREA: Thank you. I’m just going to -- speakers, do you have 

any other thing you want to add to any of those?

DEAN: This is Dean from Reading Partners. I want to add one 

other thought around the question of dissemination. 

Absolutely agree that it’s highly beneficial to have 

one-on-one sit-down meetings or potentially over the 

phone. We did at the conclusion of our NDRC 

evaluation, our first SIF grant, in that rigorous 

evaluation we held some webinar conversations as well 

and invited some of our key donors and stakeholders to 

those and had conversations in which we presented some 

information at a Q&A session much like this is going.

But the other thing we’ve learned in the last few 

years as well is very few people like to read a 50, 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

55

100, 150-page report and so having an executive 

summary and even more so if possible, if you could 

find the resources in your community where someone is 

good at data visualization and can create a -- I’m 

blanking on the word. If someone could help me out 

here. It’s a visual -- thank you, an infograph. We 

found that infographs have been helpful as well. 

ANDREA: Thank you. So, we’re going to take a few questions 

from people here in the audience. Could you say your 

name?

SINEAL: Yeah, hi. This is Sineal Eingarn [phonetic]. I just 

was wondering, this is touching on something Mary said 

about the organizations of how not only was a big part 

of all these presentations, the collection of evidence 

or data on evidence of effectiveness and impact. But 

also, there was quite a bit about expansion and moving 

into other areas. And so, I was wondering I think the 

three presentations in the middle, not necessarily the 

first or the last, all kind of smoke about the degree 

to which demand kind of played in their theory of 

change or the way they structured their data 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

56

collections, evidence collections. And so, I’m kind of 

wondering -- and one of them actually talked about 

branding. And, I guess, is there a tension between and 

how do you resolve that kind of tension that may occur 

in making course corrections and on the one hand, 

changing programming based on data about the 

implementation process or data about the evidence or 

the impact versus, on the other hand, maybe whether 

people want it to happen anyway? And whether there’s a 

demand for that particular service? And whether you 

should grow and expand even absent some of that harder 

data about efficacy? I don’t know if that is something 

that anyone feels like speaking to.

ADAM: This is Adam from Citizen Schools. I mean, that’s 

certainly the absence of understanding and 

distinguishing between what was robust support for a 

capital campaign that sort of facilitated that growth 

initially and differentiating between what was demand 

and also sort of available in district to fund. I 

think it took us longer than we should have to have 

done a significant research project to look at 

discretionary funding within districts and middle 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

57

schools. And I think once we did that landscape 

analysis and found out the middle school principals 

have, I believe, somewhere around $1200 for all 

discretionary programming, it became very clear to us 

that there was sort of a deficiency between kind of 

our theory of what would work and what was possible. I 

think as you’re getting off and going and people are 

supporting scale and you continue to grow, there’s a 

demand for it as long as you’re paying. And so, really 

trying to do that landscape analysis team is quite 

common sensical, but for whatever reason took us a 

while to actually do in a kind of a robust way and 

that ended up informing some of our decisions. 

And then also, continue to understand, as others 

talked about, what makes a good partner? There are so 

many kinds of residual or school-based factors that 

impacts the success of the program. We needed to 

become really good at sort of understanding not just 

kind of our program quality indicators, but what 

needed to be in place in order to create the ecosystem 

for us to be successful. And I think all of those 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

58

certainly not just relate to demand but good demand, 

the right demand to allow for sustainable scale.

ANDREA: Thank you. Anyone else? Any other speakers? Any other 

speaker want to reply to that? Okay. So, another 

question from the room?

CLINT: I’m Clint Glass from the Congressional Research 

Service and speaking for myself, not for my agency. I 

have to say that or else I get fired. My question is 

about heterogeneity contacts and clients. So, 

borrowing some terminology from the medical field, 

before you come in with a therapy, you do some 

diagnosis. And I’m just wondering in how you 

structured your offerings and you talk with folks out 

there, how do you structure your conversations about 

heterogeneity to assess fit for context and clients 

not only at the outset kind of like the initial 

diagnosis, but also as things move along? A patient in 

working with a doctor or a team of doctors to decide, 

“All right, this course isn’t working well. Maybe we 

need to adapt our program model, engage in some 

adaptation, maybe get away from the core model in some 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

59

ways.” How do you structure those kinds of 

conversations? Or is that an area that’s still 

developing?

ANDREA: Would a presenter take that? Or say why not?

DEAN: This is Dean.

LARA: Go ahead, Dean.

DEAN: No, no, go ahead, Lara.

LARA: I'll take a stab at it. Yeah, I mean, I would say for 

a college [unintelligible], I think the answer to that 

is somewhere in between doing as much prep work as 

possible. I think that a lot of looking into the 

evidence to sort of define what the program is that we 

believe works helps us then define what are the sort 

of key components or the non-negotiable parts of that. 

And then I think real life application in different 

markets and environments teaches us more about the 

context in which that particular model or that 

particular approach is effective. And so, I would say 

one part of my answer is a lot of thoughtfulness and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

60

prep work and research on the frontend around what we 

believe are those sorts of key ingredients. And then 

also that sort of constant negotiation and push and 

pull as we sort of take our model into new 

environments. And it is a lot of just sort of in the 

moment discussion and sometimes debate and 

conversation and then further analysis to see how we 

need to adapt so that we’re being as responsive as 

possible to students without losing fidelity to the 

model. 

DEAN: This is Dean for Reading Partners. I'll echo that. I 

think an illustrative example for us over time has 

been the curriculum and the model as it had been 

developed. I remember a program manager sharing with 

me that her perspective was, let’s say, a secondgrader would come in and be enrolled in the program. 

And the suggestion was that this student was going to 

move through the curriculum in the plan, scope, and 

sequence that we have made for the program and that 

those steps and sequences would happen no matter what 

those individual student’s needs were. And that really 

pushed us to change and think about data collection 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

61

and analysis in an ongoing way and what is now 

certainly a big thing that happens in most of all 

schools with teachers and the data that they look at 

is progress monitoring. And so, being able to 

differentiate our approach a little bit more so for 

each individual student and individualize what their 

needs are. We’ve always had an individualized reading 

plan, but I think we’re getting better at that and I 

think we’re getting better at the local coaching and 

support adaptation that’s needed for each student. And 

we continually ask what we can change or might need to 

adapt further in our program model.

And then, there are exterior forces as well. Sometimes 

those are donors. Sometimes those are school district 

partners. It could be what else we see happening in 

the field and where there’s other evidence of success. 

And so, in our case, an example of that would be that 

Reading Partners has been looking at and pilot testing 

different program models and approaches to summer 

learning. So, rather than just looking at our core 

program throughout the school year, what can we 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

62

provide and support for students during the 

summertime?

ANDREA: Thank you. I’m going to read a few more from our chat 

box and this has to do -- more comments around the 

same issues that you all are mentioning. Someone said 

that all your presentations stressed the importance of 

process evaluations. And Jeff Sheldon writes, “It 

appears that process evaluations are based on a 

theoretical model of how the program is designed to 

work and outcome evaluations based on the theory of 

change.” Just a comment, but again, maybe you can say 

something about the importance of this process 

evaluation and how you were able to think of your 

components.

SADIE: This is Sadie O’Connor with Reading Corps. And I would 

totally agree with the importance of that process 

assessment part. When we went through our pretty 

rigorous evaluation that CNCS actually commissioned 

NORC to conduct on Reading Corps a few years ago, one 

of the first things that they did was conduct a 

process assessment. And that involved going out and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

63

observing what was happening in schools, talking with 

tutors, talking with coaches, talking with teachers to 

really understand is the program being implemented 

like the theory of change is articulating? It should 

be. And when that process assessment was completed, 

then they had the confidence to move into a more 

rigorous, randomized control trial, which was really 

looking at our outcomes. Because we knew that the 

model was being applied consistently across all of our 

locations, so then could do an assessment of students 

that were getting Reading Corps, compared to an 

assessment of students that were not getting Reading 

Corps to see what the impact of Reading Corps really 

truly was. But that process assessment, I just think, 

was really super valuable in the whole process.

ANDREA: Thanks. Anyone else? Okay, and then thinking of the 

next step, Tara Wright [sic] writes, “Thank you all 

for your presentations. But how are your project 

monitoring and maintaining fidelity and quality of 

implementation once you scale up your program and 

services?”
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

64

LARA: This is Lara. I can take a gander at that one since 

this is something that we sort of have been building 

out pretty actively in the last year or two. So, we 

have a couple of staff members at our main office, our 

Program Advisors, who directly support the supervisors 

on the campuses, who are directly in turn supervising 

the AmeriCorps members, the coaches. And so, the 

Program Advisors conduct monitoring visits on a 

quarterly basis where they physically go to the campus 

and do observations of the coachings. They also do 

weekly check-ins with the supervisors where a lot of 

it is spent sort of troubleshooting and answering 

questions. But they are also asking questions to try 

to ascertain if the program is being implemented as 

intended.

And then the other thing that we look at -- I 

mentioned this earlier -- but we look at a lot of 

outcomes, which include both, obviously, student 

outcomes like retention on that campus, but also 

things like the activities that the coaches are 

recording in our database and whether that's 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

65

reflective of the types of coaching sessions that we 

ask them to do as part of our program.

SADIE: This is Sadie from Reading Corps and I'll chime in 

too. We have a role that we call an Internal Coach, 

though it’s basically a supervisor at the school 

that’s supervising and supporting the AmeriCorps 

members. And we actually have what we call fidelity 

checklists and the coaches do those two times a month. 

So, every intervention that we have and every 

assessment that we have has a corresponding checklist. 

So, when the coach is going in to observe the tutor, 

they’re actually doing it with a checklist so we can 

actually measure and sort of monitor and support the 

tutor in getting true fidelity.

And then we also have three times a year before our 

assessment period in the fall, winter, and spring, our 

coaches are also monitoring and observing the fidelity 

of the assessment data. I think somebody else had a 

comment and sort of wondering how do you know that the 

assessment data is sort of true and accurate. So, 

making sure before they collect assessment data, that 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

66

we’re observing them to make sure that they’re 

conducting that correctly.

ANDREA: Wonderful. Anyone else? 

DEAN: This is Dean from Reading Partners. The one other 

thing I would add is if our partnerships with several 

states AmeriCorps commissions and the ongoing cycles 

with CNCS around evaluation requirements and 

expectations has [unintelligible] a grant, I really 

appreciate that we continue to do our work at Reading 

Partners in terms of evaluation where we conduct more 

studies rather than just these one-time implementation 

and impact studies. And so, we’re really proud of what 

we’ve learned in the five-year process and study that 

was in partnership with [unintelligible] Associates 

here in Colorado in the las five years. Those results 

came out in the fall. We continue to use that 

information, as well as a two-year study we just 

concluded in the fall in our four California regions, 

which was much smaller than an exploratory study, but 

built on the ATA and NBRC evaluations around certain 

processes. For us that includes volunteer recruitment 

and retention, the number of sessions we’re able to 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

67

provide to students. And looking at dosage, for 

example, those studies have had us looking in a more 

granular and specific way and it’s causing our program 

team to have conversations and think about how we 

might change operations on a weekly and/or monthly 

basis, rather than think about half a year or the full 

year in terms of total doses that a student might get 

in terms of literacy intervention and tutoring and 

more about what are they getting every week and every 

month? So, there’s a standardization and expectation 

for the program model and we’ve got program dashboards 

and regional dashboards that we follow a lot of data 

around. But those specific evaluations have pushed us 

to look at dosage in a new and different way.

So, I think ongoing work on the process and 

qualitative side can be really valuable and helpful as 

well.

ANDREA: Thanks. So, turning from that to funding, Carol Cutler 

White asks, “Have any programs been successful in 

getting state funding for long-term sustainability?” I 

think that one or two of you spoke about that.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

68

SADIE: Yes, this is Sadie from Reading Corps. And our state 

funding has really been part of our financial 

sustainability. We started with $150,000 from the 

state in 2003 and in Minnesota that’s grown to be 

about 6.5 million annually. Reading Corps is a state 

appropriation so it’s built into the base, which is 

great that it’s built into the base. Just over the 

years, we’ve continued to request an increase to that. 

This current year we also have launched our program in 

Wisconsin, but this past current year we also have 

received state funding in Wisconsin. 

ANDREA: Fantastic. Any other speakers?

LARA: College Possible has received state funding in 

Minnesota as well. And that has been, echoing Sadie, a 

pretty important part of our financial assisting 

ability there. And it is something that works 

[unintelligible] in some of other states but seems to 

be more feasible in some of our larger, more mature 

sites. 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

69

ANDREA: We’re all saying that we’re moving to Minnesota. Okay. 

Do any of you want to say a little bit more not just 

state funding but funding in general? I mean, I know 

you mentioned it in some of your presentations.

Okay. So, Sadie, just a question. Someone would love 

to hear about the platform that Reading Corps built 

their student data management system on.

SADIE: Sure. I wish it was a simple answer. It has been 

something that we’ve custom-built. So, we kind of use 

it for IDF programming frameworks and we have a team 

of developers that are working on it. So, feel free to 

reach out to me if you’d like more information, but 

it’s not a simple answer.

ANDREA: Okay, great. I’m looking at the time. It’s 3:22 and we 

have lots of questions that we may not be able to get 

to. So, I think I’m going to say it now. If we don’t 

get to your question, please email us at 

evaluation@cns.gov. Some of them might have been 

answered already as speakers have been answering other 

questions. And we will get it to the right speaker and 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

70

be able to answer that. So, let’s see, just in terms 

of -- I’m trying to see something that we haven’t 

talked about. Magdalena Bianchi says, “Great 

presentation. Sadly, we’re at 34% literacy in San 

Bernardino County. I’m a nonprofit CEO, Executive 

Director, tuning in from San Bernardino.” And I think 

it’s just for those of you who are working on it, just 

a comment that you have that low literacy rate. I’m 

sure some of you have faced that in your own schools. 

Where do you start?

MALE: AmeriCorps, naturally.

DEAN: This is Dean from Reading Partners. Yeah, I'll share 

that’s a fairly consistent rate in terms of literacy 

at the elementary level in the schools in which we 

partner. And in many of our schools, it’s actually 

substantially lower. We look at those school and 

district proficiency rates on the third or fourth 

grade state reading tests. And in some of our 

districts it can be as low as 5%. And so, we’ve 

collected and looked at those data both at the 

district and/or school level, but we’ve also looked at 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

71

the rates for the individual students that we serve. I 

would say for us over time, the importance of growing 

and expanding and scaling our program, we really 

thought about the importance of looking for resources, 

how to look for high quality resources that are 

already out there in existence as much as we have

built a proprietary curriculum and a model and a 

program that we think really works and we’ve got 

evidence to show for that. We also know the need is 

great in millions of students. And so, it’s important 

to us and I think to you in your program that you 

think about looking for partners, looking for 

coalitions, other places that bring together resources 

or high-quality resources that can be used. I would 

look to your local community and see if there are 

volunteers who would be interested in getting involved 

and getting the word out. There may be low entry kinds 

of programs to just get started. But I would certainly 

look to what’s out there in terms of resources and I 

would look for the evidence that’s there and use 

impactful models.
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

72

ANDREA: Thank you very much and thanks for the question. So, 

just turning a little bit in terms of member 

recruitment and experience, probably AmeriCorps, 

“Curious how” -- this is Robin Harris -- “Curious how” 

-- and this is to any presenter -- “solve the member

recruitment experience change during scale. What 

worked? What didn’t? Any unique member benefits? Are 

members paid above the minimum living allowance if 

full-time?”

LARA: This is Lara from College Possible. One of the really 

big benefits of our Catalyze model and the fact that 

the campuses are taking so much ownership over the 

program that’s being implemented on their campuses 

instead of doing our sort of widescale, cast a wide 

net, recruit as many members as possible. Those 

campuses do pretty targeted recruitments on their 

campuses. And so, generally, they’re able to find 

alums of their campus to serve as coaches on that 

campus. And because most of them are recruiting 

between two and five people instead of our 

organization recruiting 300 people, they’re able to be 

pretty targeted and have, I would say, an easier time 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

73

in general filling their spots. I think all of us have 

probably experienced some recruitment challenges in 

the last couple of years. And so, that’s been a really 

big benefit.

The other thing that a lot of our campuses are doing 

is because they want those numbers to be so embedded 

on their campus, they’re offering room and board in 

some cases. And so, that’s been also a really big 

benefit for our members to make the living allowance 

and all of that more doable.

ANDREA: Thank you. Anyone else? AmeriCorps recruitment and 

benefits?

SADIE: Yeah, this is Sadie from Reading Corps and I would 

echo what Lara said about really engaging partners in 

the recruitment process. We really work closely with 

our school partners so that they can recruit from 

their own communities. Parents, grandparents, 

community members that they know. And we’re still 

experiencing recruitment challenges, but that has 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

74

consistently been a really good strategy to find 

members.

ANDREA: Okay, thank you. And this is the last question I’m 

going to mention. So, Carol Cutler White asks about 

“Have any of these results been published in peer 

review journals? If so, is there a list of 

references?” So, we just said in the chat box to look 

at our web page under evaluation or evidence and 

change where we have reports and papers probably from 

all the speakers you heard from. Again, you can write 

to us at evaluation@cns.gov, and we will send the 

questions to the right individual. Anything else? Any 

other place? Okay. 

So, before I close, I just want to turn to Mary, Lily, 

Anthony, anything you want to add? We’re at 3:28.

ANTHONY: I would just like to add, I thought this was a 

wonderful conversation. And the fact that we have 

talented presenters who are willing to share their 

experiences with other grantees is remarkable. And I 

think it’s a trend that we need to foster and we need 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

75

to continue here. There’s an emerging science, at 

least as we understand it, around scaling that allows 

for all of the different variations in programs on the 

ground and circumstances on the ground, but identified 

core components. And each one of the speakers today 

touched on those core components. And I think 

rehashing them and learning them and applying them is 

a critical element to moving forward with this. It’s a 

long journey, but I believe that many of the folks who 

are listening can see a path forward if we support 

them, we provide them with this information and this 

knowledge. 

So, I’d like to see this continue and I’d like to say 

this was a wonderful introduction to our guidance 

documents that will be put together that we’ll be 

releasing shortly. So, a lot of the information is 

contained in those things. We expect high traffic 

there from all of you. And any questions, please 

direct them back to us. Thank you very much.

ANDREA: Okay. So, as we mentioned before, this is being 

recorded and will be on our website in a couple of 
Using Evidence for Scaling Community-Based Interventions

That Work – CNCS Webinar Transcript

76

weeks. So, we want to thank you again, especially to 

our speakers and our audience today for such a great 

discussion. And I saw that again all of you were 

signing in from all over the place. We had over 300 

people register and we’ve had over 160 participate. 

So, that is fabulous.

I also want to thank Emily Ranos [phonetic] of ICF 

International and our colleague, Greg Wallinger, for 

their technical support and coordination because that 

is its own very big task. And Erica Zielewski of L&B 

for helping us coordinate the workshop series. We will 

be sending out a post-webinar survey, so please let us 

know your thoughts and any webinar ideas for the 

future. And that’s it. So, thank you so much and have 

a nice evening.

END OF FILE