CNCS Webinar Transcription

The Power of National Service 

Improving Children’s Literacy Outcomes

1

MARY: Hi, everyone. Thank you so much for joining us 

today. My name is Mary Hyde and I am the Director of 

the Office of Research and Evaluation at the 

Corporation for National and Community Service. Our 

office’s objectives are to support our agency’s 

mission by building knowledge on civic engagement, 

volunteering and national service. We conduct inhouse research, but also fund research through 

competitive grants to researchers, scholars and 

dissertators at institutions of higher education.

And we support research and evaluations of our 

programs and grantees. Our webinar series is one way 

we share our ongoing research and findings. Today, we 

are very excited to post another Research and 

Evidence webinar: The Power of National Service: 

Improving Children’s Literacy Outcomes. While this 

particular body of research has not been sponsored by 

our office, we are very excited to highlight it today 

because it is an excellent example of how we’ve been 

able to replicate the findings that we, your 

presenters, and really build on an evidence base that 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

2

started a number of years ago [inaudible] talk about

later on in a good minute.

We are pretty excited to present this topic and this 

multi-side evaluation that you’re gonna hear about 

today because it’s a very good example of how 

programs can be replicated, and scaled in not only 

new locations, but also with different populations, 

which is a little bit about what we’re gonna hear 

today. This webinar is a complement to our June 2018 

Using Evidence for Community-Based Interventions That 

Work webinar. It was recorded and can be heard by 

going to our webinar section under our Office of 

Research and Evaluation webpage.

This is the second webinar of what would be a threewebinar series that focuses on how you can use 

evidence to make decisions about scaling, 

interventions and community. Before I move into 

introducing our speakers, we’d like to cover a few 

housekeeping items because this is a webinar as I 

just mentioned, and we would like to make sure that 

everyone knows how to engage with us today. With 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

3

that, I’m going to hand it over to Emily who will 

walk you through a few things. 

EMILY: Thank you, Mary. Thanks, everyone for your patience 

as we get started with Adobe Connect. We want to let 

you know that this webinar will be recorded and 

posted online following the presentation. Unlike 

previous webinars, there is no dial-up phone line. 

All audio is broadcast over the internet using your 

computer speakers. All participants will be in listen

only mode until the question and answer session 

following the presentation. 

At that time, you can either ask a question in the 

chat box at the bottom of the screen or you can use 

the raise hand feature for the menu above, which will 

enable your microphone to work. If you experience any 

technical difficulties, please let us know in the 

chat box and we’ll do our best to address that. And, 

as we mentioned earlier, this webinar is being 

recorded. I believe that takes care of all of our 

housekeeping. 

FEMALE: Thank you, Emily. I’d like to introduce our 

speakers today. We’re gonna have an introduction, 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

4

then we’re gonna have a panel of speakers talking to 

us about this excellent research and then we’re gonna 

have final comments made by the Deputy Director of 

AmeriCorps. Let me explain who you’re going to be 

hearing from today. Audrey Suker is gonna introduce

and kick us off in a minute here. Audrey joined the 

Reading Corps Minnesota Organization in 1979 and 

orchestrated its transition to independent non-profit 

in 2002.

Her experience in establishing community and 

corporate partnership, strategic planning and 

innovate program design has resulted in significant 

growth for the organization. State support increased 

from 900,000 annually to eight point one million and 

property value increased from 75,000 to point five 

million. Her vision for leveraging the power of 

AmeriCorps in solving intractable problems led to the 

of Minnesota Reading Corps, the largest AmeriCorps 

tutoring program nationally in a statewide initiative 

to ensure that all children are reading proficiently 

at the end of third grade.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

5

Audrey’s gonna share with us a bit of the history of 

how they used research over the life of the Reading 

Corps program, which has sort of led us to this 

moment today. Following Audrey’s introduction, we’re 

gonna hear from Dr. Carrie Markovitz. She is a 

principal research scientist in the Economics, 

Criminal Justice, and Society department at NORC at 

the University of Chicago. Dr. Markovich is a 

statistician with over 20 years of experience 

conducting both quantitative and qualitative 

evaluations of youth and community programs with 

foundations in government agencies.

In addition to her years of experience as an 

evaluation expert, Dr. Markovich has spent two 

decades conducting research on national service and 

volunteerism, contributing to evaluations of numerous 

CNCS programs. In addition to Dr. Markovich, you’re 

gonna hear from Dr. Marc Hernandez, who is also from 

NORC and he is one of the founding directors of the 

Early Childhood Research and Practice Cooperative at 

the same organization. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

6

He is an applied Developmental Psychologist with 

expertise in cognitive development, early childhood 

and elementary education, both in formal and informal 

educational settings. We’re finally going to hear 

from Sadie O’Connor, who is the Managing Director of 

Reading and Math, Inc., which is a national nonprofit that incubates evidence-based AmeriCorps 

programs including Reading Corps. She’s been part of 

Reading Corps’ Readership team since 2013. And she 

has helped grow it from a small, local program with 

25 tutors to one that engages 2,000 tutors annually.

Sadie was also the Vice President of Reading for 

National Replication that served Minnesota and 

focused on innovating and scaling the program 

nationally. Sadie is an AmeriCorps alumni and a 

Minnesota native. She will be commenting, I think, on 

the replication aspects of this work as well and we 

will have Jennifer Bastress Tahmasebi talk to us 

about what she’s heard today and facilitate probably 

a transition into the Q&A. As I mentioned early, 

she’s the Deputy Director of AmeriCorps. With that, I 

think I will pass this on to Audrey Suker.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

7

AUDREY: Thank you, Mary. As Minnesota’s Service 

Commission, we know that it is our responsibility to 

ensure that our programs are continually [inaudible] 

as well as continually improving. The slide that you 

see in front of you illustrates a process you use to 

do this using Reading Corps as an example. The 

program is 15 years old and in its first few years, 

you really focus on establishing the promise of the 

program to achieve its objectives. This included 

process evaluations, descriptive analysis of pre- and 

post-outcomes, and then small-scale research 

projects.

And as it became clear that the Reading Corps was a 

promising model and we moved into the Refine and 

Evaluate stage, we conducted a study that found that 

students served by the Reading Corpse were three 

times less likely to require Special Education 

services. And this research really helped us with 

policy-makers that wanted to understand that our 

lives are investing in our work it went beyond 

student goals. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

8

And during this phase, we were also very grateful to 

CNCS as they contracted with NORC to conduct a threeyear impact study on our work in Minnesota. That then 

served as a foundation for continually refining the 

model and positioned us to move into the Scale and 

Improve stage, which included the NORC multi-state 

evaluation that you’ll hear about in this webinar. 

And then along the way, we have continually shared 

our results with legislators and the philanthropic

community and that has allowed us to raise matching 

funds from those two sources that now equals about 

nine million dollars annually.

So besides supporting tutors on the ground, this has 

allowed us to build an R&D infrastructure within our 

service commission. In our early years, all of our

evaluation and research projects were done through 

contractors, but I’m happy to say that we know have 

an R&D unit of ten employees that includes two PhDlevel education researchers on our team. And this 

infrastructure then allows us to have a pipeline of 

internal R&D projects that are all designed to ensure 

that we are continually improving the model with an 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

9

eye on continuing to improve its effectiveness and 

its efficiency as we move forward. I’d like to 

conclude by thanking CNCS for hosting this webinar

and I, of course, would welcome any questions or 

advice from any of you on this webinar later on and 

after the webinar’s over. Thank you.

FEMALE: Thank you, Audrey. With that, I think I’ll 

pass it on to Carrie Markovitz.

CARRIE: Thank you, Mary. Thank you, CNCS for hosting 

this webinar and giving us this opportunity to 

present our findings. We have an outline of our 

presentation. We’re going to have an introduction 

section where we’re gonna talk about the Reading 

Corps program and about the multi-state evaluation of 

Reading Corps. Stacey O’Connor’s going to present on 

the Reading Corps program and I’m going to present on 

the evaluation. 

Then Marc Hernandez is going to present on the 

Minnesota Meeting Corps Evaluation Results. I will 

present after that on the Wisconsin Reading Corps 

evaluation results, and then Marc and I both will 

provide a summary of our major takeaways and study 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

10

implications. And, of course, we’re all available for

answer questions at the end. First, the introduction. 

I’m going to hand this over to Sadie to present on

the program.

SADIE: Thanks, Carrie. Thank you, all for being here. It’s 

really exciting for all of us to be sharing these 

results with you. I’m going to take just a few 

minutes to talk about what the Reading Corps program 

is. And I’ll essentially be using what we call a 

“visual logic model to walk through what our Reading 

Corps program level is. At a high level, Reading 

Corps is an AmeriCorps program that places tutors 

into elementary schools and pre-schools to provide 

individualized intervention services to children who 

are at risk. Today I’m going to focus more on our K3 

model because that’s what our evaluation is focused 

on. In any given school, as you know there’s all 

kinds of different students who have lots of 

different needs.

How do we identify who those students are that need 

extra reading support? What we do in Reading Corps is 

we conduct a screening. Our AmeriCorps members are 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

11

trained to do a screening assessment in the fall. 

It’s a quick one-minute assessment that they’re able 

to do with students. And that assessment allows us to 

know that based on that score that students receives 

if they are at-risk or if they’re on track. And so, 

we’re able to identify really pretty quickly those 

students who are on the trajectory of potentially 

falling through the cracks, not being a proficient 

reader at the end of third grade. We’re able to 

identify those students in the fall.

Our AmeriCorps members, who are really the heart and 

soul of our program, come from all walks of life. We 

don’t require our AmeriCorps members have formal 

degrees or training in education. We provide fairly 

robust training and coaching models to help them be 

successful. All AmeriCorps members go through three 

days of institute training at the beginning of the 

year where they learn about the interventions, 

assessments, how to make decisions using the data 

that they’re collecting so they’re very equipped at 

the beginning of the year. They also receive followup coaching throughout the year so every month 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

12

they’re getting observations and support from two 

levels of coaching: internally, at their school, they 

have what we call an “internal coach”.

That’s an individual that’s employed by the school, 

so usually a literacy specialist, maybe a reading 

teacher, somebody who has a strong background in 

reading intervention and assessment, and who is there 

onsite to provide observations, support and just 

general direction to the member. And then as a 

program, we’re providing what we call a Content 

Master Coach, somebody who goes out to the school 

once a month and checks in with the internal coach 

and the AmeriCorps member, again, conducting 

observations to make sure the model is happening with 

fidelity and also to make instructional decisions for 

those kids.

What we do, our AmeriCorps members are providing 

daily one-on-one tutoring using evidence-based 

interventions to a caseload of students every day. A 

full-time tutor, so a member that’s serving five days 

a week in a full-time capacity, will be working with 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

13

between 15 and 18 students every day. Each of those 

students receives a 20-minute session where they’re 

receiving that evidence-based intervention. Our 

interventions are focused, and really guided by, the

National Reading Panel’s big five areas of 

instruction: phonological awareness, phonics, 

vocabulary, fluency, comprehension – so really 

focused on building those foundational reading 

skills.

The AmeriCorps members are collecting data every week 

and that data is helping them make decisions with 

their coaches about whether or not the intervention 

is working and whether they need to make any changes 

to that intervention. That data also helps us to see 

what we are changing and what our AmeriCorps members 

are changing. In our K-3 model, we’re really looking 

for that accelerated literacy growth, so really 

looking for students who are making growth at a pace 

that’s higher than their peers because these students 

are starting behind.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

14

This ultimately leads to students who are proficient 

in their third grade state reading test and 

ultimately leads to students who are on track for 

college and career success. With that, I’m gonna turn 

it back over to Carrie and Marc to talk about the 

findings. 

CARRIE: Thanks, Sadie. Going back to the study, now I 

want to provide an overview of the multi-state 

evaluation of the Reading Corps. Our study was funded 

by Reading and Math, Incorporated through an 

Innovative Approaches to Literacy grant that was 

provided by the U.S. Department of Education. We also 

based our study on a previous study that we did. 

Audrey mentioned that in her presentation. From 2011-

2014, we conducted an evaluation of the Minnesota 

Reading Corps study for CNCS and in that study we 

found large impact of the program on kindergarten and 

first grade students.

We found smaller impact in the second and third 

grade. The only followed students for a single 

semester for that study. That was all we were able to 

do, and again we found large impacts of the program 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

15

on kindergarten and first grade literacy scores, but 

we weren’t able to see as large an impact in second 

and third grade, but then we think that mainly is 

because we weren’t able to follow the students for 

longer, and that experience set into some of our 

design choices for the new study, which I’ll explain 

in a moment.

There were many people involved in conducting the 

study, which was conducted by NORC at the University 

of Chicago, as we mentioned, so there were many 

individuals who were involved in the study and made 

it a success. I’ve been mentioned, and Marc

Hernandez, my co-PI, was mentioned. But I also just 

wanted to acknowledge Dr. Eric Hedberg who is Lead 

Methodologist who helped come up with our analysis 

plan and was really instrumental also in the study’s 

success.

Some key features of the Minnesota and Wisconsin 

study. I mentioned that this was titled “a multistate evaluation” so we designed the evaluation to 

evaluate the Minnesota Reading Corps program again, 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

16

but also we examined the replication of the Reading 

Corpse model in Wisconsin and in Florida. This 

presentation is focused on the results for Minnesota 

and Wisconsin. We completed those evaluations. We’re 

still working on the final report for Florida, which 

is focused on the pre-K program.

Both the Minnesota study and Wisconsin study were 

focused on the K through third program. For the 

Minnesota program, this, again as you heard, this is 

the original Reading Corps program that’s been around 

since 2003. For this iteration of the evaluation of 

the program, we set the second- and third-grade 

students for the entire school year. So as I 

mentioned, the 2014 previous study we found smaller 

impacts on second and third grade, but that was only 

after a single semester.

And we had evidence that, perhaps, if we followed the 

students longer, we would see a larger impact for the 

program, and so we decided we wanted to explore that 

with this opportunity in this most recent evaluation. 

Also, the Reading Corps program had a particular 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

17

question around whether students who were furthest 

from benchmark or in the expected literacy level 

would also benefit from the program. In a response to 

intervention model, students are organized into 

different tiers. 

A Tier 1 student would be a student who’s performing 

on benchmark, which is the expected level, or above. 

That would be a Tier 1 student. Tier 2 students are 

the students just below benchmark so they’re not 

quite on track. And then Tier 3 students are students 

that are more likely to be designated as Special Ed, 

particularly when they get older. They’re more likely 

to require more specialized services. The Reading 

Corps program focuses mainly on these Tier 2 students 

that are just below benchmark. 

The program was curious as to whether or not the 

students within that Tier 2 level, the ones that were 

further from benchmark, were benefitting the program 

as much as the ones that were closest. Many times, 

schools are instructed to work with the students who 

are closest to benchmark first because you can work 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

18

with those for a shorter period of time and then get 

them kind of in the program, on track and out, and 

then you can serve more students.

But what about the students who are furthest from 

benchmark and possibly also could benefit from this 

program, but they’ll be in the program potentially 

for longer to get them on track. That particular 

question was part of our study also. Then for the 

kindergarten and first grade students, we pretty much 

did the same study again. We assessed the 

kindergarten and first grade students for a single 

semester and we wanted to see if we would be able to 

show the same high impacts of the program on the 

kindergarten and first grade students.

In the Wisconsin study, the Wisconsin program’s a 

replication site of the Reading Corps model. It’s 

only been around for a few years, and so they have 

enough students for us to be able to assess the 

kindergarten and the first grade students. 

Unfortunately, although we tried very hard, were not 

able to identify enough eligible second- and third-
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

19

grade students to be able to do a study on them. But 

we were able to assess the kindergarten and first 

grade students. 

And so, again, we assessed them during a single 

semester because we had evidence that a single 

semester was enough time in order to show an impact 

in the program. These are our research questions 

based on all our understanding, all of the 

information to feed into this design. Our first 

research question is: For second- and third-grade 

Tier 2 students who are farthest from the fall 

reading proficiency benchmark, what is the impact of 

a full school year of Minnesota Reading Corps on 

program participants compared to students who did not 

receive Reading Corps?

And then, a second question we had involved whether 

the program impacts varied by participant 

characteristics such as demographics, program dosage 

and program attendance. For the purposes of this 

presentation, we’re gonna focus on presenting the 

findings on how the program impacted certain students 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

20

based on demographics. Then our last research 

question had to do with the kindergarten and first 

grade students.

For kindergarten and first grade students, what is 

the impact of a single semester of the Reading Corps 

on program participants compared to similar students 

who did not receive Reading Corps? Before we get into 

the results, I wanted to provide a little bit of an 

overview on the outcomes that we measured. Sadie 

mentioned that there is this screening that’s done. 

We refer to those as “benchmarks”. Benchmarks are 

taken usually three times per schoolyear so they’re 

usually taken in the fall, the winter and the spring, 

and you can see that in the table below.

Fall is usually taken in September at the beginning 

of the school year and that’s our beginning benchmark 

or the “screener” that Sadie was referring to. The 

winter benchmark is taken in January and the spring 

in May. For second- and third-grade students, there’s 

a one-minute assessment that’s conducted on oral 

reading fluency. What that essentially is, is the 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

21

AmeriCorps member is to show the student a paragraph, 

and they’re to read it out loud, and the number of 

words they read correctly in a minute is their score.

For first grade, that oral fluency assessment is 

conducted in the second semester. So there’s a winter 

benchmark and spring benchmark that are taken for 

those students because it’s considered too early to 

do it at the beginning of first grade. But, at the 

beginning of first grade, what is taken is a test of 

letter sounds in nonsense words. So they take a made 

up word and they put sounds together like, “Kep. K-EP. Kuh-ep [phonetic]” and they ask the student to 

pronounce the sound. So if they pronounce that 

correctly, they get a score of three, so the number 

of sounds they get correctly identifying in a oneminute period, letter sounds in nonsense words as I 

said.

And then for kindergarten it’s quite simple. Students 

are shown cards of letters and they are to produce 

the sounds. So: A, uh, B, buh [sic], C, cuh [sic], 

and the number of sounds that they can correctly 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

22

pronounce within a one-minute assessment is their 

score. So that’s explains the types of assessments 

that both Reading Corps uses and we use to assess 

effects on the students. To give a little bit of an 

overview of the evaluation 24 kindergarten through 

third-grade schools participated during the 2017-2018 

school year.

This was a representative standpoint that we hold 

within Minnesota. It was stratified by urban and 

rural schools so we had a representative sample. And 

schools had to meet the IAL eligibility criteria. So 

this is what we did for the Minnesota Reading Corps 

evaluations. For the Wisconsin Reading Corps, all of 

the schools participated. It was a smaller program. 

There were less schools so we just had all the 

schools participate so there was no sampling that 

took place there.

For both Wisconsin and Minnesota we conducted a 

randomized trial. We randomly assigned programeligible students to either a program group or 

control group and that was done within each of the 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

23

schools that was selected. And then we tracked 

reading schools in Minnesota 622 program and control 

students across the 24 schools that were selected. 

And then from fall to winter, we assessed the 

kindergarten and the first grade student.

Fall was their big, blind score and winter was their 

follow-up school and we looked at the increase and 

compared them between the program and the control 

group. And then we did the same thing for second and 

third, except we did it for the whole year. We did it 

between the fall and the spring benchmark that takes 

place in May. Then for the Minnesota Reading Corps, 

we used a randomization process. We created matched 

pairs of program-eligible students based on similar 

fall benchmark scores.

If a student had a score of six in kindergarten and 

another student had six, we might pair them together 

and then randomly assigned one of those kids to the 

program group and one of the kids to the control 

group, so that’s a paired randomization. That was 

very helpful because then we’re able to ensure that 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

24

we have a similar distribution of kids by scores in 

both groups. And we randomly assigned students, as I 

said, within pairs, and then the program group 

received the program and the control group was 

embargoed from receiving Reading Corps services for 

the study period.

So, for kindergarten and first grade, that would be 

one semester. For second- and third-graders, that 

would mean until May until they were assessed in the 

spring. Next, my college Marc Hernandez is going to 

present on the findings from the Minnesota Reading 

Corps study. 

MARC: Good afternoon, everyone. Can you hear me okay? 

FEMALE: Yes.

MARC: Good. I’m going to present the results of the 

Minnesota study from this past year. And so I’m going 

to start, we’re going to do this by grade, we’re 

gonna start with the older grades and work our way 

younger. And so, first we’re gonna look at together 

the combination of second- and third-graders. A 

little bit about second- and third-graders: we had 

190 second-grade and 212 third-grade students who 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

25

participated again, to reiterate, in carrying out 

this year-long study.

For those kids who first started in September and 

then they concluded their participation in the study 

after the spring benchmark in May. We had, in terms 

of the English Language Learner distribution, 23% of 

both second- and third-graders were designated as 

English Language Learners, about a quarter. And in 

terms of free and reduced price lunch eligibility, an 

indicator of socioeconomic status, about half of the 

kids - these are schoolwide averages. 

We do not have individual student level information 

on this, but we do have schoolwide average

information. About half are free and reduced price 

lunch eligible. If you look here at the pie charts, 

you can see very similar distributions by race 

ethnicity. The majority of these students we tested 

were white, between 58-62% were third and fourth

graders. The most common race ethnicity was African 

American at around 17%, then Hispanic at around 10-
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

26

12. We also have representations of Asians, American 

Indian, Alaskan Native and multi-racial participants.

The second- and third-graders were very similar. 

You’ll see something a little bit different for the 

kindergarteners in a moment. The question again that 

we were answering with this second and third grade 

study was: for second- and third-grade Tier 2 

students farthest from the fall benchmark, what’s the 

impact of a full year of Minnesota Reading Corps on 

program participants compared to similar students who 

did not receive the Reading Corps?

And what we found in summary here, and I’ll show you 

in graphs in a moment more specifically, was a 

substantial effect for the oral reading fluency 

measure that was used to assess impact to the secondand third-graders, that a 6.4 word difference – we’ll 

show you what that looks like both in terms of number 

and in terms of effects of on the next slide. The 

magnitude of the programs effect on second- and 

third-graders’ oral fluency is equivalent to about 

47% or an additional half-year of an average 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

27

student’s annual growth and reading proficiency per 

the literature. 

Again, it’s important to note that this study focused 

on second- and third-grade children who were furthest 

from fall benchmark. It was really a question to us 

whether or not AmeriCorps members in this program 

would be capable, could with this program produce 

these types of outcomes in kids who might require 

professionals who are providing specialized services 

to those kids. In fact, what we find out is they can 

provide these services quite effectively.

I’m hoping that you can see a little, maybe this 

pointer will work. Can you see a pointer on the 

screen? I’m gonna walk you through these graphs 

because we’re going to use these graphs for all of 

the outcomes both for Minnesota and for Wisconsin so 

you know what you’re looking at. Here on the left, 

we’ll start with the second grade and take a look 

here. On the Y axis here, you’re gonna see the score, 

and this is in terms of the outcomes that we measured 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

28

so for oral reading fluency, this is the number of 

words read aloud in the minute period of time.

So we are showing you scale from zero to 120. When 

you go to third grade, you’re gonna see that scale 

goes higher because the kids were expected to be able 

to read more words fluently in a minute period of 

time from zero to 140. On the X axis down here you’re 

going to see a period of time in which we collected 

data. This will start with the fall data and it ends 

with the spring data. In terms of the lines that 

you’re seeing here, we have four that you can see.

The first is the solid line. This is the fall 

benchmark. This is the score at which when Carrie

noted that they were program-eligible, you have to 

score below this score to be eligible for Reading 

Corps services so you can see here that our two lines 

representing our participants down here are both 

below this line. This just confirms that the sample 

that we selected was eligible and, in fact, these 

kids are pretty far from their benchmark.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

29

This benchmark is around 62-63 and you can see that 

kids are performing closer to 40. This is the fall 

benchmark that shows eligibility. I’ve noted these 

two lines. You have two lines here: an orange line 

and a grey line. These two represent the two groups: 

the orange line is our program group; these are the 

kids that were randomly assigned to receive Reading 

Corps services. The grey line are the children who 

were randomly assigned to not receive services.

And importantly you see these two groups start at the 

same place at the beginning of the year, which we

would expect, particularly a matched-pair 

randomization where we used the pre-test score, the 

fall score, as a way to match them. This is great 

they were starting at the same point. And int ease

results, if you follow the lines up, you see they end 

up in different places. The children who received the 

program ended up moving from about 40 to be able to 

produce 92, read aloud 92 words in a minute period of 

time, whereas those who did not receive the program 

grew as well, but they grew to 84.8 words.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

30

And this is that about eight point difference that we 

mentioned before. The final line that you see here is 

the dotted line at the top. This is a high-bar spring 

benchmark where with much work that the Reading Corps 

has done through data analysis over the years has 

found that if children can meet or exceed this highlevel benchmark that they would be on track for 

scoring well on the Minnesota state reading 

proficiency tests. 

And so, this is an ultimate goal that we’d like kids 

to be able to hit this and exceed this. In each of 

the graphs that we’ll show you, you’ll see these four 

lines: the eligibility requirement, the high-level 

target and then the two lines representing our 

program and control. You can see over here for third 

grade that we have, again, the kids starting at about 

the same place, not surprisingly since these kids are 

a year older. 

They start higher than the second-graders did at 

about 70 words read aloud. No difference at the start 

of the year. At the end, we do see a higher score 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

31

among those in the third grade. This is 112 words 

read aloud versus 106, a six point difference here. 

And because the children were engaging in the same 

type of activity using the same assessment, we pooled 

the data and we looked the effects of the program on 

second- and third-graders combined and we converted 

these results from the metric that is oral reading 

fluency, number of words read aloud to in effect 

size.

Effect size for this particular was .28 and these two 

stars mean that they were statistically significant. 

A note about effect size: effect sizes are ways to 

standardize or to convert the metric that you’re 

measuring into something that can be compared apples 

to apples so later I’m going to be showing you letter 

sounds within nonsense words. In kindergarten I’ll be 

showing you letter sounds by themselves. 

These are all different types of outcomes. They’re 

all different types of metrics. You can’t compare 

what they score across grades, but what I can do is I 

can convert those into this effect size unit. And 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

32

basically effect sizes, the way you can think about 

them is zero there is no effect and as you get larger 

and larger, you get bigger effect. An effect size of 

one, for example, is a very large effect regardless 

of who you are and what age you are. 

There’s a bit of nuance to effect sizes and so it 

used to be a little bit more straightforward: people 

just figured if you had a .2 effect size that was 

small, if you had a .8 effect size or above that was 

big. But we’ve learned over time that depending upon 

the domain that you’re operating in, here we’re 

talking about education, and within education 

depending on how old a child is, where they are in 

development, what’s big and small varies.

Generally, kind of the rule of thumb is the older you 

get, the harder it is to produce large effect sizes. 

And so, in fact, an effect size that’s closer to a .2 

or a .3 is actually quite large for a child who’s 

later in high school versus a child who’s 

kindergarten. Kindergarteners you can create effects 

on the scale of .6, .8, 1.0. Those are more 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

33

reasonable in a younger age group. And just one last 

note about the reason for that, particularly with 

kindergarteners and preschoolers tends to be with 

respect to exposure.

Some children might come to school not having been in 

a pre-school, not having experienced a lot of 

academic instruction either formally or informally 

from their parents, and so you put them in front of 

high-quality instruction, let’s say around literacy, 

and you can see huge gains because they just haven’t 

been exposed, they’re ready to learn and they learn 

it. Once you become a third-grader, you don’t have 

these massive effects because of exposure anymore; 

kids have been exposed for a number of years. 

And so now it’s something about their ability to take 

in the information that they’re learning, and so the 

effect sizes – what’s big, decreases over time. So 

the point as it regards this slide is to say that a 

.28 effect size, not only is it statistically 

significant, meaning we’re really confident that this 

effect is not a fluke, that it’s a real effect, but 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

34

that that’s actually quite substantial for secondand third-graders who have been learning to read for 

some time.

Next slide. Back one. Another question that we asked 

in regard to the second- and third-graders was: does 

the program impact vary by participant demographics 

such as gender, race and English language learner 

status. And in fact, what we found was that there 

were larger impacts among these key participant 

demographic groups in the second and third grade, and 

I just pointed out too for the full sample. Here, 

just to reiterate, the full sample effect size was 

.28, but if we look at some of these groups, like in 

gender we’re looking at males, the size of that 

subgroup was 198 and the effect size for males was 

.47, much larger than the .28 effect, which was 

already quite substantial. 

For Black or African-American students, there were 66 

of those students for this sample. The effect size 

was even bigger than that. It was .50, again, 

statistically significant. And then finally, for 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

35

English Language Learners, there were 94 of them in 

our sample. Their effect size was .53. It was the 

largest effect we saw of the program. Again, the more 

stars there are, the more significant it is, so that 

just says that we’re really confident in that 

outcome.

And so the point here is that not only does this 

program work for everyone, but the groups that we 

might be most concern about or those that we often 

consider at-risk are actually benefitting more from 

the program. We did not, and were unable to conduct 

these types of analyses - I’m not going to be able to 

answer this question for kindergartener and firstgraders, the sample sizes were much smaller for those 

two groups by design, and so because the sample sizes 

were small, we weren’t able to slice the dice the 

data by these different subgroups.

Now I want to present to you the kindergarten and 

first-grade students as a reminder as Carrie noted 

the design of this study was it was a semester-long 

study because the previous studies had shown that we 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

36

had conducted that a semester was long enough to 

detect the effect of the program. These children 

participated between fall of 2017 and winter of 2018, 

collected their baseline data in September and their 

post-test data, if you will, in winter in January.

There were 60 kindergarteners in the study and 160 

first-graders. In terms of the demographics of this 

group: for English Language Learners, there were 37% 

of the kindergarteners and 31% of first graders, 

about a third. For free and reduced price lunch 

eligibility, about two-thirds of kindergarteners, 

65.5% and a little bit over half of the first 

graders. If you look first here at the first graders, 

the rate ethnicity distribution of the first graders 

looked quite similar to what you saw earlier for the 

second and third-graders majority white followed by 

African-American and Hispanic.

The kindergarten race ethnicity distribution was a 

bit different than for first graders or the other age 

groups. Here there was more equal representation 

across the three primary groups we saw before: White, 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

37

Hispanic, African-American and then the Asian 

population a higher percentage. We can talk a bit 

more about why we think that is. Some of it has to do 

with the smaller sample size and the fact that we 

were randomly selecting a smaller number of kids out 

of eligible schools we collected data from.

The question here was: for kindergarten and firstgrade students, what was the impact of a single 

semester of the Reading Corps on program participants 

compared to similar students who did not receive 

Reading Corps? I’ll give you the summary of the 

effects and then I’ll show you those graphs for each 

of these. For kindergarteners where we assessed 

letter sound fluency, we found a large effect: a 10.9 

letter sound difference.

For first-grade nonsense word fluency, again, a large 

effect: 16.3 letter sounds within words. And then we 

also looked at the first grade oral reading fluency. 

This is not really a skill that is targeted in the 

first semester, but the idea, the instructional 

theory here for the Reading Corps is that what’s 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

38

learned around the nonsense word fluency should have 

some transfer to oral reading fluency. And so we 

assessed and looked at that oral reading fluency for 

the first graders.

This was the first time they received that 

assessment, and in fact, we found another large and 

statistically significant effect for that outcome. 

Let me show you what that looks like in terms of 

graphs. I’m gonna do one slide for each of these. I 

won’t go back over since we went over how to read 

these. What you can see here is the target for this 

skill is about seven or eight, and the kids here 

start quite low, less than five. 

I’d say there’s about two letter sounds upon 

kindergarten entry. And you can see that they both 

start, the treatment and control, at the same place 

so again, the groups started equally and you can see 

how differently they grew. Again, everyone’s in 

school, they’re learning, which is great. The control 

goes from about two to 16, whereas the kids who 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

39

received the Reading Corps program: all the way up to 

27.

And it so happens that 27 is that high-level 

benchmark that was set by the program that suggested 

these kids would be on track to have third-grade 

reading comprehension proficiency so this is very 

exciting. The effect size, when we convert this into 

effect size units, is .85 and the start says that’s 

statistically significant so as I mentioned before by 

any way that you think about effect size is .85 is a 

large effect for any time of intervention.

We’re gonna look at first graders. This is nonsense 

word fluency. Again, the target here for eligibility 

is let’s say about 35. You can see that the kids 

start, again, at the same place, around 25. Kids who 

don’t receive the Reading Corps grow. They grow to do 

about 45 letter sounds within nonsense words by 

winter. The children in the program grow to about62 

and that’s within measurement of that target. I think 

that it is 63 is the target at that time.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

40

When you look at the effect size units again, this is 

.81, highly significant, all those starts. This is 

another really great finding. Large effects for both 

our kindergarteners and first graders. The final 

slide looks at the oral reading fluency results. This 

graph look a little different because we did not 

collect any oral reading fluency data on the first 

graders at all because it really isn’t 

developmentally appropriate for the average student 

at that point in time.

Winter is about the time that we can do this and it 

would be appropriate for most kids. So you’re only 

seeing a point in time picture here and you’re seeing 

the kids who are in the control group scoring 27 or 

28. The kids in the program group scoring 41. And 

there isn’t a baseline benchmark here, but there is 

this target and the initial target here is 52. The 

important thing to look at is this difference here 

between the 41 and 27. 

On a scale that isn’t directly being intervened upon, 

and the effect size for this is a .61 effect that’s 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

41

also statistically significant. For those who are 

interested, particularly those who are experts in 

literacy instruction or early literacy instruction, 

about how much does nonsense word fluency instruction 

potentially transfer to oral reading fluency, which 

is the more complicated task. There is some evidence 

here that, in fact, it does given that the kids in 

the program do perform significantly better on this 

when they weren’t even directly instructed in the 

skill. Those are the results of Minnesota. Again, 

that was k-3 and I’m gonna transfer to Carrie who 

will share similar results from Wisconsin for 

kindergarten and first graders. 

CARRIE: Thanks, Marc. As Marc said, he presented on 

the findings of the study of the original Reading 

Corps program in Minnesota. We also did another study 

on the Wisconsin Reading Corps program, which is a 

replication of the Minnesota Reading Corps model. It 

takes place in Milwaukee public schools, 2017-2018, 

when we conducted the study marks the third year of 

the Reading Corps program in Wisconsin. You have to 

keep in mind the program’s been around three years in 

Wisconsin versus 15 years in Minnesota.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

42

And the program has about, as of last school year, 18 

AmeriCorps members who served as literacy tutors in 

ten schools in Milwaukee. And all of the schools are 

located in the city of Milwaukee. For the study, we 

replicated the Minnesota Reading Corps evaluation 

design. Again, all ten schools participated that met 

the IAL eligibility criteria. So we didn’t have to 

take a sample. There just weren’t that many schools 

so we just included all of the eligible schools.

And the scope of the study was limited to 

kindergarten and first grade because as I said we 

just didn’t have enough program-eligible second- and 

third-grade students for this study to do the study. 

Again, we did a randomized controlled trial and we 

tracked reading skills. We had about 176 program and 

control students in kindergarten and first grade and 

we looked at the differences between their fall 

benchmark and their winter benchmarks. 

Again, fall is in September; Winter is in January. 

The demographics in Wisconsin are quite different 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

43

than in Minnesota. We had 64 kindergarten students 

and 112 first-grade students participated in this 

study. A very small percentage for English Language 

Learners: only 6% for kindergarten and maybe about 

10% of first-grade students. But what we do see is 

that the percentage of free and reduced price lunch 

is quite high: 81-82% in first grade and 

kindergarten.

And then in terms of demographics, the population’s 

primarily African-American so about 80% in first 

grade and 90% in kindergarten. And then there are 

small percentages, around less than 10% Asian, 

Hispanic and a very small percentage of White 

students: around 2%. Again, we present a similar 

research question: for kindergarten and first-grade 

students, what is the impact of a single semester of 

the Wisconsin Reading Corps on program participants 

compared to similar students who did not receive the 

Reading Corps program?

We did find positive effects: about six and a half 

letter sound difference for kindergarten students and 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

44

a positive effect also in first grade, so about 8.7 

letter sounds within nonsense word differences. 

Showing the graphics for that, here is kindergarten. 

You can see, as Marc pointed out on the other side, 

the fall benchmark is the bottom horizontal line and 

then the winter benchmark is the higher horizontal 

line. 

And as you can see the program group, which is the 

orange line: they met the benchmark after a single 

semester. The benchmark was 27 and you can see both 

groups started out very low, below the fall 

benchmark. But then the program group achieved the 

benchmark we would like them to be at after a single 

semester. Compared to the control group. And again, 

it’s a difference of about six and a half letter 

sounds. The effect size for this is .55.

As we would expect, it’s a little lower than 

Minnesota Reading Corps, but it’s quite impressive, a 

substantial effect. And then for first grade we have 

similar findings. It’s important to point out the 

students started very low in first grade. They’re way 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

45

below fall benchmark. By the end of the semester, 

neither group, the control nor the program, had 

reached the winter benchmark. But you can see a 

substantial improvement, much more so in the program 

group.

So the difference in the trajectory for the program 

group and the control group is pretty obvious from 

the graph and there’s an effect size of .46, so 

again, a substantial effect for the program group. 

Next, we’re going to talk about our conclusions based 

on these findings. Marc is going to present on the 

major takeaways from the Minnesota study and I will 

discuss the major takeaways from the Wisconsin study 

and then some final thoughts. 

MARC: In summary, for Minnesota, really we’ve replicated 

the 2014 study that we had conducted for the K3 

results in 2018. We found statistically significant 

meaningful effect in all grades k-3. We found that 

Black and ELL students in particular, often 

considered at risk, particularly benefitted from the 

program so the effects of the program were larger for 

those groups. This is what was different from the 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

46

other study: we found that second and third grade 

students who we selected for the study because they 

furthest from benchmark in fall did significantly 

benefit from having the Reading Corps intervention.

And it supports our previous study’s hypothesis that 

more than one semester of tutoring can produce 

significant and positive effects on second and third 

grade students’ oral reading fluency. As Carrie had 

noted in the opening slides, our earlier study in 

2014 was a one-semester study for all grades, and so 

the findings from this one were one semester, 

kindergarten and first, and we found similar, 

positive effects. 

We really felt that we were cutting off the study too 

early; kids were still receiving intervention in 

second and third grade. We could model what we 

thought they might be able to do had we stayed with 

them longer and had some support from our modeling 

that these kids would have done quite well. And so, 

in this study we followed them for the full year and 

we found, in fact, that they did perform 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

47

statistically significantly better than their 

counterparts that did not receive the Reading Corps 

when they had access to the services over the full 

course of the year. I think that was Minnesota. 

CARRIE: Yes. And then for Wisconsin, again, we found 

positive impacts to the program among kindergarten 

and first-grade students, but I think the important 

takeaway is that this demonstrates that the Reading 

Corps program can be successfully replicated in other 

locations. So now that we’ve done a study on one of 

the replication sites. So now we have multiple 

studies: the 2014 study, the 2018 study of Minnesota 

and of course the 2018 study of Wisconsin that show 

that Reading Corpse is an effective and a replicable 

program that helps accelerate student learning and 

places them on a trajectory towards grade-level 

reading proficiency by the end of third grade.

It’s important to note that the program builds 

foundational emergent literacy and early reading 

skills by using data-informed and evidence-based 

interventions and this is facilitated through strong 

organizational support; they have a strong 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

48

 

organizational structure that supports the entire 

program and its intervention. Just some final 

thoughts about the study implications: we have 

positive impacts, but not just on your typical 

student, but also typically at-risk students.

We found that the program is very effective with 

Black and African-American students, with English 

Language Learner students, and this suggests that 

these students can particularly benefit from Reading 

Corps. And although we did observe impacts in older 

grades in the second and third grade, the program did 

produce its biggest effects most quickly among 

younger students. And so, that’s can have important 

program implications because more students can be 

impacted on early grades so that’s something for the 

program to consider.

And then, of course, as a research, I think us 

researchers think always about the next study, and so 

I think the only kind of question left is possibly a 

longitudinal study that could follow students who 

received Reading Corps early or at different times 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

49

even. There’s also a pre-K program which we didn’t 

discuss today, but it would be great to follow those 

students longitudinally and figure out: what are the 

long-term impacts of the program on student 

performance? We’re happy to answer any questions, but 

I didn’t know if you want to do a wrap-up first.

MARY: I think we’ll hand it to Jennifer to give us some 

final reflections here and then we’ll open it up to 

questions and answers. So thank you, Carrie, Marc, 

and Sadie and Jennifer.

JENNIFER: Thanks, Mary. On behalf of Chester and all of 

AmeriCorps State National, we want to appreciate all 

the amazing work that is being done by all of the 

presenters. We deeply value the relationship we have 

with NORC and just listening to all of the findings. 

I think it validates for us as funders that if you 

find a great partner, in this case it was the 

Minnesota Reading Corpse and the Minnesota 

Commission. And if you set the frame, and provide the 

resources and get out of the way, amazing impactful 

work occurs. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

50

I also think that Minnesota Reading Corps and its 

replication is both instructive and reassuring to 

everybody in the AmeriCorps portfolio in that the 

thoughtful and measured steps that they took to 

assess their program model. Before they became the 

juggernaut that they currently are, they didn’t just 

jump into a randomized controlled trial. They took 

deliberate small steps to assess their model. And 

that is something that should be reassuring to all of 

us regardless of where we are on the evidence 

continuum and the level of resources that we have is 

that they are instructive in taking bitable chunks to 

move ever forward with what they’re doing.

I also want to underling that for anyone who’s 

listening who still feels that we are asking you to 

do evidence for evidence sake, you may have heard at 

the beginning that Audrey talked about how this 

process has allowed them to get nine million dollars 

in match, which is indeed not evidence for evidence 

sake, so just a loving share. I am told that 

professional athletes psych themselves up by 

listening to music; I am told that because I am not 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

51

athletic, but I get myself focused and amped up with 

quotes.

And one of my favorite quotes is a Chinese proverb 

that says, “A journey of a thousand miles begins with 

a single step.” I know this to be true related to my 

own work. When I work on writing the notice for 

funding opportunity, my first step is to change the 

date from the previous year to current year, that is 

highly satisfying. I am not a researcher, I do not 

have fancy language, I went to business school and 

the most fancy we got in business school was we 

learned to say finance instead of finance.

That was the big fancy thing we learned in business 

school. So I don’t know fancy language, but I think 

what blows my mind about this presentation is not 

just the incredible impact of some of the hardest to 

serve that this program model has done, but to the 

single step that the AmeriCorps members who served 

1200 hours and the single step for the k through two 

students who participated in the program. The single 

step is a one-minute assessment that is done that 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

52

really sets the stage for the level of success in the 

model.

To borrow from my stepdaughter, “it helps you know 

where you at” because right at the beginning with one 

minute you know through just a minute that really 

begins to layer on what we’re able to see through 

this. And the good news is for AmeriCorps State and 

National, and the challenge for many of you is what 

are the single step that you are taking or what is 

the single step that you have committed to take in 

the application that submitted to us related to this, 

And specifically in the education space, we’re lucky 

that we have an incredible amount of programs that 

can take that first single step. More than half of 

our program grants and budget supports educationrelated programs, so people working in this space. We 

have more than 44,000 AmeriCorps members and nearly 

45,000 Senior volunteers that are supporting at-risk 

youth and more than 540,000 students in communities 

across the country. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

53

So there is incredible opportunity for people both to 

consider replicating this model, but also really 

thinking about: what are the lessons they can take 

from the journey that the Minnesota Commission, and 

Minnesota Reading Corps and Reading and Math have 

taken that are germane to the work you are doing. So 

I really appreciate the Research and Evaluation 

department for putting this together and offering 

this to our field, so thank you.

FEMALE: Thank you, Jennifer. With that, I’m gonna turn 

this over to Andrew who’s going to facilitate some 

questions and answers, so, Andrea.

ANDREA: Thank you so much. This was very exciting. 

Just a few things for our in-person audience: please 

state your name before asking your question. For our 

online audience, you can ask questions by typing it 

in the Q&A box, and you can also ask a question using 

the microphone on your computer by selecting the 

raise hand feature from the menu above and we will 

grant you microphone rights in the order questions 

come in.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

54

But for now we’ve been getting a couple in the chat 

box, so here’s the first one: Is Reading Corps 

looking at expanding the study in other states or had 

any dialogue with other professionals who can conduct 

this in their state? 

SADIE: This is Sadie. I’m happy to answer that question. 

Reading Corps is currently being replicated in 12 

states and Lindsay Dolce commented I think on that 

chat box and just a couple people below you and so 

we’re always looking to expand our research and our 

scope across the country. And she shared her contact 

information also and could connect with you offline 

about that. 

FEMALE: Here’s another one: Can you please explain the 

calculation behind the statistic that second- and 

third-graders made 47% more growth in half a year?

SADIE: That was the Minnesota finding. Marc, are you able 

to speak a little bit about how that’s translated 

into the 47%?

MARC: Yes. This Is one instance I wish Eric Hedberg was 

with us. I will channel him as best as I can. What we 

did is there’s a paper that he loved to share with 

folks, it’s meta-analysis that very nicely lays out 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

55

what the expected effect sizes in particular academic 

outcomes, what the expected amount of growth is in a 

particular year on particular outcomes. And so we 

used, for second and third grade students – I don’t 

know if he chose third grade or if he combined the 

two.

But we used that and given the growth we saw in ours, 

converted it into how much growth is expected in a 

year given how much growth we had in ours, and the 

difference in growth between the two groups: what 

does that equal in terms of the amount of growth over 

the course of a year? We can actually give you, and 

we’d be happy to have Eric send you exactly how we 

computed that, but that’s in the short form where 

that came from. I don’t know, Carrie, if you have any 

more to add to that.

CARRIE: I think that was a good explanation. It’s 

beyond the typical growth that you would see. Julia 

Brown asked: Would the Literacy Corps model translate 

to middle school? I’m thinking it’s the Reading 

Corps. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

56

FEMALE: It’s the Reading Corps model. That may be a 

programmatic question. I don’t think there’s any 

plans right now to convert it to middle school, but 

maybe that would be more a question for Sadie or 

Audrey. 

FEMALE: So the Reading Corpse model currently is 

kindergarten through third grade and we haven’t 

really expanded beyond third grade. Our interventions 

in the design of it is really focused on that 

specific age group. With that said, we do also have a 

Math Corps model – not focused on literacy, of 

course, focused on math, but it’s a very similar 

model with intervention and assessment that is 

focused on the fourth through eight grade. And we’d 

be happy to talk with you offline if that was 

something that you’d be interested in learning more 

about. 

FEMALE: I will add to that: in terms of the 

interventions that would be used that would obviously 

have to be tailored toward the correct age group, but 

I think there are a lot of lessons learned about how 

the program is structured, the training, the coaching 

that they provide, the relationships they have with 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

57

the school where the school provides and onsite coach 

as a partner in the process, and I think there’s a 

lot to be learned about how they structure the 

program that I think contribute to the positive 

results that we see. 

But obviously you have to have the right intervention 

too. Even though Reading Corps is not doing this 

intervention with middle schools, I think there’s a 

lot to be learned about how you implement a program 

in schools that works with students one-on-one.

FEMALE: These lessons learned, are they in a paper and 

how can people follow up with some of the things you 

just said?

FEMALE: The 2014 study that I mentioned, so the 

previous study that we did for CNCS, there is an 

implementation, we did a process evaluation, and in 

the final chapter and in the executive summary, we 

describe the things about the program that make it a 

success. Obviously there’s the evidence-based 

interventions that are important. And Sadie mentioned 

there’s a three-day training with AmeriCorps members, 

but there are other things about it: there’s a multi-
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

58

tiered supervision instruction that also includes 

regular coaching.

So it’s not just about training people and placing 

them in the program; it’s also about providing 

regular coaching and doing regular check-ins to make 

sure that the intervention is being implemented 

correctly and consistently. There’s the partnership 

with the schools where they have buy-in also into the 

process. And there’s the data-driven nature of the 

program so they identify students who need the 

program through this fall benchmark assessment so 

that’s really critical. 

It’s not just based on teacher recommendations or 

what kids are available, but it’s based on a data 

point that has to be taken and determines which 

students are eligible. And also, in some cases, where 

they provide intervention, especially to the older 

students, they do a weekly assessment on the kids and 

so, sometimes that weekly assessment helps them 

figure out what’s working and what isn’t with 

different students, so it’s very data-driven also.
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

59

And, of course, I have to add they also do regular 

evaluations of their program, which I think is also 

critical and I would say the program has evolved over 

time and those changes to the program have all been 

driven by the evidence, the evaluation, the data 

they’ve been collecting, too, for program 

improvement. I don’t know Marc, or Sadie or Audrey if 

you had anything else to add.

MARC: I think that’s a great answer and I would say 

definitely to give a shout-out all of the materials 

from the 2014 evaluation that were funded by CNCS are 

on the CNCS website and so you can go there and 

there’s executive summary school reports, it’s the 

impact evaluations, that actually includes the pre-k 

studies as well. That’s all available right now on 

CNCS’s website.

FEMALE: And that’s through the evidence exchange. So 

here we have lots of comments coming in. This is a 

comment: We love a longer-term study. We have an RSVP 

tutoring program with a similar program design and 

see great outcomes, but would love to know the longterm effects of our intervention. That’s just a 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

60

comment. Here we go: Do you consider matching 

students in demographics in addition to the fall 

benchmark scores? Can you talk through that decision?

FEMALE: It’s a randomized control trial so the 

randomization process theoretically takes care of 

differences in demographics. We should, 

theoretically, through the randomization process end 

up with two groups that are similar, but I always 

recommend that a baseline analysis be done to confirm 

that that’s the case. So after we did the 

randomization, we then did some statistical

comparisons between the groups on demographics. 

We looked at gender, we looked at race, we looked at 

ELL percentage, we looked at free and reduced price 

lunch. We look at all of these different demographics 

between the program and the control group and we 

found that it confirmed for us that the groups were 

similar, that the randomization process did work, 

however, I will add that sometimes just because 

things are random, sometimes randomly you can end up 

with some demographics that do end up being different 

between groups. It just happens sometimes. 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

61

And if it does, you can take care of that in your 

statistical model, but you have to do that baseline 

analysis in order to identify those differences. In 

our case, like I said, we did the baseline analysis. 

It confirmed that the randomization process works. We 

ended up with two groups that weren’t similar.

FEMALE: Great.

MARC: I might add to that. A benefit of doing this that 

can sometimes be a weakness, particularly of 

randomized control trials or if you’re familiar with 

IES, Institute of Education Sciences What Works 

Clearinghouse, one of the things that can knock you 

off having really strong evidence is if there’s 

unequal attrition between groups in a program so kids 

dropping out, or moving or what have you and we know 

particularly in authentic education settings and with 

programs that are serving at-risk kids that that can 

be a real problem just in general, independent of the 

program. 

And so, one of the benefits of having a matched pair 

randomization was we knew if a kid dropped out in one 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

62

of those conditions who their counterpart was in the 

other conditions and you could eliminate them from 

the sample and maintain equal groups and random 

groups, and so that was a benefit to doing a matched 

pair randomization. I think it was one of the main 

reasons we did it. Also, just to speak to what’s more 

important maybe in a way, and the literature clearly 

shows that the most predictive variable related to 

later student growth is a child’s own pre-test score 

more than any other demographic or characteristic 

that we know.

If you don’t have pre-test scores, then we do know 

that some of the other demographic variables like 

race, ethnicity, ELL status, et cetera. are 

predictive of later scores, but the most predictive 

thing is a pre-test score and so that’s why we 

focused on that.

MARY: I actually have a question. This is Mary Hyde. This 

is actually a question for either Audrey, Sadie or 

Marc and Carrie: I’m just curious about any lessons 

learned you’d like to share around the process of 

actually replication, and taking this to scale in 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

63

different states and communities across the country. 

Clearly, you’re data-driven, you’re using evidence as 

once of your lead criteria for making good decisions 

to replicate in other communities, but I’m just 

curious maybe from a programmatic or a policy just 

practical terms, what are some of the most important 

things you’d want to share with others who are 

thinking about either maybe replicating your model or 

some other model to Jennifer’s earlier point.

SADIE: This is Sadie, again. I think the first thing that 

came to mind as you were asking that question, Mary, 

is just being really clear about what’s essential to 

the model. I think we were pretty clear, even in 

Minnesota, as we were scaling the program statewide 

here. And that same principle applied when we started 

scaling the program nationally, but what are things 

that are just essential to the model? If you were to 

go into a school in Minneapolis, if you were to go to 

a school in [unintelligible], if you were to go to a 

school in Denver, Colorado, what things would you see 

consistently being applied in all of those settings?
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

64

So having consistency with the interventions, 

consistency with the assessments, consistency with 

the implementation of those things and then measures 

of fidelity to go with it so we could actually have 

confidence that the interventions, and the 

assessments and the model was being implemented as it 

was being designed. I think that’s just been a really 

important component like I said even here in 

Minnesota, but even more so as we started to 

replicate outside of Minnesota.

CARRIE: I agree with Sadie about that and I will add: 

you should a thorough process evaluation in order to 

identify what the essential items are so I definitely 

agree with that. Something that I’ve observed both 

with Reading Corps, but also with other programs that 

I’ve seen either success of fail at scaling is that I 

think you always need a local champion for your 

program. So, in Minnesota there are key individuals 

who are responsible for both its success and its 

sustainability. 

And I think that Reading Corps did a good job of 

identifying what are the important organizations or 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

65

people that they need to have onboard and involved in 

their other locations in order to be champions for 

this program. You have to identify who are the key 

players, and make sure that they’re onboard and part 

of that process. When I’ve seen scaling not work out, 

it’s because somebody’s trying to implement a program 

and there isn’t an advocate there.

There isn’t somebody who’s spearheading it, or 

whoever is the big player in that area, in that 

location they’re not part of implementing the program 

and that’s so critical I think to success.

FEMALE: Thank you. Here’s just a question which Sadie 

took up, but I’d like to hear other people, 

especially those of you listening: Are there any 

plans to share the three-day training AmeriCorps 

tutors receive with the foster grandparent 

volunteers? I don’t know if there’s someone. 

FEMALE: That’s an interesting thought.

FEMALE: Yeah, and I commented too we have done some 

work with foster grandparents in the past, although 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

66

it was in our pre-K model so feel free, anybody can 

feel free to reach out to me, I’ve plugged in my 

email if you’re interested in learning more. We do 

replicate our program model nationally as I said 

earlier so if there is any component of this model 

that is really seems interesting, we’re happy to have 

conversation and field any questions about that. 

Our training, I will say, is really focused on those 

specific interventions and on those specific 

assessments so we’d be happy just to share kind of 

like our philosophy with how we do that training and 

what that kind of looks like.

FEMALE: Thank you. I see people are typing.

MARC: I might add as an education research, when we first 

engaged through the CNCS evaluation with the Reading 

Corps, Carrie, myself and our team attended their 

training and as part of the process assessment 

observed some of the ongoing coaching and 

professional development that is a core component to 

this program. And even as someone who has a PhD in 

Child Development it was just incredibly impressive 

the amount of information that the program had put 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

67

together in such a tight package that was highly 

aligned with the four objectives this program was 

trying to achieve. 

They tried to give as much information to these 

folks, the coaches, the onsite coaches, the 

[unintelligible] coaches, the actual AmeriCorps 

members themselves in a short period of time. And one 

of the things I’ll note that often people said was, 

“There was so much there. I learned so much, but I 

was worried, ‘How am I gonna do this?” And that that 

wasn’t the end. I think Carrie noted this too, which 

was unique about this program: that build into this 

is a wonderful model for ongoing professional 

development through ongoing trainings, coaching, the 

materials they produce. 

I still have a copy of the huge literacy handbook 

that’s a three-inch ring binder full of materials 

that they learn about and can reference later, online 

resources that have videos that show you best 

practices so you can refresh yourself or improve. It 

really is a very comprehensive set of resources that 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

68

not just an AmeriCorps member, but I think we’ve also 

heard from teachers themselves, the researchers are 

really helpful and so the concept of being able to 

take some of the what the Reading Corps’ doing and 

maybe transfer it, maybe the material itself couldn’t 

be because it’s so highly aligned with what the 

Reading Corps’ trying to do, but this structure that 

has been put in place about how you train folks who 

for the most part know nothing about literacy 

development to be able to sit down with children or 

produce these types of effects as quickly as they do 

is something that I think is a real opportunity to 

learn from in terms of what the Reading Corps has 

been able to put together. 

FEMALE: Thank you, Marc. That’s very important. One 

other thing that Lindsay Dolce adds is: in terms of 

replication, I would add that having the time to work 

through planning process prior to start-up is 

important to long-term success. So, training, the 

planning process as well. I just want to give all our 

speakers thirty seconds each to say anything that 

you’d like to add. Carrie, do you want to begin?
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

69

CARRIE: First of all, I just want to thank everyone, 

CNCS, Reading Corps, Reading and Math, the state 

commission in Minnesota. I just want to thank 

everyone for given us the opportunity to be able to 

study this program and again, I would just go back to 

the fact that there’s a lot of evidence that this 

program is effective in having an impact with 

students, and that it’s both effective and highly 

replicable. I’m very pleased to have been involved 

with this program since 2011 and watch it evolve, 

watch it be replicated in 12 states and Washington 

D.C., I think, now. 

So I do think that there’s a lot to learn from this 

program. It’s a good model and now there’s a lot of 

research that I think points to what are the 

effective practices, what are the key elements that 

make it replicable, and there’s a lot to learn from 

that so thank you, everyone for giving us this 

opportunity to study what I think is a very promising 

program. 

FEMALE: Thank you, Carrie. Sadie or Audrey?
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

70

AUDREY: I would just say to everybody on the webinar 

and CNCS and I would extend an invitation to anybody 

who’s listening: if you do have additional continuous 

improvement ideas or research questions that you’d 

want to put in to our R&D pipeline, but all means 

send them our way. We obviously want to and need to 

keep learning. 

MARC: I think Carried covered everything. I’d just say that 

this is such a great example thinking in the 

education space: how do you take service, folks who 

want to provide a year of time to help improve 

educational system and integrate them in a way that 

can produce really big impacts above and beyond in 

collaboration with the schools with which they’re 

working. It’s been a pleasure to be able to study 

this program, learn from it and hopefully in our 

presentations and our work going forward, share some 

of the best parts, the active ingredients of this 

program with others. 

FEMALE: Thank you, so much. I’m just going to turn it 

to Mary Hyde.

MARY: Great. And I believe we’re out of time so thank you 

again to all of our speakers and for those of you in 
CNCS Webinar Transcription

The Power of National Service

Improving Children’s Literacy Outcomes

71

the audience who joined today for this exciting 

discussion. I also want to thank ICF, our 

communications contractor, for their technical 

support and coordination. Thank you, Emily. We will 

be sending out a post-webinar survey so please let us 

know your thoughts on the webinar or ideas for future 

webinars. And we will be posting this recording in 

about a month for those of you who weren’t able to 

join live or had to hop off early. Thank you, all, 

and have a great afternoon or evening depending on 

your time zone. 

END OF FILE