Research & Evidence Webinar Series:

The Power of National Service: Improving 

Children’s Literacy Outcomes

November 2018
Reading Corps Ongoing Improvement and Evidence Building

Establish Promise

Internal evaluation of 

alignment between 

design and application.

Key Questions

Is the program model 

stable? 

Is it promising? 

+ Process evaluations

+ Descriptive analyses 

of pre-post outcomes

+ Small Scale Research 

Projects

Refine and 

Evaluate

External evaluation of 

causal impact and 

iterative improvement

Key Questions

Is Reading Corps 

effective? 

Is the infrastructure 

strong?

+ Special education 

cost analysis

+ NORC Evaluation 

(MN)

Scale and Improve

External evaluation of 

causal impact of the 

program at scale. 

Key Questions

Is Reading Corps effective 

across implementation 

environments?

+ NORC Multi-State 

Evaluation

Ongoing internal 

research projects to 

improve programming

Continuous 

Improvement

Key Questions

Reading Corps works—

now, how can the program 

become more effective and 

efficient?

+ Multiple internally 

lead projects
Multi-State Impact Evaluation of 

the Reading Corps Program

(2017-18)

November 2018

Carrie E. Markovitz, NORC

Marc W. Hernandez, NORC

Sadie O’Connor, Reading & Math, Inc.

Minnesota and Wisconsin Evaluation Results
4

Presentation Outline

 Introduction 

 About the Reading Corps Program

 About the Multi-State Reading Corps Evaluation

 MN Reading Corps Evaluation Results

 WI Reading Corps Evaluation Results

 Takeaways and Study Implications

 Question & Answer
Introduction

Sadie O’Connor, Reading & Math, Inc.

Carrie Markovitz, NORC
6

The Reading Corps Program
7

The Reading Corps Program (continued)
8

The Reading Corps Program (continued)
9

Overview of Multi-State Reading Corps Evaluation 

 Funded by Reading & Math, Inc. and an Innovative Approaches to 

Literacy (IAL) Grant through the U.S. Department of Education

 Based on Previous Study: 2014 Minnesota Reading Corps K-3 

Impact Evaluation

 Conducted by NORC at the University of Chicago

 Drs. Carrie Markovitz and Marc Hernandez, Co-PIs

 Dr. Eric Hedberg, Lead Methodologist
10

Overview of Multi-State Reading Corps Evaluation 

 Key features of MN and WI study:

 Part of a multi-state evaluation that includes MN, WI, & FL

 Minnesota

– Original Reading Corps Program (since 2003)

– Assessed year-long outcomes for 2nd and 3rd grade students; focused on 

students furthest from the benchmark 

– Assessed single semester outcomes for K and 1st grade students

 Wisconsin

– Replication site of the Reading Corps K-3 model

– Assessed K and 1st grade students

– Assessed single semester outcomes for K and 1st grade students
11

Overview of 2017-2018 MN Reading Corps Evaluation 

 Research Questions:

1. For 2nd and 3rd grade Tier 2 students who are farthest from the Fall reading 

proficiency benchmark, what is the impact of a full school year of Minnesota 

Reading Corps on program participants compared to students who did not 

receive Reading Corps?

2. Does program impact vary by participant characteristics, such as demographics, 

program dosage, and program attendance?

3. For Kindergarten and 1st grade students, what is the impact of a single semester 

of the Minnesota Reading Corps on program participants compared to similar 

students who did not receive Reading Corps?
12

Reading Corps Evaluation Outcome Measures

 Second and Third Grade

 Oral Reading Fluency

 First Grade

 Letter sounds within nonsense words

 Oral Reading Fluency

 Kindergarten

 Letter sounds

Grade Measure Fall Winter Spring

Third CBM-Reading 100 122 135

Second CBM-Reading 63 97 116

First Test of Nonsense Words 36 63 N/A

First CBM-Reading N/A 52 82

Kindergarten Test of Letter Sounds 8 27 48
13

Overview of 2017-2018 MN Reading Corps Evaluation 

 24 K-3 schools participated during the 2017-2018 school year

 Representative sample

 Stratified by urban and rural

 Schools had to meet IAL eligibility criteria 

 Randomized Controlled Trial (RCT) 

 Randomly assigned program eligible K-3 students to program and control groups

 Tracked reading skills for 622 program and control K-3 students

 From Fall 2017 to Winter 2018 for Kindergarten and 1st grades

 From Fall 2017 to Spring 2018 for 2nd and 3rd grades (year-long study)
14

MN Reading Corps Randomization Process

 Created matched pairs of program eligible students based on the 

similarity of their Fall benchmark scores

 Pairs of students (within each school and within each grade) 

selected for participation in the study

 Random selection of pairs for K-1st grade students 

 Purposive selection of pairs furthest from benchmark for 2nd and 3rd grade students

 Randomly assigned students within each pair to program or control 

group 

 Program group (received Reading Corps services first at beginning of the school 

year)

 Control group (embargoed from receiving RC services for duration of study period)
Impact Evaluation of the Minnesota Reading Corps

Overview of Study and Results

Marc Hernandez, NORC
16

MN Reading Corps Sample Description (2nd and 3rd grade students)

 190 2nd grade and 212 3rd grade students participated Fall 2017 to 

Spring 2018 

 English Language Learners: 23.2% 2nd and 23.6% 3rd grade 

students

 FRPL eligible (school average): 52.3%

Second Graders   Third Grade
17

MN Reading Corps Second and Third Grade Evaluation Results

For 2nd and 3rd grade Tier 2 students farthest from the Fall benchmark, what is 

the impact of a full school year of Minnesota Reading Corps on program 

participants compared to students who did not receive Reading Corps?

 Substantial effect (6.4 word difference) for oral reading fluency

 Magnitude of program’s effect on 2nd and 3rd graders’ oral 

reading fluency is equivalent to 47% or an additional half-year of 

the average student’s annual growth in reading proficiency

Notable context: 

 Purposely included 2nd and 3rd grade Tier 2 students furthest from the Fall 

benchmark 
18

m

MN Reading Corps Second and Third Grade Reading Fluency Results

Second GradeThird Grade

Pooled 2nd and 3rd grade effect size: 0.28**
19

MN Reading Corps Evaluation Results by Major Demographic Group

Does program impact vary by participant demographics, such as gender, 

race, and English Language Learner (ELL) status?

 Larger impacts were found among key participant demographics 

in the 2nd and 3rd grades

Subgroup Subgroup 

N

Subgroup Effect

size

Full sample Effect 

size

Male 198 0.47* 

0.28**

Black or AfricanAmerican 66 0.50*

English Language 

Learner 94 0.53***

 Demographic subgroup analyses were not conducted on K and 1st grades due to small 

subgroup sample sizes 
20

MN Reading Corps Sample Description (Kindergarten/1st grade 

students)

 60 Kindergarten and 160 1st grade students participated Fall 2017 to 

Winter 2018

 English Language Learners: 36.7% K and 31.3% 1st grade students

 FRPL eligible (school average): 65.5% K and 55.4% 1st grade

Kindergarteners  First Graders
21

MN Reading Corps Kindergarten and First Grade Evaluation Results

For Kindergarten and first grade students, what is the impact of a single 

semester of the Minnesota Reading Corps on program participants 

compared to similar students who did not receive Reading Corps?

 Large effect (10.9 letter sounds difference) for Kindergarten letter 

sound fluency

 Large effect (16.3 letter sounds within words difference) for 1st

grade nonsense word frequency

 Large effect (13.3 word difference) for 1st grade oral reading 

fluency
22

MN Reading Corps Kindergarten Letter Sounds Fluency Results
23

MN Reading Corps First grade Nonsense Words Fluency Results
24

MN Reading Corps First grade Reading Fluency Results
Impact Evaluation of the Wisconsin Reading Corps

Overview of Study and Results

Carrie Markovitz, NORC
26

Overview of Wisconsin Reading Corps K-3 Program

 Replication site of the Minnesota Reading Corps K-3 model

 Milwaukee Public Schools

 2017-18 marked year three of the Reading Corps program in

Wisconsin

 18 AmeriCorps members served as literacy tutors in 10 schools in

Milwaukee 
27

Overview of 2017-2018 Wisconsin Reading Corps Evaluation 

 Replication of 2017-2018 Minnesota Reading Corps evaluation 

design

 Included all 10 Wisconsin schools that met IAL eligibility criteria (not a sample)

 Scope of study limited to Kindergarten and 1st grades (too few program-eligible 

second and third grade students for study)

 Randomized Controlled Trial (RCT) 

 Tracked reading skills for 176 program and control K-1 students 

from Fall 2017 to Winter 2018
28

WI Reading Corps Sample Description (Kindergarten and 1st grade 

students)

 64 Kindergarten and 112 1st grade students participated Fall 2017 to 

Winter 2018

 English Language Learners: 6.3% K and 9.8% 1st grade students

 FRPL eligible (school average): 82.2% K and 81.1% 1st grade

Kindergarteners

First Graders
29

WI Reading Corps Kindergarten and First Grade Evaluation Results

For Kindergarten and first grade students, what is the impact of a single 

semester of the Wisconsin Reading Corps on program participants 

compared to similar students who did not receive Reading Corps?

 Positive effect (6.5 sounds letter difference) for Kindergarten letter 

sound fluency

 Positive effect (8.7 letter sounds within word difference) for 1st

grade nonsense word frequency
30

WI Reading Corps Kindergarten Letter Sounds Fluency Results
31

WI Reading Corps First Grade Nonsense Words Competency Results
Conclusions

Carrie Markovitz, NORC

Marc Hernandez, NORC
33

2017-2018 Evaluation Take-Aways

 Replicated the 2014 Minnesota K-3 results in 2018

 Significant, meaningful effects in all grades K through 3rd grade

 Black and ELL students, often considered at-risk, particularly benefited from the 

program

 2nd and 3rd grade students furthest from benchmark in Fall also significantly 

benefited

 Supports previous study’s hypothesis that more than one semester of tutoring can 

produce significant, positive impacts on 2nd and 3rd grade students’ oral reading 

fluency
34

2017-2018 Evaluation Take-Aways (continued)

 Found positive impacts of the program among K and 1st grade 

students in Wisconsin

 Demonstrates that the program can be successfully replicated in other locations

 Multiple studies now show that the Reading Corps is an effective, 

replicable program that helps accelerate student learning, and 

places them on a trajectory towards grade-level reading proficiency 

by the end of 3rd grade

 Builds foundational emergent literacy and early reading skills through datainformed and evidence-based interventions, facilitated by strong organizational 

supports
35

Study Implications

 Positive impacts on typically at-risk students (ELL, Black) suggest 

that these students can particularly benefit from Reading Corps

 Since the program produces big effects quickly among younger 

students (K-1), more students can be impacted at earlier grades

 Longitudinal studies following students who received Reading 

Corps early or at different times PreK-3 would provide powerful 

new evidence of long-term program impact
Question & Answer
Thank You!