Evaluation Report of the 

Impact of Post-Exit 

Progress Monitoring on 

Literacy Outcomes in 

the Reading Corps 

Program 

Prepared by 

Ethan R. Van Norman PhD – School Psychology 

December 2020 
Post-Exit Progress Monitoring Evaluation December 2020 2 

Executive Summary ...................................................................................... 3

Reading Corps ............................................................................................ 5

Background.................................................................................................. 5

Theory of Change (Present Project) .................................................................. 8

Purpose of Evaluation................................................................................... 9

Study 1 – Initial Observations of Regression of Literacy Skills.......................... 10

Rationale ....................................................................................................10

Description of Dataset & Method ....................................................................10

Results .......................................................................................................11

Conclusions ................................................................................................12

Study 2 – Quantifying Typical Rates of Regression of Student Skills after Exiting 

the Reading Corps Program ........................................................................ 14

Rationale ....................................................................................................14

Method.......................................................................................................14

Results .......................................................................................................15

Conclusions ................................................................................................16

Study 3 – Randomized Control Trial of the Effects of Post-Exit Progress Monitoring 

on Preventing the Regression of Student Skills ............................................... 17

Rationale ....................................................................................................17

Method.......................................................................................................17

Results .......................................................................................................18

Conclusions ................................................................................................19

Future Directions ....................................................................................... 20

Recommendations for Reading Corps..............................................................20

Points for Advancement ................................................................................20

Appendix .................................................................................................. 22
Post-Exit Progress Monitoring Evaluation December 2020 3 

Executive Summary 

According to the 2019 National Assessment of Educational Progress, approximately 65% of 

fourth grade students in the United States read at a level considered “not proficient.” Only 

36% of fourth grade students say that they “definitely can” explain the meaning of 

something they have read. The impact of literacy skills acquired in elementary grades on 

success in middle school, high school, and post-secondary settings has been documented 

for decades. The value of early intervention services in preventing the onset of later 

difficulties and ultimately improving student outcomes long-term is also well known. Yet, the 

systematic and strategic implementation of large scale supplemental preventive reading 

interventions still eludes many schools, particularly in areas that persistently experience 

economic hardship. The Reading Corps model is one approach that has demonstrated 

success in “scaling-up” the delivery of supplemental literacy interventions for at-risk 

students in schools, particularly in settings with limited resources. 

There are a number of evidence based interventions with robust findings targeting 

foundational early reading skills, reading fluency, and reading comprehension. However, 

there is much less information available in regards to the achievement levels of students 

after those supports are discontinued. In schools that have limited resources, it is likely, and 

prudent, to discontinue supplemental supports after students have shown significant 

progress and closed the gap with their peers. This is done so that other students in need are 

able to receive help. Acknowledging this reality, Reading Corps trains tutors to engage in 

frequent progress monitoring and data-based decision making to continually evaluate 

student performance. When students show exceptional progress, and have closed the gap 

with their peers, they are exited from the program so that other students at-risk for 

academic difficulties can participate in the program. 

Tutors and researchers in Reading Corps noticed that a number of students that showed 

substantial growth and ultimately exited the program went on to fail to reach proficiency on 

end of year assessments. In response, three studies were undertaken to (1) quantify the 

probability that students whom met exit criteria for the program would go on to achieve 

proficiency after supports were removed, (2) precisely measure the rate in which students 

typically regress after exiting the program, and (3) evaluate the impact of a brief low-cost

program modification to prevent the regression of literacy skills after students exit the 

program. The main findings from each study are presented in this executive summary and 

discussed in more detail in the body of the report. 

To better understand the impact of the Reading Corps program among students that have 

shown significant improvement, and were exited from the program, an extant analysis was 

conducted (Study 1) using outcomes from the 2015-2016 and 2016-2017 school years. The 

probability that students who met exit criteria from the Reading Corps intervention during 

the 2015-2016 school year and went on to achieve proficiency on end of year 2016 
Post-Exit Progress Monitoring Evaluation December 2020 4 

benchmark literacy assessments and beginning of year 2016 benchmark literacy 

assessments was compared to students that did not meet exit criteria for the program in 

2014-2015. Across kindergarten (n = 1,824), first grade (n = 2,523) and second grade (n = 

2,481) students, those that met exit criteria for Reading Corps program had, on average, a 

higher probability of meeting grade level benchmarks at the end of year and beginning of 

the following year. However, even amongst students that met exit criteria for the program, 

the probability that kindergarten students did not meet grade level expectations was equal 

39% and 58% in the Spring and Fall, respectively. Amongst first grade students,

probabilities of not meeting grade level benchmarks were equal to 40% and 45% in the 

Spring and Fall, respectively. Finally, the average probability of not meeting grade level 

benchmarks was equal to 38% in the Spring and 59% in the Fall amongst second grade 

students. 

To gain a more precise estimate of the rate in which students regressed in reading skills 

after exiting the program, a second study was undertaken (Study 2). In this study, a subset of 

students (n = 91) during the 2017-2018 school year that meet exit criteria for the program 

were assessed once a week for one min using curriculum-based measurement (CBM) 

procedures after leaving the program. Growth (number of correct response improvement 

per week) while students participated in the program was then compared to growth after 

students exited the program. Of the 91 students, 75 (82%) experienced decelerated growth 

after the intervention stopped. Across second and third grade students, the average rate of 

weekly improvement in words read correct per minute (WRCM) scores was equal to 3.50 

(SD = 1.65) during receipt of intervention and 0.09 (SD = 1.63) after exiting the intervention. 

A final study was conducted during the 2018-2019 program year to determine whether 

engaging in progress monitoring after students exited the program (i.e., post-exit progress 

monitoring; Study 3) offset the regression of skills acquired in Reading Corps program after 

interventions ceased. School sites were randomly assigned to a condition in which students 

were assessed once a week with CBM after meeting exit criteria (n = 261) or business as 

usual (n = 269) in which students were not assessed weekly after exiting. Students that 

participated in the post-exit progress monitoring condition relative to the business as usual 

group demonstrated greater growth, on average, between Winter and Spring benchmark 

assessments. In addition, more students that participated in post-exit progress monitoring 

achieved proficiency on end of year CBM benchmark assessments. Amongst grade three 

students, more students that participated in the post-exit progress monitoring condition 

achieved proficiency on the end of year state achievement tests than those participated in

the business as usual condition. Collectively, the results from Study 3 support programmatic 

implementation of post-exit progress monitoring. 

Overall, there seems to be promise in engaging in post-exit progress monitoring to prevent 

the regression of academic skills. However, more work is proposed to better pinpoint the 

classroom environmental factors as well as underlying student skill deficits that are likely to 

predict the acquisition and ultimate retention of literacy skills built through the program. 
Post-Exit Progress Monitoring Evaluation December 2020 5 

Reading Corps 

Background1 

Reading Corps was founded in 2003 and is the largest AmeriCorps tutoring program in the 

country. The program trains and places AmeriCorps members that serve as tutors in their 

local community school-based settings. Tutors deliver research-based literacy interventions 

for students at-risk of reading failure in pre-kindergarten through third grade. Tutors are 

supported by internal or site-based coaches as well as external master coaches. As of 2020, 

the Reading Corps model has been replicated in 12 states as well as Washington D.C. Over 

210,000 students and 8,834 tutors have participated in the program. An abbreviated logic 

model of Reading Corps program is presented below. 

Figure 1. Reading Corps Logic Model2

The Reading Corps program is consistent with a response to intervention (RtI) or multitiered systems of support (MTSS) delivery model. Within the Reading Corps program, an 

emphasis is placed on the early identification and delivery of supplemental preventive 

support to struggling readers. Consistent with best practices associated with MTSS and RtI, 

Reading Corps: 

1 A general overview of the Reading Corps program is provided here. More information about the 

program is available at their website: https://minnesotareadingcorps.org/ 2 Logic model taken from Impact Evaluation of the Minnesota Reading Corps Program (2017-2018). 
Post-Exit Progress Monitoring Evaluation December 2020 6 

• Engages in universal screening at least three times a year and identifies students 

whose performance falls below clearly defined grade level benchmarks 

• Uses evidence-based literacy interventions to provide supplemental support (1:1 

tutoring 20 mins per day 5 days per week)

• Collects brief measures of academic performance at least once a week 

• Makes treatment decisions to continue, intensify, or terminate interventions based 

upon student progress on weekly assessments 

• Ensures intervention and assessment fidelity through high quality training, frequent 

coaching, and on-site observations 

The use of assessment data within Reading Corps program is important to highlight for the 

present report. CBMs are used to make screening and progress monitoring decisions. CBMs 

are brief, standardized, indicators of academic performance. A typical CBM asks a student 

to complete a task that they would normally be asked to do in a classroom (e.g., read aloud 

from a grade level passage) and the number of correct responses per unit of time (e.g., 

words read correct per minute; WRCM) is the primary datum of interest. Drawing upon 

automaticity theory, the ability to fluently perform the critical academic behaviors identified 

by CBMs is highly predictive of broader academic skills. As such, CBM has a robust 

research base as a means to both identify students in need of additional help as well as an 

outcome measure to monitor the effects of interventions across time. 

Universal Screening / Benchmark Assessment: As stated above, a large pool of students in 

a participating school engage in universal screening at the beginning of the school year 

(Fall), immediately around winter break (Winter), and at the conclusion of the school year 

(Spring). Tutors administer three grade level CBM probes distributed by FastBridge 

Learning, and calculate the number of correct responses in one minute on each probe. 

Students that earn a median score below pre-determined thresholds, or benchmarks, (see 

below) are considered at-risk for reading difficulties and are invited to participate in the 

program. 

Table 1. Reading Corps Universal Screening Overview

Grade CBM Metric Fall Winter Spring

Kindergarten Letter Sound 

Fluency

Correct Letter 

Sounds per min

8 27 48

Grade 1 Nonsense 

Word 

Fluency

Correct Letter 

Sequences per 

min

36 63

Grade 1 Oral Reading 

Fluency

Words Read 

Correct per min

52 82

Grade 2 63 97 116

Grade 3 100 122 135

 
Post-Exit Progress Monitoring Evaluation December 2020 7 

Progress Monitoring: After obtaining parental consent, students begin receiving 

supplemental support, and tutors collect at least one CBM score per week. As data are 

collected they are plotted and compared to a weekly rate of improvement, or goal line, to 

evaluate progress. The goal line is drawn from the student’s baseline score to the end of 

year target and quantifies the expected rate of improvement a student needs to make per 

week to reach the end of year target. In all circumstances, performance that exceeds the 

goal line is desirable. Tutors, with support from internal and external coaches, engage in 

ongoing evaluation to make treatment decisions for individual students. Of relevance to this 

report, explicit exit criteria have been established by Reading Corps. To be exited from the 

program, 3 of the most recent 5 consecutive data points must be above the goal line and two 

of those scores must be above the upcoming seasonal benchmark. The figure below 

demonstrates instances in which both students showed substantial progress. The top student 

showed improvement, but not enough to warrant exiting the intervention. The other student

(bottom) showed substantial progress and two of those observations exceed the next 

seasonal benchmark. Upon verification from the tutor and coaches, the student would be 

exited from the Reading Corps program. 

Figure 2. Example Application of Exit Decisions for Reading Corps
Post-Exit Progress Monitoring Evaluation December 2020 8 

Theory of Change (Present Project) 

Delivering educational interventions on a large scale is a deceptively complex process. 

Previous impact evaluations have demonstrated the effectiveness and scalability of the 

Reading Corps model. However, a key assumption with most intervention research is that 

students will continue on their assumed upward trajectory of academic performance once 

initial skill deficits are remediated and interventions end. Unfortunately, efforts to remediate 

academic skill deficits may prove to be futile if students are unlikely to retain these newly 

acquired skills. The figure below summarizes the conceptualization of factors that may 

influence long-term intervention outcomes. 

Figure 3. Theory of Change Post-Exit Progress Monitoring

Here pre-intervention literacy contexts refers to the exposure students have had 

cumulatively up until the point in which they enter the Reading Corps program. The 

student’s access to literacy at home, the quality of core instruction in their classroom, and 

any other supplemental literacy activities the student is exposed to would constitute preintervention literacy contexts. In addition, persistent skill deficits that students bring with 

them to the Reading Corps program would fall within this realm. The next arrow, skills 

acquired via intervention, constitute the historical focus of evaluations of the Reading Corps

program. That is, the main mechanism in which Reading Corps has helped students toward 

their path of reading for understanding, has been through the delivery of supplemental 

evidence based literacy interventions. The purpose of this report is to both explore and 

intervene upon the last arrow, post-intervention literacy contexts. Specifically, it is 

hypothesized that if the Reading Corps program can better understand and ultimately 

intervene upon and improve the contexts in which students access literacy after they have 

acquired skills to remediate initial deficits, the probability that they will successfully 

continue on the path towards reading for understanding will greatly improve. 
Post-Exit Progress Monitoring Evaluation December 2020 9 

Purpose of Evaluation 

The purpose of this evaluation is to summarize three research studies undertaken to (1) 

better understand the performance of students following an exit from the Reading Corps

intervention, (2) precisely measure the rate in which students that exit Reading Corps

regress in academic skills, and (3) determine the effectiveness of a low-cost brief follow-up 

program modification to prevent the regression of academic skills. 

It should be noted that in Study 1 and Study 3, a key emphasis is placed on the proficiency 

status of students on CBM benchmark assessments and end of year state tests where 

appropriate. Much of the research that has evaluated the long-term impact of literacy 

interventions have done so using metrics such as group-based effect sizes (which shed light 

on the standardized difference between average performance of the intervention group and 

the comparison group) or follow-up performance from single-case experimental designs, 

often a few weeks after the conclusion of the intervention. In reality, an intermediary 

measure of student performance is more meaningful to help educators make decisions in 

schools. It is argued that establishing whether students are likely to achieve proficiency at 

the end of the school year, after removing supplemental supports, is an extremely

meaningful outcome inform resource allocation decisions in schools. The point is especially 

relevant for Reading Corps and schools as both are able to serve a limited number of 

students at a given time. 
Post-Exit Progress Monitoring Evaluation December 2020 10

Study 1 – Initial Observations of Regression of Literacy Skills 

Rationale 

For the purposes of the current report, Reading Corps researchers and staff noticed that 

many students whom participated in the intervention program, and were ultimately exited 

from the program, went on to not pass subsequent universal screening assessments. To 

better understand the likelihood, or probability, that students that did or did not meet exit 

criteria would go on to pass subsequent universal screening assessments, an analysis of 

extant data was undertaken. 

Description of Dataset & Method 

Data were available from 1,824 kindergarten 2,532 first grade and 2,481 second grade 

students for the study. All students participated in the Reading Corps program during the 

2016-2017 school year. Across grades the demographic breakdown of the sample was 51% 

female, 67% White, 14% Black, 10% Hispanic, 5% Asian, 2% American Indian, and 2% 

identified as belonging to more than one race. All states participating in the Reading Corps 

program were included in the study. 

Universal screening data were available from the Spring of 2017 and Fall of 2017 (see Table 

1). Kindergarten students in the sample read on average 66.63 (SD = 24.50) letter sounds 

per min in Spring 2017 and 31.01 (SD = 18.13) correct letter sounds per minute on measures 

of nonsense word fluency in Fall 2017. First grade students read on average 66.63 (SD = 

24.50) WRCM in Spring 2017 and 51.44 (SD = 25.73) in Fall 2017. Second grade students 

read on average 101.84 (SD = 20.87) WRCM in Spring 2017 and 83.77 (SD = 21.30) in Fall 

2017. 

Kindergarten, first grade, and second grade students spent on average 13.83 (SD = 9.42), 

18.61 (SD = 11.59), and 20.56 (SD = 11.44), weeks enrolled in the program, respectively. 

To estimate the likelihood that students would go on to meet Spring and Fall benchmarks 

after participating in the Reading Corps program, generalized linear mixed modeling 

(GLMM) was used to estimate the log-odds of achieving proficiency on subsequent universal 

screening assessments as a function of relevant predictors. The primary predictors of 

interest in the present study included: grade level, whether the student met exit criteria for 

the Reading Corps program, the number of weeks the student was enrolled in the Reading 

Corps program, and the interaction of number of weeks in the program and whether the 

student met exit criteria. Separate models were estimated to predict proficiency status on 

Spring and Fall assessments. Outcomes from GLMM analyses are converted to probabilities 

to aid in interpretation. 
Post-Exit Progress Monitoring Evaluation December 2020 11

Results 

The percentage of kindergarten students in the entire sample that were proficient on Spring 

and Fall benchmark assessments was equal to 44% and 31%, respectively. In first grade the 

percentages were equal to 28% and 22%, and in second grade they were equal to 26% and 

32%, respectively. 

The table below shows the results of the GLMMs predicting proficiency on Spring and Fall 

benchmark assessments. 

Table 2. Results of Generalized Linear Mixed Modeling for Spring and Fall Proficiency Statuses

Spring Benchmark Fall Benchmark 

Fixed Effects Log-Odds SE Log-Odds SE

Intercept -2.08*** 0.14 -2.14*** 0.13

Grade 1 -0.33*** 0.09 0.61***

Grade 2 -0.23** 0.09 0.03 0.08

Met Exit Criteria 2.80*** 0.15 1.94*** 0.13

Weeks in RC -0.02*** <0.01 -0.01** <0.01

Exit x Weeks 0.00 <0.01 0.00 <0.01

Random Effects Variance Variance 

Intercept 0.46 0.45

Deviance 6212.93 6914.56

*** p < .001; ** p < .01; * p < .05. 

Note. The intercept condition in this scenario reflected kindergarten students, who did not meet exit criteria for 

Reading Corps program and had received 0 weeks of intervention. Grade 1, Grade 2, and Met Exit Criteria were 

all dummy coded covariates. 

For interpretability, the outcomes of the GLMMs are converted to probabilities in the 

following figures. In the plot, the average number of weeks students participated in the 

Reading Corps program for a given grade is used to estimate probabilities. It should be 

noted however, that the small, but statistically significant effect of duration suggests that the 

longer students participated in the program, perhaps because they have more severe skill 

deficits that take time to remediate, the lower the likelihood they would eventually go on to 

pass the subsequent benchmark assessment.
Post-Exit Progress Monitoring Evaluation December 2020 12

Figure 4. Probability of Meeting Future Benchmark Proficiency Targets as a Function of Grade and Exit Status 

from Reading Corps

Conclusions 

Several findings are apparent from this study. First, students that meet exit criteria for 

Reading Corps program have a higher probability of going on to pass future benchmark 

assessments. 

Second, a large number of students that meet exit criteria for Reading Corps program did 

not achieve proficiency on future benchmark assessments. For instance, kindergarten 
Post-Exit Progress Monitoring Evaluation December 2020 13

students had a 39% and 58% chance of not meeting benchmark in the Spring and Fall, 

respectively, even after successfully exiting Reading Corps program. Amongst first grade

students, probabilities of not meeting grade level benchmarks were equal to 40% and 45% 

the Spring and Fall, respectively after successfully exiting the Reading Corps program. 

Finally, the average probability of not meeting grade level benchmarks in the Spring was 

equal to 38% and was equal to 59% in the Fall amongst second grade students that exited 

Reading Corps program.

The outcomes of this study suggest that it cannot be assumed that a student’s upward 

trajectory of performance will continue after they stop participating in Reading Corps

program and only receive typical classroom instruction. Based upon the exit criteria utilized 

by Reading Corps program, if this assumption held true, all students that exit the program 

would far exceed the end of year benchmark target.

The results of this study confirm suspicions held by Reading Corps researchers and staff 

members. However, limitations associated with the extant dataset, namely the limitation of 

only having access to Spring and Fall benchmark data, preclude a more thorough

understanding of the nature of student regression in academic skills. It would be helpful to 

better understand the rate in which students grow, week-to-week after exiting Reading 

Corps program. In addition, it would be helpful to understand whether the timing in which 

students exited the program was related to the rate in which students acquired as well as 

retained academic skills. Study 2 builds upon the study described here to address those 

concerns. 
Post-Exit Progress Monitoring Evaluation December 2020 14

Study 2 – Quantifying Typical Rates of Regression of Student Skills after Exiting the 

Reading Corps Program 

Rationale 

The purpose of Study 2 was to gain a better understanding of the typical rate in which 

students improve (or do not improve) after exiting Reading Corps. Information regarding 

week-to-week progress of students was not available in Study 1. Per program guidelines, 

after student’s successfully exit the intervention, they are not assessed on a weekly basis. 

This is understandable as students are not receiving intervention and thus typical treatment 

decisions (i.e., whether to change the intervention) do not have to be made. 

However, with the information from Study 1 in hand, it was decided that more frequent 

assessment, in the absence of interventions, would need to occur to get a more 

psychometrically sound estimate of student growth during and after the intervention. As 

such, a subset of students were recruited to be assessed after successfully exiting Reading 

Corps during the 2017-2018 school year. 

Method 

Sixteen schools agreed to allow tutors to progress monitor students after they had exited 

Reading Corps program. A total of 91 second and third grade students were eligible to 

participate in the study. To be included in the present analysis, students had to (1) have met 

exit criteria for the intervention and (2) have at least three CBM observations prior to, and 

after meeting exit criteria for Reading Corps. 

Of the 91 students, 34 were in second grade and 57 were in third grade. In second grade, 

53% of students were female, 53% were White, 29% Black, 12% Hispanic, 3% Asian, and 3% 

identified as belonging to more than one racial group. In third grade, 51% of students were 

female, 39% were White, 28% were Black, 25% were Hispanic, and 9% were Asian. In 

second grade 18% of students received services as English Learners whereas 28% of 

students in third grade received services as English Learners. 

For the purposes of the present analyses, and consistent with previous CBM progress 

monitoring research, outcomes from second and third grade students were combined. 

To estimate student growth during and after participating in Reading Corps, separate 

piecewise regression models were estimated for each student:

yyii = ββ0 + ββ1xx1 + ββ2 mmmmxx(xxii − mm, 0) + εε 

Where yi is a WRCM score for a student at week i. ββ0 and ββ1 are the intercept and weekly 

rate of improvement in WRCM the student demonstrated while participating in the program, 

respectively. ββ2 is a dummy variable that captures the difference in the weekly rate of 
Post-Exit Progress Monitoring Evaluation December 2020 15

improvement in WRCM after the week the student exited Reading Corps (a) compared to 

before the student exited the program. If week (x) is less than a (the WRCM score was 

collected during the intervention), the term defaults to 0. If x is greater than a (the WRCM 

score was collected after the student exited the intervention) the product of ββ2 and x was 

added to the regression equation. In this instance, a statistically significant ββ2 would suggest 

that the difference in growth during and post intervention was not equal to 0. In turn, weekly 

growth during the intervention is captured via ββ1 and growth after the student exited the 

intervention is equal to ββ1+ββ2. 

Of the 91 students included in the sample, 75, or roughly 82%, demonstrated growth

patterns better explained by the segmented regression model described above compared 

to a single monotonic rate of improvement across all WRCM scores. In addition, 37 students 

met exit criteria during the Fall semester, whereas 38 students met exit criteria during the 

Spring semester. Information regarding baseline performance and time spent in Reading 

Corps are presented in the table below for each group. 

Table 3. Descriptive Intervention and Baseline Information for Study Sample

All Students Fall Exit Spring Exit

M SD M SD M SD

Observations pre-exit 12.85 5.78 9.49 2.23 16.13 6.32

Observations post-exit 10.53 5.37 11.78 6.25 9.32 4.07

Weeks pre-exit 16.13 7.54 10.49 2.46 21.63 6.72

Weeks post-exit 17.56 6.88 23.57 2.37 11.72 4.27

Baseline WRCM score 75.61 17.12 77.06 17.80 75.21 17.55

Intervention Sessions 30.83 16.57 19.63 10.94 36.50 19.12

Results 

The table below presents growth in WRCM per week amongst all students in the sample, 

those that met exit criteria during the Fall semester and those that met exit criteria during 

the Spring semester during and after participation in Reading Corps. 

Table 4. Descriptive Growth Information for Study Sample

All Students Fall Exit Spring Exit

M SD M SD M SD

Growth pre-exit 3.50 1.65 4.32 1.75 2.69 1.05

Growth post-exit 0.09 1.63 0.54 0.44 -0.35 1.30

Students, on average, improved at a rate of 3.50 (SD = 1.65) WRCM per week while 

participating in the intervention. However, after exiting Reading Corps, growth was much 

less (M = 0.09, SD = 1.63). In addition, the difference in the weekly rate of improvement 

during the intervention between students that met exit criteria in the Fall (M = 4.32, SD = 
Post-Exit Progress Monitoring Evaluation December 2020 16

1.75) was much greater than students that met criteria in the Spring (M = 2.69, SD = 1.05; 

t[58.51] = 4.87, p < .001). The average difference in post-intervention growth (M = 0.54, SD = 

0.44 vs. M = -0.35, SD = 1.30) was also statistically significant (t[64.23] = -1.98, p < .001). 

Conclusions 

Several findings are apparent from this study. First, all students that participated in Reading 

Corps demonstrated weekly growth rates that far exceeded students that were not 

considered at-risk for reading difficulties based upon normative information provided by 

FastBridge Learning. 

Second, a rapid deceleration in growth was observed amongst all students after exiting 

Reading Corps. This pattern tended to be exacerbated if students were exited from the 

program in the Spring relative to the Fall. Students that exited in the Fall demonstrated a 

decelerated rate of improvement that was just below average based upon national 

normative information. However, students that exited in the Spring demonstrated, on 

average a steeper deceleration. 

Differences in baseline performance between the two groups may partially explain 

differences observed in the time taken to meet exit criteria and growth afterwards. 

However, students in each group did not display significantly different baseline scores 

during Fall screening (Fall Exit: M = 76.03, SD = 16.89; Spring Exit: M = 75.21, SD = 17.55). 

Although students scored similarly on measures of oral reading rate, follow-up 

investigations may be warranted to determine whether students differed in their profiles of 

foundational literacy skills that subsume oral reading rate, which may explain the 

differential response to interventions targeted the overarching skill. 

Considering students that exited the intervention in the Fall, it is unclear whether the act of 

engaging in weekly progress monitoring had some net positive effect. That is, compared to 

students enrolled in the broader Reading Corps program, it is unclear whether the 

observed rate of regression would have been worse if students were not assessed weekly 

after meeting exit criteria. Although the observed rates of growth after exiting Reading 

Corps program are not optimal, it is unclear whether there is any relative value in engaging 

said progress monitoring to maintain newly acquired skills. To answer this question a 

cluster randomized control trial was carried out (Study 3). 
Post-Exit Progress Monitoring Evaluation December 2020 17

Study 3 – Randomized Control Trial of the Effects of Post-Exit Progress Monitoring 

on Preventing the Regression of Student Skills 

Rationale 

Taken together, the results of Study 1 and Study 2 suggest that students that participate in 

Reading Corps show exceptional growth while in the program, but that they do not continue 

on as steep of an upward trajectory after supports cease. Although it is not feasible, nor 

appropriate, to continue delivering interventions throughout the entire school year to all 

students, a key question becomes, what can be done to promote the maintenance of these 

newly acquired skills? Several options may be possible, including delivering less intense 

interventions (e.g., 3 times per week or for 10 mins rather than 20 mins per session). As a 

baseline, the current study sought to determine what the net impact was when using an 

extremely brief (1 min once per week) intervention. If positive results are observed here, 

the rationale of scaling up or building upon said intervention would be supported. 

Method 

A cluster-based randomized control trial design was used for Study 3. During the 2018-2019 

school year 50 schools from the broader Reading Corps program (N = 480) were randomly 

assigned to the treatment group and 50 were assigned to a business as usual condition. 

Schools that were assigned to the treatment group were instructed to engage in post-exit 

progress monitoring (collecting 1 CBM assessment per week) of all students that met exit 

criteria during the Fall semester for the remainder of the school year. Schools in the 

business as usual condition did not engage in post-exit progress monitoring consistent with 

current program protocol. 

In addition to being randomly assigned, baseline equivalence of schools was tested, and 

confirmed on the following criteria: total enrollment, student race/ethnicity, percentage of 

students eligible for special education services, percentage of students eligible for English 

Learner services, percentage of students eligible for free or reduced priced lunch, and 

percentage of students in third grade achieving proficiency on state-wide achievement tests 

in reading. 

Student level inclusion criteria included (1) being enrolled in Reading Corps and meeting 

exit criteria during the Fall semester and (2) being enrolled in kindergarten, second or third 

grade. First grade students were excluded from the present study as the measures used for 

universal screening and progress monitoring switch mid-year from nonsense word fluency 

to CBM of oral reading. 
Post-Exit Progress Monitoring Evaluation December 2020 18

A total of 530 students across 76 schools (10 treatment and 14 business as usual schools did 

not have students meet exit criteria during the Fall semester) participated in the study. 

Demographic information for the students in each sample is presented in the table below. 

Table 5. Demographic Information for Business as Usual and Post-Exit Progress Monitoring Groups

Grade Business as Usual (n = 269) Treatment (n = 261)

Kindergarten 34% 33%

Second 27% 29%

Third 39% 38%

Gender

Female 55% 47%

Race

White 77% 80%

Black 6% 6%

Hispanic 6% 6%

Asian 3% 4%

Hawaiian or Pacific Islander <1% <1%

American Indian 1% <1%

Multi 2% 1%

The outcomes of interest in the present study included: average rates of weekly growth on 

CBM outcomes between Winter and Spring benchmark assessments (the span in which 

program supports were not being delivered), proficiency rates on end of year CBM 

benchmark assessments, and proficiency rates on end of year achievement tests in third 

grade. Descriptive results are presented here and the outcomes of three-level multilevel 

modeling inferential outcomes are available in the appendix. 

Results 

Table 6. Study Outcomes for Business as Usual and Post-Exit Progress Monitoring Groups

Business as Usual Treatment

Grade Outcome M SD M SD

Kindergarten Winter to Spring Growth 0.19 0.61 0.59 0.72

Second 0.82 0.68 0.84 0.68

Third 0.51 0.67 0.60 0.59

Kindergarten Met CBM Spring Benchmark 68% - 85% - 

Second 55% - 57% - 

Third 57% - 72% - 

Third Met State Test Proficiency 59% - 69% - 
Post-Exit Progress Monitoring Evaluation December 2020 19

Conclusions 

Several findings are apparent from this study. First, kindergarten students that engaged in 

post-exit progress monitoring demonstrated higher weekly rates of improvement between 

Winter and Spring benchmark assessments, and were more likely to achieve proficiency on 

end of year state tests, relative to students in the business as usual condition. 

Second grade students in the post-exit progress monitoring group showed slightly greater 

growth after exiting the intervention on average, relative to students in the business as usual 

group. Similarly, second grade students in the post-exit progress monitoring group had 

slightly more students achieve proficiency on the end of year CBM benchmark assessment. 

Amongst third grade students, growth, on average, between Winter and Spring benchmark 

assessments was slightly higher in the post-exit progress monitoring group compared to the 

business as usual group. The percentage of students that passed the end of year CBM 

benchmark assessment was substantially higher in the post-exit group relative to the 

business as usual group. Finally, more students that engaged in post-exit progress 

monitoring passed the end of year state achievement test relative to students in the business 

as usual group. 

Overall, post-exit progress monitoring appears to have variable impact across grades and 

student outcomes. Generally, most students that participate in post-exit progress monitoring 

have a higher likelihood of going on to achieve proficiency on end of year universal 

screening assessments than if they were not to engage in post-exit progress monitoring. 

Similarly, most students that engage in post-exit progress monitoring experience less 

regression in academic skills across the school year. 

The results of Study 3 suggest that there is some relative value in engaging in follow-up 

assessment, as a means of a low intensity intervention, after students have exited Reading 

Corps. Given this, it would be worthwhile to follow-up to better understand the individual 

student factors (e.g., pre-existing literacy skills) as well as ecological factors (e.g., quality of 

classroom instruction) that predict maintenance of intervention effects. 
Post-Exit Progress Monitoring Evaluation December 2020 20

Future Directions 

Recommendations for Reading Corps 

Based upon the results of the three studies described in this report, it is recommended that 

Reading Corps continue to engage in post-exit progress monitoring and consider 

implementing it with more school sites. It may even behoove the organization to adopt it as 

standard practice. 

Outcomes suggest that expending less than 2 mins per week has a net positive effect for 

most students and at the very least confers no negative consequences. Further, in addition to 

improving student outcomes, the program may consider using post-exit progress 

monitoring data to further engage in data-based decision making. That is, with further 

research, Reading Corps may identify a point in which student’s may be re-enrolled in the 

program if their performance drops below a certain level after a pre-determined number of 

weeks. 

Although this may pose logistical complications with Reading Corps in enrolling and reenrolling students, those students that initially meet exit criteria for the program may be 

treated differently than those that are enrolling for the first time. Given that students who 

were exited showed positive response to intervention initially, it may be that those students 

do not need the same intensity (e.g., two days a week vs. five days a week), or type 

(comprehension intervention vs. reading fluency intervention), of supports they received 

previously. 

Although this model presents its own logistical issues, the net benefit of preventing students 

from engaging in intensive interventions only to re-enroll after showing initial success,

similar to the fabled Sisyphus, will likely promote greater student outcomes and more 

effective resource allocation in the long run. 

Points for Advancement 

In addition to considering or expanding their model of post-intervention surveillance and 

intervention programming, it may benefit Reading Corps to further explore aspects of the 

proposed theory of change in the current report. Namely, it is likely that there are other 

post-intervention contexts that influence the maintenance of newly acquired intervention 

skills and ultimately impact the capacity for those students to read for understanding. The 

most apparent in school-based settings is the quality of regular education instruction. After 

students exit Reading Corps, for many the only other time in the school day dedicated to 

structured literacy instruction occurs in the regular education classroom. 
Post-Exit Progress Monitoring Evaluation December 2020 21

It would be beneficial to understand whether key indicators of high quality instruction as 

well as the frequency of active student engagement in reading in regular education 

classroom settings predict the maintenance of literacy skills acquired through participating 

in Reading Corps. Better understanding the interplay between intervention programming 

and classroom instruction may enable the development of more nuanced intervention 

packages and follow-up supports to promote maintenance of literacy skills. In addition, 

supports provided by Reading Corps tutors could be expanded to help classroom teachers 

deliver more effective literacy instruction. 

Similarly, there are characteristics of students that may contribute to pre-intervention 

contexts that predict the maintenance of literacy skills after exiting Reading Corps. Primary 

outcomes used to identify students for participation in the program and monitor intervention 

effectiveness are considered general outcome measures. More specifically, the data used to 

make instructional decisions in Reading Corps program are composed of many lower and 

higher order skills that are not directly assessed. For instance, if a student struggles with 

oral reading fluency, they may struggle with issues related to phonological processing or 

even comprehension—both of which are not directly assessed on CBM oral reading probes.

Thus, if a student receives interventions directly targeting oral reading rate, the underlying 

component skill causing the deficit may not have been fully remediated, which in turn may 

lead to renewed difficulties as classwork becomes more challenging. 

It may be worthwhile for Reading Corps to administer additional measures of reading 

performance before and after students are exited from the program, even on a small-scale 

experimental basis at first. This would in turn allow for the development of specific skill 

profiles that may help customize intervention programs (e.g., multi-component reading 

interventions) for specific students. Presumably, if the intervention better targets all facets of 

what are causing the underlying reading difficulty, the more likely students will retain their 

newly acquired literacy skills as demands for completing academic tasks continue to 

increase. 
Post-Exit Progress Monitoring Evaluation December 2020 22

Appendix3 

Table 7. Multilevel Longitudinal Modeling Results

3 Versions of the tables in the Appendix also appear in a manuscript currently under review for 

publication (Nelson, Klingbeil, Van Norman, & Parker, 2020: A Cluster Randomized Control Trial of Brief 

Follow-up Practice Sessions on Intervention Maintenance). 
Post-Exit Progress Monitoring Evaluation December 2020 23

Table 8. Generalized Linear Mixed Modeling Results Predicting Proficiency Status on End of Year 

Assessments