NCEE EVALUATION BrIEf february 2009 

After-School reading 

A reading program, adapted for use in the 

after­school setting, resulted in 48 more 

hours of reading instruction during the 

school year (20 percent more) for reading 

program students than for their counterparts, but it did not increase student 

achievement in reading. The impacts did 

not vary significantly in different grades 

or for students with different prior performance. Nor did the reading program students and their counterparts differ significantly on other in­school behaviors. 

After­school programs receive federal support through the 

21st Century Community Learning Centers, established in 

1999. A primary purpose of the program is to provide opportunities for academic enrichment to help students meet 

state and local standards in core content areas. Findings 

from a previous national evaluation of the program indicate 

that the program grants awarded between 1999 and 2002 

had, on average, a limited academic impact on the academic 

achievement of participating elementary school students. 

A possible factor is that most academic activities at the 

evaluation sites consisted of homework sessions in which 

students received limited additional academic assistance 

(such as instruction or assistance with homework). In addition, attendance was limited and sporadic. But analyses 

comparing the academic outcomes of frequent and infrequent participants suggest that increasing attendance alone 

is unlikely to improve the academic findings. So, the limited academic effects and the low levels of formal academic 

assistance offered in these programs highlight the need for 

better academic programming. In response, the Institute of 

Education Sciences supported the development and evaluation of instructional resources in core content areas that 

could be used in after­school programs. 

This brief presents findings for the first of two years of program operations (school year 2005­06) on a study for reading. 

The reading program 

The curriculum developer—Success for All Foundation— 

was selected through a competitive process to adapt its 

school­day reading materials for use in the after­school 

setting. The developer was asked to create material that is 

engaging for students, tied to academic standards, appropriate for students from diverse economic and social 

backgrounds, and fairly easy for teachers to use with a 

small amount of preparation time. 

The Success for All Foundation adapted its existing schoolday reading program to create Adventure Island, a structured reading model with daily, 45­minute lessons that 

involve switching quickly from one teacher­led activity to 

the next. It includes the key components of effective reading instruction identified by the National Reading Panel 

and builds cooperative learning into its daily classroom 

routines, which also include reading a variety of selected 

books and frequent assessments built into lessons to monitor individual student progress. 
2 After-School reAding 

The study 

The reading program was implemented in 25 after­school 

centers, which were chosen based on their expressed 

interest and their ability to implement the program and 

research design. The study sample of 1,828 students, in 

grades 2 through 5, were identified by local staff as in need 

of supplemental academic support to meet local academic 

standards and were enrolled in the after­school programs. 

Students were assigned by lottery to either the Success for 

All Foundation’s Adventure Island reading program or to 

the regular after­school program. The regular program consisted primarily of help with homework or locally assembled 

materials that do not follow a structured curriculum. 

The evaluation examines four primary questions: 

•	 Does the Adventure Island after­school reading program improve reading proficiency over what students 

would achieve in regular after­school programs, as 

measured by test scores? 

•	 What are the impacts of the after­school reading 

instruction for subgroups of students based on their 

prior academic performance and grade? 

•	 Does the after­school reading instruction affect other 

in­school academic behavior outcomes, as measured 

by reports from regular­school­day teachers of student 

engagement, behavior, and homework completion? 

•	 What does program implementation look like, and was 

it implemented as intended? 

The second and fourth questions address information to 

better target and implement the intervention. The third 

question addresses whether extended learning leads to additional positive or negative student academic behaviors. 

Impact findings from the first year are based on data collected from students, regular­school­day teachers, and 

school records. The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for reading, was administered to students at the beginning and end of the school 

year to measure the gains in achievement. For students 

in grades 2 and 3, the Dynamic Indicators of Basic Early 

Literacy Skills (DIBELS) was also administered to measure 

fluency. A survey of regular­school­day teachers was used 

to measure student academic behavior. The study also 

collected information about program implementation and 

student attendance. 

The study’s findings after one year 

In the first year of the study, Adventure Island, the reading 

model put in place in 25 after­school centers, produced the 

following interim findings. 

Student reading skills and other academic behaviors 

Students in the reading program did not experience a 

statistically significant impact on their performance on the 

SAT 10 reading test, overall or on any of the three subcomponents. There were positive and statistically significant program impacts on one of the two measures in the 

DIBELS fluency test, but after accounting for multiple comparisons, this finding is no longer statistically significant. 

The program had no positive or negative effects on teacher 

reported student behaviors during the school day. 

•	 The average total SAT 10 reading score of the Adventure Island group increased over the school year 

by 22.6 scaled points, compared with an increase of 

23.2 scaled points for the other group, resulting in a 

difference that is not statistically significant. There 

were also no statistically significant impacts on the 

three subcomponents of the SAT 10—vocabulary, reading comprehension, and word study skills (figure 1). 

•	 The estimated difference between the Adventure Island 

group and the other group was a statistically significant 3.7 points (effect size = 0.12) in the nonsense 

word fluency subtest of DIBELS. But after accounting 

for multiple comparisons, this finding is no longer 

statistically significant. For the oral fluency subtest of 

DIBELS, the difference of 2.3 points (effect size = 0.07) 

was not statistically significant. 

•	 Reading score impacts were not statistically significant 

within or between any of the subgroups—by prior 

achievement or by grade. 

•	 The reading program did not produce statistically significant impacts on any of the three academic behavior 

ncee evAluAtion Brief 
figure 1 

Student growth on reading test scores 

Growth from baseline (scaled score points) 

50 Reading program group (n = 1,048) 

Regular program group (n = 780) 

40 

30 Impact
=
0.3



Impact
=
–0.6

 Impact
=
–0.6


20 

10 

0 

Total Vocabulary Reading 

comprehension 

Impact
=
–1.9


Word study 

skills 

After-School reAding 3 

measures: homework completion, attentiveness in 

class, or classroom disruptiveness. 

Program implementation 

The strategies supporting the reading intervention were implemented as intended, but teacher feedback and program 

observations indicated some teacher implementation challenges. Student program participation led to an estimated 

increase in reading instruction of 20 percent more hours of 

reading instruction (48 hours) during the school year. 

•	 Of the reading program staff, 99 percent were certified 

teachers, most often with three or more years of teaching experience. More than 80 percent reported daily 

preparation of at least 30 minutes. The average studentto­staff ratio of 9:1 was as intended. 

•	 All 50 after­school teachers interviewed reported experiencing some challenges related to pacing. Seventy­four percent of observed program observations 

indicated that teachers did not complete all the components of the lessons in the allotted time; 84 percent of 

the observed teachers did not model or practice fluency 

as was intended during the lesson. 

•	 On average, the reading program students attended the 

after­school program for 70 days (73 percent of the instruction days offered), which was 7 more days than the 

other group. This translated into an increase in reading 

instruction of 48 hours (55 hours for reading program 

students compared with 6.5 hours for other students, 

who received normal after­school programming). 

Upcoming report 

The study was expanded to include a second year of implementation and data collection in 12 of the original participating reading centers. This sample includes students who 

were part of the study in the first year and students who 

were new to the study in the second year, allowing the new 

wave of data collection to shed light both on the cumulative 

impact of the enhanced after­school program on returning students and on the impact of a more mature program 

on new students. The results will be presented in the final 

report of the evaluation. 

For the full report, please visit: 

http://ies.ed.gov/pubsearch/pubsinfo. 

asp?pubid=NCEE20084021 

Black, A.R., Doolittle, F., Zhu P., Unterman, R., and Grossman, J. B. (2008). The Evaluation of Enhanced Academic 

Instruction in After­School Programs: Findings After the 

First Year of Implementation (NCEE 2008­4021). Washington, DC: National Center for Education Evaluation and 

Regional Assistance, Institute of Education Sciences, U.S. 

Department of Education. 

NCEE developed the Evaluation Briefs to offer short 

synopses of complex technical evaluation reports. This 

brief was not prepared by the study authors. U.S. Department of Education 

ncee 2009–4056