NCEE 2010-4016 U.S. DEPARTMENT OF EDUCATION

Effectiveness of Selected 

Supplemental Reading 

Comprehension Interventions: 

Findings from Two Student 

Cohorts 

Executive Summary 
Effectiveness of Selected 

Supplemental Reading 

Comprehension Interventions: 

Findings from Two Student 

Cohorts 

Executive Summary 

May 2010 

Susanne James-Burdumy 

John Deke 

Julieta Lugo-Gil 

Nancy Carey 

Alan Hershey 

Mathematica Policy Research 

Russell Gersten 

Rebecca Newman-Gonchar 

Joseph Dimino 

Kelly Haymond 

RG Research Group 

Bonnie Faddis 

RMC Research Corporation 

Audrey Pendleton 

Project Officer 

Institute of Education Sciences 

NCEE 2010-4016 

U.S. DEPARTMENT OF EDUCATION 
U.S. Department of Education 

Arne Duncan 

Secretary 

Institute of Education Sciences 

John Q. Easton 

Director 

National Center for Education Evaluation and Regional Assistance 

John Q. Easton 

Acting Commissioner 

May 2010 

The report was prepared for the Institute of Education Sciences under Contract No. ED-01-C0039/0010. The project 

officer is Audrey Pendleton in the National Center for Education Evaluation and Regional Assistance. 

IES evaluation reports present objective information on the conditions of implementation and impacts of the programs 

being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions 

policymakers or practitioners should take in light of the findings in the reports. 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to 

reprint this publication is not necessary, the citation should be: James-Burdumy, S., Deke, J., Lugo-Gil, J., Carey, N., 

Hershey, A., Gersten, R., Newman-Gonchar, R., Dimino, J., and Haymond, K. (2010). Effectiveness of Selected 

Supplemental Reading Comprehension Interventions: Findings from Two Student Cohorts—Executive Summary (NCEE 

2010-4016). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 

Sciences, U.S. Department of Education. 

To order copies of this report, 

• Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 1398, Jessup, MD 

20794-1398. 

• Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, call 800-872-

5327 (800-USA-LEARN). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter 

(TTY) should call 800-437-0833. 

• Fax your request to 301-470-1244. 

• Order online at www.edpubs.org. 

This report also is available on the IES website at http://ies.ed.gov/ncee. 

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For 

more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113.
iv 

DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST 

The research team for this evaluation consists of a prime contractor, Mathematica Policy 

Research, and two major subcontractors: RG Research Group and RMC Research Corporation. 

None of these organizations or their key staff members have financial interests that could be 

affected by findings from the study. None of the members of the Technical Working Group, 

convened by the research team to provide advice and guidance, have financial interests that 

could be affected by findings from the study. 
v 

 

EXECUTIVE SUMMARY 

EFFECTIVENESS OF SELECTED SUPPLEMENTAL READING COMPREHENSION 

INTERVENTIONS: FINDINGS FROM TWO STUDENT COHORTS 

Improving the ability of disadvantaged students to read and comprehend text is an important 

element in federal education policy aimed at closing the achievement gap. Title I of the No Child 

Left Behind Act (NCLB) calls on educators to close the gap between low- and high-achieving 

students using approaches that scientifically based research has shown to be effective. Such 

rigorous research is relatively scarce, however, so it is difficult for educators to determine how 

best to use Title I funds to improve student outcomes. Identifying interventions that improve 

reading comprehension is part of this challenge. 

There are increasing cognitive demands on student knowledge in middle elementary grades 

where students become primarily engaged in reading to learn, rather than learning to read (Chall 

1983). Children from disadvantaged backgrounds often lack general vocabulary, as well as 

vocabulary related to academic concepts that enable them to comprehend what they are reading 

and acquire content knowledge (Hart and Risley 1995). They also often do not know how to use 

strategies to organize and acquire knowledge from informational text in content areas such as 

science and social studies (Snow and Biancarosa 2003). Instructional approaches for improving 

comprehension are not as well developed as those for decoding and fluency (Snow 2002). 

Although multiple techniques for direct instruction of comprehension in narrative text have been 

well demonstrated in small studies, there is not as much evidence on the effectiveness of 

teaching reading comprehension within content areas (National Institute of Child Health and 

Human Development 2000). 

The Institute of Education Sciences (IES) of the Department of Education (ED) has 

undertaken a rigorous evaluation of curricula designed to improve reading comprehension as one 

step toward meeting that research gap. In 2004, ED contracted with Mathematica Policy 

Research and its subcontractors to conduct the study.1

 The study team worked with ED to refine 

the study design and select the curricula to be tested, and then recruited districts and schools, 

collected data on implementation and outcomes in two consecutive school years, and analyzed 

the data. The study was conducted based on a rigorous experimental design for assessing the 

effects of four reading comprehension curricula on reading comprehension in selected districts 

across the country, where schools were randomly assigned to use one of the four treatment 

curricula in their fifth-grade classrooms or to a control group. The four curricula included in the 

study are: (1) Project CRISS, developed by CRISS (Santa et al. 2004), (2) ReadAbout, developed 

by Scholastic (Scholastic 2005), (3) Read for Real, developed by Chapman University and 

Zaner-Bloser (Crawford et al. 2005), and (4) Reading for Knowledge, developed by the Success 

for All Foundation (Madden and Crenson 2006). 

1

These subcontractors were RMC Research Corporation, RG Research Group, the Vaughn Gross Center for 

Reading and Language Arts at the University of Texas at Austin, the University of Utah, and Evaluation Research 

Services. 
The experimental design ensures a valid basis for answering the study’s key research 

questions: 

1. What is the impact of the reading comprehension curricula as a whole on reading 

comprehension, and how do the impacts of the individual curricula compare to one 

another? 

2. How are student, teacher, and school characteristics related to impacts of the 

curricula? 

3. Which instructional practices are related to impacts of the curricula? 

4. What is the impact of the curricula on students one year after the end of the 

intervention implementation? 

5. Are impacts larger after schools and teachers have had one year of experience with 

the curricula? 

The study’s first report—based on the first year of data collected in 2006-2007 for the first 

cohort of fifth-grade students and released in May 2009 (James-Burdumy et al. 2009)—focused 

on the first three research questions. The findings indicated that, after one school year, there were 

no statistically significant positive impacts of the interventions, based on comparisons of fifthgrade student test scores in schools that were randomly assigned to use the interventions and 

schools that were randomly assigned to not use the interventions. Four statistically significant 

negative impacts of the curricula were observed. There was no clear pattern to the relationship 

between student, teacher, and school characteristics and the effectiveness of the interventions. 

SECOND YEAR STUDY COMPONENTS AT A GLANCE 

• Fifth-grade component – In this component, a second cohort of fifth-grade students from a subset of 

the study’s original schools was added to the study, maintaining the original treatment assignments. 

Fifth-grade teachers in treatment schools implemented their assigned interventions and fifth-grade 

teachers in control schools continued teaching reading using methods they would have used in the 

absence of the study. Pre-tests and post-tests administered to students were used to assess the impact of 

the interventions on the second cohort of students. The rationale for including this component in the 

study is that impacts may be larger after schools and teachers have had one year of experience using the 

curricula. 

• Sixth-grade component – In this component, the first cohort of students (all but 64 of whom were in 

sixth grade in the study’s second year) was tracked for one additional year and follow-up tests were 

administered at the end of the school year to assess whether the interventions had statistically significant 

impacts one year after the end of their implementation. Fourteen sixth-grade students (0.2 percent) had 

the same teacher in sixth grade as in fifth grade, but the study interventions were not implemented in the 

second year when first cohort students were in sixth grade. There are two main rationales for including 

this component in the study: (1) it is possible that impacts of the interventions could emerge in the 

second year even after the intervention implementation has ended and (2) to examine whether the 

negative effects of Reading for Knowledge observed in the first year continued into the second year. 

vi 
vii 

This report focuses on the fourth and fifth research questions, based on a second year of data 

collected for the study. The second year of the study focuses on (1) the impact of the 

interventions on Cohort 2 fifth-graders after one school year of implementation and (2) the 

impact of the interventions on Cohort 1 sixth graders one year after the end of the intervention 

implementation. In particular, it presents findings related to whether the curricula had an impact 

on students one year after the end of the intervention implementation based on follow-up student 

assessment data collected in spring 2008 for the first cohort of students (enrolled in the study in 

the 2006-2007 school year). The component of the study addressing this research question is 

referred to as the sixth-grade component of the second year of the study throughout the report 

(see box). This report also presents findings related to whether impacts are larger after teachers

and schools had one year of experience using the curricula (the distinction between teachers and 

schools is due to mobility of teachers – some teachers in the second year are new to the study 

schools, but they might still benefit from the experience of their colleagues who had previously 

implemented the curricula). These findings are based on data collected for a second cohort of 

fifth-grade students (enrolled in the study in the 2007-2008 school year, after treatment schools, 

and some treatment teachers, had one year of experience using the curricula). The component 

addressing this research question is referred to as the fifth-grade component of the second year of 

the study throughout the report. 

The main findings are: 

• The curricula did not have an impact on students one year after the end of their 

implementation. In the second year, after the first cohort of students was no longer 

using the interventions, there were no statistically significant impacts of any of the 

four curricula. 

• Impacts were not statistically significantly larger after schools had one year of 

experience using the curricula. Impacts for the second cohort of students were not 

statistically significantly different from zero or from the impacts for the first cohort of 

students. (Treatment students in the second cohort attended schools that had one prior 

year of experience using the study curricula, while treatment students in the first

cohort attended schools with no prior experience using the study curricula. Reading 

for Knowledge was not implemented with the second cohort of students.) 

• The impact of one of the curricula (ReadAbout) was statistically significantly 

larger after teachers had one year of experience using the curricula. There was a 

positive, statistically significant impact of ReadAbout on the social studies reading 

comprehension assessment for second-cohort students taught by teachers who were in 

the study both years (effect size: 0.22). This impact was statistically significantly

larger than the impact for first-cohort students taught by the same teachers in the first 

year of the study. 

In summary, our findings do not support the hypothesis that these four supplemental reading 

comprehension curricula improve students’ reading comprehension, except when ReadAbout 

teachers have had one prior year of experience using the ReadAbout curriculum. 
viii 

 

Curricula Included in the Second Study Year 

The curricula included in the two second-year study components differed. The design of the 

study did not call for the interventions to be implemented in the sixth-grade component of the 

study, and, indeed, the interventions were not implemented in that component.2

 Rather, the 

design called for following first-cohort students for one additional year after the end of the 

implementation of the interventions in the study’s first year, to assess whether implementation in 

the study’s first year had longer-term effects on students’ outcomes (measured in the study’s 

second year when first-cohort students were in sixth grade). Therefore, the sixth-grade 

component focused on examining the impacts of the interventions implemented in the study’s 

first year, which include Project CRISS (developed by CRISS) (Santa et al. 2004), ReadAbout 

(developed by Scholastic) (Scholastic 2005), Read for Real (developed by Chapman University 

and Zaner-Bloser) (Crawford et al. 2005), and Reading for Knowledge (developed by the 

Success for All Foundation) (Madden and Crenson 2006). 

Three of the four curricula (Project CRISS, ReadAbout, and Read for Real) were included in 

the fifth-grade component of the second year, which involves a new cohort of fifth-grade 

students. Reading for Knowledge was not included in this component because 9 of the 18 

schools that had been assigned to implement Reading for Knowledge elected not to continue 

implementing the intervention in the second year. 

Study Design 

The study’s second year (2007-2008) design builds on the study’s first year design (2006-

2007). Before the start of the first year, schools in districts that agreed to participate were 

randomly assigned to one of the five study arms (four intervention groups and one control 

group). In both years of the study, fifth-grade teachers in schools assigned to an intervention 

group developed their own strategies for incorporating the assigned reading comprehension 

curriculum into their daily schedules and their core reading instruction. (The curricula being 

evaluated in this study were designed to supplement—not replace—the core curriculum being 

used by each teacher.) Teachers in control group schools continued to teach reading using 

whatever methods they had been using before the study began. Due to the experimental design, 

differences in outcomes of students in the treatment and control groups are attributable to the 

curricula being tested.3

2

Thirty percent of sixth-grade students attended the same school in sixth grade as they did in fifth grade 

(because their school’s grade structure included sixth grade). Very few sixth-grade students (0.2 percent) had the 

same teacher in sixth grade as in fifth grade. As noted above, none of the sixth-grade students received instruction in 

the study interventions in sixth grade. 

3

The study design just discussed is also described in James-Burdumy et al. (2006). Early study design 

proposals are laid out in Glazerman and Myers (2004). 
SUMMARY OF FIRST- AND SECOND-YEAR EVALUATION DESIGN 

Intervention: 

• First Year: Four reading comprehension curricula (Project CRISS, ReadAbout, Read for Real, and 

Reading for Knowledge) were implemented with first-cohort students. 

• Second Year: 

o First-cohort students: Interventions were not implemented with first-cohort students. 

o Second-cohort students: Due to attrition of schools assigned to the Reading for Knowledge 

group, only three curricula (Project CRISS, ReadAbout, and Read for Real) were 

implemented with second-cohort students. 

Participants: 

• First Year: 10 districts, 89 schools, 268 teachers, and 6,349 fifth-grade students in the study’s first 

cohort. Districts were recruited from among those with at least 12 Title I schools, and schools were 

recruited only if they did not already use any of the four selected curricula. Students in those schools 

were eligible to participate if they were enrolled in fifth-grade classes as of January 1, 2007. Students 

in combined fourth-/fifth- or fifth-/sixth-grade classes were excluded, as were those with language 

barriers or in special education classes, although special education students mainstreamed in regular 

fifth-grade classes were eligible. 

• Second Year: 

o First-cohort students: In the second year, the 6,349 students from the first year attended 252 

schools, 176 of which agreed to permit follow-up testing of students. 

o Second-cohort students: 10 districts, 61 schools, 182 teachers, and 4,142 fifth-grade 

students in the study’s second cohort. The same eligibility and exclusion restrictions were 

used with the first and second cohorts of students. 

Research Design: 

• First Year: Within each district, schools were randomly assigned to an intervention group that would 

use one of the four curricula or to a control group that did not have access to any of the curricula being 

tested. Control group teachers could, however, use other supplemental reading programs. The study 

administered tests to Cohort 1 students near the beginning and end of the 2006-2007 school year, 

observed classrooms, and collected data from teacher questionnaires, student and school records, and 

the intervention developers. 

• Second Year: Schools and students maintained the same treatment (or control) group status in the 

second year. The study administered tests to Cohort 1 students at the end of the 2007-2008 school year 

and to Cohort 2 students near the beginning and end of the 2007-2008 school year, observed 

classrooms, and collected data from teacher questionnaires, student and school records, and the 

intervention developers. Cohort 2 impact analyses examined the effect of one year of exposure to the 

interventions after treatment schools and teachers had one year of experience using them. Cohort 1 

impact analyses examined the longer-term effects of the implementation of the interventions in the 

first study year. 

Outcomes: Impact estimates in both years focused on student reading comprehension test scores. 

Schools participating in the fifth-grade component of the study’s second year were in the 

same treatment or control group in the second year as in the first year. Students in the study’s 

sixth-grade component were classified according to their treatment or control status from the 

study’s first year. See box for a summary of the evaluation design. 

There were three key distinctions between the first and second years of the study. First, 

fewer curricula were included in the fifth-grade component of the study’s second year due to the 

attrition of schools assigned to implement Reading for Knowledge. Project CRISS, ReadAbout, 

and Read for Real were included in this component in the second year, while Reading for 

ix 
x 

 

Knowledge was not.4

 Second, fewer schools participated in the fifth-grade component of the 

study’s second year (61 of the 89 schools that participated in the first year continued 

participating in Year 2).5

 Third, more schools participated in the study’s second year than in the 

first year due to the study’s sixth-grade component, in which follow-up tests were administered 

to Cohort 1 students at the end of the 2007-2008 school year in a total of 176 schools. 

This study provides educators with a sense of the effectiveness of these curricula when used 

by teachers in “real-world” conditions. Although the study team worked to facilitate study 

activities such as the collection of data in study schools, the developers provided teacher training 

and follow-up support to teachers throughout the two study years, and teachers and schools could 

discontinue use of the curricula during the study period if they believed they were ineffective or 

too challenging to use. Therefore, the study conditions may be comparable to those many 

districts might face if they implemented these curricula in their schools. 

Collecting Data 

Addressing the study questions required information about the curricula and how they were 

implemented, study participants, and students’ performance outcomes. Information about 

teaching and implementation of the curricula was collected to support an examination of the 

fidelity of implementation to each curriculum design, the ways the curricula affected more 

general (non-curriculum-specific) teaching practices related to comprehension and vocabulary 

instruction, the resources required to implement the curricula, and the way in which the curricula 

affected teachers’ allocation of time during the school day. Data on all three “levels” of study 

participants—schools, teachers, and students—were collected as a basis for describing their 

characteristics as they entered the study. Outcomes for the first cohort of students were measured 

through assessments administered towards the end of the 2006-2007 and 2007-2008 school 

years. Outcomes for the second cohort of students were measured through assessments 

administered towards the end of the 2007-2008 school year. More information on the study’s key 

data sources is provided below. 

Information About Teaching and Implementation of the Curricula. Five data collection 

activities focused on teachers, teaching, and implementation of the four reading comprehension 

curricula. Two of these involved classroom observations, conducted in spring 2007 and spring 

2008 for two purposes. To support interpretation of the impact estimates, intervention-specific 

“fidelity” observations of fifth-grade classes taught by treatment group teachers were conducted 

to determine the extent to which the teachers adhered to the curriculum content and procedures 

prescribed by each developer. To describe more general teacher practices related to 

comprehension and vocabulary instruction (as opposed to practices linked to a specific 

4

Reading for Knowledge was examined as part of the sixth-grade component of the study, because the sixthgrade component focused on examining the longer-term effects of the four curricula implemented in the study’s first 

year with Cohort 1 students (all four study curricula, including Reading for Knowledge, were implemented in the 

first year). 

5

Of the 28 schools that left the study, 18 were assigned to Reading for Knowledge, 2 were assigned to Project 

CRISS, 2 were assigned to ReadAbout, 5 were assigned to Read for Real, and 1 was assigned to the control group. 
xi 

intervention) and determine whether these practices were correlated with intervention impacts, 

Expository Reading Comprehension (ERC) observations were carried out in both treatment and 

control group fifth-grade classrooms to record the frequency with which teachers engaged in 

behaviors that research suggests are effective comprehension and vocabulary teaching practices. 

The third data collection activity that addressed the implementation of the curricula was a survey 

of developers on the cost of their curriculum to school districts. The fourth data collection 

activity related to teaching was a survey of fifth-grade teachers in the study’s second year, 

administered to collect data on the amount of time students spent using informational text in a 

typical week. The last data collection activity related to teaching was a time allocation form 

administered to fifth-grade teachers in the second study year to collect data on teachers’ 

allocation of time during the school day. 

To help summarize the large amount of ERC observation data collected on general (nonintervention-specific) teaching practices related to comprehension and vocabulary instruction, 

the following three summary scales were created (for details on these scales, see Chapter II and 

Appendix F): 

• Traditional Interaction. This scale captures interactive teaching practices, primarily 

focused on vocabulary instruction and drawing inferences from text, that have been 

in use for many decades in American schools (Durkin 1978-1979; Brophy and 

Evertson 1976). 

• Reading Strategy Guidance. This scale captures teachers’ use of aspects of strategy 

instruction (such as using text structure and generating summaries to improve 

comprehension) to build students’ comprehension ability. 

• Classroom Management and Student Engagement. This scale captures teaching 

practices related to the management of student behavior and students’ engagement. 

Data on Teacher Characteristics. The fifth-grade Teacher Survey, conducted in early fall 

2006, was used to create two scales for examining the relationship between teacher 

characteristics and impacts (see Appendix F for details): 

• School Professional Culture. The School Professional Culture scale is intended to 

capture conditions in schools that affect the quality of instruction (Consortium on 

Chicago School Research 1999; Carlisle 2003). The scale’s 35 items—which were 

included in the Teacher Survey developed for this study—reflect teachers’ 

perceptions of the culture in their school, including relationships with colleagues, 

access to professional development, experiences with changes being implemented in 

their school, and leadership support in their school. 

• Teacher Efficacy. The Teacher Efficacy scale is intended to capture teachers’ ability 

to benefit from professional development (Sparks 1988; permission to use scale 

provided by Hoy and Woolfolk 1993). The scale’s 12 items, included in the Teacher 

Survey developed for this study, ask about teachers’ attitudes concerning student 

engagement, instructional strategies, and classroom management. 
Data on Students’ Baseline Achievement Levels. Two student assessments administered at 

the start of the 2006-2007 and 2007-2008 school years allowed the study team to characterize the 

achievement level of the two cohorts of study students at baseline: 

• Passage Comprehension subtest of the Group Reading Assessment and Diagnostic 

Evaluation (GRADE). This assessment, published by Pearson Learning Group, 

measures a student’s ability to comprehend text passages (Williams 2001). 

• Test of Silent Contextual Reading Fluency (TOSCRF). This assessment yields a 

score that reflects skills such as word identification, word meaning, and sentence 

structure, all of which are important skills for reading comprehension (Hammill et al. 

2006). 

Data on Student Outcomes. Data on students’ post-test outcomes were collected from two 

sources at the end of the fifth-grade year (spring 2007 for Cohort 1 and spring 2008 for Cohort 

2). First, students were retested using the GRADE (Williams 2001). In addition, students were 

tested for comprehension of social studies and science informational text, using assessments 

specially developed by the Educational Testing Service (ETS) for the study (Educational Testing 

Service 2007a and 2007b). To reduce burden, half the students were randomly assigned to take 

the science test and half to take the social studies test. Data on students’ follow-up outcomes 

were collected from these same assessments at the end of the sixth-grade year (spring 2008) for 

the first cohort of students. 

Cohort 1 Students Cohort 2 Students 

Study Year 1 

(2006-2007 

school year) 

• Cohort 1 students enter study as 

fifth graders 

• Interventions implemented with 

Cohort 1 treatment students 

• Administer pre-tests and post-tests 

• Not yet included in study 

Study Year 2 

(2007-2008 

school year) 

• Cohort 1 students remain in study as 

sixth graders 

• Interventions are not implemented 

with Cohort 1 students 

• Administer follow-up tests 

• Cohort 2 students enter study as 

fifth graders 

• Interventions implemented with 

Cohort 2 treatment students 

• Administer pre-tests and post-tests 

Summary of Findings from the Study’s First Year 

The key findings from the first year of the study focus on curriculum implementation and 

impacts on student achievement. The implementation analyses document treatment teachers’ 

training and feelings of preparedness to implement the curricula, adherence to their assigned 

curriculum, and teaching practices observed among teachers in the treatment and control group 

classrooms. The impact analyses examine how student outcomes were affected by the curricula 

and how the impacts relate to conditions and practices in study schools and classrooms. The key 

findings from the first year of the study were: 

xii 
xiii 

• At the time of the classroom observations in spring 2007, over 80 percent (81 to 

91 percent) of treatment teachers reported using their assigned curriculum. 

Eighty-one percent of Read for Real teachers, 83 percent of Reading for Knowledge 

teachers, 87 percent of ReadAbout teachers, and 91 percent of Project CRISS teachers 

reported using their assigned curriculum. 

• Classroom observation data from the first year of intervention implementation 

showed that teachers implemented 55 to 78 percent of the behaviors deemed 

important by the developers for implementing each curriculum. ReadAbout and 

Project CRISS teachers implemented, on average, 71 and 78 percent of such 

behaviors, respectively. Reading for Knowledge teachers implemented 58 and 

65 percent of the behaviors deemed important for the two types of instructional days 

that are part of the curriculum. Finally, Read for Real teachers implemented 55 and 

71 percent of the behaviors deemed important for the two types of instructional days 

that are part of that curriculum. 

• Two of three teacher practice scales were not statistically significantly different 

between the treatment and control groups. There were no statistically significant 

differences in the Reading Strategy Guidance and Classroom Management and 

Student Engagement scales. Scores on the third scale, Traditional Interaction, were 

statistically significantly lower for the treatment group than the control group (effect 

size: -0.52). 

• No statistically significant positive impacts of the curricula on student outcomes 

were observed in the study’s first year. Reading comprehension test scores were 

not statistically significantly higher in schools using the selected reading 

comprehension curricula than in control schools. 

• There was some evidence of statistically significant negative impacts on student 

test scores in the study’s first year. The treatment group as a whole scored lower 

than the control group on the GRADE assessment (effect size: -0.08), and the 

Reading for Knowledge group scored lower than the control group on the ETS 

science comprehension assessment (effect size: -0.21). On the composite test score, 

the treatment group as a whole scored lower than the control group and the Reading 

for Knowledge group scored lower than the control group (effect sizes: -0.08 and 

-0.14, respectively). 

Summary of Implementation Findings from the Study’s Second Year 

 The second year implementation analyses focused on documenting treatment teachers’ 

training, adherence to their assigned curriculum, teaching practices observed among teachers in 

the treatment and control group classrooms, and understanding teachers’ allocation of time 

during the school day. The key implementation findings from the study’s second year are: 

• During summer and early fall 2007, 50 to 91 percent of treatment teachers were 

trained to use the curricula. Fifty percent of Read for Real teachers, 89 percent of 

Project CRISS teachers, and 91 percent of ReadAbout teachers were trained in the use 

of the curricula. 
xiv 

• In the spring of the second year of the study, over 80 percent (83 to 96 percent) 

of treatment teachers reported using their assigned curriculum. Eighty-three 

percent of Read for Real teachers, 92 percent of Project CRISS teachers, and 

96 percent of ReadAbout teachers reported using their assigned curriculum. The 

percentage of teachers who reported using each of the three interventions did not 

differ significantly between the first and second years. 

• Classroom observation data from the second year of intervention 

implementation showed that teachers implemented 65 to 94 percent of the 

behaviors deemed important by the developers for implementing each 

curriculum. Project CRISS and ReadAbout teachers implemented, on average, 

65 and 94 percent of such behaviors, respectively, and Read for Real teachers 

implemented 75 and 76 percent of the behaviors deemed important for the two types 

of instructional days that are part of that curriculum. There were no statistically 

significant differences in average fidelity levels between the first and second study 

years. 

• Two of three teacher practice scales were not statistically significantly different 

between the treatment and control groups. There were no statistically significant 

differences in the Reading Strategy Guidance and Classroom Management and 

Student Engagement scales. Scores on the third scale, Traditional Interaction, were 

statistically significantly lower for the Project CRISS treatment group than the control 

group (effect size: -0.54). 

• Project CRISS teachers were statistically significantly less likely than control 

teachers to report engaging in enrichment activities (such as art, music, or 

physical education), non-curricular activities (such as lunch, recess, or 

arrival/dismissal activities), and other activities. Similar patterns were observed for 

ReadAbout and Read for Real, but those differences were not statistically significant. 

What Is the Impact of the Curricula on Students One Year After the End of the 

Intervention Implementation? 

No effects of the curricula on Cohort 1 students were observed in comparisons of outcomes 

measured one year after the end of the intervention implementation (in the study’s second year). 

For the three intervention groups that had no effect in the first year, effects in the second year 

remained indistinguishable from zero. For the intervention group that had evidence of a negative 

effect in Year 1 (Reading for Knowledge), the effect in the second year was indistinguishable 

from zero. Figures 1 to 4 show impacts of the curricula on Cohort 1 students’ follow-up test 

scores from spring 2008 (impacts on spring 2007 post-test scores are also shown for comparison 

purposes). Follow-up reading comprehension test scores in spring 2008 were not statistically 

significantly higher for students who attended treatment schools in the study’s first year relative 

to students who attended control schools in the study’s first year. 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Reading for 

Knowledge

Combined Treatment 

Group

Effect Size

Figure 1. Effects of Curricula on GRADE Scores, Cohort 1 Students

Spring 2007 (At End of Fifth Grade) Spring 2008 (At End of Sixth Grade)

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Reading for 

Knowledge

Combined Treatment 

Group

Effect Size

Figure 2. Effects of Curricula on Social Studies Reading Comprehension Scores, 

Cohort 1 Students

Spring 2007 (At End of Fifth Grade) Spring 2008 (At End of Sixth Grade)

xv 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Reading for 

Knowledge

Combined Treatment 

Group

Effect Size

Figure 3. Effects of Curricula on Science Reading Comphrehension Scores,

Cohort 1 Students

Spring 2007 (At End of Fifth Grade) Spring 2008 (At End of Sixth Grade)

* Statistically different from control group at the .05 level.

*

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Reading for 

Knowledge

Combined Treatment 

Group

Effect Size

Figure 4. Effects of Curricula on Composite Scores, Cohort 1 Students

Spring 2007 (At End of Fifth Grade) Spring 2008 (At End of Sixth Grade)

Note: The composite scores are based on the GRADE scores, social studies reading comprehension scores, and science reading 

comprehension scores.

* Statistically different from control group at the .05 level.

*

*

xvi 
xvii 

Were Impacts Larger After Schools and Teachers Had One Year of Experience with the 

Curricula? 

The second key research question examined in the second year of the study was whether 

impacts of the curricula were larger after schools and teachers had one year of experience using 

the curricula. As mentioned above, we distinguish between schools and teachers due to the 

mobility of teachers in and out of study schools. (Focusing on schools that participated in the 

study in both years, 76 percent of control group teachers and 72 percent of treatment group 

teachers remained in the study in both years. There were no statistically significant differences in 

the percentage of teachers remaining in the study across the treatment and control groups.) 

Impacts were not significantly larger after schools had one year of experience using the 

curricula. Overall, we found no statistically significant impacts of the interventions on any of the 

three student test score outcomes for the second cohort of fifth grade students, and there were no 

statistically significant differences between the one-year impacts for the first and second cohorts 

of students (Figures 5 to 8). 

To address the research question related to teacher experience, the study team focused on 

post-test data (measured at the end of fifth grade) from first and second cohort students whose 

teachers were in the study in both the first and second years to assess whether the one-year 

impacts for the second group of students were larger than the one-year impacts for the first 

group. 

The impact of one of the curricula (ReadAbout) was statistically significantly larger after 

teachers had one year of experience using the curricula (see Figures 9 to 12). When focusing on 

students of teachers who participated in the study for two years, we found one positive, 

statistically significant impact among students in the second cohort. In particular, there was a 

positive, statistically significant impact of ReadAbout on the social studies reading 

comprehension assessment (effect size: 0.22; Figure 10). To put this in perspective, for a student 

at the 50th percentile, an effect size of 0.10 represents about 4 percentile points, an effect size of 

0.15 represents about 6 percentile points, and an effect size of 0.20 represents about 8 percentile 

points. To provide additional perspective, a meta-analysis by Rosenshine and Meister (1994) 

found an average effect size of 0.32 across nine studies examining the impact of multiple reading 

comprehension strategy instruction on standardized test scores (this meta-analysis focused on 

reciprocal teaching, which involves the use of guided practice and dialogue between students and 

teachers to teach students about four comprehension strategies including question generation, 

summarization, prediction, and clarification). Another meta-analysis by Rosenshine, Meister, and 

Chapman (1996) found an average effect size of 0.36 across 13 studies examining the impact of 

question generation on standardized test scores. 

The impact of ReadAbout on the social studies reading comprehension post-test assessment 

for the second cohort of students was statistically significantly greater than the impact of 

ReadAbout on this outcome for the first cohort of students taught by the same teachers in the 

first year of the study (effect size difference: 0.28). ReadAbout’s impacts on the other 

assessments (GRADE and science comprehension) were not statistically significant. 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 5. Effects of School Experience with the Curricula on Post-Test GRADE Scores of 

Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation.

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 6. Effects of School Experience with the Curricula on Post-Test Social Studies Reading 

Comprehension Scores of Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation.

xviii 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 7. Effects of School Experience with the Curricula on Post-Test Science Reading 

Comprehension Scores of Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation.

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 8. Effects of School Experience with the Curricula on Post-Test Composite Test Scores 

of Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation. The composite scores are based on the 

GRADE scores, social studies reading comprehension scores, and science reading comprehension scores.

xix 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 9. Effects of Teacher Experience with the Curricula on Post-Test GRADE Scores of 

Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation.

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 10. Effects of Teacher Experience with the Curricula on Post-Test Social Studies 

Reading Comprehension Scores of Fifth-Grade Students

Cohort 1 Cohort 2

*

Note: These effects represent impacts of the interventions after one year of implementation.

* Statistically different from control group at the .05 level.

xx 
‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 11. Effects of Teacher Experience with the Curricula on Post-Test Science Reading 

Comprehension Scores of Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation.

‐0.2

‐0.15

‐0.1

‐0.05

0

0.05

0.1

0.15

0.2

Project CRISS ReadAbout Read for Real Combined Treatment 

Group

Effect Size

Figure 12. Effect of Teacher Experience with the Curricula on Post-Test Composite

Test Scores of Fifth-Grade Students

Cohort 1 Cohort 2

Note: These effects represent impacts of the interventions after one year of implementation. The composite scores are based on the 

GRADE scores, social studies reading comprehension scores, and science reading comprehension scores.

xxi 
xxii 

Findings from Nonexperimental Analyses 

 For this report, the study team conducted a set of nonexperimental analyses to examine the 

relationship between students’ test scores and classroom practices, teacher efficacy in the 

classroom, teacher professional development, and time students spent using informational text. 

The study team also examined the correlation of impacts and school characteristics. These 

findings must be interpreted with caution, as they are correlational in nature and, therefore, do 

not provide causal evidence of the relationship between the variables examined. 

 The key findings from these analyses are: 

• Two of the three teacher practice scales were correlated with test scores. There is 

evidence of a positive and statistically significant relationship between post-test 

scores and Classroom Management (14 of 16 correlations were statistically 

significant) and Reading Strategy Guidance (10 of 16 correlations were statistically 

significant) scales. The Traditional Interaction scale was not statistically significantly 

related to post-test scores. 

• Three sets of individual items from the ERC were found to have the largest 

number of statistically significant positive correlations with test scores (48 of 64).

These items included teaching practices related to (1) explicit comprehension strategy 

instruction (16 of 24 correlations were positive and statistically significant), (2) 

teachers’ management and responsiveness (18 of 24 correlations were positive and 

statistically significant), and (3) student engagement (14 of 16 correlations were 

positive and statistically significant). Among the other individual ERC items, just 15 

of 344 correlations were positive and statistically significant. 

• No statistically significant relationships were found between test scores and 

teacher efficacy, hours of professional development reported by teachers, or time 

teachers spent with students in reading activities or using informational text.
xxiii

REFERENCES

Brophy, J., and C. Evertson. Learning from Teaching: A Developmental Perspective. Boston, 

MA: Allyn and Bacon, 1976. 

Carlisle, J. “Teacher’s QUEST: Self-Administered Questionnaire.” Ann Arbor, MI: Regents of 

the University of Michigan, 2003. 

Chall, J. Stages of Reading Development. Fort Worth, TX: Harcourt-Brace, 1983. 

Consortium on Chicago School Research. “Improving Chicago’s Schools: The Teachers’ Turn, 

1999; Elementary School Teacher Survey, 1999.” Chicago: CCSR, 1999. 

[http://www.consortium-chicago.org]. 

Crawford, L.W., C.E. Martin, and M.M. Philbin. Read for Real: Nonfiction Strategies for 

Reading Results. Columbus, OH: Zaner-Bloser, 2005. 

Durkin, D. “What Classroom Observations Reveal About Reading Comprehension Instruction.” 

Reading Research Quarterly, vol. 14, no. 4, 1978-1979, pp. 481–533. 

Educational Testing Service. Science Reading Comprehension Assessment (unpublished). 

Princeton, NJ: ETS, 2007a. 

Educational Testing Service. Social Studies Reading Comprehension Assessment (unpublished). 

Princeton, NJ: ETS, 2007b. 

Glazerman, S., and D. Myers. “Assessing the Effectiveness of Education Interventions: Issues 

and Recommendations for the Title I Evaluation.” Washington, DC: Mathematica Policy 

Research, May 17, 2004. 

Hammill, D., J. Wiederholt, and E. Allen. Test of Silent Contextual Reading Fluency (TOSCRF), 

Examiner’s Manual. Austin, TX: PRO-ED, Inc., 2006. 

Hart, B., and T.R. Risley. Meaningful Differences in the Everyday Experience of Young 

American Children. Baltimore, MD: Brooks, 1995. 

Hoy, W.K., and A.E. Woolfolk. “Teachers’ Sense of Efficacy and the Organizational Health of 

Schools.” Elementary School Journal, vol. 93, 1993, pp. 355–372.

James-Burdumy, S., D. Myers, J. Deke, W. Mansfield, R. Gersten, J. Dimino, J. Dole, L. Liang, 

S. Vaughn, and M. Edmonds. “The National Evaluation of Reading Comprehension 

Interventions: Design Report.” Final report submitted to the U.S. Department of Education. 

Princeton, NJ: Mathematica Policy Research, May 2006. 
xxiv

James-Burdumy, S., W. Mansfield, J. Deke, N. Carey, J. Lugo-Gil, A. Hershey, A. Douglas, R. 

Gersten, R. Newman-Gonchar, J. Dimino, and B. Faddis. “Effectiveness of Selected 

Supplemental Reading Comprehension Interventions: Impacts on a First Cohort of FifthGrade Students” (NCEE 2009-4032). Washington, DC: National Center for Education 

Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 

Education, 2009. 

Madden, N.A., and V. Crenson. Reading for Knowledge. Baltimore, MD: Success for All 

Foundation, 2006. 

National Institute of Child Health and Human Development. Report of the National Reading 

Panel, Teaching Children to Read: An Evidence-Based Assessment of the Scientific 

Research Literature on Reading and Its Implications for Reading Instruction. (NIH 

publication no. 00-4769.) Washington, DC: U.S. Government Printing Office, 2000. 

Rosenshine, B., and C. Meister. “Reciprocal Teaching: A Review of the Research.” Review of 

Educational Research, vol. 64, no. 4, 1994, pp. 479–530. 

Rosenshine, B., C. Meister, and S. Chapman. “Teaching Students to Generate Questions: A 

Review of the Intervention Studies.” Review of Educational Research, vol. 66, no. 2, 1996, 

pp. 181–221. 

Santa, C.M., L.T. Havens, and B.J. Valdes. Project CRISS: Creating Independence through 

Student-Owned Strategies. (3rd ed.) Dubuque, IA: Kendall/Hunt Publishing, 2004. 

Scholastic. ReadAbout: The Personal Reading Coach for Every Student. New York, NY: 

Scholastic, 2005. 

Snow, C.E. Reading for Understanding: Toward a Research and Development Program in 

Reading Comprehension. Santa Monica, CA: RAND, 2002. 

Snow, C.E., and G. Biancarosa. Adolescent Literacy and the Achievement Gap: What Do We 

Know and Where Do We Go From Here? New York: Carnegie Corporation of New York, 

2003. 

Sparks, G.M. “Teachers’ Attitudes Toward Change and Subsequent Improvements in Classroom 

Teaching.” Journal of Educational Psychology, vol. 80, 1988, pp. 111–117. 

Williams, K.T. Group Reading Assessment and Diagnostic Evaluation (GRADE) Technical 

Manual. Circle Pines, MN: American Guidance Service, Inc., 2001.