> Reading First-Ohio:

Year 4 Annual Report

**February 2008**

![](media/image1.png){width="1.84375in" height="1.1875in"}

Executive Summary

Introduction

Reading First-Ohio (RFO), an early literacy initiative that provides
funds to support research-based programs in reading for students in
grades K--3, is completing its fourth year of implementation. Since the
program's initiation in 2003, Westat and Learning Point Associates have
been providing services to evaluate its achievements, addressing both
formative and summative issues. This report extends this work and
provides an initial set of answers to six questions that are central to
the understanding of the maturing RFO program and its outcomes. These
questions are:

-   What is the value added to students' reading achievement if students
    are in an RFO school versus if they are not?

-   What is the value added to teacher instructional practices for
    teachers if they are in an RFO school versus if they are not?

-   What is the quality of implementation of the RFO schools and what
    explains the variance in levels of implementation?

-   What features of the RFO program are effective in terms of value
    added to student achievement?

-   What is the quality of professional development offered through the
    RFO program?

-   What happens to implementation of the RFO program in a school after
    the school's Reading First funding has been discontinued?

Approach

The evaluation uses a variety of data collection instruments and
approaches. Table E-1 shows the major primary data collections over the
first 4 years of the evaluation. In addition, we have drawn on
information collected by the state, such as student assessments in
reading using the Early Language and Literacy Classroom Observation
(ELLCO).

**Table E-1.---Primary d**ata collection activities

  -------- ---------- --------- ---------- --------- ---------- --------- ---------- -----------
  Cohort   School               School               School               School     
           year                 year                 year                 year       
           2003--04             2004--05             2005--06             2006--07   

           Survey     Site      Survey     Site      Survey     Site      Survey     Site visit
                      visit to             visit to             visit to             to case
                      case                 case                 case                 study and
                      study                study                study                exemplary
                      schools              schools              schools              schools

  Cohort 1 X          X                    X\*       X          X                    X

  Cohort 2                      X                                         X          

  Cohort 3                                           X          X                    
  -------- ---------- --------- ---------- --------- ---------- --------- ---------- -----------

\* Limited site visits were conducted in the fall; full site visits
occurred in the spring.

Last year, our report addressed fidelity of implementation for cohort 1
schools, provided baseline data for schools in cohort 3, and examined
assessment results for all three cohorts. This year, the focus of our
implementation analysis is on the cohort 2 schools just finishing their
third year of participation.[^1] In addition, the evaluation includes a
specially selected case study subsample of cohort 1 RFO schools and
comparable schools in the same districts that were not participating in
the program but that have been followed since the beginning of the
evaluation, as well as a small number of treatment schools from cohort 1
that showed large gains in achievement over the initial 3-year period.
Each year we have examined student achievement data for the RFO schools,
looking at multiple measures used to gauge program success---the DIBELS,
the TerraNova, and the Ohio Achievement Tests-Reading (OAT-R). In this
report, we expand our analyses and present OAT-R findings for schools
receiving Title I services as a comparison group for the RFO schools.

Findings

Presented below are the preliminary answers to the six key questions
based on year 4 evaluation findings.

Question 1: What is the valued added to students' reading achievement if
they are in an RFO school versus if they are not?

Our analyses address both the progress of schools participating in RFO
and how progress in these schools compares with that of similar schools
not receiving such services. The data suggest that reading achievement
in RFO schools is better now than it had been at program initiation. The
strongest gains are found for year 1 of program participation, after
which growth levels off. Initial data comparing students in Reading
First and Title I programs suggest that students in Reading First
programs perform significantly better on achievement measures than
students in Title I programs. Further, this difference increases the
longer the Reading First program is in place.

**Question 2: What is the value added to teacher instructional practices
for teachers if they are in an RFO school versus if they are not?**

The case study indicates that features of the RFO program are now
implemented in non-Reading First schools in the case study districts.
Comparison schools in the case study now use the same reading series
used in the RFO schools and do so in literacy blocks that are at least
90 minutes long. Comparison school teachers and administrators now refer
to the five components of reading. Recently, comparison school teachers
have had more access to formative assessments and growing requirements
that these assessments be administered on a schedule.

Instructional practices still differ, however, between RFO and
comparison schools in the case study. In Reading First schools, there is
more consistency in instruction. Our data suggest that this is due to
the guidance teachers have received from a literacy specialist and the
long-term and consistent professional development they have participated
in as a group. Our data also indicate that RFO teachers use assessments
to make instructional decisions more than non-RFO teachers. Related to
this, small groups were more frequently observed in RFO classrooms than
in non-RFO classrooms.

**Question 3: What is the quality of implementation of the RFO schools
and what explains the variance in levels of implementation?**

The data indicate the implementation of Ohio's model for RFO is strong
and has increased over time. In terms of the key features of the
program, professional development, instruction, and program support and
oversight have generally progressed in implementation. Teachers are
definitely focused on the five essential features, have grown more
comfortable using small group instructional and management techniques,
and have grown in the use of, and appreciation for, the value of
assessment data. ELLCO data provide confirming evidence that
implementation improves over time and by year 2 is strong.

That said, there is considerable variance in program implementation by
school according to the survey data. Features that predict level of
implementation have not yet been identified. And some variables
hypothesized to predict implementation, such as stability of
staff---principals, teachers, and literacy specialists, fail to do so
with consistency.

**Question 4: What features of the RFO program are effective in terms of
value added to student achievement?**

While our answer to this question is incomplete at this time, both the
survey and the case study data suggest the following to be critical
features:

-   Consistent and uniform use of curriculum and materials aligned with
    research on best practices;

-   The presence of a coach or literacy specialist to support teachers
    and provide individual guidance;

-   Use of assessments for instructional decisionmaking;

-   Ongoing, systematic professional development designed to move
    teachers from providing the basics of SBRR to addressing individual
    student and teacher needs; and

-   Principal engagement and leadership that includes both clear
    expectations for instruction and ongoing program monitoring.

**Question 5: What is the quality of professional development offered
through the RFO program?**

Professional development for teachers has had a strong impact on the RFO
schools, as indicated by teachers and other respondents in the Reading
First schools. Teachers stated that professional development influenced
literacy instruction in the school due to improving consistency in
teaching literacy; providing up-to-date, research-based information on
literacy and the five components; helping them develop insights into
their students' needs; giving them new and effective instructional
strategies; and improving collaboration among teachers.\
The role of literacy specialist in providing support is seen as very
important.

**Question 6: What happens to implementation of the RFO program in a
school after the school's Reading First funding has been discontinued?**

In nearly all of the case study districts and schools, plans were made
to maintain the Reading First instructional model: using the same
reading materials, emphasizing the five components, using data to inform
instruction, and forming small groups. However, most of the districts
and schools will not maintain a high level of instructional support
personnel due to fiscal constraints that all districts but one have
encountered.

Only two schools plan to retain a full-time literacy specialist. Two
others hope to, but a fiscal crisis may make it impossible.

Conclusions

The data suggest that RFO is making positive strides toward enhancing
the teaching and learning of early literacy skills in participating
districts. The key strategies embedded in the program---special support
staff, research-based reading strategies and materials, and data-driven
teaching---are recognized as making important contributions to the
changes that are occurring. Data from student assessments provide
encouraging evidence that what is being taught is being learned.

**Table of Contents**

**Section Page**

Executive Summary iii

I Background 1

Introduction 1

The Reading First-Ohio Program 2

The Instructional Approach 3

Assessment 3

Staffing 3

Professional Development 3

Reading First Program Changes in Year 4 4

The Reading First-Ohio Evaluation 4

Summary of Previous Findings 6

Reference 6

II Program Implementation Changes Over Time---Survey Data 7

Chapter 1. Introduction and Methodologies 7

Surveys of Program Implementation 7

Characteristics of the Survey Samples 9

Data Analysis 12

Case Study Component 12

The Year 4 Case Study Schools 12

Data Collection Methods and Sources 13

Analysis of Interview Data 14

Analysis of the Observation Data 14

References 16

Chapter 2. Professional Development and Technical Assistance 17

Engaging Teachers in Professional Development 17

Principals' Professional Development 20

The Trio 21

**Section Page**

II---continued

Chapter 3. Support Mechanisms 25

Leadership and Shared Vision 25

Coordination Among People and Programs 26

SBRR Core and Supplemental Materials 30

Scheduling Instruction 30

Chapter 4. Instructional Approaches 32

What Are Teachers Using to Teach Reading? 32

How Are Teachers Teaching Reading? 35

How Often Does Teachers' Instruction Address the Five\
Essential Components of Reading? 38

Why Does Instruction Look the Way It Does? 45

Reference 48

Chapter 5. Program Monitoring and Use of Data for\
Program Improvement 49

Chapter 6. Factors That Explain Differences in Levels of\
Implementation 51

Implementation as Measured by Teacher and Principal Surveys 51

Cohort 1 52

Cohort 2 54

Implementation as Measured the ELLCO 55

Factors That Influence the Extent to Which RFO is Implemented 58

Chapter 7. The Value Added to Teachers' Instructional Practices 61

The Reading First Program Setting 61

Cleveland 61

Springfield 62

Vinton 62

Dayton 62

Youngstown 63

Summary 63

**Section Page**

II---continued

Instructional and Organizational Support for Teachers 64

Teacher Professional Development 65

The Role of the Literacy Specialist 69

Administrative Support 73

Resources in the Reading First Schools 76

Conclusion 79

Literacy Instruction in the Reading First Schools 80

The Five Components 81

Data Use 81

Literacy Instruction in Comparison Schools 83

The Observed Classrooms 83

Grouping 84

Specific Focus 84

Materials 85

Teacher Activities 85

Student Responses 86

Conclusion 86

Chapter 8. Sustaining Change 88

District Plans to Sustain Reading First and Commitment of Resources 87

Vinton 88

Cleveland 89

Springfield 89

Youngstown 90

Dayton 90

Schools 91

Summary 91

III Achievement 93

Change in Performance in Reading First Schools Over Time 93

DIBELS Performance 93

Cohort 1 96

Cohort 2 103

Cohort 3 109

**Section Page**

III---continued

Performance on the TerraNova and the OAT-R 113

Cohort 1 115

Cohort 2 120

Cohort 3 124

Comparative Evidence 127

Case Study Findings 128

RFO-Title I Analysis 131

Implementation and Student Achievement 136

Relationship Between Implementation and Student Achievement 137

Relationship Between ELLCO and Student Achievement 138

References 140

IV Conclusion 141

**List of Appendixes**

A Responses to the Teacher Survey A-1

> B Responses to the Principal Survey B-1
>
> C Reading First-Ohio Surveys C-1

**List of Tables**

**Section I Tables Page**

I-1 Primary data collection activities 5

**Section II Tables**

II-1 Characteristics of teacher respondents: 2005 and 2007 10

II-2 Characteristics of Reading First schools as reported by principals:
2005 and 2007 11

> II-3 Case study schools 13
>
> II-4 Coding levels for the CIERA observation system 15

II-5 Average number of minutes spent in professional development per
month by cohort 2 teachers: 2007 18

> II-6 Cohort 2 teachers' responses about the time spent/allotted for
> instruction and its impact on the school's core reading program: 2007
> 31
>
> II-7 Minutes per day spent on reading, by school: Cross-sectional
> sample 31
>
> II-8 Percent of cohort 2 schools using each core reading program: 2005
> and 2007 32
>
> II-9 Implementation scales at three levels 52
>
> II-10 Instruction growth for cohort 1, by observed area (N=227) 56
>
> II-11 Instruction growth for cohort 2, by observed area (N=122) 57
>
> II-12 Instruction growth for cohort 3, by observed area (N=402) 57
>
> II-13 Overall stability, by school 58
>
> II-14 Relationships between year 3 implementation and contextual
> factors 59
>
> II-15 Relationships between year 3 implementation and contextual
> factors with alternative specifications of stability 60
>
> II-16 Summary of changes experienced by Reading First case study
> schools 63
>
> II-17 Assessment of success of RFO in the schools, by degree and area
> of impact 64

**List of Tables**

**Section II Tables---continued Page**

> II-18 The usefulness of year 4 professional development, by school 66
>
> II-19 The influence of professional development on instruction, by
> school 67
>
> II-20 Teacher perceptions of the importance of the literacy specialist
> for implementing Reading First, by school 71
>
> II-21 Teacher awareness of principal use of data, by school 74
>
> II-22 Teacher awareness of principal monitoring instruction, by school
> 75
>
> II-23 Teacher perceptions of adequacy of instructional materials for
> meeting student needs, by school 77
>
> II-24 Supports for struggling readers, by school 78
>
> II-25 Percent of segments with specific grouping patterns, by type of
> classroom 84
>
> II-26 Percent of segments with specific foci of instruction, by type
> of classroom 85
>
> II-27 Percent of segments with specific materials used, by type of
> classroom 85
>
> II-28 Teacher activities in the observed classrooms, by type of
> classroom 86
>
> II-29 Student responses in the observed classrooms 86

**Section III Tables**

> III-1 DIBELS assessments 94
>
> III-2 Summary of DIBELS results for cohorts 1--3 95
>
> III-3 Comparison of DIBELS results for years 1--4 in RFO cohort 1: Raw
> score 97
>
> III-4 Comparison of DIBELS results for years 1--4 in RFO cohort 1:
> Percent proficient 99
>
> III-5 Comparison of DIBELS results for the cohort 1 longitudinal and
> cross-sectional groups: Percent proficient 100
>
> III-6 Comparison of DIBELS results for years 1--4 in RFO cohort 1,\
> by race/ethnicity 101

**Section III Tables---continued Page**

> III-7 Comparison of DIBELS results for years 1--4 in RFO cohort 1,\
> by special service 102
>
> III-8 Comparison of DIBELS results for years 1--3 in RFO cohort 2: Raw
> score 103
>
> III-9 Comparison of DIBELS results for years 1--3 in RFO cohort 2:
> Percent proficient 105
>
> III-10 Comparison of DIBELS results for cohort 2 longitudinal and
> cross-sectional groups: Percent proficient 106
>
> III-11 Comparison of DIBELS results for years 1--3 in RFO cohort 2, by
> race/ ethnicity 107
>
> III-12 Comparison of DIBELS results in year 1--3 in RFO cohort 2, by
> special service 108
>
> III-13 Comparison of DIBELS results in years 1--2 in RFO cohort 3: Raw
> score 109
>
> III-14 Comparison of DIBELS results in years 1--2 in RFO cohort 3:
> Percent proficient 110
>
> III-15 Comparison of DIBELS results for years 1--2 in RFO cohort 3,\
> by race/ethnicity 111
>
> III-16 Comparison of DIBELS results for years 1--2 in RFO cohort 3,\
> by special service 112
>
> III-17 Summary of TerraNova and OAT-R results for cohorts 1--3 114
>
> III-18 Comparison of TerraNova and OAT-R results for years 1--4 in
> RFO\
> cohort 1: Scale scores 115
>
> III-19 Comparison of TerraNova and OAT-R results for years 1--4 in
> RFO\
> cohort 1: Percent at benchmark 116
>
> III-20 Comparison of TerraNova results for the cohort 1 longitudinal
> and cross-sectional groups 117
>
> III-21 Comparison of TerraNova and OAT-R results for years 1--4 in
> RFO\
> cohort 1, by race/ ethnicity: Scale scores 118
>
> III-22 Comparison of TerraNova and OAT-R results for years 1--4 in
> RFO\
> cohort 1, by special service: Scale scores 119
>
> III-23 Comparison of TerraNova and OAT-R results for years 1--4 in
> RFO\
> cohort 1, by locale: Scale scores 119

**Section III Tables---continued Page**

> III-24 Comparison of TerraNova and OAT-R results for years 1--3 in
> RFO\
> cohort 2: Scale scores 120
>
> III-25 Comparison of TerraNova and OAT-R results for years 1--3 in
> RFO\
> cohort 2: Percent at benchmark 121
>
> III-26 Comparison of TerraNova results for the cohort 2 longitudinal
> and cross-sectional groups 122
>
> III-27 Comparison of TerraNova and OAT-R results for years 1--3 in
> RFO\
> cohort 2, by race/ ethnicity: Scale scores 122
>
> III-28 Comparison of TerraNova and OAT-R results for years 1--3 in
> RFO\
> cohort 2, by special service: Scale scores 123
>
> III-29 Comparison of TerraNova and OAT-R results for years 1--3 in RFO
> cohort 2, by locale: Scale score 123
>
> III-30 Comparison of TerraNova and OAT-R results for years 1--2 in
> RFO\
> cohort 3: Scale scores 124
>
> III-31 Comparison of TerraNova and OAT-R results in RFO cohort 3 in
> years 1--2: Percent at benchmark 125
>
> III-32 Comparison of TerraNova and OAT-R results in RFO cohort 3 for
> years 1--2, by race/ethnicity: Scale scores 126
>
> III-33 Comparison of TerraNova and OAT-R results for years 1--2 in
> RFO\
> cohort 3, by special service: Scale scores 126
>
> III-34 Comparison of TerraNova and OAT-R results for years 1--2 in
> RFO\
> cohort 3, by locale: Scale score 127
>
> III-35 Cross-sectional comparison of performance between treatment and
> comparison students 128
>
> III-36 Longitudinal comparison of performance between treatment and
> comparison students 129
>
> III-37 Comparison of demographic characteristics of the longitudinal
> sample 129
>
> III-38 Comparison of performance between stayers and movers 130
>
> III-39 HLM results for RFO effect 131
>
> III-40 Comparison student achievement of RFO and Title I schools at
> the baseline 132

**Section III Tables---continued Page**

> III-41 Comparison of post-program results from RFO and Title I
> students 132
>
> III-42 Comparison of post-program results from RFO and Title I
> students,\
> by exposure: Adjusted score 132
>
> III-43 Comparison of post-program results from RFO and Title I
> students by subgroup: Adjusted score 133
>
> III-44 HLM results for the overall sample 135
>
> III-45 Regression results accounting for individual mobility 136
>
> III-46 Relationships between implementation and student achievement
> change (N=51) 137
>
> III-47 Relationships between implementation and student achievement
> level (N=51) 138
>
> III-48 Relationships between ELLCO observation and student achievement
> level, by observed area (N=104) 139

**List of Figures**

**Section II Figures Page**

II-1 Principals' perceptions of teachers' knowledge in reading: 2007 20

> II-2 Principals' opinions about the various services provided by their
> schools' building Trio: 2007 22
>
> II-3 Percent of teachers receiving various instructional supports from
> the literacy specialist 23
>
> II-4 Percent of teachers and principals reporting that their schools
> promoted various strategies for involving parents 27
>
> II-5 Percent of teachers and principals reporting that various
> strategies for involving parents were very effective: 2007 28
>
> II-6 Percent of teachers and principals agreeing or strongly agreeing
> with statements about their parental involvement programs 29
>
> II-7 Teachers' reports of how they used the core reading program: 2007
> and 2005 33
>
> II-8 Principals' and teachers' perceptions of the impact of various
> factors on implementation of the core reading program 35
>
> II-9 Significant increases in teachers' use of strategies for
> struggling readers: 2007 and 2005 37
>
> II-10 Frequency of cohort 2 teachers' engaging students daily in
> phonemic awareness activities: 2007 39
>
> II-11 Frequency of cohort 2 teachers' engaging students daily in
> phonics activities: 2007 39
>
> II-12 Frequency of cohort 2 teachers' engaging students daily in oral
> reading fluency activities: 2007 40
>
> II-13 Frequency of cohort 2 teachers' engaging students several days a
> week in vocabulary development activities: 2007 41
>
> II-14 Frequency of cohort 2 teachers' engaging students several days a
> week in reading comprehension activities: 2007 42
>
> II-15 Significant differences in teachers' engaging students daily in
> specific areas relative to the five essential components: 2007 and
> 2005 43

**Section II Figures---continued Page**

> II-16 Factors reported by a majority of cohort 2 teachers as having
> substantial effect on reading instruction: 2007 46
>
> II-17 Factors reported by at least a quarter of cohort 2 teachers as
> having no effect on reading instruction: 2007 47
>
> II-18 Feedback received through the Coaching for Effective Classroom
> Instruction: Close-Ups of Teaching Tools 48
>
> II-19 Changes in program implementation and the three program
> standards for cohort 1 53
>
> II-20 Changes in 12 scales of program implementation for cohort 1 53
>
> II-21 Changes in program implementation and the three program
> standards for cohort 2 54
>
> II-22 Changes in the 12 scales of program implementation for cohort 2
> 55

**Section III Figures**

> III-1 Trend for Phoneme Segmentation Fluency (grade 1 winter), by
> cohort 96
>
> III-2 Four-year trends for RFO cohort 1 TerraNova assessments 116
>
> III-3 Comparison of 4-year trends for OAT-R assessments of RFO cohort
> 1 and the state average 117
>
> III-4 Three-year trends for RFO cohort 2 TerraNova assessments 120
>
> III-5 Comparison of 3-year trends for OAT-R assessments of RFO cohort
> 2 and the state average 121
>
> III-6 Two-year trends for RFO cohort 3 TerraNova assessments 124
>
> III-7 Comparison of 2-year trends for OAT-R assessments of RFO cohort
> 3 and state average 125
>
> III-8 Trend comparison for percentage of students receiving special
> education 134
>
> III-9 Trend comparison for percentage of students with mental
> retardation 134
>
> III-10 Trend comparison for percentage of students with specific
> learning disabilities 135

Section I. Background

Introduction

Reading First-Ohio (RFO), an early literacy initiative that provides
funds to support research-based programs in reading for students in
grades K--3, is completing its fourth year of implementation. Since the
program's initiation in 2003, Westat and LPA have been providing
services to evaluate its achievements, addressing both formative and
summative issues. To date we have examined:

-   Initial program implementation for all schools in each of the three
    participating cohorts of schools (based on data collected in year 1
    of their participation);

-   Changes over time in implementation for all cohort 1 schools, based
    on both broad-based survey data and in-depth data collected through
    case studies; and

-   Changes in reading achievement for participating schools, as well as
    for a small comparison sample of nonparticipating schools.

This report extends this work by providing 1) information on changes in
implementation for a second cohort of schools over a 3-year period; 2)
follow-up data on cohort 1 case study schools and additional
high-performing schools; and 3) additional data on student reading
achievement.

These findings lay the groundwork for addressing seven critical
questions about the RFO program as it nears completion.[^2] These
questions are:

-   What is the value added to students' reading achievement if they are
    in an RFO school versus if they are not?

-   What is the value added to teacher instructional practices for
    teachers if they are in an RFO school versus if they are not?

-   What is the quality of implementation of the RFO schools and what
    explains the variance in levels of implementation?

-   What features of the RFO program are effective in terms of value
    added to student achievement?

-   What is the quality of professional development offered through the
    RFO program?

-   What happens to implementation of the RFO program in a school after
    the school's Reading First funding has been discontinued?

-   Does Ohio's 4-year cohort model 1) reach more schools, and 2) does
    it do so with similar effects on student achievement as is found in
    other states implementing RF?[^3]

Section II addresses questions related to program implementation
organized as "bottom-line" findings to the relevant evaluation
questions. Section III examines the questions related to student
achievement, and Section IV presents a summary and conclusions.
Appendices A and B provide more detailed responses to the teacher and
principal surveys, respectively, than are incorporated in the text
tables. The two surveys are included in Appendix C. In the remainder of
this background section we provide 1) a description of the Reading
First-Ohio program, 2) an overview of the evaluation design, and 3) a
summary of previous findings.

The Reading First-Ohio Program

With funding from the federal government, the state of Ohio is
participating in an ambitious effort to improve literacy among
elementary school students. Reading First, a major initiative within the
No Child Left Behind (NCLB) legislation, is designed to support K--3
literacy, drawing on what research has said about how best to support
enhanced teaching and learning. Funds flow from the federal level to
state education agencies, which, in turn, allocate them to eligible
local education agencies following a competitive grant process.

The grant provides for:

-   A 6-year effort to improve early literacy in the state's neediest
    schools;

-   A unique plan for professional development, aligned to state
    standards and scientifically based reading research (SBRR), that
    engages the talents of a variety of reading and technical assistance
    providers;

-   A Trio consisting of a literacy specialist, resource coordinator,
    and data manager that provides assistance at the school level;

-   A system of diagnostic and outcome assessments designed to provide
    critical information on what students know and can do; and

-   Comprehensive program monitoring to assure that essential features
    of the program are being adequately implemented in participating
    schools and districts.

The Ohio Department of Education (ODE) has provided funding for three
cohorts of districts to participate in RFO. The initial cohort, cohort
1, is completing its fourth and final year. Cohort 2 is completing its
third year and preparing to undertake its fourth, transitional, year.
Cohort 3 has completed 2 years of program participation.

The Instructional Approach

RFO combines what has been learned from research on SBRR instruction
with an existing state tradition of carefully crafted standards for
reading/language arts instruction. Thus, there is a deliberate
recognition of the need to integrate new approaches or instructional
emphases stemming from SBRR with existing standards and practices in a
coherent and complementary way. Key to the program, therefore, is
attention both to the five essential features of successful early
literacy programs (phonics, phonemic awareness, vocabulary, fluency, and
comprehension) and to state standards regarding professional
development, comprehensive and coherent program design, and systematic
and explicit reading instruction. Evaluation of the success of RFO must
consider the extent to which both of these anchors are honored in
implementation, as well as the extent to which they are working together
to meet common ends.

Assessment

Supporting instruction is a carefully selected battery of assessments
designed to provide information for shaping instruction and assessing
what students have learned. Initial assessment includes the Texas
Primary Reading Inventory (TPRI), given in the fall as a screening test;
the DIBELS, given three times during each of grades K--3 for program
monitoring; the TerraNova, given each spring in grades 1--3; and the
Ohio Achievement Test (OAT), given in grade 3 to measure outcomes.
Assessment is a critical feature of RFO, as is the Assess-Plan-Teach
cycle, an approach to instruction that emphasizes the use of assessment
information for diagnosis and instructional planning. Evaluation of the
success of RFO must closely examine the extent to which the assessment
component is being implemented as planned and is informing instruction.

Staffing

RFO also provides increased staff support for early literacy. Key at the
school level is the "Trio," a support group usually comprising a
full-time literacy specialist, a part-time data manager, and a part-time
resource coordinator. As originally designed, the literacy specialist
serves as the site-based expert in SBRR, providing school-wide
professional development and individual modeling and coaching (see the
description of the professional development system below); the data
manager assists in collection, analysis, and interpretation of the
system of student assessment; and the resource coordinator works to
procure and organize the materials available to support instruction.

At the district level, RFO funds a district coordinator. The full-time
district coordinator oversees the implementation of the RF program in
the participating schools throughout the district. The district
coordinator is also responsible for assuring that adequate data are
gathered for program monitoring.

Professional Development

RFO has a unique system of professional development designed to support
program implementation. This system emphasizes formal professional
development supplemented by web-based communication and targeted
technical assistance. The lead in these activities is the Reading First
Center (a consortium of higher education institutions) that works to
design and deliver the professional development, as well as to train and
support the technical assistance providers who assist in observing and
modeling instruction. The objectives of this Center include:

-   Preparation of "field faculty" for their role as trainers of
    literacy specialists in RFO schools;

-   Establishment of standards of practice for K--3 teachers that
    support reading growth in all children;

-   Provision of professional development for administrators and
    principals in RFO districts; and

-   Support for technical assistance activities of regional consultants
    assigned to RFO districts.

The system is tiered. The Center trains field faculty, who in turn
provide professional development to school-based literacy specialists,
who then provide professional development to their classroom-based
colleagues. Additional training is offered to principals to meet their
needs as managers and instructional leaders.

Reading First Program Changes in Year 4

Changes for programs in their fourth year of implementation (cohort 1)
included reduced funding levels and a new professional development
model.

**Reduced Funding.** The RFO program in year 4 operates with reduced
funding. Year 4 funding is designated for schools to continue to support
a full-time literacy specialist and to continue administering the
Dynamic Indicators of Basic Early Literacy Skills (DIBELS) on a regular
schedule, using palm pilots and wireless network services. Funding
reductions in the RFO schools have affected the following:

-   **Reading First staff support.** There are no funds for resource
    coordinators and reduced support from data managers (in some schools
    there is no data manager support).

-   **Classroom support staff.** Support staff such as instructional
    aides and intervention specialists were not funded by Reading First
    in the fourth year of the initiative.

**Professional Development.** In 2006--07, cohort 1 schools began
implementing the Instructional Leadership Team (ILT) model, including a
seven-step action research approach developed by Pearson Achievement
Solutions. The ILT model is designed to facilitate collaboration among
teachers as they work together to address specific learning objectives.
Compared to earlier years, the ILT model required a more active teacher
role and a more facilitative and less directive literacy specialist
role.

The Reading First-Ohio Evaluation

The evaluation of the Reading First-Ohio program is designed to provide
ongoing formative information about program implementation, as well as
summative data on the extent to which it is succeeding in reaching its
goals for *all* students. The design for the evaluation employs a
mixed-method approach, combining broad-based teacher and principal
surveys, targeted case studies in treatment and comparison schools, and
analyses of reading assessment data.

Over the 6 years of the study, program implementation and progress will
be examined for all three cohorts of students. Last year, our report
addressed fidelity of implementation for cohort I schools and provided
baseline data for schools in cohort 3. This year, the focus of our
implementation analyses is on the cohort 2 schools just finishing their
third year of participation.[^4] In addition, the evaluation includes a
specially selected case study subsample of cohort 1 RFO schools and
comparable schools in the same districts that were not participating in
the program but that have been followed since the beginning of the
evaluation and a small number of treatment schools from cohort 1 that
showed large gains in achievement over the initial 3-year period. In
addition, each year we have examined student achievement data for the
RFO schools, looking at multiple measures used to gauge program
success---the DIBELS, the TerraNova, and the Ohio Achievement
Tests-Reading (OAT-R). In this report, we also present OAT-R results for
schools receiving Title I services as a comparison group for the RFO
schools.

Table I-1 shows the major primary data collections over the first 4
years of the evaluation. Document reviews have been ongoing.

**Table I-1.---Primary d**ata collection activities

  -------- ---------- --------- ---------- --------- ---------- --------- ---------- -----------
  Cohort   School               School               School               School     
           year                 year                 year                 year       
           2003--04             2004--05             2005--06             2006--07   

           Survey     Site      Survey     Site      Survey     Site      Survey     Site visit
                      visit to             visit to             visit to             to case
                      case                 case                 case                 study and
                      study                study                study                exemplary
                      schools              schools              schools              schools

  Cohort 1 X          X                    X\*       X          X                    X

  Cohort 2                      X                                         X          

  Cohort 3                                           X          X                    
  -------- ---------- --------- ---------- --------- ---------- --------- ---------- -----------

\* Limited site visits were conducted in the fall; full site visits
occurred in the spring.

Although the evaluation is comprehensive, it is important to acknowledge
some of its limitations.

-   First, we could not address all program components with equal depth
    and breadth. Our look at the Trio, for example, focuses more on the
    literacy specialist than the resource coordinator and data manager.

-   Second, comparative data on the program and the practices used in
    the schools for reading instruction are gathered in a sample of case
    study schools only. While these schools were sampled to represent
    the full spectrum of cohort 1 schools, they are bound to have their
    own unique characteristics. Unfortunately, the cohort 1 sample was
    reduced when ODE did not continue funding some schools at the end of
    their second year of participation and others closed. Thus, our
    comparative data for the case study sample are more limited than we
    had desired. This is an issue we plan to address in years 5 and 6 of
    the evaluation through the addition of a Title I comparison group
    (see Year 5 Scope of Work).

-   Finally, our survey data rely on self-report by teachers and
    principals. Any self-report system may have some degree (usually
    unknown) of bias---intentional or unintentional.

Summary of Previous Findings

Data from year 3 continue to show that implementation is strong and has
increased over time.

-   There are consistent findings that after 3 years, RFO is being
    implemented in ways that are closely aligned with scientifically
    based reading research and the Ohio Department of Education's vision
    for the program. Data collected through surveys, case studies, and
    classroom observations all converge to describe a program that has
    helped teachers focus on the five essential components of
    instruction---phonemics, phonics, vocabulary, fluency, and
    comprehension---and that integrates reading in a comprehensive way
    with the existing English Language Arts (ELA) standards and the
    overall instructional program. While individual schools may vary in
    their implementation and not all program areas are equally strong,
    what emerges is a picture of a program with a clear purpose that is
    supported at all levels with the tools, materials, and leadership
    needed to bring about change. Data from both our longitudinal
    assessment of cohort 1 participants and initial assessment of cohort
    3 participants provide evidence that the program is maturing over
    time and changing in ways that are constructive and responsive to
    participants' needs.

Analyses of student achievement data present a more mixed picture.

-   Where we compare grade-level performance across years, the evidence
    from TerraNova and OAT-R is similar to what DIBELS and TRPI have
    shown in many cases, but different in others. For cohort 1, while
    DIBELS and TPRI results suggested an increasing trend from years 1
    to 3, TerraNova and OAT-R results indicated that the increase from
    year 1 to year 2 had flattened out in year 3 (Westat and Learning
    Point Associates, 2005). Assessment results from cohort 2 are
    generally consistent, showing better performance in year 2 than
    year 1. Cohort 3 began at a higher achievement level than did
    cohorts 1 and 2.

-   Analyses of the data from the treatment and comparison schools also
    show mixed results over time. We compared the assessment results
    from three testing periods between treatment and comparison students
    who began kindergarten in fall 2004. First we conducted
    cross-sectional analyses, including all of the students who were
    assessed at least once. We found that on average, although treatment
    students began at a lower level at the pretest (T1), their
    achievement was significantly higher than comparison students at the
    end of the first year (T2). However, at the end of the second year
    (T3), the advantage gained in the first year disappeared.
    Longitudinally, including only students who were assessed in all
    three periods, the results are similar to those at T3.

-   Analyses by race, gender, and special education status generally
    showed a similar pattern of results.

Reference

Westat/Learning Point Associates. (2005). *Reading First-Ohio: Phase 1
evaluation report*. Rockville, MD: Westat.

**Section II.**

**Program Implementation Changes Over Time---Survey Data**

**Chapter 1. Introduction and Methodologies**

In this section, we examine data on program implementation. We address
the following questions:

-   What is the quality of implementation of the RFO schools and what
    explains the variance in levels of implementation (Chapters 2
    through 6)?

-   What is the quality of professional development offered through the
    RFO program (Chapter 2)?

-   What is the value added to teacher instructional practices for
    teachers if they are in an RFO school versus if they are not
    (Chapter 7)?

-   What happens to implementation of the RFO program in a school after
    the school's Reading First funding has been discontinued (Chapter
    8)?

The design of our work for year 4 of the evaluation allows us to look
deeply at the first two questions; the data for the last two questions
are more limited, but raise important issues that will be examined more
fully in years 5 and 6 of the evaluation.

Data to address these questions were collected through two complementary
methodologies: surveys and case studies. In the remainder of this
section, we provide an overview of these approaches.

Surveys of Program Implementation

In spring 2007, we collected the third round of survey data from Reading
First-Ohio participants. Our focus in this section is on cohort 2
schools, which were surveyed for the second time at the end of their
third year of participation. Our discussion of the survey data examines
the status of implementation at the end of 3 years of RFO and the
changes that have occurred during that time.

As we look at the implementation findings, the concept of fidelity to
the program components and standards developed by ODE will provide the
basis for interpreting what we see. Since year 1 of our evaluation work,
we have used a consistent definition of what we mean by fidelity.
Specifically, we consider four basic, but critical, components of
instruction and the instructional system:

-   Provision of, and participation in, appropriate professional
    development designed to support the SBRR approach and Ohio's ELA
    content standards;

-   Establishment of adequate support mechanisms for program
    implementation;

-   Implementation of SBRR instructional approaches and student
    assessment strategies, using appropriate core and supplemental
    materials; and

-   Establishment of systematic approaches to program monitoring and
    program improvement.

To assess program fidelity, we developed two surveys of implementation:
a principal survey and a teacher survey. The principal survey covered
the following areas:

-   Professional development in reading;

-   The school's reading program;

-   Resources and support;

-   Program coherence and monitoring; and

-   Background information on school characteristics.

Similarly, the teacher survey addressed:

-   Classroom instruction in reading;

-   The Assess-Plan-Teach method of teaching;

-   Classroom instructional strategies;

-   Materials used for classroom instruction;

-   Professional development in reading;

-   The school's reading program; and

-   Teacher background characteristics.

In March 2007, principals received a letter informing them of the survey
and its purpose. This letter also requested the identification of a
school-level survey coordinator (who would be paid \$100 for assisting
in the survey distribution and collection process) and a list of all
reading teachers in grades K--3. Based on this list, we sent surveys to
the coordinators with instructions to distribute them to the appropriate
teachers. Coordinators were also asked to make sure that surveys were
returned within a 2-week time period. At the end of those 2 weeks, we
reviewed the responses and contacted the school coordinator to urge
teachers who had not returned the survey to do so.

The data collection period was March 27 through June 8. Of the 18
surveys sent to principals, we received 17 completed surveys, for a
response rate of 94 percent. Of the 223 cohort 2 teachers eligible for
the survey, 209 responded---a response rate of 94 percent. Copies of the
principal and teacher surveys are provided in Appendix C.

It is interesting to note that comparisons between respondents in 2005
and 2007 show that there has been considerable turnover of staff in
these schools. Of the 17 principals responding in 2007, only 7 (41
percent) were in the same schools in 2005. Of the 209 teachers that
responded, only 135 (65 percent) were represented in both the 2005 and
2007 samples. In our analyses, we examine data for both the
cross-sectional and longitudinal groups of teachers and note any
significant differences.

Characteristics of the Survey Samples

Table II-1 shows the characteristics of the cohort 1 teachers who
responded to the survey in 2005 and 2007. Data are presented only for
schools participating in the program in both 2005 and 2007.[^5] Among
2007 respondents:

-   80.4 percent are as regular classroom teachers, and 11.5 percent are
    special needs teachers;

-   94.7 percent are females;

-   57.4 percent hold professional certification, and 30.6 percent have
    provisional certification;

-   54.5 percent have a master's degree;

-   77.4 percent are certified to teach elementary education, 34.6
    percent are certified in early childhood education, and 21.6 percent
    are certified in reading; and

-   on average, they have taught for 14.2 years in general and 13.5
    years in reading or language arts in particular.

Comparison of the 2005 and 2007 samples of teachers showed no
significant differences between them.[^6]

**Table II-1.---Characteristics of teacher respondents: 2005 and 2007**

  ----------------------------------------------------- --------- --------
  Teacher characteristic                                Survey    

                                                        2005      2007

                                                        Percent   

  **Teaching role**                                               

  Regular                                               82.2      80.4

  Special needs                                         11.1      11.5

  Literacy specialist                                   0.0       0.0

  Other                                                 6.7       8.1

                                                                  

  **Gender**                                                      

  Female                                                96.0      94.7

  Male                                                  4.0       5.6

                                                                  

  **Highest degree received**                                     

  Bachelor's                                            7.6       11.0

  Bachelor's + graduate credits                         43.8      34.0

  Master's                                              19.6      20.1

  Master's + graduate credits                           29.0      34.4

  Doctorate                                             0.0       0.5

  Other                                                 0.0       0.0

                                                                  

  **Licenses**                                                    

  Professional                                          46.8      57.4

  Provisional                                           36.5      30.6

  Temporary                                             0.0       1.0

  Substitute                                            1.8       0.5

  Conditional permit                                    0.5       0.5

  Alternative                                           0.0       0.0

  Other                                                 14.4      10.0

                                                                  

  **Areas of certification**                                      

  Elementary education                                  81.7      77.4

  Early childhood education                             27.0      34.6

  Reading                                               15.5      21.6

  Special education                                     14.2      13.9

  Bilingual education                                   0.0       0.0

  Other                                                 6.8       5.3

                                                        Mean      

  **Years teaching**                                              

  Any school, grade, subject                            14.1      14.2

  Reading or language arts                              13.2      13.5

  This school                                           8.3       8.8

  Current grade                                         6.6       7.7
  ----------------------------------------------------- --------- --------

NOTE: Categorical variables are reported in frequency (percent), and
continuous variables are reported in mean and standard deviation.
Percents may not add to 100 due to missing or non-ascertained responses,
or because the respondent could choose more than one area of
certification.

Table II-2 shows the characteristics of the schools led by the 17
responding principals.

-   Average enrollment in 2007 in these schools was 345.

-   Among the students in 2007, 87.0 percent were characterized as
    economically disadvantaged.[^7] African Americans (76.1 percent) and
    whites (16.9 percent) composed the two largest racial groups.

-   About 11 teachers were assigned to grades K--3. They were assisted
    by five support personnel on average (e.g., aides, tutors).

**Table II-2.---Characteristics of Reading First schools as reported by
principals: 2005 and 2007**

  ------------------------------------------------------- ---------- ------ -------
  School characteristic                                   Response          

                                                          2005              2007

                                                          Number            

  **Number of children currently enrolled**                                 

  Kindergarten                                            48                50

  1st grade                                               50                53

  2nd grade                                               47                51

  3rd grade                                               45                52

                                                                            

  Classroom teachers assigned to grades K--3              12                11

  Reading support personnel (e.g., aides, tutors )        4                 5
  assigned to grades K--3                                                   

  Mean enrollment per school                              367               345

                                                          Percent           

  **Grade levels**                                                          

  PreK/K--2                                               0.0        0.0    

  PreK/K--3                                               0.0        0.0    

  PreK/K--4                                               0.0        0.0    

  PreK/K--5                                               12.5       12.5   

  PreK/K--6                                               68.8       62.5   

  PreK/K--8                                               18.8       25.0   

  Other                                                   0.0        0.0    

                                                                            

  **Gender of students**                                                    

  Male                                                    50.3       49.3   

  Female                                                  49.7       51.3   

                                                                            

  **Factors affecting learning**                                            

  Percent economically disadvantaged                      84.8       87.0   

  Percent limited English proficient                      0.2        2.4    

  Percent with disabilities                               15.9       10.1   

                                                                            

  **Race/ethnicity of student body**                                        

  African American                                        75.9       76.1   

  American Indian/Native Alaskan                          0.0        0.0    

  Asian/Pacific Islander                                  0.5        0.3    

  Hispanic                                                3.1        2.6    

  Multiracial                                             3.7        3.7    

  White                                                   17.0       16.9   
  ------------------------------------------------------- ---------- ------ -------

NOTE: Categorical variables are reported in frequency (percent), and
continuous variables are reported in mean and standard deviation.
Percents may not add to 100 due to missing or non-ascertained responses
or rounding.

Data Analysis

The present analysis relies on simple descriptive statistics. For both
teacher and principal surveys, data are reported in two ways. For
questions using categorical variables, the responses are reported as
frequency distributions. For questions using continuous variables, the
mean and standard deviations are provided, and cross-tabulations are
used as needed. We analyzed categorical variables with cross-tabulations
and used means and standard deviations for continuous variables. We
looked at item response rates to see if any items appear especially
problematic in terms of missing information. Most items have a response
rate over 90 percent, thereby eliminating that concern.

In the chapters that follow, we present our findings from the surveys of
cohort 2 teachers and principals organized in terms of professional
development, support mechanisms, instructional approaches, program
monitoring, and overall perceptions of success. We look at status in
2007 to assess the extent to which the program is being implemented with
fidelity at the end of 3 years of program participation with special
attention to the quality of professional development and technical
assistance provided to the schools. We also look for a change in status
across the 2-year period.

**Case Study Component**

The evaluation's case study component in year 4 focused on progress in
implementing Reading First-Ohio, impact of the implementation,
sustainability of RFO as funding decreases and eventually ends, and
factors influencing implementation, impact, and sustainability. The case
study component in year 4 also focused on the differences in
implementation, impact, and sustainability associated with an individual
school's level of performance. Thus, we identified a group of
high-performing schools both within the original sample of cohort 1
Reading First-Ohio schools and outside the sample to study in contrast
to our original sample of comparison schools.

The Year 4 Case Study Schools

The original cohort 1 case study schools and districts were selected to
ensure representation across setting---urban locales, midsize
communities, and rural locales. Although there have been some changes in
the original sample due to school closures and the Ohio Department of
Education's decision to stop funding several cohort 1 schools at the end
of the 2004--05 school year, eight original Reading First schools and
six original comparison schools have remained in the case study. These
schools were studied again in year 4. In addition, the RFO sample was
expanded to include high-performing schools from cohort 1 (two of these
schools are also in the original cohort 1 case study sample) (Table
II-3).

**Table II-3.---Case study s**chools

+------------------------+--------------+--------------+--------------+
| District/school        | Original RFO | Original     | RFO cohort 1 |
|                        | cohort 1     | comparison   | high         |
|                        |              | cohort 1     | performing   |
+------------------------+--------------+--------------+--------------+
| **Cleveland**          |              |              |              |
+------------------------+--------------+--------------+--------------+
| > Giddings             | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Fullerton            | X            |              | X            |
+------------------------+--------------+--------------+--------------+
| > Clark                | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Woodland Hills       | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Captain Arthur Roth  |              |              | X            |
+------------------------+--------------+--------------+--------------+
| > Charles Lake         |              | X            |              |
+------------------------+--------------+--------------+--------------+
| > Brooklawn            |              | X            |              |
+------------------------+--------------+--------------+--------------+
| > Kentucky             |              | X            |              |
+------------------------+--------------+--------------+--------------+
| > Mary Bethune         |              | X            |              |
+------------------------+--------------+--------------+--------------+
| **Youngstown**         |              |              |              |
+------------------------+--------------+--------------+--------------+
| > North                | X            |              | X            |
+------------------------+--------------+--------------+--------------+
| > Williamson           |              |              | X            |
+------------------------+--------------+--------------+--------------+
| > Mary Haddow          |              | X            |              |
+------------------------+--------------+--------------+--------------+
| > Harding              |              | X            |              |
+------------------------+--------------+--------------+--------------+
| **Springfield**        |              |              |              |
+------------------------+--------------+--------------+--------------+
| > Kenwood Heights      | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Lagonda              | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Horace Man           |              | X            |              |
+------------------------+--------------+--------------+--------------+
| > Warder Park          |              | X            |              |
+------------------------+--------------+--------------+--------------+
| **Dayton**             |              |              |              |
+------------------------+--------------+--------------+--------------+
| > Cornell Heights      |              |              | X            |
+------------------------+--------------+--------------+--------------+
| **Vinton**             |              |              |              |
+------------------------+--------------+--------------+--------------+
| > Allensville          | X            |              |              |
+------------------------+--------------+--------------+--------------+
| > Wilton               |              |              | X            |
+------------------------+--------------+--------------+--------------+

High-performing cohort 1 RFO schools were identified using TerraNova
performance, ranking schools according to change in average scale scores
aggregated across grades 1--3 from year 1 to year 3.[^8]

Data Collection Methods and Sources

The data for the case study were collected on site visits to the 18
schools between early April and mid-May 2007. Two methods were used:
interviews and classroom observations.

The site visit interview plan has remained consistent over the 4 years
of the case study. The plan called for conducting interviews in each
school, with eight K--3 teachers, two per grade (often all of the K--3
teachers in the school), the literacy specialists, and the principals.
In the past, we conducted interviews with the data managers and resource
coordinators; however, many districts have either dropped those
positions or spread them across many schools. The evaluation plan called
for conducting interviews with district coordinators and, when possible,
with district administrators who have some supervisory authority over
the Reading First program.

In the Reading First districts, interviews were conducted in 12 schools:
6 cohort 1 schools that were not high performing and 6 high-performing
schools, 2 from the original sample and 4 added this year. In the
Reading First schools and districts, interviews were conducted with 12
literacy specialists, 12 principals, 79 teachers, 5 Reading First
district coordinators, and 2 district administrators.

In the comparison schools, also in Reading First districts, interviews
were conducted in six schools with 6 principals and 38 teachers. Two
curriculum coaches (not literacy experts) were interviewed in two of the
schools.

Following the original design, 2nd grade classes were observed this
year, 1st grade classes were observed in year 3, and kindergarten
classes were observed in year 2. In spring 2007, 2nd grade classes were
observed in all but one of the case study schools. (In one school, one
teacher scheduled to be observed was absent, and the other declined to
be observed.) A total of 28 observations were conducted in Reading First
and comparison schools.

Thus, 14 literacy specialists, 18 principals, and 117 teachers were
interviewed and 28 classes were observed. We also interviewed five
Reading First district coordinators and two district administrators.

Analysis of Interview Data

The general analytic approach was to conduct a cross-case analysis, with
the school as the unit of analysis. We established a list of school
characteristics, consistent with the Reading First model, and determined
the degree to which the characteristics were present in each of the
schools. The approach facilitated a comparison of characteristics across
schools and, in particular, supported comparison of Reading First and
comparison schools and of Reading First cohort 1 schools and Reading
First high-performing schools.

An initial step in our analysis was to determine if the six
high-performing schools exhibited any pattern of differences in the
areas we were to examine to determine if we were to consider a
four-group sample or a three-group sample. No pattern of differences was
detected.

For the school-level analysis, summary templates were developed for each
characteristic. For example, three templates addressed utility of year 4
teacher professional development, principal practices that support
implementation of literacy programs, and data use by teachers in the
school. Each of the characteristics was selected because it was
important to the program model. For each characteristic, responses from
different school respondent groups (teachers, coaches, principals) per
school were summarized separately. Teacher responses were grouped, while
principal and coach responses were considered separately. From these
summarized responses, we determined respondent agreement on the presence
or implementation level of the characteristic. Evidence was included in
the summary template, and original data were checked frequently for
clarification and verification.

Analysis of the Observation Data

All observations were conducted in 2nd grade classrooms. RF classrooms
were observed a minimum of 90 minutes and up to 2 hours, while non-RF
classrooms were observed for 60 to 90 minutes. The observations were
guided by the classroom observation tool developed by CIERA for the
Effective School/Accomplished Teacher study (Taylor et al., 2000) and
the CIERA School Change in Reading Project (Taylor et al., 2001).  This
comprehensive system captures detailed information on seven classroom
areas: instructor, grouping, primary content focus, specific content
focus, materials used, teacher activities, and student responses. The
protocol uses a coding scheme to represent descriptive data collected
during the observation sessions. Table II-4 indicates the information
captured at each coding level for the CIERA observation system.

Table II-4.---Coding levels for the CIERA observation system

  ------------ ----------------------------------------------------------
  Level        Information

  1            Who is providing instruction?

  2            What instructional groupings are observed?

  3            What major academic area is covered?

  4            What is the specific literacy activity or activity of the
               classroom teacher?

  5            What materials are being used by the classroom teacher and
               the students for this event?

  6            What is the teacher's interaction style for this segment
               of the observation?

  7            What is the expected student response?
  ------------ ----------------------------------------------------------

We chose to use the CIERA classroom observation system because it is
comprehensive and closely aligned with the state evaluation objectives.
When we first used the system in spring 2004, we made some minor
revisions to the codes and modified the data collection forms. The CIERA
training manual provided many of the definitions of the terms we coded
while observing RFO classrooms. The RFO classroom observers followed the
observation and note-taking procedures described in the CIERA manual,
which duplicated the forms so that observers could enter information
directly into a database created by LPA.

The first step in distinguishing differences between RF and non-RF
schools, and between high-performing RF and non-high-performing RF
schools, was to summarize the codes from levels 2 through 7 at the
classroom level. These summaries allowed us to conduct an empirical
overview of the combination of materials, activities, subject matter,
and methods used for each of the observed classrooms, leading to
conclusions regarding the distinguishing features of the different
classroom types. Codes were summarized by aggregating the percentage of
total codes (within each level) recorded during an observation in which
the code appeared in the data. This aggregation provided the approximate
distribution of different types of activities, materials, groupings, and
interaction styles that occurred during the specific literacy block.

In addition to the classroom-level summaries, program-level summaries
were developed for two purposes: 1) to determine the extent to which
there appeared to be a substantial difference in the composition of the
literacy block between RF and non-RF schools, and 2) to determine the
extent to which RFO schools exhibited evidence of implementation of the
core practices associated with the program (and whether there was a
higher incidence of fidelity to the program in the high-performing RFO
schools). To address the first purpose, we identified differences among
the program groups with respect to key features---whole group and small
group arrangements, content focus, time that is not academically
focused, diversity of materials used, type of teacher activities, and
student response activities. Addressing the second purpose required
examination of the extent to which the RF classroom observations showed
high incidences of codes relating to the RF program components.
Specifically, we looked for more use of small group instruction, a
greater diversity of materials, more emphasis on student-centered
instruction (coaching, modeling, and listening/watching), and a higher
occurrence of active student response expectations (reading aloud
together, responding together, writing, and manipulating). Additionally,
when the specific literacy activity was comprehension, reading connected
text, fluency, or vocabulary, the specific interaction style of the
teacher was examined for adherence to program initiatives.

References

Taylor, B.M., Pearson, D.P., Pearson, D.S., and Rodriguez, M.C. (2000).
*Reading growth in high-poverty classrooms. The influence of teacher
practices that encourage cognitive engagement in literacy learning.*
University of Minnesota: Unpublished document.

Taylor, B.M., Pearson, D.P., Pearson, D.S., and Rodriguez, M.C. (2001).
*The CIERA school change project: Using research, data, and study groups
to improve classroom reading instruction and increase students' reading
achievement*. University of Minnesota: Unpublished document.

**Chapter 2. Professional Development\
and Technical Assistance**

In this chapter we begin to address the question of the quality of
program implementation, looking first at professional development and
technical assistance.

Professional development is a central feature of the Reading First-Ohio
program. As such, ODE---working through the Reading First-Ohio Center,
their field faculty, and mentor coaches--- provides a variety of
professional learning experiences for literacy specialists and school
staff and principals. The literacy specialist, in turn, plays a key role
in providing school-based professional development for teachers. The
literacy specialist's responsibility is to deliver RFO Center-developed
training to all K--3 teachers, as well as coaching and technical
assistance to selected staff. Teachers in Reading First schools are
expected to participate in 180 minutes of professional development each
month, during or outside of the school day.[^9]

Additional supports for program implementation are provided by the
resource specialists and data managers. Along with the literacy
specialist, this Trio is a special feature of RFO's literacy program.

**The initial surveys of teachers and principals in cohort 2 generally
found that professional development was viewed very positively. The
survey data collected at the end of year 3 of RFO program participation
indicated that professional development continues to be a strength of
the program and that more attention is being paid to providing support
for differentiated instruction, although there are some populations that
still need greater attention. Teachers rated coverage of topics such as
teaching struggling readers, small group methods, and small group
management techniques significantly higher than in 2005.**

In this section, we focus on program status for cohort 2 schools in
spring 2007. Where informative, comparisons are made with earlier
responses.

Engaging Teachers in Professional Development

First, we examined how much professional development teachers are
receiving each month (Table II-5). Eight-four percent of teachers
reported receiving more than 120 minutes of professional development per
month, a percentage similar to the initial year when 82.0 percent of the
teachers in these same schools reported that level of participation.
Unlike cohort 1 for which data were reported last year, there was no
significant decrease in participation for cohort 2 teachers. And, cohort
2 teachers report a significantly larger percentage of teachers
participating in more than 120 minutes than had the cohort 2 teachers in
spring 2006.

**Table II-5.---Average number of minutes spent in professional
development per month\
by cohort 2 teachers: 2007**

  ----------------------------------------------------------- -----------
  Number of minutes                                           Percent of
                                                              teachers

  More than 120 minutes                                       84.3

  91--120 minutes                                             10.5

  61--90 minutes                                              2.9

  1--60 minutes                                               2.4
  ----------------------------------------------------------- -----------

NOTE: Percents may not add to 100 due to rounding.

Teachers were also asked to share their perceptions about the
professional development offerings along a number of dimensions. While
about 69 percent continued to feel that professional development
repeated information already known, there was overwhelming agreement
that professional development was strong in terms of a wide range of
characteristics (see Table A-1 in Appendix A). The patterns are quite
similar to those reported last year for cohort 1.

Over 90 percent of respondents agreed or strongly agreed that
professional development activities:

-   Have been clearly linked to the ELA content standards for K--3;

-   Have provided opportunities to practice what they had learned;

-   Have been supported by district support services and material
    resources; and

-   Have been led by knowledgeable and effective instructors.

Comparison of the 2007 and 2005 responses show a significant increase in
the percentage of teachers who agree or strongly agree that they have
drawn on their own input (84.7 percent vs. 75.6 percent[^10]) and
provided adequate time to plan instruction (67.8 percent vs. 52.3
percent). Significantly fewer teachers agreed or strongly agreed that
they have taken away time better spent on planning lessons (39.3 percent
vs. 48.4 percent).

It is interesting to note that even after 3 years of participation, 78.0
percent of all teachers (and 83.0 percent of the longitudinal group)
agreed or strongly agreed that professional development included
information about reading theory that initiated change in their reading
instruction.

The item still attracting the largest disagreement was "they have
provided individualized technical assistance," with 47.8 percent of the
respondents in the cross-sectional sample and 41 percent of the
respondents in the longitudinal sample disagreeing.[^11] This is a
significantly greater percentage voicing disagreement than was found
with cohort 1 teachers.

Principals' assessments of professional development for teachers echoed
those of teachers and were uniformly positive. Indeed, when principals
were asked to rate the effect of various professional development
activities in terms of the effect they had on the support of literacy,
offerings for teachers were assessed as having a substantial effect by
62.5 percent of the respondents (see Table B-1 in Appendix B).

A second set of questions looked at various topics addressed through
professional development and asked how adequately they were covered.
These items ranged from theory and foundations, to the five essential
features, teaching different groups of children, using assessments to
plan instruction, selecting materials, and using standards. With a few
exceptions that will be noted below, over 80 percent of teachers
reported that these topics were adequately or more than adequately
covered (see Table A-2 in Appendix A). The exceptions are:

-   Teaching reading to English language learners (27.1 percent agreed
    or strongly agreed); and

-   Teaching students with diagnosed learning disabilities (34.1 percent
    agreed or strongly agreed).

Ratings of adequacy in these areas were significantly lower than those
of cohort 1 respondents, who indicated around 50 percent agreement.[^12]

Also worth noting are several areas in which significant changes in
adequacy of coverage was found when responses from 2007 and 2005 were
compared. Increases were found in the following:

-   Composition (82.8 percent vs. 73.0 percent);

-   Teaching reading to struggling readers (77.4 percent vs. 63.4
    percent);

-   Small group teaching methods (87.1 percent vs. 76.8 percent); and

-   Small group class management techniques (71.8 percent vs. 55.6
    percent).

These changes are noteworthy as early in the program a concern was that
more help was needed to transition effectively to small group methods,
and management of small groups was often cited as a problem. What is
also interesting is that these data suggest movement toward a more
differentiated mode of instruction, with methods for addressing small
group instruction for at-risk readers receiving increasingly more
attention.

The data also show a significant decrease in the percentage of teachers
in the cross-sectional group reporting that phonemic awareness
development and systematic and explicit phonics instruction are
adequately or more than adequately covered. No change was found when
only the longitudinal group was considered, which may mean that teachers
new to RFO schools are not receiving the same level of grounding in
these two areas that was provided to those who were part of the program
when it first began in the school. However, this finding should not be
over interpreted since, despite the statistically significant drop,
around 87 percent of the respondents reported adequate or more than
adequate coverage.

It is of some interest to compare the reports from teachers about how
well various topics were covered in their professional development with
principals' perceptions of how well prepared teachers are to address
various aspects of instruction.

Figure II-1 shows the percentage of principals rating teachers'
knowledge as strong, adequate, or limited in a variety of areas. In all
areas, principals gave teachers at least adequate ratings of knowledge.
The strongest ratings were for strategies for teaching the five
essential components of reading and strategies for addressing the needs
of struggling readers in K--3. Few ratings of strong were given to
knowledge of processes for aligning curriculum and instruction with
state ELA standards and SBRR recommendations (31.3 percent) and
foundational topics (18.8 percent). And there is some suggestion that
ratings provided by responding principals in these two areas are lower
in the 2007 than they were in 2005.[^13]

**Figure II-1.---Principals' perceptions of teachers' knowledge in
reading: 2007**

![](media/image3.wmf){width="5.99375in" height="4.274305555555555in"}

NOTE: Percents may not add to 100 due to rounding.

Principals' Professional Development

Professional development for principals is also an important part of the
RFO program. Principals need access to high-quality training that
assists them in monitoring program implementation, using data
effectively, and providing overall leadership to the reform effort. Our
surveys of principals did not delve as deeply into the professional
development provided to principals as did the surveys of teachers, but a
small number of targeted question were asked to assess principals'
perception of several critical offerings. Review of their responses
indicates that responding principals had mixed reactions. While the new
leadership meeting, the Classroom Walk-Through Training (CWT), and the
advanced classroom walk-through training were all reported to have
substantial effects by approximately half the principals, the December
data summit and sustainability meeting were rated as having substantial
effect by 21.4 and 27.3 percent, respectively. These activities received
higher ratings from the respondents in 2005, with 53.3 and 46.2 percent,
respectively, rating the activities as having a substantial effect (see
Table B-2 in Appendix B).

Because the classroom walk-through model has the potential to provide
the principals with effective and efficient tools for monitoring and
supporting instruction, the survey asked several different questions
about the perceived efficacy of this professional development support.
Responses suggest that this is a tool that is working (see Table B-3 in
Appendix B).

-   100 percent of principals said that the tool is easy to use.

-   93.3 percent said that the professional development sessions covered
    what to look for in a CWT and how to develop a reflective prompt for
    conversations with teachers.

-   93.3 percent said that the timeframe (2-4 minutes) was adequate for
    recording the "look-for's."

-   80.0 percent agreed or strongly agreed that the CWT strengthened
    their role as an instructional leader.

These responses are consistent with the reports of cohort 1 principals
in spring 2006.

The Trio

**In 2005, the data indicated that the Trio was a strong asset for the
program and that its work was valued. The data collected in 2007
continue to support the conclusion that the work of the members of the
Trio is held in high regard. Further, the literacy specialist appears to
have taken on a greater range of duties. Teachers report significant
increases in interactions with the literacy specialist in 9 out of 16
areas examined. There are significant increases in the percentage of
teachers who report receiving various kinds of individualized support
from the literacy specialist.**

Nearly 90 percent of teachers feel that the Trio has strengthened the
reading program in the school, has provided services that previously
were not available, and has provided these services in a timely manner.
Figure II-2 presents responses from principals to a series of questions
regarding the importance of the Trio's services. In 2007, every feature
we asked about, with the exception of assisting teachers in creating
lesson plans, was rated as very important or important by at least 93
percent of the principals. Responses were virtually unchanged from 2005.
When cohort 1 responses were compared to those from cohort 2 principals
(not shown in figures), we found that the cohort 2 principals rated the
services of the Trio significantly higher in the following areas:

-   Assistance in completing required reports and documents (76.5
    percent vs. 51.2 percent, very important).

-   One-on-one coaching within teachers (82.4 percent vs. 53.7 percent,
    very important).

-   Help in working with instructional aids or paraprofessionals (56.3
    percent vs. 29.7 percent, very important).

**Figure II-2.---Principals' opinions about the various services
provided by their schools' building Trio: 2007**

![](media/image4.wmf){width="5.99375in" height="4.415972222222222in"}

NOTE: Percents may not add to 100 because of rounding.

Survey responses from teachers also provide some more specific
information about the strengths of the Trio's three individual members.

Looking first at the literacy specialist, the data show large changes in
a number of important areas (Figure II-3). Teachers showed significant
increases in reports of help received in various aspects of lesson
planning, such as:

-   Interpreting assessment data (82.2 percent vs. 57.0 percent);

-   Planning, coordinating, and implementing the reading program (66.5
    percent vs. 39.5 percent);

-   Identifying students with severe literacy learning problems (51.4
    percent vs. 40.0 percent); and

-   Helping to reflect on lesson plans and ways to improve them (48.5
    percent vs. 34.8 percent).

**Figure II-3.---Percent of teachers receiving various instructional
supports from the literacy specialist**

![](media/image5.wmf){width="5.995833333333334in"
height="7.470833333333333in"}

Another area of change relates to intervention materials. There are
significant increases in reports that help was received in the following
areas:

-   Help in obtaining intervention materials (80.8 percent vs. 54.2
    percent); and

-   Help in reflecting on teaching and improving instruction (66.8
    percent vs. 48.0 percent).

A somewhat surprising decrease was reported in help in using
intervention materials (62.3 percent vs. 77.2 percent). This decrease
appears in the data for both the cross-sectional and longitudinal
samples of teachers.

Finally, there are noteworthy increases in the percentage of teachers
receiving individualized supports of various kinds from the literacy
specialist. Significant increases in reports of assistance were found
for:

-   Learning about literacy instruction through having lessons modeled
    (59.4 percent vs. 40.8 percent);

-   Learning about literacy instruction through having one-on-one
    coaching or conference sessions (60.9 percent vs. 35.3 percent); and

-   Learning about literacy instruction through having formal or
    informal observations of a teacher's classroom (67.0 percent vs.
    24.9 percent).

It is encouraging that over half the teachers in the 2007 sample
reported receiving these individualized supports.

Questions about the resource coordinator and the data manager also
yielded very favorable evaluations. Ninety-two percent of the teachers
felt the resource coordinator helped teachers secure and/or shared
materials necessary for teaching (see Table A-3 in Appendix A).
Eighty-six percent reported that the data manager helped teachers
monitor student progress.

**Chapter 3. Support Mechanisms**

In this chapter we examine the question of fidelity with regard to the
presence of support mechanisms. We look at the extent to which the
program has a comprehensive and coherent design. Specifically, we
examine:

-   Leadership and shared vision;

-   Coordination among people and programs;

-   SBRR core and supplemental materials; and

-   Scheduling instruction.

**Fidelity to this standard has continued to remain high for the cohort
2 schools. In each of the four areas examined, we find generally strong
patterns toward implementation. There is a school-wide commitment to the
program and a shared vision for its outcomes. Leadership is seen as
supportive and teachers as collaborating to enact the goals of the
program. Teachers have adequate textbooks and materials aligned with
SBRR and report using them. Importantly, the coordination between the
core and supplemental programs has increased. One area, establishing
home-school partnerships that are supportive of the core reading
program, still trails behind the others in implementation. Approximately
60 percent of teachers spend more than 90 minutes a day in reading
instruction.**

Leadership and Shared Vision

For RFO to be successful, it is important to have strong leadership and
a commitment of all staff to its goals and objectives. The data from the
2007 surveys (see Table A-4, Appendix A) indicate that these
characteristics are in place, with teachers agreeing or strongly
agreeing with the following statements:

-   There is strong and clear leadership for the reading program (89.0
    percent).

-   There is a school-wide commitment to beginning reading instruction
    grounded in SBRR (88.0 percent).

-   Our school has a shared vision of the school's reading program (85.5
    percent). [^14]

-   Teachers support the direction of the school's reading program (90.9
    percent).

It should be noted, however, that the cross-sectional data show a
significant drop in agreement with regard to the school-wide commitment
compared to 2005, when 93.8 percent of the teachers voiced such
agreement.[^15]

The item eliciting the largest disagreement was, "Our school is
characterized by a high degree of collaboration among teachers who teach
reading." However, the absolute extent of disagreement was modest; 20.9
percent of the teachers disagreed/strongly disagreed.

Coordination Among People and Programs

Several questions were asked regarding the extent to which the teachers
perceive that a consistent and coordinated program is in place. In 2005,
both principals and teachers gave relatively high marks to RFO in these
areas. In 2007, the picture was the same.

In all areas, close to 90 percent or more of the teachers indicated
agreement with the statements about coordination (see Table A-5,
Appendix A):

-   The reading program is closely coordinated across instructional
    staff in the K--3 grades (89.9 percent) and closely coordinated
    across teachers at the same grade level (92.3 percent).

-   Our intervention programs for struggling readers are coordinated
    with the core reading program (87.1 percent).

-   The reading program is closely aligned with Ohio's English Language
    Arts content standards for grades K--3 (97.6 percent).

Questions on the principal survey asked about the extent to which
various aspects of coordination and integration have been implemented.
Responses indicated that the principals perceived implementation to be
fully or mostly complete in a wide range of areas (see Table B-4,
Appendix B). For example,

-   Linking student expectations for student learning to the Ohio
    English Language Arts academic standards for grades K--3 (87.5
    percent);

-   Linking expectations for student learning to the reading curriculum
    for grade K--3 (93.8 percent); and

-   Linking Reading First-Ohio to the overall school improvement plan
    (87.5 percent).

Only one area stands out as being very low in implementation:
coordinating reading instruction with "feeder" preschool programs. In
2005, 50 percent of the principals indicated that there was no
implementation of this feature; in 2007, 56.3 percent of the principals
responded similarly.

Because of the strong interest in parental involvement on the part of
many RFO stakeholders, the principal and teacher surveys also asked some
questions about parent involvement activities and the extent to which
parents have been engaged as partners in the reading program. The
findings are mixed, and teachers and principals see things somewhat
differently.

When asked about communication with parents, teachers' responses
indicate a significant increase in those agreeing that their "school
communicates regularly with parents and families about our reading
program" (79.3 percent vs. 69.1 percent) (not shown in tables). At the
same time, principals' responses about the extent to which home-school
partnerships exist to support children's reading progress showed a
modest shift toward less complete implementation (50.0 percent vs. 62.5
percent reporting full or mostly implemented).

To look further at this topic, teachers and principals were asked a
series of questions about the literacy-related strategies used to
improve parental involvement (Figure II-4). While the exact percentage
of teachers and principals selecting a response differed, their
responses showed surprising agreement when rank order was examined.

**Figure II-4.---Percent of teachers and principals reporting that their
schools promoted various strategies for involving parents**

![](media/image6.wmf){width="5.99375in" height="4.274305555555555in"}

The four most frequently implemented strategies were as follows:

-   Informing parents of their child's reading progress (teachers 97.1
    percent, principals 100 percent);

-   Providing ideas for parents to encourage reading at home (teachers
    94.2 percent, principals 94.1 percent);

-   Discussing literacy at parent open houses and/or parent conferences
    (teachers 95.7 percent, principals 100 percent); and

-   Providing books and materials for parents to read with their
    children (teachers 89.8 percent, principals 100 percent).

Further, three strategies showed significant increases when cohort 2 and
cohort 1 data are compared:

-   Publishing a parent newsletter (72.3 percent vs. 59.0 percent).

-   Giving parents an opportunity to volunteer in classrooms (56.0
    percent vs. 42.6 percent).

-   Holding family literacy nights (79.6 percent vs. 61.7 percent).

When asked about the effectiveness of these strategies, teachers and
principals differed widely in their assessments of effectiveness (Figure
II-5). This was especially true for

-   Seeing improvement in parental involvement (34.1 percent teachers,
    88.2 percent principals); and

-   Parental involvement is a product of parent background and
    education, not necessarily what the school does to enhance it (76.6
    percent teachers, 43.8 percent principals).

**Figure II-5.---Percent of teachers and principals reporting that
various strategies for involving parents were very effective: 2007**

![](media/image7.wmf){width="5.99375in" height="4.274305555555555in"}

The area rated as least effective by both teachers and principals was
giving parents an opportunity to volunteer in reading classrooms. This
strategy was reported as not effective by 63.6. percent of the teachers
and 33.3 percent of the principals.[^16]

A second set of questions asked teachers and principals their opinions
of impacts related to parental involvement. Again, teachers were more
negative than principals, and they saw the relative impact of many of
the strategies somewhat differently (Figure II-6).

-   Our current reading program includes activities and strategies to
    enhance parental involvement in literacy (78.0 percent of teachers
    and 88.2 percent of principals agreed or strongly agreed).

-   Parental involvement is easier to encourage in reading than it is in
    other subject areas (59.7 percent of teachers and 64.7 percent of
    principals agreed or strongly agreed).

-   The Reading First-Ohio building Trio has positively impacted
    parental involvement (49.5 percent of teachers and 76.5 percent of
    principals agreed or strongly agreed).

-   I have seen some improvement in parental involvement during the last
    year as compared to the previous year (34.1 percent of teachers and
    88.2 percent of principals agreed or strongly agreed).

It is also worth noting that 76.6 per percent of teachers and 43.8
percent of principals agreed or strongly agreed that parental
involvement (or lack of) is a product of parent background and
education, not necessarily what the school does to enhance it.

**Figure II-6.---Percent of teachers and principals agreeing or strongly
agreeing with statements about their parental involvement programs**

![](media/image8.wmf){width="5.99375in" height="4.274305555555555in"}

Further, across the board, cohort 2 teachers' ratings were significantly
less positive than the ratings given by cohort 1 teachers.

SBRR Core and Supplemental Materials

Several questions were asked about the availability of adequate SBRR
materials and resources Teachers and principals reported that SBRR
materials were available and used. It is noteworthy that the responses
from teachers showed a significant improvement from 2005 to 2007 on the
following items:

-   Our school has adequate resources to effectively implement our
    reading curriculum (92.0 vs. 89.7 percent strongly agreed or
    agreed); and

-   The impact of "appropriate instructional resources and other
    materials or resources" on the implementation of the core reading
    program was also rated higher (93.3 percent vs. 91.6 percent
    indicating a positive impact).

Ninety-four percent of the principals also reported that there is
alignment of intervention programs and materials with the core reading
program, up from the 75 percent of principals who gave that answer in
2005.

Scheduling Instruction

Finally, we looked at how time was used for instruction and whether use
of time had changed from year 1 and year 3. Our principal and teacher
surveys asked a number of questions regarding scheduling.

Principals were asked a series of questions regarding changes made as a
result of the RFO program. Two of the questions that address scheduling
issues appear to show some important changes. First, 87.5 percent said
that they implemented additional before- and/or after-school
intervention programs for struggling readers in grades K--3 (including
volunteer programs), an increase over the 68.8 percent who reported
initiating these supports in 2005. Second, in 2007, 62.5 percent said
that they had decreased time devoted to subjects other than English
language arts, in contrast to the 93.8 percent who said that had
occurred in 2005. We cannot tell whether this apparent drop means that
schedules were readjusted to allow more time for other subjects or,
given the large turnover in principals serving these schools, all
changes occurred before the current respondents assumed responsibility
for these cohort 2 schools.

When asked how much time they allotted for instruction to their classes
and whether or not instructional time had changed, teachers reported
spending an average of 118 minutes on reading each day. These data are
quite similar to those reported in 2005, when the average was 116
minutes. Indeed, the modal response to the question of whether time has
increased, decreased, or stayed the same is that it stayed the same.
However, as Table II-6 shows, some teachers exceeded 118 minutes, with
12.9 percent reporting that they spent between 145 and 360 minutes
teaching reading/language arts to students and 42.6 percent reporting
spending 90 minutes or less. The percentage reporting 90 minutes or less
is statistically significant from cohort 1, where only 30 percent of the
sample gave that response.

**Table II-6.---Cohort 2 teachers' responses about the time
spent/allotted for instruction and its impact on the school's core
reading program: 2007**

  ---------------------------------------------------- ------------------
  Response category                                    Percent of
                                                       teachers

  **Minutes per day teaching reading/language arts to  
  your class**                                         

  90 minutes or less                                   42.6

  91--110 minutes                                      8.1

  111--144 minutes                                     34.9

  145--360 minutes                                     12.9

                                                       

  **Average number of minutes teaching                 
  reading/language arts compared to last year**        

  Increased                                            34.6

  Remained the same                                    63.9

  Decreased                                            1.4
  ---------------------------------------------------- ------------------

NOTE: Percents may not add to 100 due to missing or non-ascertained
responses or rounding.

Because the 42.6 percent reporting 90 minutes or less was a large change
from the data reported last year for cohort 1, we examined the data by
school to get a more fine-grained look at minutes per day spent on
instruction. Table II-7 shows the wide variation in the average number
of minutes spent on reading instruction during the school day, ranging
from 90 to 152 minutes per day. And there is a mixture of increases and
decreases in time spent when comparing data from 2007 to that of 2005.
While many changes were small, we do see a handful of more substantial
changes in both directions.

Table II-7.---Minutes per day spent on reading, by school:
Cross-sectional sample

  -------------------------- ----------- ----------- ----------- -----------
  School                     2005                    2007        

                             90 minutes\ Average     90 minutes\ Average
                             or less                 or less     

  Barrett Elementary         33.3        113         61.9        96

  Blairwood Elementary       35.7        130         50.0        111

  Broadmoor Elementary       42.9        120         58.3        148

  Caledonia Elementary       30.0        108         22.2        109

  Chambers Elementary        42.1        111         50.0        126

  Chase Elementary           36.4        115         11.1        116

  Imani Learning Center      75.0        93          100.0       90

  Margaret Park Elementary   37.5        106         16.7        111

  Nathan Hale Elementary     38.9        107         56.3        116

  Pope John Paul II          66.7        92          33.3        131

  Prospect Elementary        28.6        126         75.0        98

  Riverside Elementary       23.8        123         13.6        152

  Rozelle Elementary         53.3        117         50.0        109

  Shilohview Elementary      38.5        123         45.5        125

  Superior Elementary        33.3        117         33.3        122

  Townview Elementary        25.0        124         22.2        117

  Westbrooke Village         33.3        123         36.4        124
  Elementary                                                     

  Western Elementary         44.4        116         62.5        93
  -------------------------- ----------- ----------- ----------- -----------

**Chapter 4. Instructional Approaches**

In the previous chapters, we focused on features of Reading First-Ohio
that lay the groundwork for an effective literacy program. In this
chapter, we explore some very basic but essential questions about the
extent to which the teaching of reading has fidelity to the RFO program
design. We address the following questions: 1) What are teachers using
to teach reading? 2) How are teachers teaching reading? Also, *how
often* does their instruction address the five essential components of
reading? and 3) Why does instruction look the way it does?

What Are Teachers Using to Teach Reading?

Most educators agree that the reading curriculum and materials that are
used by teachers in the classroom widely influence what students are
taught. In this chapter, we discuss the core reading programs and
materials used by schools in cohort 2, as well as the factors that
influence implementation of these programs.

**Data show consistency in the choice of reading programs in cohort 2.
Compared to 2005, significantly more teachers use the program as a
framework for a scope and sequence of skills and report that the program
is supported by common assessments. A majority (65.1 percent) report
using narrative decodable or leveled books to teach reading.**

The choice of a core reading program obviously influences the type of
instruction that students receive and how they ultimately learn to read.
We used data from the principal survey to identify the core reading
programs that are being used. Similar to teachers' reports,[^17]
principals' reports found no variation between reporting years for the
core reading programs used (Table II-8). Harcourt Brace programs were
the overwhelming choice in cohort 2 schools, with 68.8 percent of
principals identifying this series in both 2005 and 2007.

**^Table\ II-8.---Percent\ of\ cohort\ 2\ schools\ using\ each\ core\ reading\ program:\ 2005\ and\ 2007^**

  ------------------------------------------------ ----------- -----------
  Core reading program                             2005        2007

  Harcourt Brace^1^                                68.8        68.8

  Open Court                                       31.2        31.2
  ------------------------------------------------ ----------- -----------

^1^Harcourt Brace includes Harcourt Signatures, Harcourt Trophies, and
other unspecified Harcourt Brace series.

NOTE: Sixteen principals responded to this question in 2005 and 2007,
though only seven remained in the same schools in both reporting years.

The choice of these programs also appears well known to teachers, with
93.8 percent agreeing that there is a core curriculum that all teachers
follow. Yet, how teachers follow this curriculum varies somewhat, and
two significant differences were found between the 2007 and 2005
reporting years (Figure II-7). Specifically, more teachers in 2007 than
2005 reported that they use the core reading program as a framework for
a scope and sequence of reading content/skills (94.2 percent vs. 88.3
percent) and that they use it as a resource for a few key lesson plans
(43.6 vs. 31.8 percent). These increases would appear to suggest less
reliance on the core reading program as the sole driver of instructional
content, but the percentage of teachers indicating they use the program
as their primary tool for instruction remained high---and
unchanged---from 2005 (roughly 91 percent in both years).

**Figure II-7.---Teachers' reports of how they used the core reading
program: 2007 and 2005**

![](media/image9.wmf){width="5.996527777777778in"
height="4.490277777777778in"}

\*Indicates statistically significant from 2005 at p \< .05.

In addition to the choice and prominence of the core reading curriculum
in teachers' instruction, we were also interested in how often teachers
used various primary and/or supplementary instructional materials to
teach reading. Materials usage tended to reflect similar patterns
reported last year for cohort 1. Also, just as we saw significantly more
teachers using computers for vocabulary instruction, we found a similar
trend in using computers for reading programs and games (see Table A-6,
Appendix A).

Over 40 percent of teachers responded that they used the following
always/almost always:

-   Narrative decodable or leveled books (65.1 percent);

-   Phonics workbooks (47.3 percent);

-   Computer reading programs and reading games (45.5 percent);

-   Narrative textbooks (42.0 percent); and

-   Children's narrative trade books (40.3 percent).

Comparing 2007 and 2005 usage reports revealed one significant
increase--- teachers' use of computer reading programs and games with
their students (45.5 vs. 31.8 percent). Whether teachers obtained new
instructional software[^18] or are simply making better use of existing
technology resources is unclear.

As was the case with cohort 1 last year, usage of several of the
materials listed were found to be statistically significant by grade
level, including phonics workbooks, student writing, computer reading
programs, overheads, and at-home extensions. Data relative to most of
these materials show a consistent pattern (see Figures A-1 through A-5
in Appendix A). In short, most materials were used always/almost always
by a higher percentage of kindergarten and 1st grade teachers, followed
by a decrease by teachers in the two higher grades. For example, roughly
57 percent of kindergarten teachers and 1st grade teachers reported
always/almost always using student writing, while roughly 29 percent of
grade 2--3 teachers did so. We see this pattern with the use of all of
the materials listed above, except for overheads, where more grade 2--3
teachers used them more frequently than did grade K--1 teachers (perhaps
because overheads are intended for whole class rather than individual
use---the former more common in the later grades).

Thus far, we have reported what reading curriculum and materials
teachers are using to teach reading. However, we know that the use of
curricular and material resources does not exist in a vacuum, and
implementation of any core reading program is affected by any number of
issues. We asked teachers and principals, then, what type of impact
various factors had on implementation of the reading program in their
schools. Responses about the six factors common to the teacher and
principal surveys are presented in Figure II-8.

Similar to findings in Chapter 3, principals' responses are more
positive overall than those of teachers. For example, higher percentages
of principals versus teachers saw the following factors as having a
positive impact on the implementation of the core reading program:

-   Amount of time allotted for instruction (100.0 percent vs. 86.5
    percent);

-   Influence of district leadership (81.3 percent vs. 59.0 percent);
    and

-   Reading initiatives other than Reading First (75.0 percent vs. 52.8
    percent).

Only 12.5 percent of principals viewed staff mobility and turnover as
having a negative impact on implementation of the core reading program,
while 25.4 percent of teachers felt the same way. Twenty percent of
teachers also reported that the amount of time allotted for teacher
planning negatively impacted implementation, an improvement over the
2005 findings. That is, significantly more teachers in 2007 reported
that planning time had a positive impact on implementation (55.8 percent
vs. 33.0 percent).

Figure II-8.---Principals' and teachers' perceptions of the impact of
various factors on implementation of the core reading program

+--------------------------+-------------+----------------------------+
| ![](media/image          | >           | ![](media/image12.wmf){w   |
| 10.wmf){width="2.3875in" | Non-reading | idth="2.446527777777778in" |
| heigh                    | >           | heig                       |
| t="3.752083333333333in"} | initiatives | ht="3.7840277777777778in"} |
|                          | > (such as  |                            |
|                          | > math or   |                            |
|                          | > science   |                            |
|                          | > programs) |                            |
|                          | >           |                            |
|                          | > Reading   |                            |
|                          | >           |                            |
|                          | initiatives |                            |
|                          | > other     |                            |
|                          | > than      |                            |
|                          | > Reading   |                            |
|                          | > First     |                            |
|                          | >           |                            |
|                          | > Staff     |                            |
|                          | > mobility  |                            |
|                          | > and       |                            |
|                          | > turnover  |                            |
|                          | >           |                            |
|                          | > Amount of |                            |
|                          | > time      |                            |
|                          | > allotted  |                            |
|                          | > for teach |                            |
|                          | > planning  |                            |
|                          | >           |                            |
|                          | > Amount of |                            |
|                          | > time      |                            |
|                          | > allotted  |                            |
|                          | > for       |                            |
|                          | >           |                            |
|                          | instruction |                            |
|                          | >           |                            |
|                          | > Influence |                            |
|                          | > of        |                            |
|                          | > district  |                            |
|                          | >           |                            |
|                          |  leadership |                            |
+--------------------------+-------------+----------------------------+

NOTE: Percents may not add to 100 due to rounding.

How Are Teachers Teaching Reading?

It is obviously not enough to ask teachers what they are using to teach
reading; we must also know how they go about teaching reading on a daily
basis. In this section, we examine both general and specific teaching
strategies, as well as how teachers monitor whether their students'
reading skills are improving.

The Assess-Plan-Teach (APT) instructional model has been adopted by Ohio
as a key component in the delivery of scientifically based reading
instruction. It is a "whole picture" approach to instruction rather than
an array of discrete strategies. The model builds on the belief that
successful implementation of the five essential components requires 1)
information on what students know (assess), 2) use of this information
to develop appropriate instructional programs (plan), and 3) delivery of
instruction using a variety of strategies that effectively scaffold the
learning process (teach).

**Similar to previous reports, the overall findings suggest that the
model is well understood and generally adopted. There were very few
statistical differences between responses in 2004 and 2006 in terms of
cohort 1 teachers' reported usage of various APT components, and that
picture remains unchanged for teacher reports in 2005 and 2007. In fact,
the only difference was a significant increase in teachers' reporting of
daily use of assessment data to decide what SBRR content needs to be
taught (19.8 vs. 14.7 percent). Clearly this finding affirms a welcome
outcome of RFO, that is, increasing use of data for classroom-based
instruction. It appears that teachers are becoming more savvy in terms
of understanding and interpreting data and using this information to
drive content-related decisions.**

In general, the majority of teachers engaged in various teaching
activities on a daily basis. For example, 90 percent or more of teachers
responded they engaged daily in explaining the task (95.2 percent),
modeling the task (91.4 percent), and assisting students in performing
the task (91.3 percent).

Planning and assessment activities occurred more often on a weekly
basis. Specifically, over half of teachers responded they did the
following planning and assessment activities weekly:

-   Interpret assessment results and apply to materials selection (61.1
    percent);

-   Interpret assessment results and apply to classroom instruction
    (60.1 percent);

-   Plan interventions to be delivered by the teacher inside or outside
    of the class (56.9 percent);

-   Based on assessment data, decide what SBRR content needs to be
    taught (56.0 percent); and

-   Identify best ways to assess student performance and to analyze data
    (53.6 percent).

Again, these data reinforce teachers' use of assessment to inform
instruction. In fact, one of the strongest messages to come out of our
examination of instructional practices is that the reading program is
supported by systematic assessment and that teachers pay attention to
and use the results of assessments. Specifically:

-   86.6 percent of teachers agreed that there is a common set of
    assessments in reading that all teachers use, other than the TPRI,
    DIBELS, and TerraNova.

-   94.2 percent agreed that diagnostic assessment guides instructional
    planning.

-   93.8 percent of principals echoed teachers in their agreement that
    there is a student assessment system in place that uses reliable and
    valid measurement tools to measure growth in reading.

Similar to findings last year for cohort 1, significant grade-level
differences were found relative to the use of the APT model with cohort
2. Most of them are to be expected, given the cognitive development of
students at particular levels. For example, grade K--1 teachers were
more likely than their grade 2--3 colleagues to diagnose students who
fall behind daily in order to pinpoint reading strengths and weaknesses.
They were also more likely to monitor, on a daily basis, independent
student performance of a task than were the teachers of older students
in grades 2--3. These and other grade-level differences are presented in
Appendix A (Figures A-6 through A-10).

Next, we asked teachers about specific instructional strategies they use
with struggling readers. As has been the case in previous years,
teachers continued to make rather frequent use of special strategies for
this group. However, teacher responses in 2007 reveal significant
increases from 2005 in several areas (Figure II-9). [^19]

Specifically, the data show increases relative to K--3 teachers' use
always/almost always of various differentiated instructional techniques
and literacy staff. For example, significant increases occurred in:

-   Selecting materials for students that target individual needs (64.4
    percent vs. 49.8 percent);

-   Delivering individual or group interventions inside or outside of
    class (62.0 vs. 45.8 percent); and

-   Using the literacy specialist to provide instructional feedback
    (19.6 vs. 11.6 percent).

**Figure II-9.---Significant increases in teachers' use of strategies
for struggling readers: 2007 and 2005**

![](media/image13.wmf){width="5.999305555555556in"
height="4.095138888888889in"}

NOTE: All items are statistically significant from 2005 at p \< .05.

We examined the strategies by grade level and found one significant
difference: more K--1 than grade 2--3 teachers report always/almost
always using trained paraprofessionals to deliver interventions. More
than likely, this difference is due to the increased assignment of
paraprofessionals to the earlier elementary grades.

An important part of how teachers teach reading is how they monitor
students' improvement in literacy. The obvious corollary is that
evaluating student progress in reading equates to teachers' increased
effectiveness in targeting instruction to student need. When asked how
often various tools and sources were used to monitor student
improvement, teachers responded that they use a variety of monitoring
tools always/almost always (see Table A-7 in Appendix A). In addition,
data show significant increases in the use of DIBELS data to assess
student improvement**.**

High or moderate percentages of teachers indicated the following
regarding the monitoring of student improvement:

-   68.9 percent of K--3 teachers always/almost always used student work
    products;

-   50.5 percent of K--3 teachers always/almost always used assessments
    that are part of the core reading series/other classroom tests; and

-   40.2 percent of K--3 teachers often use other diagnostic assessments
    (e.g., Rigby benchmarking, running records).

On the other hand, 51.7 percent of teachers reported rarely/never using
the TerraNova to assess student improvement, and 34.8 percent reported
rarely/never using the TPRI. More than likely, these lower percentages
relate to the grades in which these tests are administered, and
grade-level analyses support this claim. Specifically, fewer K--2
teachers (roughly 10--28 percent) report rarely/never using the TPRI
assessment results to assess reading improvement than do teachers in
grade 3 (65.2 percent). This finding makes sense given that TPRI is last
administered in the fall of grade 2 and no new results are available to
3rd grade teachers. Similarly, 94.9 percent of kindergarten teachers
(compared to roughly 39 percent of grade 1--3 teachers) reported they
rarely/never use TerraNova assessment data---obviously since there are
none for their students.[^20]

Comparison of the 2007 and 2005 responses show significant increases in
the percentage of teachers who reported using the DIBELS assessment data
always/almost always to assess student improvement (44.7 percent vs.
37.8 percent). This was a significant increase that we also saw last
year with cohort 1 and, coupled with similar findings presented earlier
in this chapter, speaks favorably for how teachers are growing in their
understanding of data-driven instruction.

Reading First-Ohio was conceived, in part, upon the pillars of the five
essential components of reading. It is important, then, to move away
from the macro question of how teachers teach reading and monitor
student growth toward a deeper micro question, as follows in the next
section.

How Often Does Teachers' Instruction Address the Five Essential
Components of Reading?

Teachers were asked how often they engage their students in activities
in each of the five essential components of reading: phonemic awareness,
phonics, fluency, vocabulary, and comprehension. This section looks
first at the overall trends in the cross-sectional data relative to each
of the five essentials, then examines more closely grade-level
differences and differences between the 2007 and 2005 teacher reports.

**Overall, as in years past, the data show a majority of teachers at all
grade levels engaging students in each of the five components daily or
several days a week. Developmentally appropriate grade-level differences
still appear in grades K--1 and 2--3. Also, significantly more teachers
are making use of computers for vocabulary instruction; however,
specific phonemic awareness and phonics strategies are less prevalent.**

**Phonemic awareness.** Teachers continue to make frequent use of
phonemic awareness activities. In fact, a quarter or more of teachers
engage students daily in each of these activities, and over half report
engaging students daily in two specific activities---blending phonemes
to form words (56.0 percent) and identifying phonemes (51.4 percent)
(Figure II-10).

**Figure II-10.---Frequency of cohort 2 teachers' engaging students
daily in phonemic awareness activities: 2007**

![](media/image14.wmf){width="5.979861111111111in"
height="3.654166666666667in"}

**Phonics.** Teachers were asked about three primary activities related
to phonics instruction. Generally, data show that over 50 percent of
K--3 teachers engaged students in phonics activities daily (Figure
II-11).

**Figure II-11.---Frequency of cohort 2 teachers' engaging students
daily in phonics activities: 2007**

![](media/image15.wmf){width="5.9944444444444445in"
height="3.654861111111111in"}

**Oral reading fluency.** Similar to findings for phonics, percentages
are high for K--3 teachers' engaging students daily in oral reading
fluency (over 50 percent report everyday use of each oral fluency
activity) (Figure II-12). Reports are particularly high (66.2 percent)
for engaging students in reading books at their own independent level of
reading ability to complement classroom instruction.

**Figure II-12.---Frequency of cohort 2 teachers' engaging students
daily in oral reading fluency activities: 2007**

![](media/image16.wmf){width="5.9944444444444445in"
height="3.654861111111111in"}

**Vocabulary development.** In general, a majority of teachers engaged
students in vocabulary development several days a week versus on a daily
basis. Specifically, over 40 percent of teachers engaged students
several days a week in word analysis (49.3 percent), learning specific
words before reading (47.6 percent), and actively engaging with words
such as making up sentences with words (46.4 percent) (Figure II-13).

**Figure II-13.---Frequency of cohort 2 teachers' engaging students
several days a week in vocabulary development activities: 2007**

![](media/image17.wmf){width="5.9944444444444445in"
height="4.321527777777778in"}

^1^For example, making up sentences with words and brainstorming
examples of words in categories, such as types of occupations.

^2^For example, substituting easy words for more difficult words.

^3^For example, conversing with adults, being read to, and reading on
their own.

**Reading comprehension.** Teachers were given a list of seven reading
comprehension strategies deemed most effective by the National Reading
Panel. As was the case with vocabulary development, most teachers
engaged students several days a week in activities designed to promote
reading comprehension. More specifically, close to or over half of
teachers engaged students several days a week in using graphic and
semantic organizers to focus on concepts, recognizing story structure,
summarizing text, and predicting what might occur next in the text
(Figure II-14).

**Figure II-14.---Frequency of cohort 2 teachers' engaging students
several days a week in reading comprehension activities: 2007**

![](media/image18.wmf){width="5.99375in" height="3.701388888888889in"}

^1^For example, identifying key ideas, condensing and generalizing
information.

^2^For example, leading discussion by posing a question, discussing text
meaning, and clarifying group misunderstanding of text meaning.

As has been the case in previous examination of the five essential
features of reading, grade-level differences remained in all areas. And
again, these differences were largely developmentally appropriate
distinctions between grade K--1 and grade 2--3 students regarding time
spent in phonics and phonemic awareness. Specifically, data show K--1
students participating in most of these activities at higher levels than
students in grades 2 and 3. In addition, certain phonemic awareness
activities require more sophisticated literacy skills (i.e., deleting,
adding, blending, segmenting phonemes), so we see slightly greater daily
use at grade 1 than grade K.

Regarding oral reading fluency, greater daily use was found in grades
1--3 as opposed to grade K in terms of reading and re-reading passages
orally as students receive guidance. This grade-level difference makes
sense, of course, since fluency is an emergent skill. In relation to
vocabulary instruction, more 1st grade teachers tended to report daily
use of vocabulary activities than did teachers in the other grades.
These activities include actively engaging students with words and
having students participate in word analysis.

Finally, with reading comprehension, significantly more kindergarten
teachers reported asking students daily to recognize story structure,
predict what might occur next in the text, and engage in reciprocal
teaching conversations about the meaning of the text---all foundational
skills that one would expect to be emphasized in kindergarten
particularly. (See Figures A-11 through A-27 in Appendix A for
presentation of grade-level differences.)

There were three significant differences between the 2007 and 2005
survey responses relative to instruction in the five essential areas
(Figure II-15).

**Figure II-15.---Significant differences in teachers' engaging students
daily in specific areas relative to the five essential components: 2007
and 2005**

![](media/image19.wmf){width="5.9944444444444445in"
height="3.3208333333333333in"}

First, there was a significant increase in the number of teachers who
reported that they use computers daily in vocabulary instruction (22.7
percent vs. 17.2 percent). This finding mirrors what we found with
cohort 1, where teachers' use of computers for vocabulary instruction
rose significantly from 2004 to 2006. As we alluded before, these
findings are positive, particularly since the National Reading Panel
(NRP) identified computer-assisted vocabulary instruction as a promising
practice.

Second, significantly fewer teachers reported that they asked children
daily to identify phonemes (51.4 percent vs. 60.4) and to apply
knowledge of phonics as they read words, sentences, and text (71.6
percent vs. 80.4 percent). It is not entirely clear why fewer teachers
responded in this manner, but the finding appears related to a similar
finding (presented in the next section) where fewer teachers in 2007
than in 2005 report agreed that they have put more emphasis on
instruction in phonemic awareness.

In order to understand whether the reports of usage discussed above were
common practice or were somehow related to Reading First, we asked a
series of questions about the extent to which specific changes in the
five essentials had occurred during the 2006--07 school year. Overall,
the data show that teachers have increased their emphasis or approach to
the five essential components, with the greatest number of teachers
reporting they emphasize oral fluency and reading comprehension**.**

Specifically, over 80 percent of teachers strongly agreed/agreed with
the following statements:

-   I have put more emphasis on helping students become fluent readers
    (93.8 percent).

-   I have put more emphasis on explicit skill and strategy instruction
    for reading comprehension (92.3 percent).

-   I have put more emphasis on vocabulary instruction (87.1 percent).

-   I have put more emphasis on systematic phonics instruction (80.9
    percent).

-   I have put more emphasis on instruction in phonemic awareness (80.3
    percent).

Interestingly enough, even though all teacher reports are high, the
slightly lower percentages of agreement with the statements about
phonics and phonemic awareness appear to reinforce similar findings and
sentiment from cohort 2 teachers in general for the 2006--07 school
year. For instance (as previously stated), there was a statistically
significance decrease between 2007 and 2005 in the percentage of
teachers agreeing or strongly agreeing that more emphasis was placed on
phonemic awareness.

We included one area that the research base identifies as less effective
than explicit and intensive phonics instruction (i.e., literature-based
embedded phonics) and continue to examine its use by teachers.
Specifically, the NRP report states: "Comparison of PASP (e.g., explicit
and intensive instruction in PA) and EP (embedded phonics) revealed
superior performance by PASP on measures of phonological awareness,
phonemic decoding accuracy and efficiency, and word reading
accuracy....Clearly, the effects of synthetic phonics instruction
persisted more strongly over the grades than effects of embedded phonics
instruction" (National Institute for Literacy, 2000, p. 2-128).

Both cohort 1 teachers from last year and cohort 2 teachers this year
(roughly 79 percent each) agreed that they have put more emphasis on
literature-based programs that emphasize reading and writing activities
with embedded phonics. However, there is a statistically significant
decrease in the cohort 2 longitudinal sample, with 75.6 percent of
teachers in 2007 compared to 85.2 percent in 2005 agreeing with that
statement.[^21] How RFO teachers understood and interpreted the concept
of embedded phonics in unclear; however, given the high percentage of
teachers that continue to utilize this type of instruction, it is an
area that we will keep following.

We also examined changes in the usage of the five essential components
by grade level (see Figures A-28 through A-30 in Appendix A). As was the
case with cohort 1 last year, significant changes in phonemic awareness
and phonics instruction were greater at the earlier than the later grade
levels (e.g., more K--1 than grade 2--3 teachers strongly agreed that
they have put more emphasis on phonemic awareness and phonics during the
current school year). The inverse is true for vocabulary
instruction---more grade 2--3 than K--1 teachers strongly agree that
they have put more emphasis on vocabulary instruction during the current
school year. These findings support the earlier grade-level differences
reported for each of the five essential components.

[]{.mark}

[]{.mark}

Why Does Instruction Look the Way It Does?

Our last question is a contextual one. Just as the implementation of the
core reading program is influenced by a number of issues (see above), so
it goes with reading instruction. We asked teachers to what extent
various factors (e.g., state standards, conference participation, and
courses taken) influence their reading instruction.

**Data show that high percentages of cohort 2 teachers, similar to
teachers in cohorts 1 and 3, found factors related to materials,
standards, and assessment to have substantial effect on their reading
instruction. Data show feedback received through the ELLCO and
Close-Ups, as well as principal recommendations, cast a smaller net of
influence. Significant increases were found in teachers' reliance upon
other teachers for impacting their reading instruction.**

Teachers ranked in effect (i.e., substantial, some, no) how 16 different
sources impacted their reading instruction (see Table A-8 in Appendix
A). Figure II-16 presents the eight factors reported by most K--3
teachers to have substantial effect on their reading instruction.[^22]
Specifically:

-   78.8 percent of teachers said that materials, books, and resources
    have substantial effect on instruction.

-   69.6 percent of teachers said state and/or district standards have
    the same type of effect, as do similar district curriculum
    frameworks (51.7 percent).

-   63.0 percent said TPRI, DIBELS, and TerraNova data have substantial
    effect.

**Figure II-16.---Factors reported by a majority of cohort 2 teachers as
having substantial effect on reading instruction: 2007**

![](media/image20.wmf){width="5.996527777777778in"
height="3.8881944444444443in"}

There was a significant increase in the percentage of teachers reporting
a substantial effect of one factor on reading instruction from 2005 to
2007. This factor, recommendations from other teachers (50.0 vs. 35.6
percent), was also significant in the longitudinal sample. Although we
cannot be certain why this change emerged, it speaks positively for the
increased collaboration and teamwork that RFO hopes to foster in school
buildings, as well as for sustainability.

As for factors reported to have no effect on their reading instruction
by at least a quarter or more of teachers, the following were identified
(Figure II-17):

-   State-sponsored SIRI institutes (42.8 percent);

-   Feedback received through the Close-Ups teaching tool (36.8
    percent);

-   Recommendations from the school principal (33.8 percent); and

-   Feedback received through the Early Language and Literacy Classroom
    Observation (ELLCO) (25.5 percent).

[]{.mark}

**Figure II-17.---Factors reported by at least a quarter of cohort 2
teachers as having no effect on reading instruction: 2007**

![](media/image21.wmf){width="5.999305555555556in"
height="3.8041666666666667in"}

The feedback teachers received through ELLCO and Close-Ups were areas we
noted as lacking widespread impact on teacher instruction initially in
cohort 1. However, we noted significant increases in teacher reports in
those same areas last year. We should note that data for cohort 2 do
show increases as well in these areas (roughly 5 percent more teachers
in 2007 than in 2005 reporting substantial effect of each), but these
increases are not significant. Certainly, lack of resources prevents
across-the-board utilization of Close-Ups and ELLCO coaching with all
K--3 teachers, and these limitations must be considered in interpreting
these data. Moreover, the lack of effect relative to the impact of the
SIRI institutes may be due to nonparticipation.

Grade-level analyses revealed two significant differences. First,
kindergarten and grade 2 teachers and grade 1 and 3 teachers appeared to
feel similarly about the relative effect of the Close-Ups coaching
(Figure II-18), with the former group expressing greater benefit from
the tool. For instance, roughly 60 percent of grade K and grade 2
teachers felt that the Close-Ups coaching has had some effect on their
instruction, while roughly 30 percent of grade 1 and grade 3 felt
similarly. Perhaps these data speak to the manner in which the
instrument is targeted and used with teachers at various grade levels.

Second, grade-level analyses show that kindergarten teachers appeared to
profit more from reading research presented at conferences than did
teachers in the other grades. Specifically, over a third (35.0 percent)
of kindergarten teachers reported that reading research at conferences
has a substantial effect on their instruction versus roughly 10--19
percent of grade 1--3 teachers (not shown in figures).

**Figure II-18.---Feedback received through the Coaching for Effective
Classroom Instruction: Close-Ups of Teaching Tools**

![](media/image22.wmf){width="5.999305555555556in"
height="3.147222222222222in"}

NOTE: Percents may not add to 100 due to rounding.

Reference

National Institute for Literacy. (2000, December). *Report of the
National Reading Panel: Teaching children to read.* Washington, DC:
Author.

**Chapter 5. Program Monitoring and\
Use of Data for Program Improvement**

The final area of implementation examined is program monitoring. Program
monitoring supports the Reading First-Ohio program in two distinct ways.
First, there is the program monitoring that is carried out by principals
and other administrative personnel as part of their management duties.
Second, there is the monitoring that occurs through the use of the
Program Monitoring Tool. This tool, developed by the Reading First-Ohio
Center, is intended to provide information to the state on the
implementation of the three state standards. To distinguish between the
two in this report, we refer to the work of the principal as "program
monitoring" and use the term "Program Monitoring" to refer to the
state's tool (which is not addressed in this chapter).

**Principal's responses suggest that procedures for program monitoring
have been developed and are being used. The data from the varying
monitoring tools contribute to decisions at the school level as well as
to the accountability requirement. Overall, the data show little change
from 2005, when use of monitoring tools was also reported to be strongly
in place.**

The principal survey contained several questions about the status of
program monitoring (Tables B-5 and B-6 in Appendix B). One hundred
percent of cohort 2 principals agreed or strongly agreed with the
following statements in 2007:

-   Procedures have been developed for collecting and sharing data.

-   Procedures have been implemented for collecting and sharing data.

-   Results from program monitoring help to evaluate student needs and
    inform interventions.

-   Results from program monitoring help make our school make better use
    of instruction materials.

-   The monitoring and program improvements specific to our school's RFO
    program are linked to our overall plan.

Over 80 percent also agreed or strongly agreed with the following:

-   There is one uniform reporting system used at both district and
    school levels.

-   Results from program monitoring help to evaluate teacher
    performance.

Additionally the quality of the information received through monitoring
was uniformly judged to be high, with 93.8 percent of the principals
agreeing or strongly agreeing with the following:

-   Our school monitoring system yields information sufficient to inform
    building-level decision-making (i.e., personnel, scheduling,
    resource decisions).

-   Our school monitoring system yields information sufficient to inform
    classroom-level or instructional decisionmaking.

-   Our school monitoring system yields information sufficient to
    document and demonstrate accountability for Reading First-Ohio state
    and federal requirements.

The surveys also asked about the frequency with which various types of
data were used. Student assessments from standardized instruments were
cited by 75 percent of the principals as always or almost always being
used as a monitoring strategy. Other strategies mentioned with
relatively high frequency were:

-   Discussions with literacy specialist (56.3 percent);

-   Classroom observations (including walk-throughs) (56.3 percent); and

-   Teacher attendance at professional development sessions (56.3
    percent).

> Least frequently used were:

-   Discussions with parents (37.6 percent sometimes or never); and

-   Teacher self-assessment of effectiveness in teaching the five
    essential reading components (37.6 percent sometimes or never).

Relatively little change is found when responses for 2005 and 2007 are
compared, with one interesting exception: In response to the choice
"other student diagnostic data (e.g., Rigby benchmarking, other reading
tasks)," the percentage of principals saying that these tools were
always or almost always used for monitoring increased from 13.3 percent
to 43.8 percent.

**Chapter 6.**

**Factors That Explain Differences in\
Levels of Implementation**

Chapters 2--5 have focused on implementation, examining the fidelity of
implementation of cohort 2. Generally, the data show that for the
program as a whole, implementation is strong, and by year 3 of the
program, both cohorts 1[^23] and 2 have in place key features of the RFO
design, including professional development, instructional practices, and
a supportive and coherent program structure. In this section we look at
implementation further, examining individual schools' level of
implementation as determined by scores on scales specially constructed
for RFO and by the ELLCO, a measure used by the program. The chapter
presents information on the factors believed by our cohort 2 respondents
to be key, and concludes with exploration of possible predictors of
program implementation.

Implementation as Measured by Teacher and Principal Surveys

As described in the Westat/LPA 2005 report, we have developed
implementation scales based on teacher and principal surveys to
empirically characterize the level of program implementation among RFO
schools. Table II-9 shows the three levels of implementation scale:
level 1 represents overall program implementation; level 2, the three
program standards---Professional Learning (PL), Comprehensive and
Coherent Program Design (CCPD), and Systematic and Explicit Reading
Instruction (SERI); and level 3, 12 scales designed to assess different
components of the three standards.[^24] Scales 1--7 were derived from
the teacher survey, and scales 8--12 were from the principal
survey.[^25]

**Table II-9.---**Implementation scales at three levels

  ---------------- ---------------- -----------------------------------------
  Level 1          Level 2          Level 3

  Program          Professional     Scale 1: Professional development
  Implementation   Learning (PL)    activities
  (overall)                         

                                    Scale 3: TRIO and literacy specialist

                                    Scale 8: Principal's view of literacy
                                    specialist

                   Comprehensive    Scale 4: Teachers' perspective of
                   and Coherent     comprehensive and coherent program design
                   Program Design   
                   (CCPD)           

                                    Scale 9: Monitoring program effectiveness

                                    Scale 10: Implementation of formal
                                    procedures

                                    Scale 11: Resource and time allocation

                   Systematic and   Scale 2: Assess-plan-teach
                   Explicit Reading 
                   Instruction      
                   (SERI)           

                                    Scale 5: Five essentials-frequency of
                                    instruction

                                    Scale 6: Use of strategies for struggling
                                    readers

                                    Scale 7: Five essentials-increased
                                    emphasis

                                    Scale 12: SBRR
  ---------------- ---------------- -----------------------------------------

Each school received an overall implementation score as well as score
for each of the three program standards. The following procedures were
used to derive the implementation scores.

-   The level 3 scales were summed to create each level 2 score for
    program standards. The scales were weighted by the proportion of
    items the scale contributed to the program standard. The overall
    scale was created in a similar way.

-   Scale scores for the three levels of scales were calculated based on
    year 1 and year 3 surveys from cohort 1 using the factor structure
    from the year 1 survey and aggregated to the school level.

The unit of analysis is the school. Fifty-one schools (36 from cohort 1
and 15 from cohort 2) are included in the analysis. These schools have
year 1 and year 3 data from teacher and principal surveys as well as
student achievement data from TerraNova (grades 1--3) and/or OAT-R
(grade 3) administration. We did not include data from cohort 3 because
the cohort is in its second year of implementation and we do not have
its year 3 data on implementation.

Cohort 1

Figure II-19 presents the average scores for overall program
implementation and the three program standards for the 36 schools from
years 1 and 3. For presentation, we have standardized the scale scores
on a range of 0 to 1 with 1 representing the highest possible score. The
average overall program implementation score is 0.79, and the scores for
the three program standards range between 0.78 to 0.79 in year 3. In
addition, the data also show that overall program implementation
increased by 0.06 from year 1 to year 3. For the three program
standards, the largest increases (0.08 to 0.09) were in Professional
Learning and Comprehensive and Coherent Program Design, while Systematic
and Explicit Reading Instruction increased by only 0.02. The improvement
for overall implementation scales and the three standards is
statistically significant at the .10 level.

**Figure II-19.---Changes in program implementation and the three
program standards for cohort 1**

![](media/image23.wmf){width="5.999305555555556in"
height="2.829861111111111in"}

\*Statistically significant at p \< .10.

Figure II-20 shows implementation scores for all 12 specific scales for
the 36 schools from years 1 and 3. The average levels of implementation
for all of the 12 specific scales in year 3 range from 0.70 to 0.93. In
addition, scales 1--4, 6, and 10--11 show statistically significant
growth from year 1 to year 3.

**Figure II-20.---Changes in 12 scales of program implementation for
cohort 1**

![](media/image24.wmf){width="6.004166666666666in" height="2.48125in"}

  ------------------------------------------- ---------------------------
  Scale 1: Professional development           Scale 7: Five
  activities                                  essentials-increased
                                              emphasis

  Scale 2: Assess-Plan-Teach                  Scale 8: Principal's view
                                              of literacy specialist

  Scale 3: Trio and literacy specialist       Scale 9: Monitoring program
                                              effectiveness

  Scale 4: Teachers' perspective of           Scale 10: Implementation of
  comprehensive and coherent program design   formal procedures

  Scale 5: Five essentials-frequency of       Scale 11: Resource and time
  instruction                                 allocation

  Scale 6: Use of strategies for struggling   Scale 12: SBRR
  readers                                     
  ------------------------------------------- ---------------------------

\*Statistically significant at p \< .10.

Cohort 2

Figure II-21 presents the average scores for overall program
implementation and the three program standards for the 15 cohort 2
schools for years 1 and 3. The average overall program implementation
score is 0.77, and the scores for the three program standards range
between 0.74 to 0.78 in year 3. In addition, the data also show that
overall program implementation stayed the same from year 1 to year 3.
Similarly, the changes for the three program standards were minimal,
with PL increasing by 0.01, CCPD decreasing by 0.01, and SERI dropping
by 0.3. However, the decrease in SERI was statistically significant.

**Figure II-21.---Changes in program implementation and the three
program standards for cohort 2**

![](media/image25.wmf){width="5.999305555555556in"
height="2.7180555555555554in"}

\*Statistically significant at p \< .10.

Figure II-22 shows the implementation scores for all 12 specific scales
from years 1 and 3. Scales 4, 5 and 7 had significant decreases from
year 1 to year 3. Changes in other scales were modest.

**Figure II-22.---Changes in the 12 scales of program implementation for
cohort 2**

![](media/image26.wmf){width="6.004166666666666in" height="2.60625in"}

  ------------------------------------------- ---------------------------
  Scale 1: Professional development           Scale 7: Five
  activities                                  essentials-increased
                                              emphasis

  Scale 2: Assess-Plan-Teach                  Scale 8: Principal's view
                                              of literacy specialist

  Scale 3: Trio and literacy specialist       Scale 9: Monitoring program
                                              effectiveness

  Scale 4: Teachers' perspective of           Scale 10: Implementation of
  comprehensive and coherent program design   formal procedures

  Scale 5: Five essentials-frequency of       Scale 11: Resource and time
  instruction                                 allocation

  Scale 6: Use of strategies for struggling   Scale 12: SBRR
  readers                                     
  ------------------------------------------- ---------------------------

\*Statistically significant at p \< .10.

Comparisons of implementation between cohorts 1 and 2 show that
implementation fidelity in year 1 was lower for cohort 1 than cohort 2
in overall scores, two standards (CCPD and SERI), and scales 2 and
10--11, but higher in scales 3 and 6. However, by year 3, cohort 1 had
higher level of implementation in scales 1 and 4--7 and fell behind only
in scale 2. It is clear that cohort 1 has made greater progress in
program implementation (not presented in tables).

Implementation as Measured the ELLCO

The ELLCO protocol is designed to address pre-K to grade 3 literacy and
language practices and materials as well as environmental factors in
early childhood classroom. In general, each teacher was observed twice
each year by a team of two literacy specialist observers from the RFO
Center in September/October 2006 and April/May 2007. Teachers were rated
on a scale of 1 to 5, with 5 being the highest.

The results show a high degree of inter-rater reliability with
statistically significant Kappa coefficients above .90 in all 14
observed areas. Given the high correlation between the observers, we
then computed the mean observation scores of fall and spring
observations. We conducted pairwise statistical tests to see if, on
average, teacher instruction practices improved from fall to spring both
overall and by RFO cohort.

Table II-10 shows that teachers from cohort 1 experienced statistically
significant growth during the year in all 14 observed areas. In
addition, the level of rating in spring is high across the board, with
all scores above 3.8 on a scale of 1 to 5.

Table II-10.---Instruction growth for cohort 1, by observed area (N=227)

  ------------------------------------------------ ----------- ------------
  Observed area                                    Fall        Spring

  Organization of the classroom                    4.18        4.41\*

  Contents of the classroom                        4.01        4.30\*

  Presence and use of technology                   3.85        4.18\*

  Opportunities for child choice and initiative    3.55        3.91\*

  Classroom management strategies                  4.24        4.47\*

  Classroom climate                                4.31        4.57\*

  **General Classroom Environment Subtotal**       **4.03**    **4.31\***

  Oral language facilitation                       3.91        4.17\*

  Presence of books                                3.86        4.24\*

  Approaches to book reading                       4.05        4.32\*

  Approaches to children's writing                 3.89        4.19\*

  Approaches to curriculum integration             3.51        3.86\*

  Recognizing diversity in the classroom           3.74        4.12\*

  Facilitating home support for literacy           3.96        4.29\*

  Approaches to assessment                         4.16        4.55\*

  **Language, Literacy, and Curriculum Subtotal**  **3.89**    **4.22\***

  **Total**                                        **3.96**    **4.26\***
  ------------------------------------------------ ----------- ------------

\*Indicates that the score in spring is significantly higher than in
fall at the .05 level.

NOTE: Ratings are based on a scale of 1 to 5, with 5 being the highest.

Table II-11 shows that teachers from cohort 2 also experienced
statistically significant growth during the year in 7 of 14 observed
areas, although the level of classroom organization decreased. In
addition, the level of rating in spring was relatively low, with all of
the scores below 3.5; in four areas (i.e., opportunities for child
choice and initiative, oral language facilitation, approaches to
curriculum integration, and recognizing diversity in the classroom) the
scores were below 3.

Table II-11.---Instruction growth for cohort 2, by observed area (N=122)

  --------------------------------------------- ------------ ------------
  Observed area                                 Fall         Spring

  Organization of the classroom                 3.52         3.48\^

  Contents of the classroom                     3.06         3.18

  Presence and use of technology                3.05         3.07

  Opportunities for child choice and initiative 2.49         2.77\*

  Classroom management strategies               3.30         3.42

  Classroom climate                             3.29         3.43

  **General Classroom Environment Subtotal**    **3.12**     **3.22**

  Oral language facilitation                    2.71         2.90\*

  Presence of books                             3.16         3.23

  Approaches to book reading                    3.15         3.36\*

  Approaches to children's writing              2.89         3.17\*

  Approaches to curriculum integration          2.42         2.86\*

  Recognizing diversity in the classroom        2.54         2.81\*

  Facilitating home support for literacy        2.92         3.04

  Approaches to assessment                      2.88         3.17\*

  **Language, Literacy, and Curriculum          **2.83**     **3.07\***
  Subtotal**                                                 

  **Total**                                     **2.98**     **3.15\***
  --------------------------------------------- ------------ ------------

\* Indicates that the score in spring is significantly higher than in
fall at the .05 level; \^ indicates that the score in spring is
significantly lower than in fall at the .05 level.

NOTE: Ratings are based on a scale of 1 to 5, with 5 being the highest.

Table II-12 shows that teachers from cohort 3 experienced statistically
significant growth during the year in all 14 observed areas. The ratings
in spring are all between 3.4 and 4.0 on a scale of 1 to 5.

Table II-12.---Instruction growth for cohort 3, by observed area (N=402)

  --------------------------------------------- -------------- ------------
  Observed area                                 Fall           Spring

  Organization of the classroom                 3.55+          3.89\*

  Contents of the classroom                     3.35           3.74\*

  Presence and use of technology                3.07           3.62\*

  Opportunities for child choice and initiative 2.94           3.46\*

  Classroom management strategies               3.61           3.93\*

  Classroom climate                             3.63           4.02\*

  **General Classroom Environment Subtotal**    **3.36**       **3.78\***

  Oral language facilitation                    3.13           3.59\*

  Presence of books                             3.20           3.67\*

  Approaches to book reading                    3.48           3.85\*

  Approaches to children's writing              3.13           3.66\*

  Approaches to curriculum integration          3.03           3.46\*

  Recognizing diversity in the classroom        3.08           3.55\*

  Facilitating home support for literacy        3.31           3.72\*

  Approaches to assessment                      3.36           3.83\*

  **Language, Literacy, and Curriculum          **3.21**       **3.67\***
  Subtotal**                                                   

  **Total**                                     **3.29**       **3.72\***
  --------------------------------------------- -------------- ------------

\* Indicates that the score in spring is significantly higher than in
fall at the .05 level.

NOTE: Ratings are based on a scale of 1 to 5, with 5 being the highest.

**Factors That Influence the Extent to Which RFO is Implemented**

In the section above we examined program implementation and the extent
to which implementation, measured through various means, predicts
student achievement. A related question, one that we have just begun to
explore this year, is what factors explain variance in implementation?
To explain the variance of level of implementation, we hypothesized that
implementation is influenced by a variety of contextual factors
including 1) student demographic characteristics, 2) teacher quality,
3) stability (among students, teachers, principals, literacy
specialists), 4) district economic factors, and 5) other programs.[^26]
Our first set of analyses examines variables from categories 1 and 3
above.

The implementation data are survey based and drawn from 51 schools in
cohorts 1 and 2---the schools for which we have both teacher and
principal data. Data on school enrollment and disadvantaged are from
ODE's Education Management Information System (EMIS). Stability data
come from various sources: principal and literacy specialist data were
provided by John Carrol University based on the yearly records from
2004--07; teacher retention data are based on Westat's survey receipt
control records, which were generated from the district-provided class
rosters. We decided not to include Cleveland State University's student
stability data because unlike the other stability data, they were
calculated at the district level.[^27]

Because the stability measures are on different scales, we standardized
each of the measure and derived an overall stability index in which all
three stability measures were weighted equally. The standardized score
has a mean of 0 and standard deviation of 1, which measures the school's
relative standing in the sample. Only 34 of the 51 schools have three
complete stability measures, all of which are from cohort 1. Table II-13
ranks these schools by overall stability index from high to low.

Table II-13.---Overall stability, by school

  ----------------------------------------- -------------------- --------
  School                                    District             Score

  Captain Arthur Roth Elementary            Cleveland            1.69

  Clifton Elementary                        Cincinnati           1.65

  Sheridan Elementary                       Youngstown           1.42

  Jefferson Montessori Elementary           Dayton               1.32

  Wilton Elementary                         Vinton               1.29

  Fullerton Elementary                      Cleveland            1.19

  Hamden Elementary                         Vinton               0.65

  Lagonda Elementary                        Springfield          0.65

  Marion-Sterling Elementary                Cleveland            0.63

  Iowa-Maple Elementary                     Cleveland            0.53

  Perrin Woods Elementary                   Springfield          0.50

  Williamson Elementary                     Youngstown           0.48

  Kenwood Heights Beacon EC Center          Springfield          0.44

  Empire Computech Elementary               Cleveland            0.36

  Allensville Elementary                    Vinton               0.31

  Zaleski Elementary                        Vinton               0.17
  ----------------------------------------- -------------------- --------

Table II-13.---Overall stability, by school---continued

  ----------------------------------------- -------------------- --------
  School                                    District             Score

  Marion C. Seltzer Elementary              Cleveland            0.17

  Clark Elementary                          Cleveland            0.02

  Corlett Elementary/Charles Dickens        Cleveland            -0.01

  Irving Elementary                         Lorain               -0.03

  Van Cleve McGuffey Elementary             Dayton               -0.05

  Woodland Hills Elementary                 Cleveland            -0.10

  Paul Laurence Dunbar Elementary           Cleveland            -0.22

  Giddings Elementary                       Cleveland            -0.61

  Crooksville K-8 Elementary                Crooksville          -0.66

  North Elementary                          Youngstown           -0.67

  Larkmoor Elementary                       Lorain               -0.88

  Coolville Elementary                      Federal Hocking      -0.88

  Cornell Heights Elementary                Dayton               -1.00

  Longfellow Elementary                     Lorain               -1.52

  Harvey Rice Elementary/Harvey Rice@Jesse  Cleveland            -1.59
  Owens                                                          

  McNary Elementary                         Dayton               -1.59

  Louis Pasteur Elementary                  Cleveland            -1.62

  South Avondale Elementary                 Cincinnati           -2.06
  ----------------------------------------- -------------------- --------

Table II-14 presents the results of the analysis of the relationship
between year 3 implementation fidelity and the various contextual
factors that we are able to explore at this point. In the analytic
model, we specified year 3 implementation as a function of year 1
implementation (baseline), school overall stability, school size, and
percentage of students who are disadvantaged. In the table, the
relationship between year 3 implementation and unexplained variables is
captured by intercept. We found that year 1 implementation is positively
associated with year 3 implementation, but the relationship is not
statistically significant. In addition, none of the other contextual
factors such as overall stability, enrollment, and percentage of
students who are disadvantaged are significantly associated with year 3
implementation.

Table II-14.---Relationships between year 3 implementation and
contextual factors

+-----------------+---------+---------+---------+---------+---------+
| Variable        | Unstand | Std.    | Stand   | t       | Sig.    |
|                 | ardized | error   | ardized |         |         |
|                 |         |         | coeff   |         |         |
|                 | coef    |         | icients |         |         |
|                 | ficient |         |         |         |         |
+-----------------+---------+---------+---------+---------+---------+
| Intercept       | .62     | .13     |         | 4.73    | .00     |
+-----------------+---------+---------+---------+---------+---------+
| Y1              | .21     | .19     | .21     | 1.09    | .29     |
| implementation  |         |         |         |         |         |
+-----------------+---------+---------+---------+---------+---------+
| Stability       | -.02    | .01     | -.30    | -1.64   | .11     |
+-----------------+---------+---------+---------+---------+---------+
| Enrollment      | .00     | .00     | .20     | 1.10    | .28     |
+-----------------+---------+---------+---------+---------+---------+
| Percent FRL     | .00     | .00     | -.13    | -.67    | .51     |
+-----------------+---------+---------+---------+---------+---------+

FRL = free and reduced-price lunch.

We substituted the overall stability variable with different
specifications including teacher stability, literacy specialist
stability, and principal stability. Table II-15 summarizes the results
of these additional analyses. When stability is defined by teacher
stability (Model 1), we found that year 1 implementation and teacher
stability are both statistically and positively associated with year 3
implementation. When stability is defined by literacy specialist
stability (Model 3), we found that year 1 implementation is
statistically and positively associated with year 3 implementation, but
the stability of literacy specialists is negatively related to year 3
implementation. Across three models, variations among school enrollment
and percentage of students who are disadvantaged are not related with
year 3 implementation.

Table II-15.---Relationships between year 3 implementation and
contextual factors with alternative specifications of stability

  -------------------------- ------------------- --------------------- --------------------
  Variable                   Model 1:            Model 2:              Model 3:
                             Stability=teacher   Stability=principal   Stability=literacy
                             stability           stability             specialist stability

  Intercept                  .60\*\*             .59\*\*               .56\*\*

  Y1 implementation          .25\*               .24                   .31\*\*

  Stability                  .01\*               -.00                  -.01\*\*

  Enrollment                 .00                 .00                   .00

  Percent FRL                .00                 .00                   .00
  -------------------------- ------------------- --------------------- --------------------

FRL = free and reduced-price lunch.

\*Statistically significant at .10 level; \*\* significant at .05 level.

We replicated these analyses using the ELLCO as our predictor of
implementation level. The pattern of findings was the same. That is, the
stability measures do not have a strong relationship to implementation
level.

These initial findings raise more questions than they provide answers.
While the general lack of relationship between the stability measures
and implementation suggest that stability taken by itself is not a
critical determiner of implementation, it would be unwise to reject
stability as a variable to consider. And, it is likely that other
factors, in conjunction with stability, such as strategies in place for
integrating new teachers into the program or the competence level of the
staff member make a difference in how change affects the level of
implementation of the program.

**Chapter 7. The Value Added to Teachers' Instructional Practices**

In this chapter we discuss what we have learned about the value added to
teacher instructional practices if they are in a RFO school versus if
they are not. To do so, we consider the following factors:

-   **The Reading First program setting.** The district setting and
    changes experienced by the Reading First districts and schools.

-   **Instructional and organizational support for teachers.** The major
    and current sources of support for teachers: professional
    development, literacy specialists, resources, and administrative
    support from principals. Reading First and non-Reading First schools
    are compared with respect to the sources of support; Reading First
    cohort 1 schools are compared to Reading First high-performing
    schools.

-   **Instructional change.** The implementation of the Reading First
    instructional model, synthesizing interview responses and
    observation data. Reading First and non-Reading First schools, and
    Reading First cohort 1 schools and Reading First high-performing
    schools are compared.

**The Reading First Program Setting**

The Reading First-Ohio case study schools and districts, with the
exception of those in Vinton County, have experienced major changes
since 2003--04, when they first participated in the program. The
changes, which are largely interrelated, include funding cuts, moves and
mergers, teacher turnover, administrative changes, turnover of Reading
First personnel (literacy specialists and district coordinators), and
reduced funding. The district and school contexts for the program are
important for understanding its implementation and impact. This chapter
briefly describes the changes in resources in the Reading First schools.

Cleveland

Five Cleveland schools participated in the year 4 case study. Just as
the Reading First year 1 ended (2003--04), over 1,000 teachers in the
Cleveland Municipal School District were laid off or transferred because
of a large budget shortfall. In addition, many support services were
cut. As a result, during year 2 (2004--05), class sizes in the Reading
First schools became larger, some schools experienced high teacher
turnover, and few, if any, services were offered to low-achieving
students outside of the regular classroom setting.

During year 3 (2005--06), the elementary schools were restructured,
becoming K--8 rather than K--5 or K--6 schools. Last year's case study
respondents said the change had a significant impact on the culture of
the schools. In some schools, class sizes---already large because of the
large-scale teacher layoffs just before the 2004--05 school
year---increased. Student disciplinary issues were more prominent in the
newly configured schools and occupied more of the principals' time than
in past years. Several respondents felt that principals had to assume a
less active role in supporting and monitoring the implementation of the
Reading First program model.

A major change in the Reading First program in year 4 (2006--07) was the
district's decision to have Reading First literacy specialists divide
their time between two schools, spending 2 or 3 days a week in each. Of
five case study schools, literacy specialists in four served two
schools.

Springfield

During 2004--05, after participating in RFO for 1 year, the Springfield
school district was in a state of fiscal emergency, and the state
financial oversight commission took over district finances. A large
number of teachers were laid off or transferred, and many nonteaching
staff were cut. District and school personnel who were interviewed said
they had fewer resources than before.

The financial pressure continued in 2006--07. The district also
experienced high turnover of key district personnel. Each year, the
district hired a new superintendent and replaced the Reading First
district coordinator. The district implemented short-cycle assessments,
which are administered ever quarter in every subjects, in 2006--07.

Vinton

Vinton County implemented Reading First in all of its elementary
schools, setting up a uniform literacy program for K--3 students that
generated high levels of district support. Vinton's Reading First
program has benefited from consistent leadership at the district and
school levels. Unusual for the districts participating in the case
study, the district and schools have retained the superintendent,
principals, literacy specialists, and RF district coordinator for the
entire 4 years.

From the start, the district relied less on Reading First funds than
other funding streams, such as Title I. The Reading First schools had
the same level of support---combining reduced state funding with
increased local funding---in 2006--07 as prior years.

Dayton

One Dayton Reading First school was added to the case study component
this year as a high-performing school. At the time of the site visits,
the city was to vote on a bond issue that would help the district close
a deficit. (It did not pass.) The Reading First case study school has
had a recent influx of new teachers, including one special education
teacher, according to the principal. The district has had three Reading
First district coordinators, and the case study school has had two
principals during the program tenure. The literacy specialist, however,
has been with the school for all 4 years. Reduced Reading First funding
in year 4 resulted in the loss of paraprofessionals who provided
after-school tutoring in the school.

Youngstown

In 2003--04, two district schools participated in the case study. One of
the two schools closed at the end of the 2005--06 school year, and its
students and teachers were merged into another Reading First school. The
latter school was selected to participate in the Reading First study as
a high-performing school. The former, also a high-performing school
although one that had been in the case study since 2003--04, moved to
temporary quarters while its new building was under construction. The
temporary building was miles from the original school neighborhood, and
as a result, 100 students transferred to different schools.

Budget concerns were evident in the district in 2004--05, when the
district cut staff positions, though these cuts were not as severe as
anticipated. At the end of the 2006--07 school year, the district faced
a \$16 million budget shortfall and was taken over by the state
financial oversight commission. The district coordinator and
administrative office were concerned about implications and uncertain
that their Reading First sustainability plan would be funded. In year 4,
due to reduced funding, the Reading First schools no longer had the
services of intervention specialists: teachers who were hired using
Reading First funds were tasked to work with low-performing students.

Summary

Table II-16 illustrates the major changes experienced by the Reading
First schools in the case study. Most schools have experienced multiple
changes. Almost all of the schools are in districts that have had a
budget crisis in the last 4 years. Reading First staff changes have been
fairly low. Only 3 of the 12 Reading First schools do not have their
original literacy specialist. Three schools are in districts where there
has been high turnover in the district coordinator position.

**Table II-16.**---Summary of changes experienced by Reading First case
study schools

  ------------- ------------- ------------ ------------- -------- ----------- -------------- --------
  District      School        Number of    Number of     School   School      Number of      Budget
                              principals   literacy      move or  structure   district       crisis
                                           specialists   merger   change      coordinators   

  Cleveland     Clark         1            2             No       Yes         1              Yes

                Giddings      2            2             No       Yes                        

                Woodland      2            1             No       Yes                        
                Hills                                                                        

                Fullerton     1            1             No       Yes                        

                Capt Roth     1            1             No       Yes                        

  Springfield   Kenwood       1            1             Yes      No          4              Yes

                Lagonda       1            1             No       No          4              Yes

  Vinton        Allensville   1            1             No       No          1              No

                Wilton        1            1             No       No          1              No

  Dayton        Cornell Hts   2            1             Yes      No          3              Yes

  Springfield   North         1            3             Yes      No          1              Yes

                Williamson    2            1             Yes      Yes         1              Yes
  ------------- ------------- ------------ ------------- -------- ----------- -------------- --------

Instructional and Organizational Support for Teachers

Reading First schools have several sources of instructional and
organizational support for teachers, including consistent, extensive
professional development for teachers; the support of literacy
specialists; support from building principals; and resources,
particularly those that assist them in meeting the needs of struggling
readers.

Table II-17 summarizes respondent perceptions on the degree of impact
the Reading First initiative has had in their school, and the area where
that impact was most evident. Per school, the assessment of impact by
teachers, literacy specialists, and principals are noted in the left
column. For example, in school A, teachers' opinions were mixed on the
impact of Reading First, as was the principal's opinion. The literacy
specialist said impact was low. The right three columns indicate the
area of impact respondents identified. Again, for school A, no
respondents said the impact was in student learning, while responses
most frequently cited impact with respect to resources and support.

Table II-17.---Assessment of success of RFO in the schools, and
assessment of area and degree of impact

  ----------------- ------------ ------------ ----------- ----------- ------------- ----------
  School            Assessment                            Area of                   
                    of impact                             impact                    

                    Teachers     Literacy     Principal   Resources   Instruction   Student\
                                 specialist               and support               learning

  Cohort 1 schools                                                                  

  A                 Mixed        Low          Mixed       √√√         √√            ---

  B                 High         Mixed        Mixed       √√√         √             √

  C                 Mixed        Mixed        High        √           √             √

  D                 Mixed        Mixed        Low         √           √             √

  E                 Mixed        Mixed        Mixed       √√          ---           √

  F                 High         High         High        √√          √             √√√

                                                                                    

  High-performing                                                                   
  schools                                                                           

  G                 High         High         High        √           √             √√√

  H                 Mixed        High         High        √√√         √√√           √√√

  I                 Mixed        High         High        √√          √√√           √√√

  J                 High         High         High        √√          √√√           √√√

  K                 Mixed        High         High        √√√         √√            √√

  L                 High         High         Mixed       ---         √√√           √√
  ----------------- ------------ ------------ ----------- ----------- ------------- ----------

--- No impact identified.

√ = mentioned by a few respondents.

√√ = mentioned by multiple respondents but not the majority.

√√√ = mentioned by a majority of respondents.

Reading First impact, from the teachers' point of view, does not seem to
differ noticeably by performance level. However, the ratings of
principals and literacy specialists do vary by performance level, with
those in high-performing schools generally more positive. Further,
respondents from high-performing schools generally cited student
learning as an area of impact. That was not the case among cohort 1
schools, in which respondents from only one school stated that student
learning had improved as a result of Reading First.

In the majority of cohort 1 schools, responses did not emphasize student
learning as a major area of impact due to participation in Reading
First. When student learning was mentioned, respondents mostly referred
to specific instances, such as improved reading by kindergarten students
or better reading results due to emphasis on phonemic awareness and
phonics. In most of the cohort 1 schools, respondents expressed mixed
opinions about the success of Reading First in their school. Usually,
teachers did not agree about the degree of success, and in very few
cases did literacy specialists and principals rate the initiative as a
clear success. When identifying how Reading First was successful, the
weight of responses was on the resources and support the school had
received: materials, services of a literacy specialist, and, to a lesser
extent, professional development.

In the high-performing schools, more respondents considered Reading
First to be a success, and far more responses cited student learning and
improved instruction as major areas of impact. In addition, the majority
of respondents in all of the high-performing schools said Reading First
had a positive impact on instruction.

The remainder of this chapter will describe the influence of the
following factors on instruction in the Reading First schools and
comparison schools: teacher professional development, support from the
literacy specialist (or coach in the comparison schools), administrative
support, and instructional resources, particularly those that target
struggling readers.

Teacher Professional Development

Teacher professional development is a key component of the Reading First
program. Many teachers in Reading First schools have received 3 hours of
professional development each month while working in a Reading First
school (for a number of them, for 4 years). Teachers were exposed to the
same professional development content. This section discusses the
influence of professional development on teacher practice in the Reading
First schools and, briefly, the comparison schools.

Professional development requirements for teachers in Reading First
schools remained high. Teachers were to attend approximately 3 hours of
professional development sessions each month for 4 years, and literacy
specialists participated in sessions facilitated by field faculty twice
a month. Moreover, RF teachers in an individual school have received the
same professional development each month, attending sessions that cover
the same topics and that is delivered by the literacy specialist, who
has shared the state curriculum learned over the course of the RF
program.

In looking back at teacher professional development, interview
respondents said it influenced their own instruction and that of their
colleagues in several ways:

-   Improving consistency in teaching literacy;

-   Providing up-to-date, research-based information on literacy and the
    five components;

-   Helping them develop insights into their students' needs;

-   Giving them ideas and strategies to apply in the classroom; and

-   Improving collaboration among teachers.

Overview of Reading First Professional Development

The RFO professional development is a centrally driven curriculum
designed by the Reading First-Ohio Centers, a consortium of colleges and
universities under contract with the Ohio Department of Education. In
the first year, the curriculum focused on explaining the literacy
framework and providing specific information on the five essential
components of literacy (phonemic awareness, phonics, fluency,
vocabulary, and comprehension). For Reading First schools in the first
cohort, the first-year curriculum was being developed during the first
implementation year. Overall, teachers gave this professional
development mixed reviews, due in part to implementation issues (the
fast pace of implementation created delays and problems), and to what
some teachers described as "too much theory" and/or information being
"too basic."

By the second year, although professional development topics still were
driven by the central curriculum, content was modified to address school
and teacher needs, and sessions included more instructional strategies
for teachers to apply in their classrooms. Schedules and settings for
professional development also changed in the second year. In most
schools, professional development sessions were delivered in small group
sessions, usually grade-level teams, rather than to all K--3 teachers.
The 180 minutes of professional development were typically scheduled
over several sessions. Teachers were more favorable about professional
development in the second year than the first, saying it had a definite
impact on their knowledge and understanding of the literacy.

In 2005--06, the literacy specialists, rather than the Centers, selected
teacher professional development topics. Topics were selected based on
teacher preferences and needs; thus, they varied from school to school.
In one school, for example, the emphasis was on comprehension,
reciprocal teaching, and peer coaching, while in another school the
focus was on including students and struggling readers, setting up
learning centers, and using literature that was appealing to boys.

In 2006--07, professional development for teachers focused on learning
and implementing the Instructional Leadership Team (ILT) model, a
seven-step action research approach developed by Pearson Achievement
Solutions. The ILT model is designed to facilitate collaboration among
teachers as they work together to address specific learning objectives.
Compared to earlier years, use of the ILT model required a more active
teacher role. As several literacy specialists said, use of this model
makes their role more facilitative than directive.

Assessment of Year 4 Professional Development

In general, teachers at Reading First schools said that the fourth year
professional development was useful. As Table II-18 shows, teachers in 5
of the 12 Reading First schools had favorable opinions about the
usefulness of professional development. These teachers were assigned to
four of the six high-performing schools and one of the schools in the
cohort 1 group.

Table II-18.---The usefulness of year 4 professional development, by
school

  ------------------------ -------- --- --- --- --- --- ----------------- --- --- --- --- ---
  Usefulness               Cohort 1                     High-performing                   

                           A        B   C   D   E   F   G                 H   I   J   K   L

  High agreement that year                          √                     √   √   √       √
  4 PD was useful                                                                         

  Mixed opinions on        √        √   √   √   √       √                             √   
  usefulness of year 4 PD                                                                 
  ------------------------ -------- --- --- --- --- --- ----------------- --- --- --- --- ---

Teachers offered several reasons for their opinion that the ILT model
was useful. The most common response referred to the opportunities to
collaborate with other teachers during the grade-level professional
development sessions. Another teacher elaborated, "You're sharing with
colleagues and you have a chance to look across grade levels, look at
your grade level, plan together, you know, get ideas and suggestions to
each other." Another respondent said, "I've changed some of my ways
because it works better the way I've seen done by a colleague."

Another benefit cited by teachers was that the sessions addressed topics
that they as a group selected and wanted to address. Topics varied from
school to school and included fluency, writing, reading comprehension,
differentiated instruction, and creating learning centers. Teachers said
the sessions provided them opportunities to learn new strategies related
to topics they had selected. Reflecting an opinion shared by other
teachers, one teacher said, "Working with other teachers, especially on
my grade level, gives me many ideas and strategies I can use." In some
schools, teachers said the literacy specialist was helpful in bringing
instructional strategies that focused on the professional development
topic. One teacher said, "Our literacy specialists listen to us and what
our kids need, and then they will bring strategies to us. . . we modify
them to fit what we think would work best with our students."

Several concerns about the ILT model were expressed (more strongly in
the mixed opinion schools). Several comments indicated that those making
the remarks found only some of the topics of the ILT meetings were
useful. A number of teachers did not like the model: some said it was a
burden to write out lesson plans, and one teacher said the meetings
"felt rushed" and took up her planning time. Also, in several schools,
teachers said ILT meetings were too frequent and the process repetitive.
They said the meetings worked well early in the year but should become
less frequent later on.

Influence of Four Years of Reading First Professional Development

Teachers in the Reading First case study schools were asked to consider
the extent to which professional development had influenced their
instruction over the course of the grant. In 7 of the 12 schools,
respondents had favorable opinions about the usefulness of RF
professional development over the course of the grant (Table II-19).
Those responses did not seem to be associated with performance
condition.

Table II-19.---The influence of professional development on instruction,
by school

  ----------------------- --------- --- --- --- --- --- ----------------- --- --- --- --- ---
  Influence               Cohort 1                      High-performing                   
                          schools                       schools                           

                          A         B   C   D   E   F   G                 H   I   J   K   L

  High agreement that RF            √       √       √   √                     √   √       √
  PD was useful                                                                           

  Mixed opinions on       √             √       √                         √           √   
  usefulness of RF PD                                                                     
  ----------------------- --------- --- --- --- --- --- ----------------- --- --- --- --- ---

Many teachers reported positive perceptions about the professional
development over the course of the grant, including its ability to
provide the following:

-   **Information on instructional and literacy-specific topics.** The
    most common responses regarding topics the teachers found useful
    include using flexible grouping strategies for small group
    placement, using learning centers in their classrooms, becoming
    familiar with the RF curriculum and the five essential components,
    looking at data and learning how to use it to effectively guide
    their instruction, and learning how to best support struggling
    students.

-   **New strategies and ideas.** Teachers reported enjoying receiving
    information and materials that would help them approach reading
    topics with their students. As one respondent pointed out, "We've
    attended some professional development that has been able to give us
    some really good teaching ideas, some new lessons or just some
    creative ideas of hands-on \[activities\]...That's kind of gotten me
    a little more motivated to try something new."

-   **Consistency.** Several teachers commented on the structure of the
    professional development over the course of the 4-year grant. The
    sessions were seen to be useful in keeping everyone addressing the
    same topics and focusing on the same instructional goals. As one
    teacher reported, "It's been good because it brought everyone
    together to one... We're all trying to achieve the same goal and
    that's good. It gives everybody a focus." On a similar note, some
    respondents also noted that the professional development sessions
    helped them stay up to date on the "latest information" and research
    regarding effective literacy instruction.

Suggestions for Improvements

Several literacy specialists and principals offered suggestions for
improving the teacher professional development model to make it more
compatible with school schedules and more focused on individual teacher
needs.

-   **Schedules.** Scheduling professional development was a problem for
    a number of schools. In some schools, professional development
    schedules changed from month to month. One literacy specialist said
    schools needed "a definite set aside time for \[teachers\] to have
    professional development; that would be the most valuable piece of
    improvement that they could make so that they have time." On a
    similar note, some literacy specialists and principals pointed out
    scheduling professional development during the school day requires
    funds for substitute teachers and pulls teachers out of the
    classroom too frequently. In one district where there were problems
    hiring qualified substitute teachers, nearly every teacher said it
    was difficult to get classes back on track when they returned from
    professional development sessions. A number of respondents suggested
    offering professional development on early dismissal days, similar
    to what one district in the case study does.

-   **New teachers.** The number of new teachers in schools and
    districts had a strong influence on teacher professional development
    over the course of the grant. Teacher professional development was
    based on a multiyear model, in which teachers acquired knowledge
    each year and built on previous years' experiences as the program
    progressed. When new teachers without any experience or background
    in Reading First came to RF schools, it was a difficult and
    sometimes overwhelming process to catch them up on the program and
    strategies that other teachers had been using for several years.
    Especially in the latter years of the program, some teachers had 3
    years of experience, whereas others had no prior experience with
    Reading First. Since professional development topics were not
    necessarily differentiated, this discrepancy impacted how
    effectively professional development could be provided to all
    teachers in a school or district.

-   **Differentiating professional development.** Several respondents
    thought professional development for teachers should be
    differentiated to address differences in teachers' experience and
    skills. Although that was done in some schools in year 4, "getting
    input \[about topics\] from the people you're in-servicing" would
    have been helpful from the start, and would have allowed the
    teachers who felt the professional development was repetitive and
    useless to find more value in the sessions. A few respondents also
    suggested providing new teachers with separate professional
    development sessions to get them up to speed and to avoid having
    more experienced RF teachers sit through sessions they may find
    redundant.

-   **The ILT model.** As noted earlier, a number of teachers thought
    the ILT model was demanding and repetitive. One of the literacy
    specialists said teachers liked the ILT meetings when we started.
    Teachers met in grade levels twice a month. "Now," the literacy
    specialist said, "teachers are a little tired of coming twice a
    month." One of the literacy specialists said the ILT model should
    have been introduced earlier. She said several teachers reported
    enjoying working with other teachers during the learning team
    sessions and suggested that using this approach earlier might have
    helped the literacy specialists more effectively provide
    professional development. (She may have been referring to teacher
    resistance to professional development in the first RF year.)

-   **The Teachscape videos.** In several schools, a number of teachers
    expressed mixed opinions about the Teachscape videos. Some of the
    videos were acknowledged to be useful, but teachers did not think
    that videos that represented unrealistic situations, that were
    lectures, and/or that did not depict a classroom setting were
    especially worthwhile.

Professional Development in Comparison Schools

Respondents in the six comparison schools (in three of the Reading First
districts) also were asked about teacher professional development to
determine if there were similarities between their professional
development and that for teachers in Reading First schools.

Teachers in the six comparison schools generally reported their
professional development this year to have had only a minor impact on
their literacy instruction. In only one of the six comparison schools
did the majority of teachers report that the professional development
had a major impact on their instruction this year. The eight teachers
who were interviewed at the school also had an average of over 18 years'
teaching experience. At this particular school, teachers reported
attending State Institute for Reading Instruction (SIRI) sessions, which
they found helpful and influential in their literacy instruction.

The professional development received by teachers at comparison schools
reportedly included sessions on the five essential components, decodable
books, the use of assessments and interpreting data, differentiated
instruction, and how to use the curriculum materials (e.g., Trophies
series). The vast majority of professional development sessions attended
by teachers were provided or sponsored by the district; very few
teachers remarked on sessions provided within their buildings, usually
due to a lack of funding and/or a literacy coach to provide such
school-based training.

The Role of the Literacy Specialist

This section examines the role of the literacy specialist and the
effectiveness of the literacy specialist in the school. In the fourth
year, most of the literacy specialists stated they were increasingly
comfortable in their position and felt more effective in helping
teachers. They noted that their role had changed somewhat, as veteran
Reading First teachers required less support. Nearly all literacy
specialists said their workload had increased in the past year due to
district reassignment decisions (for four literacy specialists) and
increased administrative work. A number of literacy specialists
prioritized their time, mainly working with new teachers and a few
teachers who needed more support.

Overwhelmingly, teachers and principals perceived the literacy
specialists to be an important component of Reading First and described
the many responsibilities the literacy specialists fulfilled, including
coaching and mentoring teachers, providing professional development,
keeping the literacy program going, and providing support for teachers
in many informal ways.

In the fourth year, teachers perceived the literacy specialist to be
less effective than in earlier years, but reasons for this assessment
varied. In several schools, it was due to the literacy specialists being
assigned by the district to cover two schools rather than one; in other
schools, it was because teachers were familiar with the literacy model
and needed less support.

Overview of the Literacy Specialist Role

As the literacy curriculum has changed, so has the literacy specialist
role in the school. In the first year, the literacy specialists learned
and delivered a scripted professional development curriculum on a set
schedule, focusing on the five components and other essential literacy
concepts. They delivered the curriculum to all K--3 teachers, usually in
a single 3-hour session. Literacy specialists also helped teachers
understand and fulfill the Reading First requirement, including formal
observations of teachers by the literacy specialist (with follow-up),
administration of DIBELS on a schedule, adherence to the core reading
series and a commitment to using materials based on scientifically based
reading research, and completion of field work assignments that
accompanied their professional development sessions.

After the first year, the Reading First curriculum became more flexible
and was altered so that literacy specialists could adapt it to meet the
needs of their school. It was in the second year that literacy
specialists became more comfortable in their role. Professional
development began to be delivered in smaller groups, usually grade-level
teams. Professional development focused on instructional strategies that
teachers were asked to try out in their classrooms. Literacy specialists
then observed how teachers were implementing the strategies and offered
support as needed. The focus on strategies helped address a problem
cited by many of the literacy specialists in year 1: their uncertainty
about how to work with individual teachers. By the end of year 3,
literacy specialists described being increasingly comfortable observing
teachers and helping them learn or refine instructional strategies. Most
literacy specialists were in the classroom, often informally observing.
They provided high levels of support to teachers, when asked, and
several literacy specialists were direct in encouraging and facilitating
instructional change.

The Literacy Specialist in Year 4

More than previous years, the literacy specialists expressed ease and
purposefulness in working with teachers individually: observing them in
the classroom, getting them materials, observing teachers using formal
protocols, and modeling instructional strategies. Literacy specialists
indicated that they were benefiting from the relationships they had
established over the previous years and referred to "wonderful working
relationships," "trust," and "forming a bond with teachers."

With respect to the year 4 professional development, the literacy
specialists had a more facilitative role than in the past. The literacy
specialist taught and guided the ILT model, but because the model
encouraged teacher collaboration, the literacy specialist's role in the
meetings was not so focused on providing information as in the past. One
of the literacy specialists described herself as "a capable other"
rather than a "presenter." Another literacy specialist described her
role in professional development sessions as "asking the questions," not
giving information, adding that "teachers were the ones who did the
work."

In several schools, teachers and literacy specialists noted that the
teachers, particularly those who had been participating in Reading First
for several years, required less of the literacy specialist. A literacy
specialist described teachers as "less dependent" and showing "growth
and independence." "They've gotten the routine of Reading First," she
said.

In most schools, literacy specialists said they spent more time working
with new teachers (if there were new teachers) and a few teachers who
needed support and strategies, especially in working with lower level
students. Literacy specialists stated that it was increasingly difficult
to orient new teachers to the reading program because they did not have
the professional development offered to Reading First teachers in
earlier years.

Nearly all literacy specialists said they spent more time on
administrative duties than in the past, and several literacy specialists
said they had too little time to support teachers. The challenge was
greatest for literacy specialists from one district who were now serving
two schools rather than one. As they said, not only were they
responsible for more teachers, but they were also responsible for
administrative duties in each school. Most literacy specialists assumed
responsibilities that were once assumed by data managers and resource
coordinators, including filling out the Program Monitoring Tool for the
state three times a year, scheduling and helping with student
assessments, and making sure materials are distributed to teachers.

Literacy Specialist Effectiveness

Overall, teachers and principals indicated that the literacy specialist
has been important for implementing the Reading First model in the
school. Table II-20 conveys the responses of teachers to close-response
questions on how important the literacy specialist was to their school.

Table II-20.---Teacher perceptions of the importance of the literacy
specialist for implementing Reading First, by school

  ----------------------- --------- --- --- --- --- --- ----------------- --- --- --- --- ---
  Perception              Cohort 1                      High-performing                   
                          schools                       schools                           

                          A         B   C   D   E   F   G                 H   I   J   K   L

  High agreement that     √                 √   √   √                         √   √   √   √
  literacy specialist is                                                                  
  important                                                                               

  Mixed opinions that               √   √               √                 √               
  literacy specialist is                                                                  
  important                                                                               
  ----------------------- --------- --- --- --- --- --- ----------------- --- --- --- --- ---

As the table suggests, in most schools in both groups of schools, most
teachers perceived the literacy specialist to be important. All four
schools in which teacher opinions were mixed had a literacy specialist
only half time in year 4. In most of the schools, the respondents said
they would like to continue to have a literacy specialist once funding
ends. As a teacher explained:

> *It would be hard to lose her and keep the program going, because you
> can come to her with DIBELS problems, questions, concerns. "How do we
> do this? What affect does this have?" Like I said, the resources that
> she's loaned me, or shown me, have been really great. That saved me a
> lot of time going out, and trying to do a lot of research.*

Principals and teachers were asked about the literacy specialist's role
in the school and what she has done to support literacy in the school.
The major areas of responsibility include the following:

-   **Directing the literacy program.** A major role of the literacy
    specialist, mentioned by teachers and principals in several schools,
    is keeping the literacy program running. A teacher described the
    role as very important, saying that the literacy specialist "keeps
    the schedule going, keeps the school going." Two principals said
    that the literacy specialists have gradually taken on more
    responsibility for monitoring lessons and addressing instructional
    problems quickly. One principal said, "I think it is an essential
    piece of the whole grant. As an administrator I couldn't keep that
    motivation going. They were trained, so they had the plan. They
    helped me keep a pulse on what is going on."

-   **Keeping teachers current on instructional practices.** Literacy
    specialists were perceived by many teachers to bring knowledge and
    information into the school, and to disseminate this information
    through professional development sessions, as well as sharing
    materials. As one teacher said:

> *I couldn't think of any way to get them* \[*my students*\] *to
> improve in fluency so* \[*the literacy specialist*\] *did some
> research and looked into the things that she has and she gave me
> materials that I can work with to help them. Any questions that I may
> have concerning the program, she's always willing to help.*

-   **Providing help with instruction.** Literacy specialists helped
    teachers in their classroom with specific problems and provided some
    tangible, material support. One of the respondents said, "They're
    not only coaches, they support you and help you out if you have
    trouble connecting to a student or you can't reach him, then he has
    other ways and you work together." Another teacher described the
    literacy specialist as "a second set of eyes that often may see
    things that I might miss." One teacher pointed out the value of
    getting feedback and additional options on instructional strategies,
    saying, "If I try a strategy, and maybe it doesn't work, or I'm not
    sure if I'm doing it correctly, just going to her, and having
    meetings with her, and her maybe re-explaining something, or talking
    me through something."

-   **Providing materials.** Teachers said literacy specialists were
    helpful because they identified materials to meet special
    instructional needs. "She's also been able to give me materials at
    the reading level, to help those kids who are highly at risk," a
    teacher said. Another teacher said, "Anything you want, she'll get
    it for you. She'll hunt it down."

-   **Logistic support.** Teachers in all schools said the literacy
    specialist provides logistic support, i.e., making copies, keeping
    track of assessment schedules, helping with assessments, making sure
    materials are distributed, communicating policies and mandates from
    the central office, and so forth.

All principals described the literacy specialist as very important to
the school. Some of the Reading First principals do not have a
background in literacy, and their other responsibilities in the school
prevent them from completely taking charge of the literacy program.
Principals stated that it is important to have someone whose sole
responsibility is implementation of literacy program.

Several dissenting opinions were expressed about the importance of the
literacy specialist. Two teachers said the literacy specialist was
needed in the first 2 years of the grant, but the role is not as
important now. Two other teachers said that because the literacy
specialist is only in the school part time, the role is less important
and influential. One teacher said that it would be better for the
literacy specialist to work directly with students rather than teachers.

Literacy Coaches in the Comparison Schools

Only one of the six comparison schools had a dedicated literacy coach on
staff. Another school had a lead teacher that assumed some of the role
of a literacy specialist, two schools had a curriculum coach who was
responsible for all subjects (not just literacy instruction), and two
schools reported not having a coach or literacy coach at all. At the
school with the literacy coach, two of the three teachers interviewed
perceived the coach to have had only a minor influence on literacy
instruction in the school, and the other teacher interviewed reported
that she had somewhat of an influence. The principal said the literacy
coach had a major impact on instruction.

Three of the five teachers interviewed at the school with the lead
teacher in literacy reported either no impact or a minor impact on
instruction (two of these teachers had never worked with her). One
teacher in her first year teaching an early elementary grade found her
to be a "wonderful resource" and reported her to have a major impact on
the teacher's literacy instruction, and the lead teacher herself
perceived that she had a major impact on instruction in the school this
year by helping teachers plan and acting as a mentor.

The two schools with general curriculum coaches also reported the
coaches had only a moderate level of influence on literacy instruction
in their schools. At one school, the coach herself said she had little
influence on instruction because of the teachers' resistance to change.
However, the principal stated that the coach had a major impact on
instruction. Teacher reports at this school were mixed, with two
teachers reporting that the coach had a strong influence on instruction,
two teachers reporting she had a minor influence on instruction, and one
teacher indicating that she knows the coach is there if she needs help
but did not meet with her this year. At the other school with a general
curriculum coach, two teachers said that she had a major influence on
literacy instruction this year, especially in terms of working with data
and providing ideas for strategies and using materials. The remaining
four teachers perceived a minor impact, although most of them
acknowledged having worked with the coach on data and getting materials
this year.

Administrative Support

In year 4, we examined the principal role primarily by looking at two
practices that are central to Reading First-Ohio: administrator use of
data to communicate expectations in the school and approaches to
monitoring, both of which keep the principal informed about instruction.

In several high-performing schools, teachers offered detailed accounts
on how the principal uses data and what he or she expects in terms of
student performance objectives. They also were aware of decisions the
principal makes based on data. In other schools (all cohort 1 schools),
there was little or no teacher awareness of the principal's expectations
or how the principal uses data. Similarly, monitoring of instruction
appeared to be greater in the high-performing schools than the cohort 1
schools.

Overview of the Principal Role

Early in the RFO initiative, principal support was mainly reflected by
principals' accommodating the Reading First program. In most of the RF
schools, responsibility for implementation was turned over to RF staff,
particularly the literacy specialist. Several literacy specialists said,
"The principal supports the program by letting me do my job." Many of
the principals were slow to become involved with the Reading First
initiative, offering such explanations as being reluctant to impose
demands on teachers when they were already overwhelmed, having to
address many other concerns in the high-poverty schools, and not fully
understanding their role in the initiative.

The role of the Reading First principals became more clearly defined as
they became more familiar with the program and as the state professional
development more effectively addressed their needs and concerns.
Compared to earlier years, by year 3, principals were more active in
supporting and communicating to literacy specialists and teachers about
the Reading First model and student results. On the other hand, support
varied across the schools: some of the Reading First principals
continued to let the literacy specialists do their job while they
themselves remained relatively uninvolved, and others more actively
moved the Reading First program forward through monitoring,
communicating, and planning.

Continued Variation in the Principal Role

Common to all schools was principal support in ensuring the Reading
First program is being implemented with fidelity through, for instance,
an uninterrupted literacy block, time set aside for professional
development, and availability of the literacy specialist. In most of the
schools, respondents indicated the principal made sure that teachers
have the resources they need such as instructional materials and, in
some cases, tutors. However, principal support differed among schools in
other ways, such as the principal's using data to communicate
expectations throughout the school and monitoring implementation through
classroom observations, lesson plan review, or other means.

**Principal use of data.** When asked how the principal uses data,
teachers provided a variety of responses (Table II-21). In several
schools, all or nearly all teachers offered many examples of principal
data use, suggesting fairly high levels of communication between the
principal and teachers. In other schools, teachers offered mixed
responses, while in a few schools, most teachers said they did not know.

Table II-21.---Teacher awareness of principal use of data, by school

  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----
  Awareness              Cohort 1                           High-performing                       
                         schools                            schools                               

                         A         B    C    D    E    F    G                 H    I    J    K    L

  High awareness among                       √         √                                √         √
  teachers.                                                                                       

  Mixed awareness among                           √         √                 √    √         √    
  teachers                                                                                        

  Little or no awareness √         √    √                                                         
  among teachers                                                                                  
  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----

In four schools, all or nearly all teachers said they were aware of how
the principal used data and the decisions made based on student data.
Two of the schools were high-performing schools and two were cohort 1
schools. All of the schools in which there was little or no awareness
among teachers regarding principal use of data were cohort 1 schools.

In schools where awareness of this practice was high, teachers said the
principal talks about data frequently and uses data to set school-wide
goals and make administrative decisions that affect teachers and
students. In one of the schools, nearly all teachers said the principal
meets with teachers as a group to review the results of student
assessment data, offering suggestions on how to improve instruction;
meets with students to discuss their test scores; and uses data to
communicate with parents. That principal was described as "very
involved," and "a data-driven person." In another school, teachers said
the principal reviews all the data in the school, working with teachers
to identify weak areas. The principal makes personnel assignments based
on data and works with the literacy specialist on modifying the literacy
program. Major administrative decisions are data based, according to
teachers in this school, including a decision to reduce class sizes. In
another school, the principal was new but has been active in discussing
data with teachers. "He talks about data all the time," the literacy
specialist said, as do the teachers, though with some complaints that
the principal "holds it over our heads," letting teachers know which
teachers are doing better than others. The principal used data to form
student groups, make placements, and evaluate teacher performance. In
another school, the principal presented data at teacher grade-level
meetings and weighs in on conversations about how to use the data to set
up classrooms, monitor student progress, and select additional reading
materials. The principal indicated that the key to school progress in
raising student reading levels is "they truly use data to drive
instruction."

In other schools, principal communication about expectations and data
was far less extensive. The principal reviewed data with the literacy
specialists but generally limited data-based communication with
teachers. For example, in one school, teacher responses indicated that
the principal communicates individually with certain teachers, but not
with all teachers. In another school, teachers said that principal
communication is limited to weekly postings. The principal makes
decisions about grouping and peer tutoring but does not communicate
expectations and goals.

In a number of schools, all or nearly all teachers said they had no idea
how the principal uses data or what decisions the principal makes based
on data. Principals and literacy specialists in those schools noted
principal appreciation and/or growing awareness of data. However, none
of the respondents (teachers, literacy specialists, or principals)
offered examples of principals' communicating with teachers or making
data-based decisions.

**Monitoring.** Another administrative practice that was examined was
principal monitoring of instruction. Teachers were asked a general
question about principal support for the Reading First program, and in
most schools, teachers described high levels of monitoring (Table
II-22). Because the question was not asked directly, the data are based
on multiple teachers stating that principals engaged in observations and
other monitoring practices or did not do so. As noted earlier,
respondents in all schools said the principal ensured that the literacy
block and Reading First professional development were scheduled and that
resources for teachers were available.

Table II-22.---Teacher awareness of principal monitoring instruction, by
school

  ------------------------ --------- --- --- --- --- --- ----------------- --- --- --- --- ---
  Awareness                Cohort 1                      High-performing                   
                           schools                       schools                           

                           A         B   C   D   E   F   G                 H   I   J   K   L

  High awareness among                       √       √   √                     √   √   √   √
  teachers                                                                                 

  Mixed/low awareness      √         √   √       √                         √               
  among teachers                                                                           
  ------------------------ --------- --- --- --- --- --- ----------------- --- --- --- --- ---

In the majority of the schools, respondents indicated that the principal
monitors instruction by conducting walk-throughs, reviewing lesson
plans, and, in some cases, reviewing student work. Monitoring by the
principals was more prevalent among high-performing schools than cohort
1 schools.

In schools where teachers were aware of principal monitoring routines,
the teachers indicated the principals were aware of instruction, student
performance, and the differences between individual classrooms. A
teacher stated that in the school, "We have to turn in our activities to
her so that she's sure that we are doing grouping. She also asks for our
list so she sees that the kids move up the groups or down." One teacher
said of the principal, "She likes us to have things visible and in the
room of students' work and to have if it's graded and up." In another
school, a teacher said the principal is "one of the few people that will
actually get in there and teach a skill for you." Finally, in one
school, the principal was described as being "everywhere . . . checking
lesson plans, doing classroom walk-throughs . . .assessing our data,
checking on the results of all the DIBELS, the data meetings."

Monitoring was not discussed by respondents in three schools and was
perceived as irregular in two schools, one of which had a new principal.
In most of the schools, where monitoring was noted as low or mixed,
teachers offered few examples of interactions with the principals. In
one school, a literacy specialist said visible principal support for
Reading First was lower in year 4 than in the past, and the principal
seldom visits classrooms. In another school, respondents indicated the
principal mainly leaves monitoring up to the literacy specialist. In one
of the schools, the principal was out of the building for part of the
year.

The Principal Role in Comparison Schools

In the six comparison schools, the principals' use of data varied. In
three schools, teachers described only administrative decisions made by
the principal but no regular communication and strategizing by
principals and teachers. In one school, however, the principal was in
regular contact with teachers to discuss achievement data and strategies
about ways to improve student learning. This school had a leveled
reading program, and formative assessments were a key part of it. In two
schools, administrators attended what respondents referred to as data
meetings with teachers, but a number of teachers were not aware of these
meetings. Comparison school respondents were not specifically asked
about principal monitoring; therefore, we do not have direct
information.

Resources in the Reading First Schools

The section examines the use of instructional materials and support from
personnel in the Reading First schools. Because the schools have
implemented Reading First with reduced funding, the section focuses on
the adequacy of resources for meeting the needs of all students,
particularly struggling readers.

Each Reading First school had a high level of materials received through
the program, including supplemental materials that target struggling
readers. The use of the materials varied in the school and appeared to
influence teacher opinions about whether the materials are adequate for
meeting the needs of struggling readers. In four cohort 1 schools,
teachers were not using all of the materials available. Typically, in
these schools, teachers had mixed opinions about the adequacy of
materials. On the other hand, in eight schools, including all six of the
high-performing schools, teachers indicated materials were adequate for
meeting the needs of struggling readers.

In many of the schools, teachers had less support from personnel for
meeting the needs of struggling readers than in earlier years. This was
due primarily to reduced Reading First funding, which prompted schools
to let go of paraprofessionals and intervention specialists who worked
with struggling readers. In year 4, four schools, three of which were
high-performing schools, maintained fairly high levels of personnel
support, sometimes by replacing staff support with support from tutors.
Other schools, however, had low levels of personnel support, and some
relied on inconsistent support from tutors.

Instructional Materials

Reading First schools in each district have the same core reading
program, and teachers have been required to use the program
consistently. The reading program has shaped day-to-day reading
instruction (see the Year 2 Case Study Report) and is viewed as critical
for aligning literacy instruction across grades. Across all of the case
study schools, having plentiful, varied, and research-based
instructional materials was seen as a benefit of their schools
participation in Reading First.

When asked about how well the materials met the needs of their students,
the responses of teachers within each school were generally positive
(Table II-23). In most of the schools, all or nearly all of the teachers
indicated that the materials met students needs "well" or "very well."
In four schools, teachers expressed mixed opinions on the adequacy of
materials, with more than one teacher indicating the materials only meet
the needs "fairly well" or "not at all."

**Table II-2**3.---Teacher perceptions of adequacy of instructional
materials for meeting student needs, by school

  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----
  Adequacy               Cohort 1                           High-performing                       
                         schools                            schools                               

                         A         B    C    D    E    F    G                 H    I    J    K    L

  High agreement among             √                   √    √                 √    √    √    √    √
  teachers on adequacy                                                                            
  of materials                                                                                    

  Mixed opinions among   √              √    √    √                                               
  teachers on adequacy                                                                            
  of materials                                                                                    
  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----

In the majority of schools, including all the high-performing schools,
respondents stated that they have adequate materials to individualize
instruction for all student groups, even struggling readers. This was
one of the sharper differences between the two school groups. In the
high-performing schools and two cohort 1 schools, materials were
perceived to be adequate and they were all used. Not all materials were
used in the other cohort 1 schools.

In schools where teachers said materials were adequate, teachers
primarily used the core reading series and occasionally pulled in other
materials to focus on specific student needs. Teachers in these schools
said they have materials for all students, and lack of materials schools
was not a problem. Teachers said they were very familiar with their
instructional materials and used them all. One teacher said, "We use the
materials as we were taught in the first years. It is difficult not to
succeed---there is enough stuff to choose from." In another school, a
teacher said, "We use them all," and another teacher said, "The
materials are used." With materials in her school, a teacher said, "You
really can reach everyone."

In each of the schools where teachers expressed mixed opinions about the
adequacy of materials, several respondents indicated the materials are
not adequate for meeting the needs of struggling readers. Many of the
materials are not used by teachers, according to respondents from the
schools. One of the literacy specialists said there are still materials
that are not used in her school: "Our teachers are not using some
materials yet, because they feel overwhelmed." In another school, a
teacher said, "We have to go outside the core to meet their \[higher and
lower students\] needs." In one school, a respondent said, "Many of the
supplemental materials have not been used at all. They are still sitting
in cellophane." The respondent attributed this situation to the district
decision to not bring in trainers from the publisher, saying, "I think
we received less than half of the support needed from the publishers for
the teachers to understand these materials and how to use them."

Instructional Materials in the Comparison Schools

All of the comparison schools now have access to the same core reading
series as used in the Reading First schools in their district. However,
in most of the comparison schools, there was little direction or
guidance for how teachers use the materials. In most of the schools,
teachers said they have a high level of discretion. In some schools,
that is consistent with the reading series approach (for example,
leveled readers) while in others, teachers complained about getting
materials and having little introduction or professional development on
how to use them.

Support From Personnel

Meeting the needs of struggling readers was challenging for some of the
Reading First schools in the fourth year. Because of reduced funding,
many of the resources teachers relied on, particularly extra personnel
support, were cut. Four of the schools, including three high-performing
schools, have maintained consistent support for teachers in working with
struggling readers. In several schools, teachers said they have little
or no support, usually relying on irregular volunteer tutors and/or
community-based programs, as well as the support they themselves
provide.

Table II-24 summarizes interview responses regarding the type of
personnel support teachers use to meet the needs of struggling readers.
Consistent tutoring programs use paid tutors or volunteer tutors in a
structured program where the school pays at least administrative costs
to the organization providing the tutors, and tutors are available to
the same students at least two times a week. Limited tutoring refers to
tutoring that is available only once a week or less, or is provided by
nonprogram tutors such as older students or volunteers. (For example, a
teacher said, "I have just a few volunteers that come in, but they're
not a regular approach.") External community-based programs provide
academic support services for students.

Table II-24.---Supports for struggling readers, by school

  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----
  Support                Cohort 1                           High-performing                       
                         schools                            schools                               

                         A         B    C    D    E    F    G                 H    I    J    K    L

  In-school staff with   √                   √    √         √                 √                   
  reading expertise                                                                               

  Regular instructional                                √                                √    √    √
  aides                                                                                           

  Consistent tutoring                                  √                      √         √         √

  Limited tutoring       √                   √    √                                √              

  Community-based                            √    √                                               
  programs                                                                                        
  ---------------------- --------- ---- ---- ---- ---- ---- ----------------- ---- ---- ---- ---- ----

Four of the schools offered two consistent sources of support for
teachers in working with struggling readers. Each of those schools had
acquired tutor services from an organization, and at least 40 percent of
the teachers had an instructional aide or access to an in-school staff
person with reading expertise. Three of the four schools are
high-performing schools. In these schools, teachers said they relied on
this support, especially because they themselves usually do not have
time in the literacy block to give struggling readers the instruction
they need. They appreciated extra support with struggling readers. A
teacher said, "For the most part, a classroom teacher cannot meet their
needs." Teachers said the following about the support from tutors,
aides, and other personnel:

> *This year we were lucky enough to have resource teachers. That low
> group works more intensively with the resource teacher.*
>
> *One of our biggest pluses was all of the tutors or intervention
> tutors that the principal was able to get to help us. I have the tutor
> that I only share with one other class. So she* \[the tutor\] *can
> pull every student every day who is struggling and work with them one
> on one.*
>
> *Every student that I have that is below level 1 send to a tutoring
> program that we have.*

In many of the schools, teachers, literacy specialists, and principals
said they have fewer and/or less available qualified personnel to help
with lower achieving students. In eight schools, additional personnel
designated to work with lower level students were cut due to reduced
Reading First funding. The personnel included intervention specialists,
paraprofessionals, and paid tutors. In two schools, tutors (in a
structured tutoring program) replaced the paid staff who were cut. In
one of the schools, the principal said the tutors did not have the same
skills as the former staff, who were highly trained intervention
specialists; in fact, one of the intervention specialists became the
school's literacy specialist.

Personnel Support for Struggling Readers in Comparison Schools

Comparison schools were similar to the Reading First schools in that
several had little or no regular resources for struggling readers, while
others had at least structured tutoring programs and some in-school
support from a remedial teacher. Three comparison schools appeared to
have low levels of support from peer tutors or volunteer college tutors.
However, three other schools had more support from tutors who work with
students several times a week (two schools) and a remedial teacher.

Conclusion

The conclusion briefly describes differences between the two Reading
First groups, i.e., the six cohort 1 schools and the six high-performing
schools, and differences between those Reading First schools and the six
comparison schools.

Cohort 1 and High-Performing Schools

When considering instructional supports, the respondents in cohort 1 and
high-performing schools differed with respect to the principal's use of
data and communication about expectations with teachers. The main
difference is that in three of the cohort 1 schools, communication
around data was minimal, with most teachers saying they had little or no
idea about how the principal uses data.

Another difference between cohort 1 and high-performing schools was the
use of instructional materials. In all of the high-performing schools
and only two of the cohort 1 schools, respondents indicated that
materials were adequate and were all used. In other schools, many in the
same districts with the same materials, not all instructional materials
were used. Finally, in high-performing schools, teachers had more
support from additional personnel to work with struggling readers.

While these differences are specific, they suggest that building
leadership communicates expectations about the literacy program to
teachers in a consistent way; as teachers in the high-performing schools
said, the principal talks about data, reviews data with teachers, and
attends meetings with teachers in which student progress is reviewed.
The principal is likely to be responsible for teachers getting support
from personnel such as tutors or instructional aides. Use of materials
may reflect clear expectations for meeting student needs and
implementing the core reading program. Other factors may be high levels
of support from literacy specialists who help teachers learn about and
use materials, as well as teacher willingness to explore newly available
materials without becoming overwhelmed.

Instructional Support in Non-Reading First School Literacy Programs

Comparison schools did not have the same level of support for teachers
as Reading First schools. They did not have a high level of consistent
professional development, and most comparison schools in this sample did
not have a literacy coach. Both the professional development and support
teachers have received from curriculum coaches and lead teachers have
had, according to respondents, a minor influence on their instruction.

In one school, the principal used data in a way similar to what some
teachers described in Reading First schools, i.e., tracking student
achievement data on a regular basis and conferring with teachers about
strategies for improving achievement scores.

Although all non-Reading First schools in the districts have the same
reading series as the Reading First schools, in most of the schools
there has been little or no professional development on how to use the
materials, and there are few guidelines about whether and to what extent
teachers have to follow the series. In terms of support, this is a major
difference between Reading First and comparison schools.

Literacy Instruction in the Reading First Schools

Analysis of the interview data did not show differences between the
high-performing Reading First schools and the cohort 1 schools.
Therefore, we report findings for the Reading First schools as a group
in this section.

When asked about how instruction had changed as a result of
participating in Reading First, principals overwhelmingly said it had
become more consistent and focused. Prior to Reading First, the
principals said there was no focus at all. Literacy instruction was
described as "haphazard," and instruction was "all over the board."
"Everyone was at a totally different place," a principal said. Another
change cited by principals was that prior to Reading First, literacy
instruction was mostly whole group and based on the basal text. There
was no data use. One principal said, "Before, you just followed the
basal. There was no understanding as to why." Reading First has resulted
in a more systematic literacy program throughout the schools, according
to principals and teachers. One principal said Reading First was "really
well laid out" for accomplishing instructional change and consistency.

The Five Components

Overwhelmingly, the Reading First teachers have indicated that they
incorporate the five components into their literacy instruction. With
only a few exceptions, Reading First teachers said the components were
addressed adequately in their reading program. When not addressed (in
one school, teachers said phonics was not emphasized enough), teachers
chose supplemental materials that focused on the skill.

Teachers said the focus on the components has been beneficial to their
students. "It makes them become better readers," a teacher said. "It is
exactly what students need," another teacher stated. This opinion was
expressed across all of the Reading First schools. Teachers indicated
that after several years in Reading First, they see how students in
grades 2 and three 3 benefit from the program. "The advantage is
evident," a teacher said. Another teacher said, "We are building skills
from one grade to another."

Responses indicate that it has been helpful to consider literacy as a
set of skills (the five components and other skills). As a teacher said,
a focus on literacy skills makes teaching literacy "very doable," even
for inexperienced teachers. A number of teachers said distinguishing one
skill from another helps them structure and organize their lessons, and
identify more specifically areas where students are struggling.

Teachers in several schools indicated that though the five components
are addressed adequately, writing is not. This concern had been
mentioned in the first year of Reading First, but most schools have not
added a writing component to the literacy block. Several schools have
added such a component, and two others plan to do so in the coming year.
In a few of the Reading First schools, the Instructional Leadership Team
meetings have identified "extended response" writing as a student skill
that needs to improve --- primarily because of low student performance
on the Ohio Achievement Tests.

Data Use

Overall, Reading First teachers were positive about the utility of the
assessments used. The assessments mentioned included DIBELS, TPRI, and
TerraNova, but comments regarding utility strongly focused on DIBELS,
which was used to monitor student progress. DIBELS was administered at
least quarterly in the Reading First schools (per the requirements of
the grant) and more frequently by teachers in some schools for students
who were not achieving proficiency. The teachers were consistent in
saying that using formative assessments had been beneficial. There were
more varied opinions regarding whether DIBELS was the best assessment to
use.

Overwhelmingly, teachers indicated that the use of a formative
assessment and readily available results had opened their eyes not only
to which students were struggling, but also to what skills they were not
mastering. One teacher said, "I'm not frustrated that they aren't
getting it \[mastering a skill\] because we are working together. I
realize I need to change the procedure instead of just moving on."
Another teacher described how DIBELS fits with her own assessment of
student skills.

> *It confirms what you thought and shows the weakness. . . sometimes
> there are surprises. Sometimes they \[students\] are higher than what
> you thought, usually not lower. Maybe they don't need as much in a
> particular area. You have a roomful of kids, so you're not always
> right.*

Teachers said they used DIBELS to track the progress of struggling
students, sometimes at 1- or 2-week intervals. They formed small groups
based on DIBELS data, and re-formed those groups with every DIBELS
administration. Several teachers also said that because DIBELS was used
throughout the K--3 grades, teachers could anchor their discussions
about student learning and instructional strategies on comparable data.
Within grades and across grade levels, DIBELS allowed teachers discuss
student progress and track individual students. Many teachers said when
students themselves tracked their progress using DIBELS, they became
more engaged in working on specific skills. Finally, the teachers said
DIBELS charts were helpful for communicating with parents about student
reading progress and needs.

DIBELS was questioned by teachers in more than half of the schools.
Third grade teachers, in particular, indicated DIBELS was not as helpful
for them as for teachers in the earlier grades. An opinion shared by a
few teachers was that administering DIBELS was accomplished at the
expense of instructional time. "It's a time management issue," a teacher
said. "It really is difficult." One teacher in a school that plans to
continue using DIBELS next year said, "I don't think so \[on using
DIBELS next year\] because I really don't like losing out on
instructional time." Teachers greatly appreciated the palm pilots for
entering DIBELS information and getting instant results. A number of
teachers said they would be reluctant to administer DIBELS without the
palm pilots, because it would take more time to administer and to assess
results. A third concern expressed was that even though it is helpful to
learn about individual student needs, it was hard to address those needs
at the individual level. A teacher said, "It is hard to differentiate
instruction to so many different levels. If we had a pullout program and
went back to Title I, we could probably use that data more effectively."

All of the literacy specialists said DIBELS has had a strong influence
on instruction, support of positive changes in teacher instructional
approaches, how they think about instruction, and how they talk with one
another about instruction and learning. Literacy specialists confirmed
that teachers formed ability-based, flexible groups and, more than in
the past, selected materials for different levels. In the past, a
literacy specialist said, "teachers just tossed the data to the side."
Now it is examined and used.

Five literacy specialists stated that there was room for improvement. In
one school, the literacy specialist said data use "has changed
instruction, but it \[instruction\] could change a lot more." In these
schools, literacy specialists said use of data varied from teacher to
teacher.

Eight of the literacy specialists questioned whether DIBELS was the best
assessment for their school. Five literacy specialists were concerned
because DIBELS does not measure comprehension, and a number of these
literacy specialists (plus one of the principals in the schools) said
they would like to identify an assessment tool that could be used as
efficiently as DIBELS but address more literacy skills, especially
comprehension. Three other literacy specialists said DIBELS was a good
assessment for kindergarten and 1st grade students but was not adequate
for 2nd and 3rd graders.

In six schools, literacy specialists expressed concerns about the time
needed to administer required assessments. This concern is due, in part,
to the number of assessments that are required, including those mandated
by the state. One literacy specialist said teachers also have to
administer a state diagnostic test, Rigby Benchmarking, TPRI, and the
standard achievements tests: "There are too many tests in the first
month of school," she said, adding that teachers are "overwhelmed with
data."

Literacy Instruction in Comparison Schools

The teachers in the comparison schools indicated that the five essential
components have a presence in their instruction. In four of the
comparison schools, most (though not all) of the teachers described
literacy lessons that included a focus on a specific literacy component
(phonics, phonemic awareness, etc.), as well as reading, spelling, and
writing.

More than in the past, teachers in comparison schools indicated they use
assessments to form small groups, decide what to reteach, and determine
how to meet individual student needs. In one district, teachers
indicated they rely most on the assessments that accompany the reading
series (which they began using in 2005--06), such as story tests and
unit tests. Teachers in another district were using district short cycle
assessments, which they said were aligned with the state standards and
the Ohio Achievement Tests, and which they administered several times a
year. A school in another district had been using Rigby assessments and
Star reading assessments several times a year for the past several
years, and teachers indicated that the assessments were well integrated
into the school's literacy program.

Teachers in the comparison schools said they had great discretion in
developing lesson plans and deciding on the focus of lessons and
materials. In several schools, discretion seems to be more a lack of
direction, as teachers indicated that they have had little guidance on
how to use the core reading series, a number of teachers in the school
have decided they will not use the series, or both.

Whereas the Reading First teachers had high levels of support for
learning how to interpret assessments and instructional strategies to
address specific needs, that was not the case in the comparison schools
(except for one that had had a literacy coach for several years, though
not in 2006--07).

The Observed Classrooms

Twenty-eight classrooms in the RF and non-RF schools were observed: 19
Reading First classrooms in 11 schools (in one school, the proposed
observations were cancelled due to a teacher absence, and a teacher
refusal to be observed), and 9 classrooms in the 6 comparison schools.
Of the Reading First classrooms, 9 were observed in high-performing
schools and 10 in the cohort 1 schools. For each observation, narratives
were developed for up to 14 five-minute observation segments. Each
narrative was coded across the following levels or features of the
classroom: instructor (who was teaching), grouping (whole group, small
group, pairs, or individual), major focus (always coded as reading),
specific focus (specific literacy area addressed), materials, teacher
activities, and student response activities.

The section compares the differences in the classrooms across five of
these seven levels: grouping, content focus, materials used, teacher
activities, and student responses. The instructor and major literacy
focus are not discussed due to lack of variation. The classroom teacher
was the instructor in all of the observed classrooms, and reading was
the focus.

For each of the five levels, four school groups are represented: 1) all
Reading First classrooms (n=19); 2) non-Reading First classrooms in the
comparison schools (n=9); 3) the Reading First classrooms in the
original cohort 1 schools that are not high-performing schools (n=10);
and 4) the Reading First classrooms that are in the high-performing
schools (four schools new to the case study and two schools from our
original case study cohort) (n=9). Observation data are compared between
classrooms in Reading First and non-Reading schools, and between
classrooms in Reading First cohort 1 and Reading First high-performing
schools.

Grouping

Small ability-based groups are a key feature of Reading First, and
teachers in comparison schools indicated they too use data to form small
ability-based groups. As shown in Table II-25, the Reading First and
non-Reading First classrooms that were observed differ with respect to
grouping. Among Reading First classrooms, small groups were observed in
37 percent of the segments. In the non-Reading First schools, such
groups accounted for only 6 percent of the segments.

**Table II-25.**---Percent of segments with specific grouping patterns,
by type of classroom

  ------------ -------------- -------------- -------------- -----------------
  Group        All Reading    Non-Reading    Reading First  Reading First
               First          First          cohort 1       high-performing
               classrooms\    classrooms\    classrooms\    classrooms (n=9)
               (n=19)         (n=9)          (n=10)         

  Large group  57             94             58             56

  Small group  37             6              32             44

  Pairs        4              0              8              0

  Individual   0              0              1              0
  ------------ -------------- -------------- -------------- -----------------

NOTE: Percents may not add to 100 because of rounding.

Twelve of the 19 Reading First classrooms observed had some time
dedicated to small groups. In eight of the classrooms, over 40 percent
of the segments were coded for small group sessions. Among the
non-Reading First classrooms, two of the nine had dedicated time to
small group sessions. In each of these two classrooms, over 30 percent
of the segments were coded small group. As shown, the percentage of
codes indicating small groups were observed were approximately the same
for the classrooms in the Reading First cohort 1 and the high-performing
schools.

Specific Focus

Specific focus was the most important level observed, as it conveys the
intent of the instruction. As shown in Table II-26, comprehension was
emphasized more in all observed Reading First classrooms than in the
non-Reading First classrooms (17 percent of the Reading First segments
and 8 percent of non-Reading First segments). Comprehension was also
emphasized more in the high-performing Reading First classrooms than the
Reading First cohort 1 classrooms (20 percent of segments in
high-performing Reading First classrooms and 14 percent in other Reading
First classrooms).

Another difference between Reading First and non-Reading First observed
classrooms is evident in the data for other and non-literacy topics.
Instruction is coded as other and non-literacy when the focus of
instruction is not captured by the other specific foci codes, when the
focus is difficult to determine, and/or when the instruction is clearly
not literacy based. Among Reading First classrooms, 11 percent of
segments were coded as other and non-literacy topics, while 25 percent
of segments in non-Reading First classrooms were recorded as such.

A third major difference among schools conveyed in this table concerns
writing. A higher percentage of segments in non-Reading First than
Reading First schools were coded as writing.

As the table conveys, only a small percent of segments in all schools
were coded as fluency. There were no distinct differences noted in
vocabulary. Phonics is still addressed in the classrooms.

**Table II-26.---Percent of segments with specific foci of instruction,
by type of classroom**

  ----------------- ------------- ------------- ------------- -----------------
  Focus             All Reading   Non-Reading   Reading First Reading First
                    First         First         cohort 1      high-performing
                    classrooms\   classrooms\   classrooms\   classrooms (n=9)
                    (n=19)        (n=9)         (n=10)        

  Comprehension     17            8             14            20

  Fluency           1             2             3             0

  Vocabulary        14            15            12            16

  Phonics           8             7             11            10

  Reading connected 21            16            25            16
  text                                                        

  Spelling          7             6             4             9

  Writing           1             9             2             0

  Other and         11            25            12            10
  non-literacy                                                

  No instruction    15            9             13            17
  ----------------- ------------- ------------- ------------- -----------------

NOTE: Percents may not add to 100 percent because non-ascertained data
or rounding.

Materials

Table II-27 summarizes the percentage of codes indicating what materials
were used in the observed classrooms. There were relatively few
differences among the school groups, although the observation
information does not capture whether teachers within a school were using
the same materials, for example, the same narrative texts. Narrative
texts, a central classroom white board or black board, and worksheets
were coded in the highest percentages for all school groups.

**Table II-27.---Percent of segments with specific materials used, by
type of classroom**

  ----------------- ------------- ------------- ------------- -----------------
  Material          All Reading   Non-Reading   Reading       Reading First
                    First         First         First\        high-performing
                    classrooms\   classrooms\   cohort 1      classrooms (n=9)
                    (n=19)        (n=9)         classrooms\   
                                                (n=10)        

  Board/chart       20            22            24            16

  Computer          2             1             3             2

  Manipulatives     10            1             10            10

  Narrative         29            28            30            28

  Worksheet         21            18            17            25

  Student work      2             6             1             3

  Audio/visual      2             7             1             3

  Informational     0             1             0             1

  Other             2             2             2             1

  None              12            10            12            12
  ----------------- ------------- ------------- ------------- -----------------

NOTE: Percents may not add to 100 because of non-ascertained data or
rounding.

Teacher Activities

Teacher activities exemplify different instructional approaches. As
shown in Table II-28, teacher activities varied between Reading First
and non-Reading First groups in the observed classrooms. The differences
are particularly evident with respect to recitation (in which the
teacher directs instruction, usually through a question and answer
response format), listening and watching (teacher observes student work
or listens to students read but does not make instructional comments),
telling and giving information (usually explanatory information), and
lower order questioning.

**Table** II-28.---Teacher activities in the observed classrooms, by
type of classroom

  ---------------------- ------------- ------------- ------------ -----------------
  Activity               All Reading   Non-Reading   Reading      Reading First
                         First         First         First cohort high-performing
                         classrooms\   classrooms\   1 classrooms classrooms (n=9)
                         (n=19)        (n=9)         (n=10)       

  Recitation             11%           22%           10%          11%

  Listening/watching     17            22            23           10

  Tell/give information  25            16            25           25

  Lower order            13            8             11           16
  questioning                                                     

  Higher order           2             0             3            2
  questioning                                                     

  Reading aloud          7             3             6            7

  Coaching/scaffolding   7             8             7            7

  Modeling               1             1             0            1

  Checking work          2             6             1            2

  Assessments            0             3             0            0

  None                   10            9             11           9
  ---------------------- ------------- ------------- ------------ -----------------

NOTE: Percents may not add to 100 because of non-ascertained data or
rounding.

In Reading First classrooms, teacher activities included a smaller
percentage of recitation and listening and watching, and a greater
percentage of telling and giving information than in non-Reading First
classrooms.

Student Responses

Student responses were grouped as active, passive, or none (which
referred to periods when there was no instructional focus in the
classroom). Active responses were those in which the students were
reading, writing, manipulating objects, responding or discussing, while
passive responses were those in which students *took turns* to respond
or read (thus spending a lot of time waiting). As shown in Table II-29,
a somewhat higher percentage of segments were coded with active codes in
the Reading First classrooms than in non-Reading First classrooms, and
the same percentage were coded with passive codes in each.

**Table II-29.---Student responses in the observed classrooms**

  ------------- -------------- -------------- -------------- -----------------
  Response type All Reading    Non-Reading    Reading First  Reading First
                First          First          cohort 1       high-performing
                classrooms\    classrooms\    classrooms\    classrooms (n=9)
                (n=19)         (n=9)          (n=10)         

  Active        44%            39%            43%            45%

  Passive       45             45             46             45

  Other\*       2              3              2              2

  None          9              12             9              9
  ------------- -------------- -------------- -------------- -----------------

\*Responses that did not fall into the coding categories.

NOTE: Percents may not add to 100 because of rounding.

Conclusion

Respondents in the Reading First schools indicated that participation in
Reading First has had a pronounced impact on literacy instruction in the
school. Reading First has resulted in more consistent literacy
instruction across the K--3 grades, a change from what principals
described as "haphazard" instruction, in which each teacher used his or
her own approach and materials. Reading First has also resulted in
teachers' understanding literacy as a set of interrelated skills that
can be addressed systematically. The use of DIBELS in the Reading First
schools has resulted in more differentiation, as reflected in the
formation of small, ability-based groups. Teachers in the Reading First
schools indicated they were able to identify and focus more specifically
on what students did not understand.

In the comparison schools, teachers had the same reading
programs---those that are used in the Reading First schools. They
indicated they are now aware of the five components and, to an extent,
have formative assessment data that they are able to use to form
ability-based groups. Compared to Reading First schools, there was less
direction and guidance for comparison school teachers on use of the
instructional materials and use of data to differentiate instruction.
Although comparison school teachers said they formed small groups, there
was little evidence of that in the classes observed. The most obvious
difference between Reading First and comparison school classrooms was
small group instruction, which was frequently used in Reading First
observed classrooms.

**Chapter 8. Sustaining Change**

In this chapter we turn to the question of what happens to
implementation of the Reading First program in a school after the
school's funding for the program has been discontinued?

Over the course of the initiative, the respondents in Reading First
schools have talked about the changes in instruction resulting from
participation in the initiative. As noted earlier in this report, these
changes include a shared perspective on literacy, consistent use of the
same instructional materials, awareness of literacy as a set of skills,
a move away from the whole language approach, use of the same
assessments to monitor student progress and learning needs, and
instruction tailored to student needs. In the final few weeks of the
initiative's fourth year, respondents were asked about their plans for
sustainability, specifically, what practices and resources would be
sustained to support literacy instruction. This chapter describes
whether and to what extent respondents believed that the Reading First
literacy model will be sustained by their districts and schools, and
what factors have influenced and will influence decisions about the
model and the likelihood of sustaining it.

District Plans to Sustain Reading First and Commitment of Resources

The district plays a major role in determining whether the Reading First
literacy model will be sustained in the schools. Although principals
expressed hopes and plans for continuing instructional practices and
assessment techniques, many of these plans were not firm at the time of
the interview because districts had not yet determined their budget
allowances for next year. At least two of the districts are in fiscal
emergency, which puts an extra constraint on the availability of funds
to maintain the Reading First model. Even if district personnel are in
favor sustaining Reading First, they must balance literacy needs with
other initiatives. As one respondent explained, "Our district has eight
initiatives and we've addressed four of them really hard over the last
four years. And now we're ready to address the other four. I do know
that literacy was one of the first four." Whether or not they will be
able to sustain the RF model and maintain their focus on literacy will
depend on funding.

Vinton

In this district, district and school respondents said the RF program
was successful in improving literacy instruction and student reading.
From the start, this district decided to implement Reading First in all
of its elementary schools. Before the schools agreed to participate in
Reading First, the future district coordinator learned all she could
about the program and talked to all schools and district administrators
about the key program components. Once the Reading First grant was
awarded, she learned about all the positions and requirements so she
could better support schools. The district curriculum coordinator and
the district coordinator have worked closely to align the program with
the district goals. The district superintendent was very knowledgeable
about the Reading First program and met with the district coordinator
twice a week to discuss the program and student progress.

Respondents indicated that the district has had a feasible
sustainability plan in place since the first year of Reading First. The
district linked Reading First with a program funded through Title I and
developed a plan to maintain key Reading First components with a
different funding stream. In implementing Reading First, the district
planned for sustainability. Rather than adding staff, the district
diverted qualified existing staff members into RF positions. The
district also ensured that the literacy specialists, who were so
critical to teacher professional development, would not be laid off or
transferred in the case of teacher layoffs by making sure they already
had seniority in the system. A similar plan has been in place to retain
services of part-time data managers and resource coordinators, and to
ensure that the professional development piece will endure.

Next year, the role of the literacy specialist will expand to include
the upper elementary grades, since Title I money needs to be used for
school-wide purposes. The principals and literacy specialists said there
will be no change in literacy instruction after the grant, and DIBELS,
which will be retained for formative assessments, will be administered
several times a year. Professional development will be offered to upper
grade elementary teachers to familiarize them with the Reading First
literacy framework and strategies.

Cleveland

The district has provided consistent support to the Reading First
schools throughout the 4 years. However, it has had serious budget
issues during the initiative, and the central office has experienced
high turnover of top administrators. When teachers were laid off, the
district was not successful in making sure Reading First teachers were
transferred to Reading First schools. In the fourth year, the district
reduced funding to cohort 1 schools to a point where RF presence was
reduced in some of the schools. The reassignment of literacy specialists
to two schools in the last year of RF has hindered the integration of
many newly transferred teachers into the RF model.

The district intends to continue several aspects of the program next
year, but overall, it is not clear how the literacy program will be
directed. All of the district schools have now adopted the same reading
program, although as our discussion of comparison schools indicated,
expectations and support for using these materials is low in some
schools. The district is considering using DIBELS as an assessment tool,
but there may not be funding. It is anticipated that new teachers and
teachers changing grade levels will be offered professional development
that addresses the five components.

It does not appear that the literacy specialists will be retained in
their position in 2007--08. Elementary schools in the district have now
become K--8 schools, which makes it unlikely that principals can be
strong stand-ins for the literacy specialists. Finally, the district's
capacity to sustain the initiative is in doubt due to continuing fiscal
problems.

Springfield

In this district, Reading First has achieved mixed results. Respondents
noted improved and more consistent instruction but mixed results with
respect to student learning. As in other districts, the district has
adopted the same reading series used by the Reading First schools in
other district elementary schools. All district schools now have at
least a 90-minute literacy block. The district has hired curriculum
coaches for each building, and the district will retain them the coming
year. However, the curriculum coaches cover all elementary topics and
some are not experts in literacy. The district expects schools to
continue to schedule grade-level meetings, which will include data
meetings based on the ILT model, and have formative assessments that
drive instruction. The district does not endorse the use of DIBELS.

Youngstown

The Youngstown district has experienced a high number of building
closures and mergers during the Reading First years. Reading First was
perceived to be successful and to have attributed to improved
instruction and student reading. The RF model and materials were
consistent with the district's commitment to a guided reading approach
to instruction, and the district expressed hopes for continuing the
model in the future.

The district has provided consistent support to the Reading First
schools. The district coordinator's position has been filled by the same
person, who has been supportive of the RF schools. The district ensured
that new or transferring teachers have received all the professional
development in RF from the previous year or years. The district has
mandated that the formats for grade-level and department-level meetings
include a review of student achievement data, including DIBELS. One of
the district administrators said, "The district has done a good job with
the grade-level meetings, with actually looking at the data and making
decisions about what had to happen with kids in the buildings." The
district coordinated the use of the same reading series across all
district elementary schools. Throughout RF, the district curriculum
coordinator met directly with the RF literacy specialists during
regularly scheduled meetings. A district administrator added, "We've
tried to incorporate a lot of parent programs with Reading First to pull
the parents in. We try to supply students with a lot of print material,
both in their classroom and at home, and we've also tried to integrate
the Reading First processes with the processes that we've already had in
place."

In the last year, the district experienced a large budget shortfall and
went into fiscal emergency. The district plan to sustain the RF
initiative and maintain literacy coaches in each of the district schools
is in doubt, with any decisions ultimately to be made by the Fiscal
Oversight Commission.

Dayton

This district, new to the case study this year, was selected because one
of its schools is a high-performing school. According to interviewees,
the district has provided a high level of support to the Reading First
schools, including implementing a district-wide reading series. One
interviewee expressed the view that reduced funding for the Reading
First program in the last year was detrimental, saying, "As long as the
pieces are in the puzzle it works well. With less funding, therefore,
you have less pieces to the puzzle."

The district is in a budget crisis, and a critical bond issue failed to
pass shortly after the site visits were conducted. The district will not
be able to fund the wireless data collection tools, which may hinder the
administration and use of DIBELS. It is not clear that literacy
specialists will be retained.

Schools

With or without a district plan, all 12 of the Reading First principals
interviewed expressed confidence that their teachers and school will
continue to follow the Reading First model by using the instructional
materials, maintaining the 90-minute reading block, focusing on the five
components, and using data to identify student needs and form small
ability-based groups. Most principals said their schools would use
DIBELS, but four of the principals and several literacy specialists (as
well as teachers) raised concerns about teacher willingness to
administer DIBELS if they no longer had palm pilots and wireless network
services for immediate data analysis. Only a few principals were
definite about continuing to administer DIBELS in the district, while
several others said they would identify school or other funds to
continue with the assessment.

Nearly all teachers said they would continue teaching the five essential
components, and comments from a number of teachers suggest that this is
a well-established part of the curriculum that as one teacher said,
"doesn't go away because Reading First goes away." Most of the teachers
said they would continue to use data to assess student skills and form
small groups. As one teacher stated, "That's the only way we can make
the improvements we need to make---to have a clear plan in place and
focus on instruction." Whether they use DIBELS to determine student
needs depends on district and school decisions.

The 90-minute reading block is well established in the Reading First
schools and other district schools. It is unlikely that will change in
the future.

Summary

In general, it is likely that Reading First will have a sustained impact
on district literacy instruction due to the following: 1) more support
for using a district-wide and school-wide curriculum; 2) more emphasis
on formative assessments; 3) more emphasis on a literacy framework that
includes the five components; and 4) longer literacy blocks. All of the
principals and most of the teachers in Reading First schools said they
would continue to implement the Reading First literacy model. However,
it is not certain how well the literacy program can be implemented
without the support of a full-time literacy specialist. Many principals
and teachers have relied on literacy specialists to keep the literacy
program going, that is, to schedule assessments, provide professional
development, coach individual teachers, support teachers who are new to
the school, distribute materials, and provide individual and on-demand
support for teachers.

Although several districts have formal sustainability plans in place,
the likelihood of these plans being implemented depends greatly on the
availability of funding, which in most cases was not known at the time
of the interviews. If these plans are implemented, they will likely be
with reduced resources and, in most of the schools in the case study,
without a literacy specialist.

In our case study of the five districts; we found the following:

-   One district has a sustainability plan, which includes maintaining
    literacy specialists and has a high likelihood of implementing that
    plan.

-   One district has a plan to maintain literacy specialists and has
    exhibited continued support for the RF schools, but due to its
    fiscal emergency, may not have funds. On the other hand, this
    district has done well in integrating RF with a larger (and
    preexisting) district commitment to improving literacy, and a
    district administrator said the district will not back away from
    that commitment.

-   Three districts will maintain instructional components of the RF
    program but have no plans to provide school-based support.

Section III. Achievement

In this section, we present findings with regard to RFO's impact on
student achievement and assess the value added by being in a RF school.
We examine the data from three perspectives.

The first approach is largely descriptive, whereby we compare student
achievement in different cross-sections over time and against
test-specific benchmarks. The analysis addresses all students from the
RFO cohorts 1--3 schools by grade level. Specifically, cohort 1 has 4
years of data, cohort 2 has 3 years of data, and cohort 3 has 2 years of
data. The comparison, or "status," approach is similar to what most
states are using in NCLB accountability reporting. Results from this
type of analysis provide evidence on whether student achievement in RFO
schools has been improved.

The second approach is comparative, whereby we examine differences of
student achievement between RFO treatment and comparison schools. The
comparison groups derive from two sources: 1) 8 similar non-RFO schools
from the case study sample, and 2) 85 non-RFO schools receiving
school-wide Title I that we pair-matched with RFO schools based on
student reading achievement in 2002. Results from this type of analysis
provide evidence on value added of RFO to student reading achievement as
compared to other or no specific services.

Finally, we explore the relationship between various measures of
implementation and student achievement. The first measure is based on
teacher and principal surveys that aim to get at implementation of three
program standards underlying RFO intervention; the second measure is
based on observation of teachers' instructional practices as a result of
participating in RFO. The findings provide evidence to the question of
what features of the RFO program are effective in terms of value added
to student achievement?

Changes in Performance in Reading First Schools Over Time

In this section we address the question of how achievement changes in RE
schools over time. We examine changes in achievement outcomes as the
program becomes more mature, using both cross-sectional and longitudinal
student data.

RFO has been using a test battery including DIBELS (K--3), TerraNova
(1--3) and OAT-R (3) to measure student reading achievement in all RFO
schools. The data we analyzed come from 51 schools in cohort 1, 18
schools in cohort 2, and 41 school in cohort 3. Because the assessments
are different, we discuss the results for DIBELS and TerraNova, OAT-R
separately in the sections that follow.

DIBELS Performance

The DIBELS is a widely used assessment of early reading skills.
Administered three times a year, the DIBELS provides developmentally
appropriate information on the extent to which students are attaining
what are believed to be critical standards or benchmarks in early
literacy. Table III-1 presents a summary of the assessments used in
grades K--3.

Table III-1.---DIBELS assessments

+-------------------------------------+------+------+------+------+----+
| Instrument                          | Nu   |      |      |      |    |
|                                     | mber |      |      |      |    |
|                                     | of   |      |      |      |    |
|                                     | ass  |      |      |      |    |
|                                     | essm |      |      |      |    |
|                                     | ents |      |      |      |    |
+-------------------------------------+------+------+------+------+----+
|                                     | K    | 1    | 2    | 3    | T  |
|                                     |      |      |      |      | ot |
|                                     |      |      |      |      | al |
+-------------------------------------+------+------+------+------+----+
| **DIBELS**                          |      |      |      |      |    |
+-------------------------------------+------+------+------+------+----+
| > Initial Sound Fluency             | 2    | 0    | 0    | 0    | 2  |
+-------------------------------------+------+------+------+------+----+
| > Letter Naming Fluency             | 3    | 1    | 0    | 0    | 4  |
+-------------------------------------+------+------+------+------+----+
| > Phoneme Segmentation Fluency      | 2    | 3    | 0    | 0    | 5  |
+-------------------------------------+------+------+------+------+----+
| > Nonsense Word Fluency             | 2    | 3    | 1    | 0    | 6  |
+-------------------------------------+------+------+------+------+----+
| > Oral Reading Fluency              | 0    | 2    | 3    | 3    | 8  |
+-------------------------------------+------+------+------+------+----+

We looked at student performance on the DIBELS in a variety of ways. For
the three cohorts of participating schools, we performed the following
analyses:[^28]

-   We examined student achievement data by year in raw scores to
    illustrate trends over time. These are essentially cross-sectional
    comparisons. For each cohort, we estimated both the statistical
    significance using a two-level hierarchical linear modeling (i.e.,
    students nested with schools) and the effect size of differences
    between 2007 and year 1 achievement.

-   We looked at the descriptive statistics for students achieving
    grade-level RFO proficiency benchmarks.

-   In addition to analyzing the data for the overall population, we
    also examined subgroups by student race/ethnicity, receipt of free
    or reduced-price lunch, English proficiency level, and special
    education status.[^29]

**Cross-sectional analyses for all cohorts show significant increases in
DIBELS performance over time. Both students' raw scores and the
percentage of students that are proficient increased for each grade
level. For the great majority of the subtests, the improvement was
continuous over time. In addition, more than half of the students were
proficient in 22 out of 25 subtests. Only with regard to Oral Reading
Fluency (ORF), does there appear to be a consistent pattern of large
numbers of students failing to reach proficiency. However, even for this
subtest, we see large improvements over time in the percentage who
attain proficiency. Changes in performance by race/ethnicity and special
services showed no consistent pattern with variations occurring by
cohort, grade level and specific subtest.**

Table III-2 provides an overall summary of DIBELS findings showing
changes in raw scores over time. Figure III-1 provides an illustration
of how proficiency scores changed over time. The pattern shown on the
selected subtest is reflective of patterns on the subtests in general.
Details on findings by cohort are presented below.

Table III-2.---Summary of DIBELS results for cohorts 1--3

  --------------------------------------- ---------- ---------- ----------
  Assessment                              Cohort 1   Cohort 2   Cohort 3

  **Kindergarten fall**                                         

  DIBELS Initial Sound Fluency            \+         \+         \+

  DIBELS Letter Naming Fluency            \+         \+         \+

  **Kindergarten winter**                                       

  DIBELS Initial Sound Fluency            \+         \+         \+

  DIBELS Phoneme Segmentation Fluency     \+         \+         o

  DIBELS Nonsense Word Fluency            \+         \+         o

  DIBELS Letter Naming Fluency            \+         \+         o

  **Kindergarten spring**                                       

  DIBELS Phoneme Segmentation Fluency     \+         \+         \+

  DIBELS Nonsense Word Fluency            \+         \+         o

  DIBELS Letter Naming Fluency            \+         \+         \+

  **Grade 1 fall**                                              

  DIBELS Phoneme Segmentation Fluency     \+         \+         \+

  DIBELS Nonsense Word Fluency            \+         \+         \+

  DIBELS Letter Naming Fluency            \+         \+         \+

  **Grade 1 winter**                                            

  DIBELS Phoneme Segmentation Fluency     \+         \+         \+

  DIBELS Nonsense Word Fluency            \+         \+         \+

  DIBELS Oral Reading Fluency             \+         \+         o

  **Grade 1 spring**                                            

  DIBELS Phoneme Segmentation Fluency     \+         \+         \+

  DIBELS Nonsense Word Fluency            \+         \+         \+

  DIBELS Oral Reading Fluency             \+         \+         o

  **Grade 2 fall**                                              

  DIBELS Nonsense Word Fluency            \+         \+         \+

  DIBELS Oral Reading Fluency             \+         \+         \+

  **Grade 2 winter**                                            

  DIBELS Oral Reading Fluency             \+         \+         \+

  **Grade 2 spring**                                            

  DIBELS Oral Reading Fluency             \+         \+         \+

  **Grade 3 fall**                                              

  DIBELS Oral Reading Fluency             \+         \+         \+

  **Grade 3 winter**                                            

  DIBELS Oral Reading Fluency             \+         \+         \+

  **Grade 3 spring**                                            

  DIBELS Oral Reading Fluency             \+         \+         \+
  --------------------------------------- ---------- ---------- ----------

\+ = Statistically significant in favor of current year; o = no
significant difference.

**Figure III-1.---Trend for Phoneme Segmentation Fluency (grade 1
winter), by cohort**

![](media/image27.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

Cohort 1

Table III-3 presents comparisons of data for years 1--4 (2004--07) from
RFO cohort 1. Each row represents a different assessment. For example,
the first row provides results for Initial Sound Fluency. The table
shows that the average raw score (i.e., number of correct answers in
each test) for the year 1 group is 12.1. The sample size of the group
(in parenthesis) is 2,184. The average raw score for the year 2 group of
2,586 students is 12.3. In year 3, the average score from 2,519 students
is 13.0, and the mean score from 2,143 students in year 4 went up to
15.0. The difference between year 4 group (current status) and year 1
group (baseline) is statistically significant at .05 level in favor of
the year 4 group and the effect size of the difference is 0.28.

In general, students in RFO schools had higher levels of performance in
year 4 than in year 1 across all grades and assessments. The differences
were statistically significant, and the effects sizes were moderate to
large (i.e., between 0.14 to 0.53).[^30] In addition, the trend was
continuous for half of the outcomes; for the rest, the improvement
became stable by year 3. We describe the patterns by grade level as
follows:

-   Kindergarten: student achievement has improved over time. In the 10
    assessments administered, the largest change occurred between year 1
    and year 2 of RFO program participation. The effect sizes between
    year 4 and year 1 differences were around 0.30.

-   Grade 1: student achievement has demonstrated similar improvement in
    all nine assessments administered at this level. The effect sizes
    for the differences between year 4 and year 1 were above 0.40 for
    five assessments. While student performance on the fall assessments
    showed continuing increases over time, the trend flattened out by
    year 3 for assessments administered in winter and spring.

-   Grade 2: student performance has increased continuously in all four
    assessments over time.

-   Grade 3: continuous increases in scores were found for the 3rd
    graders. The effect sizes for the differences between year 4 and
    year 1 were around 0.30.

Table III-3.---Comparison of DIBELS results for years 1--4 in RFO cohort
1: Raw score

  ----------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment        Year 1           Year 2           Year 3           Year 4           Sig\     ES\
                                                                                        (Y4-1)   (Y4-1)

                    Raw     Number   Raw     Number   Raw     Number   Raw     Number            
                    score            score            score            score                     

  **Kindergarten                                                                                 
  fall**                                                                                         

  DIBELS Initial    12.1    2,184    12.3    2,586    13.0    2,519    15.0    2,143    \+       0.28
  Sound Fluency                                                                                  

  DIBELS Letter     20.0    2,108    21.5    2,582    21.5    2,521    24.2    2,146    \+       0.25
  Naming Fluency                                                                                 

  **Kindergarten                                                                                 
  winter**                                                                                       

  DIBELS Initial    21.8    2,454    23.6    2,513    26.0    2,583    26.8    2,392    \+       0.30
  Sound Fluency                                                                                  

  DIBELS Phoneme    18.4    2,457    20.4    2,510    24.2    2,579    22.9    2,383    \+       0.28
  Segmentation                                                                                   
  Fluency                                                                                        

  DIBELS Nonsense   16.5    2,457    20.2    2,505    21.9    2,576    21.1    2,378    \+       0.29
  Word Fluency                                                                                   

  DIBELS Letter     31.8    2,458    35.3    2,515    36.5    2,582    36.7    2,391    \+       0.28
  Naming Fluency                                                                                 

  **Kindergarten                                                                                 
  spring**                                                                                       

  DIBELS Phoneme    30.4    2,633    37.3    2,552    36.6    2,354    37.1    2,025    \+       0.35
  Segmentation                                                                                   
  Fluency                                                                                        

  DIBELS Nonsense   24.4    2,627    29.4    2,552    28.5    2,349    30.2    2,022    \+       0.31
  Word Fluency                                                                                   

  DIBELS Letter     39.6    2,635    42.7    2,556    43.0    2,357    44.7    2,025    \+       0.28
  Naming Fluency                                                                                 

  **Grade 1 fall**                                                                               

  DIBELS Phoneme    34.6    2,346    32.5    2,744    38.1    2,550    41.2    2,244    \+       0.37
  Segmentation                                                                                   
  Fluency                                                                                        

  DIBELS Nonsense   30.0    2,329    31.1    2,747    36.3    2,553    38.2    2,245    \+       0.41
  Word Fluency                                                                                   

  DIBELS Letter     45.0    2,403    47.2    2,747    49.1    2,538    51.1    2,185    \+       0.35
  Naming Fluency                                                                                 

  **Grade 1                                                                                      
  winter**                                                                                       

  DIBELS Phoneme    36.5    2,439    41.7    2,656    45.6    2,537    46.6    2,428    \+       0.57
  Segmentation                                                                                   
  Fluency                                                                                        

  DIBELS Nonsense   39.3    2,454    44.9    2,675    52.7    2,566    50.2    2,469    \+       0.47
  Word Fluency                                                                                   

  DIBELS Oral       25.0    2,409    27.3    2,670    30.7    2,551    28.4    2,472    \+       0.14
  Reading Fluency                                                                                

  **Grade 1                                                                                      
  spring**                                                                                       

  DIBELS Phoneme    41.3    2,526    46.3    2,668    47.8    2,334    50.0    2,118    \+       0.53
  Segmentation                                                                                   
  Fluency                                                                                        
  ----------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------

Table III-3.---Comparison of DIBELS results for years 1--4 in RFO cohort
1: Raw score---continued

  ----------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment        Year 1           Year 2           Year 3           Year 4           Sig\     ES\
                                                                                        (Y4-1)   (Y4-1)

                    Raw     Number   Raw     Number   Raw     Number   Raw     Number            
                    score            score            score            score                     

  DIBELS Nonsense   47.9    2,512    54.5    2,669    59.1    2,335    58.9    2,116    \+       0.42
  Word Fluency                                                                                   

  DIBELS Oral       35.3    2,527    37.3    2,661    40.6    2,334    39.7    2,119    \+       0.16
  Reading Fluency                                                                                

  **Grade 2 fall**                                                                               

  DIBELS Nonsense   44.7    2,277    47.1    2,327    53.0    2,515    59.2    2,119    \+       0.49
  Word Fluency                                                                                   

  DIBELS Oral       48.4    2,284    47.6    2,420    48.4    2,539    53.8    2,193    \+       0.18
  Reading Fluency                                                                                

  **Grade 2                                                                                      
  winter**                                                                                       

  DIBELS Oral       61.7    2,401    66.4    2,396    68.5    2,552    69.7    2,409    \+       0.23
  Reading Fluency                                                                                

  **Grade 2                                                                                      
  spring**                                                                                       

  DIBELS Oral       69.3    2,435    75.3    2,384    75.4    2,334    80.3    2,034    \+       0.30
  Reading Fluency                                                                                

  **Grade 3 fall**                                                                               

  DIBELS Oral       68.5    2,344    66.6    2,521    74.8    2,389    77.7    2,186    \+       0.28
  Reading Fluency                                                                                

  **Grade 3                                                                                      
  winter**                                                                                       

  DIBELS Oral       74.9    2,346    81.2    2,407    85.5    2,376    84.7    2,347    \+       0.28
  Reading Fluency                                                                                

  **Grade 3                                                                                      
  spring**                                                                                       

  DIBELS Oral       84.0    2,463    92.8    2,363    94.9    2,203    96.5    2,038    \+       0.34
  Reading Fluency                                                                                
  ----------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y4.

Table III-4 presents a comparison of benchmark data over time for
cohort 1. Students in the "low risk" category are identified as
proficient. The table shows that the proficient percentages have gone up
considerably over time, although the increase became stable by year 3 in
some cases. As of year 4, more than half of the students were proficient
in 19 of the 25 assessments. This is especially impressive for Letter
Naming Fluency (LNF) for grade 1 fall and Phoneme Segmentation Fluency
(PSF) for grade 1 winter and spring, all of which where above 80 percent
proficient. While similar increases were found for Oral Reading Fluency,
the percentages of students achieving proficiency in grades 1--3 were
below 50 percent in many cases.

Table III-4.---Comparison of DIBELS results for years 1--4 in RFO cohort
1: Percent proficient

  --------------------------------------- --------- ------- ------- -------
  Assessment                              Year 1    Year 2  Year 3  Year 4

  **Kindergarten fall**                                             

  DIBELS Initial Sound Fluency            62.2%     65.9%   70.1%   74.6%

  DIBELS Letter Naming Fluency            70.9      72.0    71.3    77.1

  **Kindergarten winter**                                           

  DIBELS Initial Sound Fluency            34.9      42.9    48.9    50.6

  DIBELS Phoneme Segmentation Fluency     45.9      51.8    58.4    55.9

  DIBELS Nonsense Word Fluency            50.8      61.5    66.5    64.5

  DIBELS Letter Naming Fluency            62.4      69.1    72.4    71.2

  **Kindergarten spring**                                           

  DIBELS Phoneme Segmentation Fluency     45.9      60.9    60.5    62.9

  DIBELS Nonsense Word Fluency            47.1      59.6    55.8    60.8

  DIBELS Letter Naming Fluency            51.5      60.8    59.0    64.0

  **Grade 1 fall**                                                  

  DIBELS Phoneme Segmentation Fluency     55.3      48.5    63.4    70.8

  DIBELS Nonsense Word Fluency            61.9      62.4    72.8    76.9

  DIBELS Letter Naming Fluency            69.1      72.8    78.3    81.1

  **Grade 1 winter**                                                

  DIBELS Phoneme Segmentation Fluency     57.9      70.6    77.5    80.8

  DIBELS Nonsense Word Fluency            28.0      37.1    52.4    47.4

  DIBELS Oral Reading Fluency             46.8      51.0    58.3    54.2

  **Grade 1 spring**                                                

  DIBELS Phoneme Segmentation Fluency     69.2      78.7    82.6    87.1

  DIBELS Nonsense Word Fluency            42.6      52.9    59.9    59.6

  DIBELS Oral Reading Fluency             35.5      39.0    44.5    43.0

  **Grade 2 fall**                                                  

  DIBELS Nonsense Word Fluency            35.7      39.2    47.5    57.7

  DIBELS Oral Reading Fluency             50.0      49.5    50.7    59.0

  **Grade 2 winter**                                                

  DIBELS Oral Reading Fluency             41.4      48.0    50.9    53.7

  **Grade 2 spring**                                                

  DIBELS Oral Reading Fluency             28.9      36.7    36.6    42.5

  **Grade 3 fall**                                                  

  DIBELS Oral Reading Fluency             37.6      38.8    47.9    49.5

  **Grade 3 winter**                                                

  DIBELS Oral Reading Fluency             29.8      37.3    42.1    40.6

  **Grade 3 spring**                                                

  DIBELS Oral Reading Fluency             23.8      35.7    36.3    37.5
  --------------------------------------- --------- ------- ------- -------

We examined performance for a longitudinal group of students (grade K in
2004) who had been with RFO for four consecutive years (N=706). In Table
III-5, comparison of the longitudinal group and the respective
cross-sectional group shows that students in the longitudinal group
began at a higher achievement level in kindergarten and remained higher
throughout the years than the cross-sectional group.

Table III-5.---Comparison of DIBELS results for the cohort 1
longitudinal and cross-sectional groups: Percent proficient

  -------------------------------------------------- -------------- -----------------
  Assessment                                         Longitudinal   Cross-sectional

  **Kindergarten fall**                                             

  DIBELS Initial Sound Fluency                       67.7%          62.2%

  DIBELS Letter Naming Fluency                       81.3           70.9

  **Kindergarten winter**                                           

  DIBELS Initial Sound Fluency                       43.4           34.9

  DIBELS Phoneme Segmentation Fluency                52.4           45.9

  DIBELS Nonsense Word Fluency                       61.1           50.8

  DIBELS Letter Naming Fluency                       73.2           62.4

  **Kindergarten spring**                                           

  DIBELS Phoneme Segmentation Fluency                52.8           45.9

  DIBELS Nonsense Word Fluency                       57.8           47.1

  DIBELS Letter Naming Fluency                       60.6           51.5

  **Grade 1 fall**                                                  

  DIBELS Phoneme Segmentation Fluency                53.1           48.5

  DIBELS Nonsense Word Fluency                       71.6           62.4

  DIBELS Letter Naming Fluency                       79.1           72.8

  **Grade 1 winter**                                                

  DIBELS Phoneme Segmentation Fluency                74.2           70.6

  DIBELS Nonsense Word Fluency                       42.9           37.1

  DIBELS Oral Reading Fluency                        61.4           51.0

  **Grade 1 spring**                                                

  DIBELS Phoneme Segmentation Fluency                81.6           78.7

  DIBELS Nonsense Word Fluency                       58.4           52.9

  DIBELS Oral Reading Fluency                        46.7           39.0

  **Grade 2 fall**                                                  

  DIBELS Nonsense Word Fluency                       53.1           39.2

  DIBELS Oral Reading Fluency                        56.3           49.5

  **Grade 2 winter**                                                

  DIBELS Oral Reading Fluency                        58.3           48.0

  **Grade 2 spring**                                                

  DIBELS Oral Reading Fluency                        43.3           36.7

  **Grade 3 fall**                                                  

  DIBELS Oral Reading Fluency                        55.4           47.9

  **Grade 3 winter**                                                

  DIBELS Oral Reading Fluency                        46.5           42.1

  **Grade 3 spring**                                                

  DIBELS Oral Reading Fluency                        41.4           36.3
  -------------------------------------------------- -------------- -----------------

Finally, we analyzed achievement data disaggregated by subgroups. Table
III-6 shows the summary results disaggregated by racial/ethnic status.
For cohort 1 between years 1 and 4, the sample size by assessment and
period ranged from 889 to 1,372 for African Americans; 210 to 315 for
Hispanics; and 754 to 871 for whites. Compared to year 1,

-   African American students in year 4 scored significantly higher on
    23 of the 25 assessments. The effect sizes were from 0.10 to 0.57.

-   Hispanic students in year 4 scored significantly higher on 14 of the
    25 assessments. No differences were found in the results from the
    other 11 assessments.

-   White students in year 4 scored significantly better on all 25
    assessments, and the effect sizes tended to be larger than the
    overall average, especially for 7 assessments where the sizes
    exceeded 0.50**.**

**Table III-6.---**Comparison of DIBELS results for years 1--4 in RFO
cohort 1, by race/ ethnicity

  --------------------------- ---------- -------- ---------- -------- -------- --------
  Assessment                  African             Hispanic            White    
                              American                                         

                              Sig\       ES\      Sig\       ES\      Sig\     ES\
                              (Y4-1)     (Y4-1)   (Y4-1)     (Y4-1)   (Y4-1)   (Y4-1)

  **Kindergarten fall**                                                        

  DIBELS Initial Sound        \+         0.21     \+         0.26     \+       0.40
  Fluency                                                                      

  DIBELS Letter Naming        \+         0.15     o          0.13     \+       0.40
  Fluency                                                                      

  **Kindergarten winter**                                                      

  DIBELS Initial Sound        \+         0.13     \+         0.43     \+       0.53
  Fluency                                                                      

  DIBELS Phoneme Segmentation \+         0.27     o          0.22     \+       0.33
  Fluency                                                                      

  DIBELS Nonsense Word        \+         0.21     \+         0.32     \+       0.34
  Fluency                                                                      

  DIBELS Letter Naming        \+         0.22     \+         0.25     \+       0.36
  Fluency                                                                      

  **Kindergarten spring**                                                      

  DIBELS Phoneme Segmentation \+         0.38     o          0.23     \+       0.32
  Fluency                                                                      

  DIBELS Nonsense Word        \+         0.22     o          0.21     \+       0.40
  Fluency                                                                      

  DIBELS Letter Naming        \+         0.21     \+         0.25     \+       0.34
  Fluency                                                                      

  **Grade 1 fall**                                                             

  DIBELS Phoneme Segmentation \+         0.33     \+         0.27     \+       0.47
  Fluency                                                                      

  DIBELS Nonsense Word        \+         0.32     o          0.19     \+       0.56
  Fluency                                                                      

  DIBELS Letter Naming        \+         0.30     \+         0.24     \+       0.41
  Fluency                                                                      

  **Grade 1 winter**                                                           

  DIBELS Phoneme Segmentation \+         0.57     \+         0.45     \+       0.65
  Fluency                                                                      

  DIBELS Nonsense Word        \+         0.47     \+         0.45     \+       0.50
  Fluency                                                                      

  DIBELS Oral Reading Fluency o          0.07     o          0.05     \+       0.23

  **Grade 1 spring**                                                           

  DIBELS Phoneme Segmentation \+         0.48     \+         0.47     \+       0.63
  Fluency                                                                      

  DIBELS Nonsense Word        \+         0.25     \+         0.43     \+       0.57
  Fluency                                                                      

  DIBELS Oral Reading Fluency o          0.00     o          0.14     \+       0.25

  **Grade 2 fall**                                                             

  DIBELS Nonsense Word        \+         0.32     \+         0.23     \+       0.75
  Fluency                                                                      

  DIBELS Oral Reading Fluency \+         0.10     o          -0.04    \+       0.33

  **Grade 2 winter**                                                           

  DIBELS Oral Reading Fluency \+         0.18     o          0.14     \+       0.32

  **Grade 2 spring**                                                           

  DIBELS Oral Reading Fluency \+         0.24     \+         0.27     \+       0.38

  **Grade 3 fall**                                                             

  DIBELS Oral Reading Fluency \+         0.25     o          0.16     \+       0.31

  **Grade 3 winter**                                                           

  DIBELS Oral Reading Fluency \+         0.26     o          0.21     \+       0.31

  **Grade 3 spring**                                                           

  DIBELS Oral Reading Fluency \+         0.24     \+         0.31     \+       0.41
  --------------------------- ---------- -------- ---------- -------- -------- --------

\+ = Statistically significant in favor of Y4; o = no significant
difference.

Table III-7 compares data for students by receipt of the special
services such as free and reduced-price lunch (FRL) and limited English
proficient (LEP), and special education instruction. For cohort 1
between years 1 and 4, the sample size by assessment and period ranged
from 1401 to 1,921 for FRL students; 16 to 134 for LEP students; and 121
to 337 for special education students. Compared to year 1,

-   FRL students in year 4 scored significantly better on 23 of the 25
    assessments. The effect sizes were moderate to high.

-   LEP students in year 4 scored significantly higher on 7 of the 25
    assessments. No differences were found in the results from the other
    assessments. The effect sizes were large for kindergarten students,
    but small for other grade levels.

-   Special education students in year 4 scored significantly better on
    20 of the 25 assessments. The effect sizes were especially large for
    kindergarten and 1st grade students.

Table III-7.---Comparison of DIBELS results for years 1--4 in RFO cohort
1, by special service

  -------------------------- -------- -------- -------- -------- --------- --------
  Assessment                 FRL               LEP               Special   
                                                                 Ed        

                             Sig\     ES\      Sig\     ES\      Sig\      ES\
                             (Y4-1)   (Y4-1)   (Y4-1)   (Y4-1)   (Y4-1)    (Y4-1)

  **Kindergarten fall**                                                    

  DIBELS Initial Sound       \+       0.30     \+       0.63     \+        0.43
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.20     \+       0.32     o         0.24
  Fluency                                                                  

  **Kindergarten winter**                                                  

  DIBELS Initial Sound       \+       0.27     o        0.78     \+        0.45
  Fluency                                                                  

  DIBELS Phoneme             \+       0.29     \+       0.72     \+        0.43
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.25     \+       1.07     \+        0.43
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.23     o        0.46     \+        0.47
  Fluency                                                                  

  **Kindergarten spring**                                                  

  DIBELS Phoneme             \+       0.31     o        1.06     \+        0.50
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.24     \+       0.90     \+        0.47
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.22     \+       0.67     \+        0.52
  Fluency                                                                  

  **Grade 1 fall**                                                         

  DIBELS Phoneme             \+       0.35     o        0.09     \+        0.35
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.35     o        -0.07    \+        0.46
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.34     o        0.37     \+        0.44
  Fluency                                                                  

  **Grade 1 winter**                                                       

  DIBELS Phoneme             \+       0.64     o        0.57     \+        0.59
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.46     o        0.16     \+        0.52
  Fluency                                                                  

  DIBELS Oral Reading        o        0.07     o        0.02     \+        0.23
  Fluency                                                                  

  **Grade 1 spring**                                                       

  DIBELS Phoneme             \+       0.56     o        0.64     \+        0.54
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.36     o        0.06     \+        0.57
  Fluency                                                                  

  DIBELS Oral Reading        o        0.09     o        0.11     \+        0.36
  Fluency                                                                  

  **Grade 2 fall**                                                         

  DIBELS Nonsense Word       \+       0.45     o        -0.11    \+        0.22
  Fluency                                                                  

  DIBELS Oral Reading        \+       0.16     o        -0.45    o         0.10
  Fluency                                                                  
  -------------------------- -------- -------- -------- -------- --------- --------

Table III-7.---Comparison of DIBELS results for years 1--4 in RFO cohort
1, by special service---continued

  -------------------------- -------- -------- -------- -------- --------- --------
  Assessment                 FRL               LEP               Special   
                                                                 Ed        

                             Sig\     ES\      Sig\     ES\      Sig\      ES\
                             (Y4-1)   (Y4-1)   (Y4-1)   (Y4-1)   (Y4-1)    (Y4-1)

  **Grade 2 winter**                                                       

  DIBELS Oral Reading        \+       0.21     o        -0.13    o         0.13
  Fluency                                                                  

  **Grade 2 spring**                                                       

  DIBELS Oral Reading        \+       0.29     o        0.03     \+        0.25
  Fluency                                                                  

  **Grade 3 fall**                                                         

  DIBELS Oral Reading        \+       0.24     o        0.05     o         0.01
  Fluency                                                                  

  **Grade 3 winter**                                                       

  DIBELS Oral Reading        \+       0.24     o        0.17     o         0.10
  Fluency                                                                  

  **Grade 3 spring**                                                       

  DIBELS Oral Reading        \+       0.27     \+       0.41     \+        0.18
  Fluency                                                                  
  -------------------------- -------- -------- -------- -------- --------- --------

\+ = Statistically significant in favor of Y4; o = no significant
difference.

Cohort 2

Table III-8 presents comparisons of data for years 1--3 (2005--07) from
RFO cohort 2. In general, students had higher levels of performance in
year 3 than in year 1 across all grades and assessments. The differences
were statistically significant and the effects sizes were moderate to
large (i.e., between 0.16 to 0.87). The improvement is continuous over
time for almost all the outcomes. We describe the patterns by grade
level as follows:

-   Kindergarten: student achievement has shown consistent increases in
    all nine assessments with effect sizes between year 3 and year 1
    difference above 0.25.

-   Grade 1: students have shown similar progress in all nine
    assessments. The effect sizes between year 3 and year 1 difference
    in six assessments were above 0.40.

-   Grades 2 and 3: students have demonstrated continuing increases in
    all assessments over time. The effect sizes for the differences
    between year 3 and year 1 were from 0.16 to 0.62.

Table III-8.---Comparison of DIBELS results for years 1--3 in RFO cohort
2: Raw score

  ----------------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment        Year 1           Year 2           Year 3           Sig\     ES\
                                                                       (Y3-1)   (Y3-1)

                    Raw     Number   Raw     Number   Raw     Number            
                    score            score            score                     

  **Kindergarten                                                                
  fall**                                                                        

  DIBELS Initial    10.4    812      14.6    952      15.5    771      \+       0.54
  Sound Fluency                                                                 

  DIBELS Letter     20.5    813      23.2    954      24.6    769      \+       0.25
  Naming Fluency                                                                

  **Kindergarten                                                                
  winter**                                                                      

  DIBELS Initial    22.4    800      26.7    927      27.9    848      \+       0.33
  Sound Fluency                                                                 

  DIBELS Phoneme    14.9    799      19.3    929      22.6    848      \+       0.54
  Segmentation                                                                  
  Fluency                                                                       

  DIBELS Nonsense   17.2    800      22.1    927      22.7    843      \+       0.37
  Word Fluency                                                                  

  DIBELS Letter     33.8    800      37.3    927      38.0    848      \+       0.26
  Naming Fluency                                                                
  ----------------- ------- -------- ------- -------- ------- -------- -------- --------

Table III-8.---Comparison of DIBELS results for years 1--3 in RFO cohort
2: Raw score---continued

  ----------------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment        Year 1           Year 2           Year 3           Sig\     ES\
                                                                       (Y3-1)   (Y3-1)

                    Raw     Number   Raw     Number   Raw     Number            
                    score            score            score                     

  **Kindergarten                                                                
  spring**                                                                      

  DIBELS Phoneme    27.8    809      34.6    926      36.0    748      \+       0.44
  Segmentation                                                                  
  Fluency                                                                       

  DIBELS Nonsense   24.9    810      29.4    926      31.1    747      \+       0.36
  Word Fluency                                                                  

  DIBELS Letter     40.2    811      43.9    925      44.9    748      \+       0.27
  Naming Fluency                                                                

  **Grade 1 fall**                                                              

  DIBELS Phoneme    25.6    911      35.8    974      39.4    781      \+       0.87
  Segmentation                                                                  
  Fluency                                                                       

  DIBELS Nonsense   27.1    911      35.4    975      41.1    783      \+       0.71
  Word Fluency                                                                  

  DIBELS Letter     44.6    924      47.8    978      52.4    793      \+       0.44
  Naming Fluency                                                                

  **Grade 1                                                                     
  winter**                                                                      

  DIBELS Phoneme    36.7    892      44.2    944      43.8    832      \+       0.45
  Segmentation                                                                  
  Fluency                                                                       

  DIBELS Nonsense   41.3    893      54.5    965      55.7    848      \+       0.62
  Word Fluency                                                                  

  DIBELS Oral       27.0    883      31.4    964      32.2    839      \+       0.21
  Reading Fluency                                                               

  **Grade 1                                                                     
  spring**                                                                      

  DIBELS Phoneme    41.7    884      45.9    947      45.4    757      \+       0.26
  Segmentation                                                                  
  Fluency                                                                       

  DIBELS Nonsense   51.1    885      60.1    947      61.7    757      \+       0.41
  Word Fluency                                                                  

  DIBELS Oral       36.6    884      41.9    945      43.3    748      \+       0.24
  Reading Fluency                                                               

  **Grade 2 fall**                                                              

  DIBELS Nonsense   41.9    901      50.4    925      58.4    782      \+       0.62
  Word Fluency                                                                  

  DIBELS Oral       47.8    895      51.9    923      55.1    768      \+       0.26
  Reading Fluency                                                               

  **Grade 2                                                                     
  winter**                                                                      

  DIBELS Oral       65.1    868      71.3    928      70.3    846      \+       0.16
  Reading Fluency                                                               

  **Grade 2                                                                     
  spring**                                                                      

  DIBELS Oral       74.4    877      79.7    905      79.9    754      \+       0.16
  Reading Fluency                                                               

  **Grade 3 fall**                                                              

  DIBELS Oral       64.7    888      73.3    946      78.7    779      \+       0.45
  Reading Fluency                                                               

  **Grade 3                                                                     
  winter**                                                                      

  DIBELS Oral       79.6    831      84.4    918      87.6    839      \+       0.24
  Reading Fluency                                                               

  **Grade 3                                                                     
  spring**                                                                      

  DIBELS Oral       87.6    816      93.2    879      98.0    752      \+       0.30
  Reading Fluency                                                               
  ----------------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y3.

Table III-9 presents a comparison of benchmark data over time for cohort
2. The percentages have gone up considerably over time. As of year 3,
more than half of the students were proficient in 21 of the 25
assessments. This is especially impressive for LNF in grade 1 fall, and
PSF grade 1 spring, where the percentage were above 80. However, the
percentages for ORF in grade 2 spring and grade 3 (winter and spring)
were below 50 percent.

Table III-9.---Comparison of DIBELS results for years 1--3 in RFO cohort
2: Percent proficient

  ---------------------------------------------- -------- ------- ------- ------- ------- --
  Assessment                                     Year 1   Year 2          Year 3          

  **Kindergarten fall**                                                                   

  DIBELS Initial Sound Fluency                   61.7%    72.2%           75.7%           

  DIBELS Letter Naming Fluency                   72.7     75.5            77.2            

  **Kindergarten winter**                                                                 

  DIBELS Initial Sound Fluency                   34.4     49.8            54.2            

  DIBELS Phoneme Segmentation Fluency            38.3     47.8            54.8            

  DIBELS Nonsense Word Fluency                   57.4     69.0            70.6            

  DIBELS Letter Naming Fluency                   67.9     74.4            77.6            

  **Kindergarten spring**                                                                 

  DIBELS Phoneme Segmentation Fluency            42.4     57.5            59.4            

  DIBELS Nonsense Word Fluency                   47.7     61.1            66.1            

  DIBELS Letter Naming Fluency                   54.5     62.5            62.8            

  **Grade 1 fall**                                                                        

  DIBELS Phoneme Segmentation Fluency            27.8     56.4            65.9            

  DIBELS Nonsense Word Fluency                   55.7     71.7            79.6            

  DIBELS Letter Naming Fluency                   69.8     74.3            82.5            

  **Grade 1 winter**                                                                      

  DIBELS Phoneme Segmentation Fluency            58.3     75.6            76.3            

  DIBELS Nonsense Word Fluency                   27.8     53.5            55.6            

  DIBELS Oral Reading Fluency                    50.2     59.1            60.9            

  **Grade 1 spring**                                                                      

  DIBELS Phoneme Segmentation Fluency            69.2     78.7            80.8            

  DIBELS Nonsense Word Fluency                   46.4     58.3            63.7            

  DIBELS Oral Reading Fluency                    37.9     46.2            46.4            

  **Grade 2 fall**                                                                        

  DIBELS Nonsense Word Fluency                   30.2     45.5            57.4            

  DIBELS Oral Reading Fluency                    51.1     58.7            63.2            

  **Grade 2 winter**                                                                      

  DIBELS Oral Reading Fluency                    47.9     56.8            53.3            

  **Grade 2 spring**                                                                      

  DIBELS Oral Reading Fluency                    35.3             38.7            43.8    

  **Grade 3 fall**                                                                        

  DIBELS Oral Reading Fluency                    33.3             44.1            53.7    

  **Grade 3 winter**                                                                      

  DIBELS Oral Reading Fluency                    34.8             40.1            43.6    

  **Grade 3 spring**                                                                      

  DIBELS Oral Reading Fluency                    27.3             31.2            39.6    
  ---------------------------------------------- -------- ------- ------- ------- ------- --

We examined performance for a longitudinal group of students (grade K in
2005) who had been with RFO for three consecutive years (N=358).
Comparison of the longitudinal group and the respective cross-sectional
group in Table III-10 shows that students in the longitudinal group
began at a higher achievement level in K and remained so throughout the
years than the cross-sectional group.

Table III-10.---Comparison of DIBELS results for cohort 2 longitudinal
and cross-sectional groups: Percent proficient

  ------------------------------------------------- -------------- -----------------
  Assessment                                        Longitudinal   Cross-sectional

  **Kindergarten fall**                                            

  DIBELS Initial Sound Fluency                      66.8%          61.7%

  DIBELS Letter Naming Fluency                      81.9           72.7

  **Kindergarten winter**                                          

  DIBELS Initial Sound Fluency                      39.5           34.4

  DIBELS Phoneme Segmentation Fluency               42.4           38.3

  DIBELS Nonsense Word Fluency                      67.1           57.4

  DIBELS Letter Naming Fluency                      76.1           67.9

  **Kindergarten spring**                                          

  DIBELS Phoneme Segmentation Fluency               45.5           42.4

  DIBELS Nonsense Word Fluency                      56.7           47.7

  DIBELS Letter Naming Fluency                      62.6           54.5

  **Grade 1 fall**                                                 

  DIBELS Phoneme Segmentation Fluency               60.6           56.4

  DIBELS Nonsense Word Fluency                      79.3           71.7

  DIBELS Letter Naming Fluency                      82.7           74.3

  **Grade 1 winter**                                               

  DIBELS Phoneme Segmentation Fluency               82.1           75.6

  DIBELS Nonsense Word Fluency                      60.4           53.5

  DIBELS Oral Reading Fluency                       68.8           59.1

  **Grade 1 spring**                                               

  DIBELS Phoneme Segmentation Fluency               83.5           78.7

  DIBELS Nonsense Word Fluency                      64.2           58.3

  DIBELS Oral Reading Fluency                       54.5           46.2

  **Grade 2 fall**                                                 

  DIBELS Nonsense Word Fluency                      63.1           57.4

  DIBELS Oral Reading Fluency                       66.5           63.2

  **Grade 2 winter**                                               

  DIBELS Oral Reading Fluency                       60.6           53.3

  **Grade 2 spring**                                               

  DIBELS Oral Reading Fluency                       49.4           43.8
  ------------------------------------------------- -------------- -----------------

Table III-11 shows the summary of the results disaggregated by whether
the student was African American or white. For cohort 2 between years 1
and 3, the sample size by assessment and period ranges from 562 to 710
for African Americans and 133 to 170 for whites. The sample sizes by
grade for Hispanic students in cohort 2 were less than 30 and were too
small for such analysis. Compared to the year 1 cohort,

-   African American students in year 3 scored significantly better on
    all 25 assessments. The effect sizes were modest to high (i.e., 0.14
    to 0.88).

-   White students in year 3 scored significantly better on 22 of the 25
    assessments and the effect sizes tended to be larger than the
    overall average, especially for 11 assessments where the sizes
    exceeded 0.50.

Table III-11.---Comparison of DIBELS results for years 1--3 in RFO
cohort 2, by race/ ethnicity

  ------------------------------------ ---------- -------- -------- --------
  Assessment                           African             White    
                                       American                     

                                       Sig (Y3-1) ES       Sig      ES
                                                  (Y3-1)   (Y3-1)   (Y3-1)

  **Kindergarten fall**                                             

  DIBELS Initial Sound Fluency         \+         0.53     \+       0.58

  DIBELS Letter Naming Fluency         \+         0.20     \+       0.52

  **Kindergarten winter**                                           

  DIBELS Initial Sound Fluency         \+         0.36     \+       0.33

  DIBELS Phoneme Segmentation Fluency  \+         0.54     \+       0.74

  DIBELS Nonsense Word Fluency         \+         0.35     \+       0.53

  DIBELS Letter Naming Fluency         \+         0.22     \+       0.53

  **Kindergarten spring**                                           

  DIBELS Phoneme Segmentation Fluency  \+         0.46     \+       0.51

  DIBELS Nonsense Word Fluency         \+         0.38     \+       0.38

  DIBELS Letter Naming Fluency         \+         0.31     \+       0.24

  **Grade 1 fall**                                                  

  DIBELS Phoneme Segmentation Fluency  \+         0.88     \+       0.90

  DIBELS Nonsense Word Fluency         \+         0.67     \+       0.80

  DIBELS Letter Naming Fluency         \+         0.44     \+       0.41

  DIBELS Phoneme Segmentation Fluency  \+         0.47     \+       0.28

  DIBELS Nonsense Word Fluency         \+         0.60     \+       0.56

  DIBELS Oral Reading Fluency          \+         0.20     o        0.15

  **Grade 1 spring**                                                

  DIBELS Phoneme Segmentation Fluency  \+         0.24     \+       0.29

  DIBELS Nonsense Word Fluency         \+         0.38     \+       0.40

  DIBELS Oral Reading Fluency          \+         0.23     \+       0.22

  **Grade 2 fall**                                                  

  DIBELS Nonsense Word Fluency         \+         0.56     \+       0.94

  DIBELS Oral Reading Fluency          \+         0.22     \+       0.48

  **Grade 2 winter**                                                

  DIBELS Oral Reading Fluency          \+         0.16     o        0.22

  **Grade 2 spring**                                                

  DIBELS Oral Reading Fluency          \+         0.14     \+       0.31

  **Grade 3 fall**                                                  

  DIBELS Oral Reading Fluency          \+         0.43     \+       0.53

  **Grade 3 winter**                                                

  DIBELS Oral Reading Fluency          \+         0.21     o        0.32

  **Grade 3 spring**                                                

  DIBELS Oral Reading Fluency          \+         0.27     \+       0.40
  ------------------------------------ ---------- -------- -------- --------

\+ = Statistically significant in favor of Y4; o = no significant
difference.

Table III-12 compares the data for students by special group status. For
cohort 2 between years 1 and 3, the sample size by assessment and period
ranged from 253 to 731 for FRL students; and 42 to 111 for special
education students. The numbers of LEP students by grade were too small
for such analysis. Compared to the year 1,

-   FRL students in year 3 scored significantly better on all 25
    assessments. The effect sizes were moderate to large.

-   Results for special education students were less impressive. Special
    education students in year 3 scored significantly better on 7 of the
    25 assessments. No differences were found in other assessments.

Table III-12.---Comparison of DIBELS results in year 1--3 in RFO cohort
2, by special service

  ------------------------------------ -------- -------- --------- --------
  Assessment                           FRL               Special   
                                                         Ed        

                                       Sig      ES       Sig       ES
                                       (Y3-1)   (Y3-1)   (Y3-1)    (Y3-1)

  **Kindergarten fall**                                            

  DIBELS Initial Sound Fluency         \+       0.50     \+        0.70

  DIBELS Letter Naming Fluency         \+       0.30     o         0.21

  **Kindergarten winter**                                          

  DIBELS Initial Sound Fluency         \+       0.14     o         0.10

  DIBELS Phoneme Segmentation Fluency  \+       0.21     o         0.19

  DIBELS Nonsense Word Fluency         \+       0.27     o         0.14

  DIBELS Letter Naming Fluency         \+       0.33     o         0.19

  **Kindergarten spring**                                          

  DIBELS Phoneme Segmentation Fluency  \+       0.16     o         0.10

  DIBELS Nonsense Word Fluency         \+       0.32     o         0.22

  DIBELS Letter Naming Fluency         \+       0.27     o         0.28

  **Grade 1 fall**                                                 

  DIBELS Phoneme Segmentation Fluency  \+       0.59     \+        0.63

  DIBELS Nonsense Word Fluency         \+       0.72     \+        0.56

  DIBELS Letter Naming Fluency         \+       0.53     o         0.15

  **Grade 1 winter**                                               

  DIBELS Phoneme Segmentation Fluency  \+       0.19     o         0.17

  DIBELS Nonsense Word Fluency         \+       0.55     o         0.17

  DIBELS Oral Reading Fluency          \+       0.28     o         -0.09

  **Grade 1 spring**                                               

  DIBELS Phoneme Segmentation Fluency  \+       0.10     o         0.25

  DIBELS Nonsense Word Fluency         \+       0.38     o         0.22

  DIBELS Oral Reading Fluency          \+       0.32     o         0.00

  **Grade 2 fall**                                                 

  DIBELS Nonsense Word Fluency         \+       0.62     \+        0.54

  DIBELS Oral Reading Fluency          \+       0.28     o         0.20

  **Grade 2 winter**                                               

  DIBELS Oral Reading Fluency          \+       0.21     o         0.03

  **Grade 2 spring**                                               

  DIBELS Oral Reading Fluency          \+       0.23     o         0.15

  **Grade 3 fall**                                                 

  DIBELS Oral Reading Fluency          \+       0.49     \+        0.68

  **Grade 3 winter**                                               

  DIBELS Oral Reading Fluency          \+       0.27     \+        0.32

  **Grade 3 spring**                                               

  DIBELS Oral Reading Fluency          \+       0.39     \+        0.54
  ------------------------------------ -------- -------- --------- --------

\+ = Statistically significant in favor of Y3; o = no significant
difference.

Cohort 3

Table III-13 presents comparison of data for years 1 and 2 (2006--07)
from RFO cohort 3. Students had higher levels of performance in year 2
than in year 1 on 19 of the 25 assessments. The differences were
statistically significant and the effect sizes were mostly moderate,
except for PSF (grade 1 fall) and NWF (grade 2 fall), when the effect
sizes exceeded 0.50.

Table III-13.---Comparison of DIBELS results in years 1--2 in RFO cohort
3: Raw score

  -------------------------- ------- -------- ------- -------- ---------- ----------
  Assessment                 Year 1           Year 2           Sig (Y2-   ES (Y2-1)
                                                               1)         

                             Raw     Number   Raw     Number              
                             score            score                       

  **Kindergarten fall**                                                   

  DIBELS Initial Sound       14.4    2,002    17.0    2,273    \+         0.25
  Fluency                                                                 

  DIBELS Letter Naming       22.7    2,006    23.9    2,274    \+         0.08
  Fluency                                                                 

  **Kindergarten winter**                                                 

  DIBELS Initial Sound       27.1    1,974    31.3    2,457    \+         0.21
  Fluency                                                                 

  DIBELS Phoneme             24.1    1,972    24.5    2,457    o          0.03
  Segmentation Fluency                                                    

  DIBELS Nonsense Word       22.4    1,965    21.0    2,455    o          -0.09
  Fluency                                                                 

  DIBELS Letter Naming       37.3    1,974    36.9    2,457    o          -0.02
  Fluency                                                                 

  **Kindergarten spring**                                                 

  DIBELS Phoneme             40.1    1,971    41.4    2,216    \+         0.09
  Segmentation Fluency                                                    

  DIBELS Nonsense Word       29.7    1,969    29.3    2,213    o          0.02
  Fluency                                                                 

  DIBELS Letter Naming       43.0    1,971    44.5    2,215    \+         0.09
  Fluency                                                                 

  **Grade 1 fall**                                                        

  DIBELS Phoneme             32.8    1,922    40.4    2,224    \+         0.51
  Segmentation Fluency                                                    

  DIBELS Nonsense Word       32.3    1,923    40.8    2,226    \+         0.39
  Fluency                                                                 

  DIBELS Letter Naming       46.4    1,924    51.1    2,226    \+         0.27
  Fluency                                                                 

  **Grade 1 winter**                                                      

  DIBELS Phoneme             44.7    1,900    46.8    2,343    \+         0.15
  Segmentation Fluency                                                    

  DIBELS Nonsense Word       54.4    1,903    57.3    2,363    \+         0.11
  Fluency                                                                 

  DIBELS Oral Reading        33.8    1,906    34.2    2,365    o          0.01
  Fluency                                                                 

  **Grade 1 spring**                                                      

  DIBELS Phoneme             47.2    1,889    48.2    2,182    \+         0.08
  Segmentation Fluency                                                    

  DIBELS Nonsense Word       62.5    1,884    64.9    2,181    \+         0.08
  Fluency                                                                 

  DIBELS Oral Reading        44.5    1,875    45.7    2,182    o          0.04
  Fluency                                                                 

  **Grade 2 fall**                                                        

  DIBELS Nonsense Word       46.0    1,973    62.4    2,088    \+         0.58
  Fluency                                                                 

  DIBELS Oral Reading        53.0    1,971    57.3    2,082    \+         0.14
  Fluency                                                                 

  **Grade 2 winter**                                                      

  DIBELS Oral Reading        71.4    1,956    74.6    2,257    \+         0.09
  Fluency                                                                 

  **Grade 2 spring**                                                      

  DIBELS Oral Reading        78.4    1,935    83.9    2,053    \+         0.15
  Fluency                                                                 

  **Grade 3 fall**                                                        

  DIBELS Oral Reading        74.2    1,878    80.6    2,187    \+         0.19
  Fluency                                                                 

  **Grade 3 winter**                                                      

  DIBELS Oral Reading        84.2    1,879    87.9    2,276    \+         0.10
  Fluency                                                                 

  **Grade 3 spring**                                                      

  DIBELS Oral Reading        95.2    1,858    100.2   2,152    \+         0.13
  Fluency                                                                 
  -------------------------- ------- -------- ------- -------- ---------- ----------

\+ = Statistically significant in favor of Y2; - = Statistically
significant in favor of Y1; o = no significant difference.

Table III-14 presents a comparison of benchmark data over time for
cohort 3. The percentages have gone up from year 1 to 2. As of year 2,
more than half of the students were proficient in 22 of the 25
assessments. The percentages were especially high (i.e., above 80
percent) for LNF (grade 1 fall) and PSF (grade 1 winter and spring) but
relatively low for ORF in grade 2 (spring) and grade 3 (winter and
spring), when less than half of the students were proficient.

Table III-14.---Comparison of DIBELS results in years 1--2 in RFO cohort
3: Percent proficient

  ------------------------------------------------ ----------- -----------
  Assessment                                       Year 1      Year 2

  **Kindergarten fall**                                        

  DIBELS Initial Sound Fluency                     76.3%       79.0%

  DIBELS Letter Naming Fluency                     76.3        78.7

  **Kindergarten winter**                                      

  DIBELS Initial Sound Fluency                     55.5        61.7

  DIBELS Phoneme Segmentation Fluency              65.4        63.7

  DIBELS Nonsense Word Fluency                     68.7        67.0

  DIBELS Letter Naming Fluency                     73.7        75.3

  **Kindergarten spring**                                      

  DIBELS Phoneme Segmentation Fluency              71.6        73.2

  DIBELS Nonsense Word Fluency                     58.6        59.4

  DIBELS Letter Naming Fluency                     61.4        62.9

  **Grade 1 fall**                                             

  DIBELS Phoneme Segmentation Fluency              49.2        70.8

  DIBELS Nonsense Word Fluency                     67.1        78.8

  DIBELS Letter Naming Fluency                     74.8        81.8

  **Grade 1 winter**                                           

  DIBELS Phoneme Segmentation Fluency              80.2        85.1

  DIBELS Nonsense Word Fluency                     53.7        58.3

  DIBELS Oral Reading Fluency                      62.7        60.4

  **Grade 1 spring**                                           

  DIBELS Phoneme Segmentation Fluency              87.6        88.9

  DIBELS Nonsense Word Fluency                     66.2        66.4

  DIBELS Oral Reading Fluency                      47.9        50.0

  **Grade 2 fall**                                             

  DIBELS Nonsense Word Fluency                     37.1        64.0

  DIBELS Oral Reading Fluency                      56.3        63.2

  **Grade 2 winter**                                           

  DIBELS Oral Reading Fluency                      53.9        58.7

  **Grade 2 spring**                                           

  DIBELS Oral Reading Fluency                      41.0        47.2

  **Grade 3 fall**                                             

  DIBELS Oral Reading Fluency                      47.2        54.8

  **Grade 3 winter**                                           

  DIBELS Oral Reading Fluency                      42.0        45.5

  **Grade 3 spring**                                           

  DIBELS Oral Reading Fluency                      38.2        44.3
  ------------------------------------------------ ----------- -----------

Table III-15 shows the summary of the results disaggregated by race and
ethnicity. For cohort 3 between years 1 and 2, the sample size by
assessment and period ranges from 473 to 572 for African Americans; 46
to 82 for Hispanics; and 1,210 to 1,626 for whites. Compared to the year
1,

-   African American students in year 2 scored significantly higher on
    16 of the 25 assessments. The effect sizes were small to moderate.

-   Hispanic students in year 2 scored significantly higher on 7 of the
    25 assessments. The effect sizes varied greatly.

-   White students in year 2 scored significantly better on 16 and worse
    on 3 of the 25 assessments. The effect sizes were small to moderate.

Table III-15.---Comparison of DIBELS results for years 1--2 in RFO
cohort 3, by race/ ethnicity

  --------------------------- ---------- -------- ----------- -------- -------- --------
  Assessment                  African             Hispanics            White    
                              American                                          

                              Sig\       ES\      Sig\        ES\      Sig\     ES\
                              (Y2-1)     (Y2-1)   (Y2-1)      (Y2-1)   (Y2-1)   (Y2-1)

  **Kindergarten fall**                                                         

  DIBELS Initial Sound        \+         0.41     \+          0.35     \+       0.20
  Fluency                                                                       

  DIBELS Letter Naming        \+         0.17     o           0.11     o        0.04
  Fluency                                                                       

  **Kindergarten winter**                                                       

  DIBELS Initial Sound        \+         0.34     o           -0.08    \+       0.18
  Fluency                                                                       

  DIBELS Phoneme Segmentation o          0.04     o           -0.24    o        0.02
  Fluency                                                                       

  DIBELS Nonsense Word        o          0.08     o           -0.06    \-       -0.15
  Fluency                                                                       

  DIBELS Letter Naming        \+         0.19     o           0.10     \-       -0.10
  Fluency                                                                       

  **Kindergarten spring**                                                       

  DIBELS Phoneme Segmentation \+         0.26     o           -0.03    o        0.01
  Fluency                                                                       

  DIBELS Nonsense Word        \+         0.22     o           -0.12    \-       -0.11
  Fluency                                                                       

  DIBELS Letter Naming        \+         0.31     o           0.20     o        -0.01
  Fluency                                                                       

  **Grade 1 fall**                                                              

  DIBELS Phoneme Segmentation \+         0.42     \+          0.60     \+       0.54
  Fluency                                                                       

  DIBELS Nonsense Word        \+         0.42     \+          0.46     \+       0.37
  Fluency                                                                       

  DIBELS Letter Naming        \+         0.22     \+          0.40     \+       0.26
  Fluency                                                                       

  **Grade 1 winter**                                                            

  DIBELS Phoneme Segmentation o          0.14     o           0.17     \+       0.18
  Fluency                                                                       

  DIBELS Nonsense Word        \+         0.17     \+          0.31     \+       0.08
  Fluency                                                                       

  DIBELS Oral Reading Fluency o          0.04     o           0.13     o        0.01

  **Grade 1 spring**                                                            

  DIBELS Phoneme Segmentation o          0.06     o           -0.15    \+       0.10
  Fluency                                                                       

  DIBELS Nonsense Word        o          0.06     o           0.13     \+       0.10
  Fluency                                                                       

  DIBELS Oral Reading Fluency o          0.03     o           0.18     \+       0.05

  **Grade 2 fall**                                                              

  DIBELS Nonsense Word        \+         0.50     \+          0.70     \+       0.59
  Fluency                                                                       

  DIBELS Oral Reading Fluency \+         0.19     o           0.05     \+       0.10

  **Grade 2 winter**                                                            

  DIBELS Oral Reading Fluency \+         0.18     o           0.12     o        0.03
  --------------------------- ---------- -------- ----------- -------- -------- --------

Table III-15.---Comparison of DIBELS results for years 1--2 in RFO
cohort 3, by race/ ethnicity---continued

  --------------------------- ---------- -------- ----------- -------- -------- --------
  Assessment                  African             Hispanics            White    
                              American                                          

                              Sig\       ES\      Sig\        ES\      Sig\     ES\
                              (Y2-1)     (Y2-1)   (Y2-1)      (Y2-1)   (Y2-1)   (Y2-1)

  **Grade 2 spring**                                                            

  DIBELS Oral Reading Fluency \+         0.23     o           0.10     \+       0.09

  **Grade 3 fall**                                                              

  DIBELS Oral Reading Fluency \+         0.20     \+          0.43     \+       0.16

  **Grade 3 winter**                                                            

  DIBELS Oral Reading Fluency o          0.12     o           0.15     \+       0.08

  **Grade 3 spring**                                                            

  DIBELS Oral Reading Fluency o          0.12     o           0.09     \+       0.12
  --------------------------- ---------- -------- ----------- -------- -------- --------

\+ = Statistically significant in favor of Y2; - = statistically
significant in favor of Y1; o = no significant difference.

Table III-16 compares the data for students by special service status.
For cohort 3 between years 1 and 2, the sample size by assessment and
period ranges from 985 to 1,574 for FRL students; 68 to 94 for LEP
students; and 162 to 347 for special education students. Compared to
year 1,

-   FRL students in year 2 scored significantly better on 18 of the 25
    assessments. The effect sizes were modest in general.

-   LEP students in year 2 scored significantly better on 8 of the 25
    assessments. The effect sizes were modest in general.

-   Special education students in year 2 scored significantly better on
    12 of the 25 assessments. The effect sizes were small to moderate.

Table III-16.---Comparison of DIBELS results for years 1--2 in RFO
cohort 3, by special service

  -------------------------- -------- -------- -------- -------- --------- --------
  Assessment                 FRL               LEP               Special   
                                                                 Ed        

                             Sig\     ES\      Sig\     ES\      Sig\      ES\
                             (Y2-1)   (Y2-1)   (Y2-1)   (Y2-1)   (Y2-1)    (Y2-1)

  **Kindergarten fall**                                                    

  DIBELS Initial Sound       \+       0.25     o        0.21     \+        0.27
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.07     o        -0.03    \+        0.28
  Fluency                                                                  

  **Kindergarten winter**                                                  

  DIBELS Initial Sound       \+       0.23     o        -0.19    \+        0.28
  Fluency                                                                  

  DIBELS Phoneme             \+       0.11     o        -0.30    o         0.18
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       o        -0.06    o        -0.07    o         0.06
  Fluency                                                                  

  DIBELS Letter Naming       o        0.03     o        0.05     \+        0.17
  Fluency                                                                  

  **Kindergarten spring**                                                  

  DIBELS Phoneme             \+       0.13     o        -0.04    \+        0.24
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       o        0.01     o        -0.09    o         0.21
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.15     o        0.21     \+        0.24
  Fluency                                                                  

  **Grade 1 fall**                                                         

  DIBELS Phoneme             \+       0.46     \+       0.64     \+        0.54
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.39     \+       0.52     \+        0.43
  Fluency                                                                  

  DIBELS Letter Naming       \+       0.26     \+       0.36     \+        0.38
  Fluency                                                                  
  -------------------------- -------- -------- -------- -------- --------- --------

Table III-16.---Comparison of DIBELS results for years 1--2 in RFO
cohort 3, by special service---continued

  -------------------------- -------- -------- -------- -------- --------- --------
  Assessment                 FRL               LEP               Special   
                                                                 Ed        

                             Sig\     ES\      Sig\     ES\      Sig\      ES\
                             (Y2-1)   (Y2-1)   (Y2-1)   (Y2-1)   (Y2-1)    (Y2-1)

  **Grade 1 winter**                                                       

  DIBELS Phoneme             \+       0.14     o        -0.06    \+        0.19
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       \+       0.12     \+       0.25     o         0.11
  Fluency                                                                  

  DIBELS Oral Reading        o        0.00     o        0.21     o         0.13
  Fluency                                                                  

  **Grade 1 spring**                                                       

  DIBELS Phoneme             \+       0.08     o        -0.08    o         0.20
  Segmentation Fluency                                                     

  DIBELS Nonsense Word       o        0.06     \+       0.27     o         0.09
  Fluency                                                                  

  DIBELS Oral Reading        o        0.02     \+       0.32     o         0.13
  Fluency                                                                  

  **Grade 2 fall**                                                         

  DIBELS Nonsense Word       \+       0.56     \+       0.57     \+        0.44
  Fluency                                                                  

  DIBELS Oral Reading        \+       0.16     o        0.11     o         0.05
  Fluency                                                                  

  **Grade 2 winter**                                                       

  DIBELS Oral Reading        \+       0.11     o        0.23     o         -0.02
  Fluency                                                                  

  **Grade 2 spring**                                                       

  DIBELS Oral Reading        \+       0.17     \+       0.31     o         0.01
  Fluency                                                                  

  **Grade 3 fall**                                                         

  DIBELS Oral Reading        \+       0.15     o        0.29     \+        0.12
  Fluency                                                                  

  **Grade 3 winter**                                                       

  DIBELS Oral Reading        o        0.08     o        0.15     o         0.06
  Fluency                                                                  

  **Grade 3 spring**                                                       

  DIBELS Oral Reading        \+       0.11     o        0.01     o         0.08
  Fluency                                                                  
  -------------------------- -------- -------- -------- -------- --------- --------

\+ = Statistically significant in favor of Y2; o = no significant
difference.

Performance on the TerraNova and the OAT-R

The TerraNova is a widely used standardized achievement test that
provides a summative assessment of reading performance. In RFO schools
it is administered at the end of grades 1, 2, and 3. The Ohio
Achievement Test in Reading was developed specifically to assess
attainment of Ohio's standards in English language arts. All schools use
the OAT-R starting in grade 3.

Similar to the analyses of DIBELS data, we looked at student performance
on the TerraNova and the OAT-R in a variety of ways. For the three
cohorts of participating schools, we performed the following analyses:

-   We examined student achievement data by year in scale scores to
    illustrate trends over time. Within each cohort, we estimated both
    the statistical significance using a 3-level HLM (i.e., students
    nested with year with schools) and the effect size of differences
    between 2007 and year 1 achievement.

-   We looked at the descriptive statistics for students' achieving
    grade-level RFO benchmarks. Two benchmarks have been used for
    examining student performance in RFO for TerraNova results: the
    percentage of students scoring at or above the 40th percentile, and
    the percentage of students scoring at or above the 50th percentile.
    For OAT-R, the cutoff scale score for "proficient" is 400.

-   In addition to analyzing the data for the overall population, we
    also examined subgroups by student race/ethnicity,[^31] receipt of
    free or reduced-price lunch, English proficiency level and special
    education status, as well as the urbanicity of the district in which
    the schools were located.

**We found statistically significant improvement in TerraNova and OAT-R
for all grade levels from baseline to 2006--07 for all three cohorts.
The improvement, although sometimes fluctuating, was moderate in
magnitude and seemed to slow down after year 2. For TerraNova, more than
half of the students in cohort 3 performed at or above the 50th
percentile. However, cohorts 1 and 2 only had an average of 40 percent
of students at or above the same benchmark. In general, no consistent
patterns were found from students in different racial/ethnic groups,
economic statuses, or locales. On the OAT-R, performance gaps among
whites, Hispanics, and African Americans were closed for cohort 1 and 3
students.**[^32] **In addition, we found clear evidence that students
with limited English proficiency and special education students have
made larger improvement than those students who did not receive special
services. Further, for cohorts 1 and 2 the gap between RFO schools and
the state overall was reduced by 5 percent and 2 percent,
respectively.**

Table III-17 provides a summary of the findings.

Table III-17.---Summary of TerraNova and OAT-R results for cohorts 1--3

  --------------------------------------------- -------- -------- --------
  Assessment                                    Cohort 1 Cohort 2 Cohort 3

  **Overall**                                                     

  TerraNova Grade 1                             \+       \+       o

  TerraNova Grade 2                             \+       o        \+

  TerraNova Grade 3                             \+       o        \+

  OAT-R Grade 3 spring                          \+       o        o

  **African American**                                            

  TerraNova Grade 1                             o        \+       o

  TerraNova Grade 2                             o        o        o

  TerraNova Grade 3                             \+       o        \+

  OAT-R Grade 3 spring                          \+       o        \+

  **Hispanic**                                                    

  TerraNova Grade 1                             o        na       \+

  TerraNova Grade 2                             o        na       o

  TerraNova Grade 3                             o        na       o

  OAT-R Grade 3 spring                          \+       na       \+

  **White**                                                       

  TerraNova Grade 1                             \+       \+       o

  TerraNova Grade 2                             \+       o        o

  TerraNova Grade 3                             \+       \+       \+

  OAT-R Grade 3 spring                          o        o        \-
  --------------------------------------------- -------- -------- --------

Table III-17.---Summary of TerraNova and OAT-R results for cohorts
1--3---continued

  --------------------------------------------- -------- -------- --------
  Assessment                                    Cohort 1 Cohort 2 Cohort 3

  **FRL**                                                         

  TerraNova Grade 1                             \+       \+       o

  TerraNova Grade 2                             \+       o        o

  TerraNova Grade 3                             \+       \+       \+

  OAT-R Grade 3 spring                          \+       \+       \+

  **LEP**                                                         

  TerraNova Grade 1                             \+       na       o

  TerraNova Grade 2                             \+       na       \+

  TerraNova Grade 3                             o        na       \+

  OAT-R Grade 3 spring                          \+       na       o

  **Special education**                                           

  TerraNova Grade 1                             \+       o        o

  TerraNova Grade 2                             \+       o        o

  TerraNova Grade 3                             \+       \+       \+

  OAT-R Grade 3 spring                          \+       o        \+
  --------------------------------------------- -------- -------- --------

\+ = Statistically significant in favor of current year; - =
statistically significant in favor of Y1; o = no significant difference;
na=not available due to small sample size at school level.

Presented below are detailed findings by cohort.

Cohort 1

Table III-18 and Figure III-2 present comparisons of data for years 1--4
(2004--07) for RFO cohort 1. In general, students had higher levels of
performance in year 4 than in year 1 across all grades. The differences
were statistically significant, and the effect sizes were around 0.20.
However, it appears that the most visible improvement occurred from year
1 to year 2, after which it became stable.

Table III-18.---Comparison of TerraNova and OAT-R results for years 1--4
in RFO\
cohort 1: Scale scores

  --------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment      Year 1           Year 2           Year 3           Year 4           Sig      ES\
                                                                                      (Y4-1)   (Y4-1)

                  Scale   Number   Scale   Number   Scale   Number   Scale   Number            
                  score            score            score            score                     

  TerraNova Grade 549.9   2,786    557.3   2,139    559.4   2,379    558.1   2,112    \+       .19
  1                                                                                            

  TerraNova Grade 585.4   2,373    592.9   1,961    588.7   2,327    592.3   2,032    \+       .18
  2                                                                                            

  TerraNova Grade 605.9   2,415    618.6   1,901    617.0   2,178    615.6   1,963    \+       .26
  3                                                                                            

  OAT-R Grade 3   397.8   2,386    404.0   1,899    400.9   2,145    402.5   2,089    \+       .17
  spring                                                                                       
  --------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y4.

**Figure III-2.---Four-year trends for RFO cohort 1 TerraNova
assessments**

![](media/image28.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

Next, we examined the percentage of students meeting certain benchmarks
for TerraNova and OAT-R. As shown in Table III-19, those percentages
fluctuated over time. In year 4, slightly more than half of the students
in RFO cohort 1 schools performed at or above the 40th percentile; the
percentages of students scoring at or above the 50th percentile were
between 36 and 44 percent for TerraNova. For OAT-R, the percentage
proficient and above was 58 percent.

**Table III-19.---Comparison of TerraNova and OAT-R results for years
1--4 in RFO\
cohort 1: Percent at benchmark**

  -------------------------------------- -------- ------- ------- -------
  Assessment                             Year 1   Year 2  Year 3  Year 4

  **At or above 40th percentile**                                 

  TerraNova Grade 1                      53.1%    58.1%   61.5%   54.4%

  TerraNova Grade 2                      48.8     57.5    51.8    52.6

  TerraNova Grade 3                      42.1     54.0    53.7    50.6

                                                                  

  **At or above 50th percentile**                                 

  TerraNova Grade 1                      42.4     48.2    50.9    43.8

  TerraNova Grade 2                      38.2     45.8    40.5    40.7

  TerraNova Grade 3                      29.9     42.0    40.6    36.3

                                                                  

  **OAT-R Grade 3 spring (percent        49.5     59.3    50.8    58.3
  proficient or above)**                                          
  -------------------------------------- -------- ------- ------- -------

As shown in Figure III-3, the gap in OAT-R between RFO cohort 1 students
and state average was narrowed by 8 percent \[(75.1-49.5)-(76.0-58.3) =
79\] as of year 4 and the improvement was statistically significant.

**Figure III-3.---Comparison of 4-year trends for OAT-R assessments of
RFO cohort 1 and the state average**

![](media/image29.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

We examined performance for a longitudinal group of students (grade 1 in
2004) who had been with RFO for three consecutive years (N=905).
Comparison of TerraNova scores for the longitudinal group and the
respective cross-sectional group indicate that although the longitudinal
group scored higher at each test point, there were no differences in the
growth rate (Table III-20).

Table III-20.---Comparison of TerraNova results for the cohort 1
longitudinal and cross-sectional groups

  ------------------------------------------------- -------------- -----------------
  Assessment                                        Longitudinal   Cross-sectional

  **Average scale score**                                          

  TerraNova Grade 1                                 564.7          549.9

  TerraNova Grade 2                                 600.8          592.9

  TerraNova Grade 3                                 624.1          617.0

                                                                   

  **At or above 40th percentile**                                  

  TerraNova Grade 1                                 65.2%          53.1%

  TerraNova Grade 2                                 65.6           57.5

  TerraNova Grade 3                                 60.4           53.7

                                                                   

  **At or above 50th percentile**                                  

  TerraNova Grade 1                                 54.1           42.4

  TerraNova Grade 2                                 53.4           45.8

  TerraNova Grade 3                                 48.0           40.6
  ------------------------------------------------- -------------- -----------------

Finally, we analyzed trends in student achievement by subgroups. Table
III-21 presents the achievement data for cohort 1 students disaggregated
by racial/ethnical group.

-   For both African American and Hispanic students, the increases were
    moderate for TerraNova. However, performance on the OAT-R appears to
    indicate more improvement.

-   For white students, the trends were very similar to the overall
    pattern for TerraNova, except that the effect sizes were larger.
    However, there was no improvement in OAT-R performance.

-   The gaps in achievement levels between minority students (African
    American, Hispanic) and white students increased for TerraNova but
    disappeared for OAT-R by year 4.

Table III-21.---Comparison of TerraNova and OAT-R results for years 1--4
in RFO\
cohort 1, by race/ ethnicity: Scale scores

  --------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------
  Group and       Year 1           Year 2           Year 3           Year 4           Sig\     ES\
  assessment                                                                          (Y4-1)   (Y4-1)

                  Scale   Number   Scale   Number   Scale   Number   Scale   Number            
                  score            score            score            score                     

  **African                                                                                    
  American**                                                                                   

  TerraNova Grade 545.2   1,329    552.1   918      552.4   1,106    548.4   897      o        0.08
  1                                                                                            

  TerraNova Grade 579.5   1,214    587.7   809      581.6   1,096    583.8   878      o        0.12
  2                                                                                            

  TerraNova Grade 599.1   1,300    609.8   810      609.5   1,016    607.8   861      \+       0.25
  3                                                                                            

  OAT-R Grade 3   394.1   1,293    399.9   815      397.0   998      402.9   869      \+       0.34
  spring                                                                                       

  **Hispanic**                                                                                 

  TerraNova Grade 538.1   336      542.5   219      549.1   259      541.3   230      o        0.07
  1                                                                                            

  TerraNova Grade 578.0   258      583.4   212      572.6   244      582.4   234      o        0.12
  2                                                                                            

  TerraNova Grade 598.3   239      612.4   187      608.9   196      603.4   202      o        0.13
  3                                                                                            

  OAT-R Grade 3   389.7   242      399.0   185      396.0   187      400.8   209      \+       0.33
  spring                                                                                       

  **White**                                                                                    

  TerraNova Grade 560.9   969      565.9   888      570.5   885      572.5   829      \+       0.27
  1                                                                                            

  TerraNova Grade 595.5   806      600.4   834      601.6   865      604.0   795      \+       0.23
  2                                                                                            

  TerraNova Grade 618.5   780      628.5   803      627.7   846      626.7   794      \+       0.22
  3                                                                                            

  OAT-R Grade 3   405.9   763      409.4   795      406.7   842      402.0   897      o        -0.14
  spring                                                                                       
  --------------- ------- -------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y4; - = statistically
significant in favor of Y1; o = no significant difference.

Table III-22 presents the achievement data for cohort 1 students
disaggregated by special service status. Compared to year 1,

-   For FRL students, the trends were similar to the overall pattern for
    TerraNova. However, results for OAT-R showed more improvement.

-   For LEP students, the increase in achievement level was substantial
    for TerraNova grades 1 and 2 and OAT-R, showing that the gaps have
    been considerably narrowed; TerraNova grade 3 results were similar
    to the overall pattern.

-   Special education students experienced large increases in
    achievement level across the board, thus decreasing the gaps in
    achievement levels with non-special-education students.

Table III-22.---Comparison of TerraNova and OAT-R results for years 1--4
in RFO\
cohort 1, by special service: Scale scores

  ---------------- ------- -------- ------- -------- ------- -------- ------- -------- ------- -------
  Status and       Year 1           Year 2           Year 3           Year 4           Sig\    ES\
  assessment                                                                           (Y4-1   (Y4-1

                   Scale   Number   Scale   Number   Scale   Number   Scale   Number           
                   score            score            score            score                    

  **FRL**                                                                                      

  TerraNova Grade  545.3   2,119    553.0   1,532    555.3   1,805    553.6   1,509    \+      0.19
  1                                                                                            

  TerraNova Grade  582.7   1,774    589.1   1,413    584.1   1,796    589.0   1,498    \+      0.17
  2                                                                                            

  TerraNova Grade  604.3   1,821    615.6   1,370    614.5   1,680    611.1   1,452    \+      0.19
  3                                                                                            

  OAT-R Grade 3    395.8   1,735    401.5   1,367    399.2   1,657    402.8   1,552    \+      0.26
  spring                                                                                       

  **LEP**                                                                                      

  TerraNova Grade  511.9   143      521.7   93       520.5   94       525.1   88       \+      0.32
  1                                                                                            

  TerraNova Grade  555.6   116      565.6   94       554.9   107      567.2   94       \+      0.32
  2                                                                                            

  TerraNova Grade  581.2   129      596.2   80       585.2   87       589.1   98       o       0.21
  3                                                                                            

  OAT-R Grade 3    376.0   121      392.0   80       380.7   82       401.9   78       \+      0.70
  spring                                                                                       

  **Special                                                                                    
  education**                                                                                  

  TerraNova Grade  519.4   294      534.0   207      537.6   235      537.5   126      \+      0.41
  1                                                                                            

  TerraNova Grade  556.6   275      565.3   209      563.5   247      571.3   162      \+      0.38
  2                                                                                            

  TerraNova Grade  583.3   310      598.8   245      600.5   259      597.4   183      \+      0.40
  3                                                                                            

  OAT-R Grade 3    382.6   343      389.8   245      389.6   240      401.7   267      \+      0.75
  spring                                                                                       
  ---------------- ------- -------- ------- -------- ------- -------- ------- -------- ------- -------

\+ = Statistically significant in favor of Y4; o = no significant
difference.

With regard to differences in student achievement by locale, Table
III-23 shows that while students in urban areas experienced improvement
from year 1 to 4 for TerraNova and the OAT-R, students from the suburbs
and towns did not. The achievement gaps among the various locales are
largely unchanged, with the achievement level in rural area being the
highest followed by suburban and town, and urban. The sample size at
school level for rural area is too small for the HLM model.

**[Table III-23.---Comparison of TerraNova and OAT-R results for years
1--4 in RFO\
cohort 1, by locale: Scale scores]{.mark}**

  ---------------- ------- -------- ------- -------- ------- -------- ------- -------- ------ --------
  Locale and       Year 1           Year 2           Year 3           Year 4           Sig.   ES\
  assessment                                                                                  Y3-Y1)

                   Scale   Number   Scale   Number   Scale   Number   Scale   Number          
                   score            score            score            score                   

  **Urban**                                                                                   

  TerraNova Grade  544.7   1,779    550.4   1,299    551.6   1,499    550.6   1,229    \+     0.14
  1                                                                                           

  TerraNova Grade  580.5   1,713    588.8   1,128    582.4   1,444    586.1   1,179    \+     0.15
  2                                                                                           

  TerraNova Grade  601.1   1,762    612.1   1,088    611.7   1,331    609.4   1,139    \+     0.23
  3                                                                                           

  OAT-R Grade 3    393.7   1,671    401.9   1,087    397.3   1,305    402.4   1,134    \+     0.32
  spring                                                                                      

  **Suburban and                                                                              
  town**                                                                                      

  TerraNova Grade  551.8   657      562.2   522      568.8   563      562.1   588      o      0.22
  1                                                                                           

  TerraNova Grade  592.7   351      592.9   511      594.1   584      596.5   543      o      0.10
  2                                                                                           

  TerraNova Grade  613.6   354      625.4   514      622.8   532      619.5   522      o      0.16
  3                                                                                           

  OAT-R Grade 3    404.2   419      403.8   523      404.5   528      403.7   547      o      -0.02
  spring                                                                                      

  **Rural**                                                                                   

  TerraNova Grade  572.8   350      577.3   318      579.5   317      581.6   295      na     0.21
  1                                                                                           

  TerraNova Grade  604.6   309      607.0   322      608.5   299      608.6   310      na     0.12
  2                                                                                           

  TerraNova Grade  625.4   299      630.8   299      629.3   315      632.1   302      na     0.19
  3                                                                                           

  OAT-R Grade 3    412.2   296      412.3   289      409.5   312      401.0   408      na     -0.40
  spring                                                                                      
  ---------------- ------- -------- ------- -------- ------- -------- ------- -------- ------ --------

\+ = Statistically significant in favor of Y4; o = no significant
difference; na=not available due to small sample size at school level.

Cohort 2

Table III-24 and Figure III-4 present comparisons for years 1--3 for
cohort 2. Compared to the baseline, 1st graders in year 3 experienced
sizable and continuing increases in TerraNova, but the improvement in
other grade levels was modest and statistically insignificant. It is
also important to note a large drop in TerraNova grade 3 scores from
year 2 to year 3.

Table III-24.---Comparison of TerraNova and OAT-R results for years 1--3
in RFO\
cohort 2: Scale scores

  ---------------- ------- -------- ------- -------- ------- -------- -------- --------
  Assessment       Year 1           Year 2           Year 3           Sig\     ES\
                                                                      (Y3-1)   (Y3-1)

                   Scale   Number   Scale   Number   Scale   Number            
                   score            score            score                     

  TerraNova Grade  552.2   803      559.5   859      563.7   779      \+       .28
  1                                                                            

  TerraNova Grade  588.3   769      593.8   871      591.1   774      o        .08
  2                                                                            

  TerraNova Grade  607.2   726      622.8   857      612.6   803      o        .15
  3                                                                            

  OAT-R Grade 3    400.8   696      399.8   798      403.0   817      o        .08
  spring                                                                       
  ---------------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y3; o = no significant
difference.

**Figure III-4.---Three-year trends for RFO cohort 2 TerraNova
assessments**

![](media/image30.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

Analyses against the 40th and 50th percentile benchmarks for TerraNova
(Table III-25) show an increasing percentage of 1st graders meeting the
benchmarks. However, for 3rd grade students, the percentage meeting
benchmarks decreased from the previous year after a large increase from
year 1 to year 2.

For OAT-R, 60 percent of the 3rd graders were at or above proficient by
year 3. Comparison with the state performance in Figure III-5 shows that
the gap was reduced by 3 percent and the reduction was statistically
significant.

**Table III-25.---Comparison of TerraNova and OAT-R results for years
1--3 in RFO\
cohort 2: Percent at benchmark**

  --------------------------------------------- ------- -------- --------
  Assessment                                    Year 1  Year 2   Year 3

  **At or above 40th percentile**                                

  TerraNova Grade 1                             55.2%   62.5%    61.9%

  TerraNova Grade 2                             51.9    58.2     53.7

  TerraNova Grade 3                             42.6    61.1     48.4

  **At or above 50th percentile**                                

  TerraNova Grade 1                             44.7    50.8     52.1

  TerraNova Grade 2                             39.5    44.9     43.7

  TerraNova Grade 3                             29.8    48.0     33.6

                                                                 

  **OAT-R Grade 3 spring (percent proficient or 56.5    47.9     60.2
  above)**                                                       
  --------------------------------------------- ------- -------- --------

**Figure III-5.---Comparison of 3-year trends for OAT-R assessments of
RFO cohort 2 and the state average**

![](media/image31.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

Then, we examined performance for a longitudinal group of students
(grade 1 in 2005) who had been with RFO for three consecutive years
(N=386). Comparison of TerraNova scores for the longitudinal group and
the respective cross-sectional group did not show differences in the
growth rate (Table III-26). Students in the longitudinal group began at
a higher achievement level in grade 1 and remained at a higher level by
grade 3 than the cross-sectional group.

[]{.mark}

Table III-26.---Comparison of TerraNova results for the cohort 2
longitudinal and cross-sectional groups

  ------------------------------------------------ -------------- -----------------
  Assessment                                       Longitudinal   Cross-sectional

  **Average scale scores**                                        

  TerraNova Grade 1                                562.1          552.2

  TerraNova Grade 2                                596.7          593.8

  TerraNova Grade 3                                616.1          612.6

                                                                  

  **At or above 40th percentile**                                 

  TerraNova Grade 1                                66.7%          55.2%

  TerraNova Grade 2                                62.4           58.2

  TerraNova Grade 3                                50.6           48.4

  **At or above 50th percentile**                                 

  TerraNova Grade 1                                54.4           44.7

  TerraNova Grade 2                                48.4           44.9

  TerraNova Grade 3                                37.0           33.6
  ------------------------------------------------ -------------- -----------------

Table III-27 shows the achievement data disaggregated by racial/ethnic
group. The pattern for African Americans was identical to the overall
pattern. For white students, the improvement seemed to be greater than
the overall. The sample size for Hispanics is too small for a
significance test. Interestingly, the typical differences between
racial/ethnic groups were not found in this cohort, i.e., the three
racial groups in cohort 2 performed at similar levels.

Table III-27.---Comparison of TerraNova and OAT-R results for years 1--3
in RFO\
cohort 2, by race/ ethnicity: Scale scores

  --------------- ------- -------- ------- -------- ------- -------- -------- --------
  Group and       Year 1           Year 2           Year 3           Sig\     ES\
  assessment                                                         (Y3-1)   (Y3-1)

                  Scale   Number   Scale   Number   Scale   Number            
                  score            score            score                     

  **African                                                                   
  American**                                                                  

  TerraNova Grade 551.1   615      559.1   649      562.7   587      \+       .29
  1                                                                           

  TerraNova Grade 590.0   586      593.9   657      590.8   585      o        .05
  2                                                                           

  TerraNova Grade 609.3   568      621.7   647      613.2   608      o        .11
  3                                                                           

  OAT-R Grade 3   402.5   556      399.6   609      403.3   592      o        .03
  spring                                                                      

  **White**                                                                   

  TerraNova Grade 556.7   148      561.3   158      565.8   145      \+       .22
  1                                                                           

  TerraNova Grade 583.0   136      592.4   161      591.6   143      o        .22
  2                                                                           

  TerraNova Grade 597.9   127      624.6   153      611.2   149      \+       .33
  3                                                                           

  OAT-R Grade 3   392.1   116      400.8   139      402.1   184      o        .36
  spring                                                                      
  --------------- ------- -------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y3; o = no significant
difference.

Table III-28 compares the data for students based on whether they
received special services such as FRL and special education
assistance.[^33] It appears that FRL students showed greater improvement
than the overall in grade 3 from both TerraNova and OAT-R. For special
education students with disabilities, we found greater improvement
across the board in terms of effect sizes. Consequently, the gaps with
non-special-education students decreased over time.

Table III-28.---Comparison of TerraNova and OAT-R results for years 1--3
in RFO\
cohort 2, by special service: Scale scores

  --------------- ------- -------- ------- -------- ------- -------- ------- -------
  Status and      Year 1           Year 2           Year 3           Sig\    ES\
  assessment                                                         (Y3-1   (Y3-1

                  Scale   Number   Scale   Number   Scale   Number           
                  score            score            score                    

  **FRL**                                                                    

  TerraNova Grade 550.2   602      559.4   748      562.4   675      \+      .31
  1                                                                          

  TerraNova Grade 587.8   498      592.5   694      589.2   674      o       .04
  2                                                                          

  TerraNova Grade 603.4   395      622.6   688      611.2   686      \+      .22
  3                                                                          

  OAT-R Grade 3   397.5   364      399.5   654      403.0   713      \+      .19
  spring                                                                     

  **Special                                                                  
  education**                                                                

  TerraNova Grade 532.2   73       541.8   66       540.4   62       o       .22
  1                                                                          

  TerraNova Grade 559.4   75       576.2   64       569.3   73       o       .26
  2                                                                          

  TerraNova Grade 583.2   79       602.7   79       595.4   78       \+      .36
  3                                                                          

  OAT-R Grade 3   388.4   60       392.0   64       399.1   123      o       .37
  spring                                                                     
  --------------- ------- -------- ------- -------- ------- -------- ------- -------

\+ = Statistically significant in favor of Y3, o=no significant
difference.

Comparison of student achievement by locale in cohort 2 (Table III-29)
shows that students in suburban areas experienced more improvement from
year 1 to year 3 for TerraNova than the overall. The sample sizes for
rural and urban areas at school level were not large enough for HLM
statistical test. The achievement gaps among locale have not been
evident for cohort 2, at least for grades 1 and 2. In addition, because
of large progress made by 3rd graders both on TerraNova and OAT-R, rural
students finally caught up with urban and suburban students in cohort 2
by the end of year 3.

Table III-29.---Comparison of TerraNova and OAT-R results for years 1--3
in RFO cohort 2, by locale: Scale score

  --------------- ------- -------- ------- -------- ------- -------- ------- -------
  Locale and      Year 1           Year 2           Year 3           Sig.    ES
  assessment                                                                 

                  Scale   Number   Scale   Number   Scale   Number           
                  score            score            score                    

  **Urban**                                                                  

  TerraNova Grade 551.8   277      554.4   324      559.8   294      na      0.21
  1                                                                          

  TerraNova Grade 589.1   253      592.9   306      588.0   280      na      -0.03
  2                                                                          

  TerraNova Grade 604.7   210      624.1   262      606.5   302      na      0.05
  3                                                                          

  OAT-R Grade 3   400.4   162      394.9   226      404.8   306      na      0.16
  spring                                                                     

  **Suburban and                                                             
  town**                                                                     

  TerraNova Grade 550.7   455      562.6   470      563.5   388      \+      0.31
  1                                                                          

  TerraNova Grade 588.2   445      594.6   460      593.4   392      \+      0.15
  2                                                                          

  TerraNova Grade 610.5   446      622.7   486      617.9   404      \+      0.23
  3                                                                          

  OAT-R Grade 3   403.4   464      402\.   4614     401.9   395      o       -0.06
  spring                                                                     

  **Rural**                                                                  

  TerraNova Grade 563.6   71       562.0   65       576.3   97       na      0.29
  1                                                                          

  TerraNova Grade 585.9   71       593.3   105      590.7   102      na      0.14
  2                                                                          

  TerraNova Grade 593.6   70       620.1   109      609.6   97       na      0.42
  3                                                                          

  OAT-R Grade 3   384.5   70       399.2   106      402.0   116      na      0.61
  spring                                                                     
  --------------- ------- -------- ------- -------- ------- -------- ------- -------

\+ = Statistically significant in favor of Y3; o = no significant
difference; na=not available due to small sample size at school level.

Cohort 3

Table III-30 and Figure III-6 present comparisons of data from years 1
and 2 for cohort 3 students. Analysis from last year found that student
performance in the baseline was higher in cohort 3 than the other two
cohorts. Results from this year show moderate improvement from year 1 to
year 2 for TerraNova grade 3 but practically no improvement for
TerraNova grades 1--2 and OAT-R.

Table III-30.---Comparison of TerraNova and OAT-R results for years 1--2
in RFO\
cohort 3: Scale scores

  ----------------------------- ------- -------- ------- -------- ------- -------
  Assessment                    Year 1           Year 2           Sig\    ES\
                                                                  (Y2-1   (Y2-1

                                Scale   Number   Scale   Number           
                                score            score                    

  TerraNova Grade 1             566.2   1,621    567.3   2,179    o       .02

  TerraNova Grade 2             600.7   1,674    603.8   2,108    \+      .08

  TerraNova Grade 3             620.7   1,594    629.8   2,033    \+      .23

  OAT-R Grade 3 spring          404.4   1,585    404.3   1,847    o       -.00
  ----------------------------- ------- -------- ------- -------- ------- -------

\+ = Statistically significant in favor of Y2; o = no significant
difference.

**Figure III-6.---Two-year trends for RFO cohort 3 TerraNova
assessments**

![](media/image32.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

As shown in Table III-31 67 percent of cohort 3 students across grades
were at or above the 40th percentile and between 56 to 59 percent were
at or above 50th percentile benchmarks for TerraNova, which were better
than those from cohorts 1 and 2. In addition, 61 percent of 3rd graders
were proficient according to OAT-R. However, Figure III-7 suggests that
the gap between RFO students and state average in OAT-R remains the
same.

**Table III-31.---Comparison of TerraNova and OAT-R results in RFO
cohort 3 in years 1--2: Percent at benchmark**

  --------------------------------------------------- --------- ---------
  Assessment                                          Year 1    Year 2

  **At or above 40th percentile**                               

  TerraNova Grade 1                                   68.4%     66.7%

  TerraNova Grade 2                                   66.0      67.5

  TerraNova Grade 3                                   58.5      67.9

  **At or above 50th percentile**                               

  TerraNova Grade 1                                   57.9      59.4

  TerraNova Grade 2                                   55.3      56.8

  TerraNova Grade 3                                   46.3      56.0

                                                                

  **OAT-R Grade 3 spring (percent proficient or       56.3      60.9
  above)**                                                      
  --------------------------------------------------- --------- ---------

**Figure III-7.---Comparison of 2-year trends for OAT-R assessments of
RFO cohort 3 and state average**

![](media/image33.wmf){width="5.995833333333334in"
height="3.7381944444444444in"}

Table III-32 presents results disaggregated by racial and ethnic group
for cohort 3. In general, student performance by group was similar to
the overall trend for TerraNova except that Hispanic students appeared
to experience greater improvement in grade 1. Results from OAT-R,
however, tell a different story. While white students saw a decrease in
their average scores from year 1 to year 2, Hispanic and African
American students made large gains to the extent that achievement gaps
between racial groups disappeared.

Table III-32.---Comparison of TerraNova and OAT-R results in RFO cohort
3 for years\
1--2, by race/ethnicity: Scale scores

  ----------------------------- ------- -------- ------- -------- -------- --------
  Group and assessment          Year 1           Year 2           Sig\     ES\
                                                                  (Y2-1)   (Y2-1)

                                Scale   Number   Scale   Number            
                                score            score                     

  **African American**                                                     

  TerraNova Grade 1             553.4   357      551.0   484      o        -.06

  TerraNova Grade 2             584.8   400      589.2   443      o        .11

  TerraNova Grade 3             603.2   371      612.8   453      \+       .25

  OAT-R Grade 3 spring          390.6   374      407.3   414      \+       .60

  **Hispanic**                                                             

  TerraNova Grade 1             534.6   51       552.8   65       \+       .45

  TerraNova Grade 2             579.5   64       585.5   68       o        .13

  TerraNova Grade 3             599.6   40       609.7   69       o        .25

  OAT-R Grade 3 spring          390.2   40       407.9   48       \+       .66

  **White**                                                                

  TerraNova Grade 1             571.3   1,105    572.8   1,495    o        .04

  TerraNova Grade 2             607.1   1,117    608.5   1,452    o        .04

  TerraNova Grade 3             627.3   1,092    636.0   1,372    \+       .24

  OAT-R Grade 3 spring          409.7   1,081    402.9   1,262    \-       -.24
  ----------------------------- ------- -------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y2; - = statistically
significant in favor of Y1; o = no significant difference.

Table III-33 shows the achievement data disaggregated by special
service. For both FRL and special education students, the trends were
similar to the overall pattern for TerraNova grades 1 and 3. However,
their improvement on TerraNova grade 1 and OAT-R was larger. On the
other hand, LEP students experienced large improvement across the board.

Table III-33.---Comparison of TerraNova and OAT-R results for years 1--2
in RFO\
cohort 3, by special service: Scale scores

  ----------------------------- ------- ------- ------- -------- ------- ------- -------- -------- --------
  Status and assessment                 Year 1                           Year 2           Sig\     ES\
                                                                                          (Y2-1)   (Y2-1)

                                        Scale           Number           Scale   Number            
                                        score                            score                     

  **FRL**                                                                                          

  TerraNova Grade 1             560.7           961              560.2           1,458    o        -.01

  TerraNova Grade 2             593.4           1,095            597.7           1,373    o        .12

  TerraNova Grade 3             613.6           1,022            622.7           1,394    \+       .24

  OAT-R Grade 3 spring          398.5           1,018            404.7           1,333    \+       .22

  **LEP**                                                                                          

  TerraNova Grade 1             530.1           74               552.6           80       o        .52

  TerraNova Grade 2             563.4           74               569.7           77       \+       .14

  TerraNova Grade 3             584.3           56               593.8           85       \+       .21

  OAT-R Grade 3 spring          380.3           57               404.2           42       o        .78

  **Special education**                                                                            

  TerraNova Grade 1             540.6           182              538.9           250      o        -.04

  TerraNova Grade 2             578.3           244              580.6           276      o        .06

  TerraNova Grade 3             601.0           237              608.9           294      \+       .22

  OAT-R Grade 3 spring          392.3           241              402.0           353      \+       .37
  ----------------------------- ------- ------- ------- -------- ------- ------- -------- -------- --------

\+ = Statistically significant in favor of Y2; o = no significant
difference.

Comparison of student achievement by locale in cohort 3 (Table III-34)
shows the following:

-   Students in different locales have made some progress in selected
    grade levels for TerraNova. In some cases, such as grade 3 in
    suburban and rural areas, the increases were large.

-   OAT-R results were mixed. While urban students experienced
    improvement, student achievement from suburban and rural areas
    decreased. In the case of rural students, the drop was large**.**

Table III-34.---Comparison of TerraNova and OAT-R results for years 1--2
in RFO\
cohort 3, by locale: Scale score

  ----------------------------- ------- -------- ------- -------- ------- -------
  Locale and assessment         Year 1           Year 2           Sig.    ES

                                Scale   Number   Scale   Number           
                                score            score                    

  **Urban**                                                               

  TerraNova Grade 1             560.7   769      558.3   1,061    o       -0.06

  TerraNova Grade 2             592.3   817      598.6   1,031    \+      0.16

  TerraNova Grade 3             613.2   743      618.3   1,002    \+      0.13

  OAT-R Grade 3 spring          398.1   742      406.0   975      \+      0.27

  **Suburban and town**                                                   

  TerraNova Grade 1             571.3   463      574.8   725      o       0.08

  TerraNova Grade 2             610.6   477      608.4   693      o       -0.05

  TerraNova Grade 3             625.1   465      638.8   665      \+      0.39

  OAT-R Grade 3 spring          407.6   461      404.3   662      o       -0.12

  **Rural**                                                               

  TerraNova Grade 1             571.1   389      577.5   393      na      0.17

  TerraNova Grade 2             606.7   380      609.6   384      na      0.09

  TerraNova Grade 3             629.8   386      644.8   366      na      0.43

  OAT-R Grade 3 spring          412.9   382      396.7   210      na      -0.62
  ----------------------------- ------- -------- ------- -------- ------- -------

\+ = Statistically significant in favor of Y2; o = no significant
difference; na=not available due to small sample size at school level.

Comparative Evidence

The analyses discussed above focused on the RFO per se, examining
cross-sectional changes in student outcomes over time. While this is a
very important question, a related and equally important question is the
extent to which the services provided through RFO add value to the
learning process that is above and beyond that provided through
alternative educational venues. We examine this issue through two means:
comparison of findings from selected cohort 1 case study schools and
comparison of performance in RFO and Title I schools.

**Comparison of performance in the cohort 1 treatment and comparison
case study schools indicates that students in these schools are
performing at about the same level. However, these findings must be
viewed with caution because of the substantial reduction in the size of
the student sample and the original set of schools. Comparisons of
performance in RFO and Title I schools indicate that overall, students
in RFO schools perform higher than those in Title I schools after
adjusting for sampling bias.**

Case Study Findings

In order to measure the impact of RFO, our evaluation design originally
included a quasi-experimental longitudinal component comparing the
achievement growth between treatment and comparison students over time.
To establish this sample, we used within-district matching to match RFO
schools with comparison schools based on student enrollment and
demographics. For both treatment and comparison schools (i.e., case
study schools), we selected students from the class cohort that began
kindergarten in 2005 and intended for follow them for 3 years.

The original sample consisted of 14 pairs of schools purposefully
selected from five districts in cohort 1. Assuming a design effect of
1.5, the sample size would allow us to detect effect sizes as small as
0.05 with a power of 0.8. However, because of programmatic changes, the
sample size for case study schools decreased from 14 pairs of school
from five districts to 8 pairs from three districts by the end of
year 1. During 2006--07, the sample was further reduced to 6 pairs
because one treatment and two comparison schools were closed. Coupled
with attrition, the number of students tested decreased from 978 (T:523,
C:455) at T1 to 235 (T:144, C:96) at T4. In other words, only 25 percent
of the students remained at the end of third year.

We compared the assessment results from four testing periods for the
treatment and comparison students. First we conducted cross-sectional
analyses, including all of the students who were at least assessed once
(Table III-35). We found that on average, although treatment students
began at a lower level at the pretest (T1), their achievement was
significantly higher than comparison students at the end of first year
(T2). However, the advantage gained in the first year in DIBELS
disappeared in years 2 (T3) and 3 (T4), when no significant differences
were detected between two groups as measured by TerraNova.

**Table III-35**.---Cross-sectional comparison of performance between
treatment and comparison students

+------------------------------+-----+-----+-----+-----+-----+------+
| Assessment                   | Tre |     | C   |     | S   | Sig  |
|                              | atm |     | omp |     | ign |      |
|                              | ent |     | ari |     |     |      |
|                              |     |     | son |     | (T  |      |
|                              |     |     |     |     | -C) |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              | M   | Std | M   | Std |     |      |
|                              | ean |     | ean |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T1 (2004 fall)**           |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Initial Sound Fluency | 1   | 8.7 | 1   | 1   | \-  | .03  |
|                              | 1.6 |     | 3.0 | 2.0 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 1   | 1   | 1   | 1   | \-  | .03  |
|                              | 7.0 | 5.4 | 9.3 | 7.0 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T2 (2005 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Phoneme Segmentation  | 3   | 1   | 2   | 1   | \+  | .00  |
| Fluency                      | 2.6 | 9.6 | 1.0 | 7.7 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Nonsense Word Fluency | 2   | 1   | 2   | 1   | \+  | .03  |
|                              | 6.9 | 6.9 | 4.3 | 8.9 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 3   | 1   | 3   | 1   | \+  | .00  |
|                              | 9.9 | 6.8 | 2.3 | 8.0 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T3 (2006 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 1            | 55  | 4   | 55  | 4   | \-  | .07  |
|                              | 2.9 | 1.0 | 8.0 | 0.4 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T4 (2007 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 2            | 59  | 3   | 59  | 3   | \-  | .76  |
|                              | 0.3 | 4.5 | 1.2 | 4.3 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+

Second, we analyzed the results longitudinally, including only students
who were assessed in all four periods (Table III-36). The data tell a
similar story.[^34] While students from both groups started at a similar
level, treatment students significantly outperformed the comparison
students in DIBELS after 1 year but have been performing at similar
levels since year 2.

Table III-36.---Longitudinal comparison of performance between treatment
and comparison students

+------------------------------+-----+-----+-----+-----+-----+------+
| Assessment                   | Tre |     | C   |     | S   | Sig  |
|                              | atm |     | omp |     | ign |      |
|                              | ent |     | ari |     |     |      |
|                              |     |     | son |     | (T  |      |
|                              |     |     |     |     | -C) |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              | M   | Std | M   | Std |     |      |
|                              | ean |     | ean |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T1 (2004 fall)**           |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Initial Sound Fluency | 1   | 9.2 | 1   | 1   | \+  | .65  |
|                              | 3.9 |     | 3.3 | 2.0 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 2   | 1   | 2   | 1   | \+  | .22  |
|                              | 3.4 | 5.2 | 0.8 | 6.5 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T2 (2005 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Phoneme Segmentation  | 3   | 1   | 2   | 1   | \+  | .00  |
| Fluency                      | 9.4 | 7.3 | 6.1 | 8.5 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Nonsense Word Fluency | 3   | 1   | 3   | 1   | \-  | .58  |
|                              | 2.7 | 5.8 | 4.0 | 9.4 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 4   | 1   | 3   | 1   | \+  | .00  |
|                              | 6.0 | 3.9 | 6.3 | 5.9 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T3 (2006 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 1            | 56  | 4   | 56  | 3   | \-  | .66  |
|                              | 6.9 | 1.4 | 9.2 | 7.3 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T4 (2006 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 2            | 59  | 3   | 59  | 3   | \-  | .88  |
|                              | 6.9 | 3.4 | 7.6 | 7.6 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+

Analysis of the demographic characteristics[^35] of the longitudinal
sample shows that the treatment and comparison group are similar in
terms of race/ethnicity and gender composition (Table III-37).

Table III-37.---Comparison of demographic characteristics of the
longitudinal sample

  --------------------------------------- --------------- ---------------
  Characteristic                          Treatment       Comparison

  **Gender**                                              

  Male                                    49.8%           50.0%

  Female                                  50.2            50.0

  **Race/ethnicity**                                      

  African American                        53.5            44.8

  White                                   37.5            45.8
  --------------------------------------- --------------- ---------------

NOTE: Percents do not add to 100 because not all racial/ethnic groups
are included.

Comparison of cross-sectional and longitudinal samples also illustrates
the issue of sample reduction. In order to shed light on the possible
impact from data reduction on the findings, we compared the performance
between stayers and movers to see whether the movers are statistically
different from the stayers (Table III-38). Stayers are students who were
in the longitudinal sample that had been tested four times from fall
2004 to spring 2007. Movers are those who had incomplete test records.
The significance test shows that for all case study schools, stayers
were performing at a significantly higher level in all assessments. When
dividing students by treatment and comparison status (not shown in the
table), we found the same pattern for students from treatment schools.
However, in comparison schools, while stayers performed at a higher
level than movers at T2, no significant differences were found in
results from other times.

Table III-38.---Comparison of performance between stayers and movers

+------------------------------+-----+-----+-----+-----+-----+------+
| Assessment                   | S   |     | Mov |     | S   | Sig  |
|                              | tay |     | ers |     | ign |      |
|                              | ers |     |     |     |     |      |
|                              |     |     |     |     | (S  |      |
|                              |     |     |     |     | -M) |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              | M   | Std | M   | Std |     |      |
|                              | ean |     | ean |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T1 (2004 fall)**           |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Initial Sound Fluency | 1   | 1   | 1   | 1   | \+  | .04  |
|                              | 3.5 | 0.2 | 1.9 | 0.4 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 2   | 1   | 1   | 1   | \+  | .00  |
|                              | 2.4 | 5.8 | 7.0 | 6.3 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T2 (2005 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Phoneme Segmentation  | 3   | 1   | 2   | 1   | \+  | .00  |
| Fluency                      | 4.1 | 8.9 | 4.3 | 9.2 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Nonsense Word Fluency | 3   | 1   | 2   | 1   | \+  | .00  |
|                              | 3.2 | 7.3 | 3.0 | 7.3 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| DIBELS Letter Naming Fluency | 4   | 1   | 3   | 1   | \+  | .00  |
|                              | 2.1 | 5.5 | 4.2 | 8.0 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T3 (2006 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 1            | 56  | 4   | 55  | 3   | \+  | .00  |
|                              | 7.5 | 0.0 | 0.2 | 9.6 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
|                              |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| **T3 (2007 spring)**         |     |     |     |     |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+
| TerraNova Grade 2            | 59  | 3   | 58  | 3   | \+  | .00  |
|                              | 7.2 | 4.8 | 6.2 | 3.4 |     |      |
+------------------------------+-----+-----+-----+-----+-----+------+

Finally, we examined the extent to which the RFO contributes to
differences in student reading achievement holding other confounding
variables. The main methodological challenge in this kind of analysis is
a lack of consistent achievement measure over time. First, the assessed
areas are different. Student performance at T1 was assessed by DIBELS
ISF and LNF; at T2 by DIBELS PSF, NWF, and LNF; and at T3 and T4 by
TerraNova. Second, the reporting metrics vary. While TerraNova results
are reported in scale scores, DIBELS subset scores are raw scores only,
measuring the number of correct answers in each assessment. Finally,
TerraNova is a norm-reference assessment, whereas DIBELS is a
criterion-referenced test.

In order to resolve these differences, we converted scores with
different reporting formats into a standardized z-score using the
statistics from the relevant technical reports (Good et al., 2002; CTB,
2003). The standardized z-score has a mean of 0 and standard deviation
of 1.[^36] The standard scores provide a measure of the child's relative
standing with the norm. For the DIBELS measures, we then created one
composite score by taking into account performance from all related
assessments at any given time. We believe that the composite score is a
more comprehensive measure of a child's reading achievement and a closer
approximate of the TerraNova score than relying a result from any single
DIBELS measure.

Standardization of scores allows us to model achievement results over
time between RFO and comparison students using hierarchical linear
modeling (HLM). We used a three-level HLM in which time is nested within
students, and students are nested within schools. Student assessment
scores were used as the dependent variable, and program status was used
as the main independent variable. Factors such as students'
race/ethnicity and gender were introduced as time-invariant covariates.
At level 1, we modeled student achievement as a function of time. Level
2 models the time-invariant variables such as race/ethnicity and gender
at the student level. Level 3 specifies student achievement as a
function of schools' RFO status (i.e., treatment, comparison).

Table II-39 presents results of the analysis. We found that although the
sign was positive in favor of RFO, the average difference of student
reading achievement between RFO and comparison students was not
statistically significant. Student ethnicity was not significantly
associated with student achievement, but female students performed
significantly higher than male students by 0.21 standard deviation.

Table III-39.---HLM results for RFO effect

+-------------------+---------+---------+---------+---------+---------+
| Variable          | Unstand | S       | T-ratio | d.f.    | P-value |
|                   | ardized | tandard |         |         |         |
|                   |         | error   |         |         |         |
|                   | coef    |         |         |         |         |
|                   | ficient |         |         |         |         |
+-------------------+---------+---------+---------+---------+---------+
| Intercept         | -.18    | .18     | -1.02   | 9       | .34     |
+-------------------+---------+---------+---------+---------+---------+
| RFO               | .03     | .08     | .36     | 9       | .73     |
+-------------------+---------+---------+---------+---------+---------+
| Time              | -.06    | .04     | -1.72   | 913     | .09     |
+-------------------+---------+---------+---------+---------+---------+
| African American  | .14     | .21     | .66     | 227     | .51     |
+-------------------+---------+---------+---------+---------+---------+
| White             | .11     | .16     | .68     | 227     | .50     |
+-------------------+---------+---------+---------+---------+---------+
| Female            | .21     | .09     | 2.35    | 227     | .02     |
+-------------------+---------+---------+---------+---------+---------+

In summary, data from fall 2005 to spring 2007 show mixed findings.
While treatment students have statistically higher levels of performance
after 1 year of intervention, the difference seems to disappear by the
end of the second year. However, the findings need to be interpreted
with caution for several reasons.

-   First, our qualitative case studies found that two out of five
    districts had distributed the RFO core reading program to all
    schools, including the comparison schools. Other districts had
    developed a districtwide literacy framework that includes features
    of RFO. Therefore, the lack of difference could be a result of
    "program contamination."

-   Second, there are fundamental differences in the developmental
    trajectories of reading skills. Most constrained reading skills are
    acquired in a relatively short developmental time frame. In general,
    letter knowledge, phonics, and concepts of print are highly
    constrained; phonemic awareness and reading fluency are less
    constrained; and vocabulary and comprehension are least constrained.
    The skills measured by DIBELS are more specific in nature as
    compared to those by TerraNova and perhaps are more aligned to RFO
    intervention.

-   Finally, in spite of the standardization, a lack of a consistent
    measure for student reading achievement at various test times may
    affect the validity of the results.

RFO-Title I Analysis

In order to strengthen our examination of the value added by RFO, we
added another quasi-experimental design this year. In this study, we
examined the impact of RFO on student achievement by comparing
performance in a matched sample of RFO and Title I schools that are not
receiving RFO services.

Matching Title I Schools With RFO Schools

Two types of matching were conducted. For districts that have a larger
pool of both RFO and Title I schools, comparison schools were chosen
within districts primarily based on scale scores from 4th grade Ohio
Reading Proficiency (ORP) achievement in 2002--03 (pre-intervention),
because OAT-R was not administered until 2003--04. In addition, we also
took into account socioeconomic status as indicated by the percentage of
students in the school receiving free and reduced-price lunch. For
districts that do not have both types of schools, we used
across-district matching. Schools with the same type of urbanicity
characteristics were matched primarily on 4th grade ORP scores in
2002--03; socioeconomic status also was considered. Incidentally, all
RFO schools are receiving Title I funding at the same time.

Eighty-five pairs of schools were matched. The RFO schools were taken
from all three RFO cohorts: 42 from cohort 1, 15 from cohort 2, and 28
from cohort 3. As expected, there were no statistically significant
differences in terms of 4th grade student achievement at the baseline in
2002--03 (Table III-40). However, on average, the sampled 85 Title I
schools have lower achievement level than the Title I population of 795
schools. This creates a situation known as regression toward the mean
(RTM), where those with extreme scores on any measure at one point in
time will, for purely statistical reasons, probably have less extreme
scores the next time they are tested. In other words, because the
sampled TI schools had lower achievement in 2002, they are likely to
have higher observed scores in the post-test than the true scores. An
adequate post-test comparison requires that the TI post-test scores be
adjusted by RTM.[^37]

Table III-40.---Comparison student achievement of RFO and Title I
schools at the baseline

  ----------------------- --------------- --------------- ---------------
  Measure                 RFO             Sampled Title I Title I
                                                          population

  ORP scale score         205.9           205.9           212.6
  ----------------------- --------------- --------------- ---------------

Results

We used results of the OAT-R grade 3 spring as measure of student
achievement for the period 2004--07 because it was the only measure
available for students in both RFO and Title I schools. Table III-41
shows that on average, RFO students had slightly lower performance on
the OAT-R than their Title I counterparts. However, if we adjusted the
RTM, the achievement level of RFO students was actually higher than the
comparison.

Table III-41.---Comparison of post-program results from RFO and Title I
students

  ---------------------- ------------ ------------ ------------ ------------
  Measure                RFO          Title I      P-value      Effect size

  OAT-R scale score      401.1        403.4        .00          -.08
  (observed)                                                    

  OAT-R scale score      401.1        395.9        .00          .19
  (adjusted)                                                    
  ---------------------- ------------ ------------ ------------ ------------

Because our RFO sample contained schools from all three cohorts and thus
varied in amount of time spent receiving RFO services, we also examined
whether or not the amount of exposure a school had to the RFO program
resulted in a performance differential. For example, we looked at
whether students from schools receiving RFO for 4 years show a greater
effect than those that received program services for 1 year.[^38] Table
III-42 data suggest that students from RFO schools that participated in
the program for 1 to 2 years had a higher average achievement than those
from their matched Title I schools. Further, the more exposure to RFO,
the more the score differentials between the two programs.

Table III-42.---Comparison of post-program results from RFO and Title I
students,\
by exposure: Adjusted score

  ----------------------- ---------- -------- ---------- -------- --------- --------
  Exposure time           RFO                 Title I             P-value   Effect
                                                                            size

                          Adjusted   Number   Adjusted   Number             
                          score               score                         

  1 year                  399.4      3,921    396.2      4,412    .05       .11

  2 years                 402.7      3,459    397.4      4,337    .00       .19

  3 years                 400.8      2,482    394.5      2,736    .00       .22

  4 years                 402.5      1,829    393.9      2,067    .00       .31
  ----------------------- ---------- -------- ---------- -------- --------- --------

Table III-43 presents the comparisons by subgroup defined by
race/ethnicity, economic status, language proficiency, and special
education status:

-   **RFO students in all race/ethnicity groups had higher achievement
    than their Title I peers. The effect sizes for African American
    students were the largest, followed by those for Hispanics.**

-   **RFO students receiving special services had higher achievement
    than Title I counterparts. The effect sizes were above the overall
    of .23, suggesting that the RFO students benefited more than the
    average.**

Table III-43.---Comparison of post-program results from RFO and Title I
students, by subgroup: Adjusted score

+----------------------+-----------+-----------+----------+----------+
| Subgroup             | RFO       | Title I   | P-value  | Effect   |
|                      |           |           |          | size     |
+----------------------+-----------+-----------+----------+----------+
| Race/ethnicity       |           |           |          |          |
+----------------------+-----------+-----------+----------+----------+
| > African American   | 398.9     | 389.7     | .00      | .33      |
+----------------------+-----------+-----------+----------+----------+
| > Hispanic           | 396.6     | 391.2     | .01      | .19      |
+----------------------+-----------+-----------+----------+----------+
| > White              | 404.5     | 402.2     | .04      | .08      |
+----------------------+-----------+-----------+----------+----------+
| Disadvantaged        | 399.7     | 393.4     | .00      | .23      |
+----------------------+-----------+-----------+----------+----------+
| LEP                  | 388.4     | 381.4     | .19      | .24      |
+----------------------+-----------+-----------+----------+----------+
| Special education    | 392.4     | 382.4     | .00      | .36      |
+----------------------+-----------+-----------+----------+----------+

Another hypothesis we explored is whether RFO has an impact on placement
in special education programs and how this outcome variable may differ
in RFO and Title I schools. We examined the trends for students in
special education in the schools, looking at both total percentage of
students in special education and the percentages classified as mentally
retarded and having a specific learning disability.[^39] Figure III-8
shows the trends for the overall data; Figures III-9 and III-10 present
the comparisons by the two subtypes of special education categorization.
Note that for the subtype comparison, ODE does not produce percentages
for schools with fewer than 10 students identified; thus, sample sizes
vary.

Figure III-8 suggests that over a 4-year period, the percentage of
students in special education declined by a faster rate in RFO schools
than in Title I schools. The difference is modest, i.e., 0.2 percent.
Figures III-9 and III-10 show no differences between the samples for
students with either mental retardation or specific learning
disabilities.[^40]

**Figure III-8.---Trend comparison for percentage of students receiving
special education**

![](media/image34.wmf){width="5.995833333333334in"
height="2.9854166666666666in"}

**Figure III-9.---Trend comparison for percentage of students with
mental retardation**

![](media/image35.wmf){width="5.995833333333334in"
height="3.361111111111111in"}

**Figure III-10.---Trend comparison for percentage of students with
specific learning disabilities**

![](media/image36.wmf){width="5.995833333333334in"
height="3.361111111111111in"}

Finally, we ran a 2-level HLM model for the overall sample to control
for confounding effects from school exposure, cohort and student
demographics, as well as the nesting structure at school level. Results
in Table III-44 shows that on average, RFO students' OAT scores are
about 6 points above those of their Title I peers.

Table III-44.---HLM results for the overall sample

  ------------------------------ ---------------- ------- -------- ------- -------
  Model                          Unstandardized           t        d.f.    Sig.
                                 coefficients                              

                                 B                Std                      
                                                  Err.                     

  Intercept                      406.36           1.00    406.57   168     .00

  RFO                            5.62             1.28    4.38     168     .00

  Exposure                       -.18             .17     -1.04    25235   .30

  African American               -6.44            .50     -12.85   25235   .00

  Hispanic                       -2.92            .85     -3.42    25235   .00

  FRL                            -6.37            .47     -13.62   25235   .00

  LEP                            -12.72           1.05    -12.06   25235   .00

  Special education              -14.36           .48     -29.92   25235   .00
  ------------------------------ ---------------- ------- -------- ------- -------

However, the results above do not take into account student attrition.
In order to control for the potential impact of student mobility, we
selected a subsample of students from both RFO and Title I schools who
were 3rd graders in 2007 and traced their school records from
kindergarten to the 3rd grade to identify the number of years they have
stayed in that school.[^41] Comparison of the data shows little
difference between RFO and Title I students. In general, 40 percent of
students have stayed in the same school for 4 years, 20 percent for 3
years, 20 percent for 2 years, and 20 percent for 1 year.

Table III-45 presents the 2-level HLM results for the subsample
including the additional variable of retention. The results show that in
addition to the 6-point differential between RFO and Title I students,
students who were in the schools longer consistently had higher
achievement.

Table III-45.---Regression results accounting for individual mobility

  ------------------------------ ---------------- ------- -------- ------- --------
  Model                          Unstandardized           t        d.f.    Sig.
                                 coefficients                              

                                 B                Std                      
                                                  Err.                     

  Intercept                      403.19           1.28    314.92   164     .00

  RFO                            6.29             1.46    4.32     164     .00

  Exposure                       -1.15            .82     -1.41    5851    .16

  Retention                      1.30             .32     4.05     5851    .00

  Black                          -3.68            .99     -3.72    5851    .00

  Hispanic                       -2.02            1.83    -1.11    5851    .24

  FRL                            -3.57            .97     -3.67    5851    .00

  LEP                            -7.49            2.23    -3.36    5851    .00

  Special education              -13.84           1.05    -13.18   5851    .00
  ------------------------------ ---------------- ------- -------- ------- --------

Comparison of student achievement in OAT-R shows that RFO students are
performing at a higher level than their Title I peers from matched
comparison schools after adjusted for RTM, showing preliminary evidence
of the RFO impact. However, the adjustment was made with an assumption
that the RTM is constant and it is unclear whether it was an
over-correction. We will continue to explore this area. In the next
year, we will redraw the Title I comparison sample by correcting the RTM
at sampling stage.[^42]

Implementation and Student Achievement

In the previous chapter, we measured the quality of implementation using
two different measures. In this section, we look at the relationship
between various measures of implementation and student achievement.

-   Implementation scores derived from the teacher and principal surveys
    are used in the RFO evaluation. Based on self-reported data, the
    implementation scale attempts to measure teachers' and schools'
    performance of three program standards underlying RFO
    intervention---Professional Learning (PL), Comprehensive and
    Coherent Program Design (CCPD), and Systematic and Explicit Reading
    Instruction (SERI). This score is the most comprehensive measure of
    program implementation among the three.

-   Early Language and Literacy Classroom Observation (ELLCO) provides
    observed data on teachers' instructional practices as a result of
    participating in RFO. Conceptually, ELLCO measures different domains
    from what was captured by implementation of program standards.

**Examination of the relationship between implementation measures and
student outcomes presents a complex picture. While the implementation
scale developed from the surveys shows some important relationships to
achievement increases and status on the OAT-R, correlations with
achievement on the TerraNova are not significant. Strategies for
struggling students, implementation of formal procedures, and SBRR
appear to explain differences in student achievement. Conversely, the
ELLCO correlates strongly with the TerraNova level but not with the
OAT-R. Teachers' approaches to book reading, children's writing, and
assessment, as well as use of** **technology, approaches to curriculum
integration, diversity, use of books, are positively associated with
TerraNova achievement level from grades 1--3.**

Relationship Between Implementation and Student Achievement

We first examined the relationship between the change of student
achievement and school-level RFO implementation using multiple
regression models by combining data from 51 schools in cohorts 1 and 2.
The student achievement results came from TerraNova and ORA from grades
1--3. To examine the relationship, we regressed year 3 achievement level
on average implementation scores between years 1 and 3 (for overall,
three standards, and 12 scales), controlling for the year 1 achievement
level as well as cohort differences.

Table III-46 presents the summary of these regressions. We found that in
general, the achievement level in year 1 significantly predicts the year
3 achievement for all TerraNova outcomes, and but year 3 student
performance in the 3rd grade is significantly lower in cohort 2 than
cohort 1. However, implementation fidelity is not significantly
associated with student achievement in most cases. Specifically, we
found no significant association between TerraNova scores and RFO
implementation. For OAT-R, we found evidence that the achievement is
positively associated with Comprehensive and Coherent Program Design in
general, and implementation of formal procedures (subscale 10) and SBRR
(subscale 12) in particular.

Table III-46.---Relationships between implementation and student
achievement change (N=51)

  ------------------------------- ----------- ----------- ----------- ----------
  Standard and scale              TerraNova   TerraNova   TerraNova   OAT-R
                                  grade 1     grade 2     grade 3     

  **Overall implementation**                                          

  **Three Program Standards**                                         

  Professional Learning (PL)                                          

  Comprehensive and Coherent                                          \+
  Program Design (CCPD)                                               

  Systematic and Explicit Reading                                     
  Instruction (SERI)                                                  

  **12 Subscales**                                                    

  1: Professional development                                         
  activities                                                          

  2: Assess-plan-teach                                                

  3: TRIO and literacy specialist                                     

  4: Teachers' perspective of                                         
  CCPD                                                                

  5: Five essential-frequency of                                      
  instruction                                                         

  6: Use of strategies for                                            
  struggling students                                                 

  7: Five essentials-increased                                        
  emphasis                                                            

  8: Principal's view of literacy                                     
  specialist                                                          

  9: Monitoring program                                               
  effectiveness                                                       

  10: Implementation of formal                                        \+
  procedures                                                          

  11: Resource and time                                               
  allocation                                                          

  12: SBRR                                                            \+
  ------------------------------- ----------- ----------- ----------- ----------

\+ suggests a significant positively relationship between the
implementation of the particular scale/standard and student achievement;
empty cells denote no significant relationship.

We then examined the relationship between the level of student
achievement in year 3 and year 3 school-level RFO implementation using
multiple regression models. The student achievement results came from
TerraNova and ORA from grades 1--3. The results on achievement level are
similar to that of the achievement change. Table III-47 shows that
implementation fidelity is not significantly associated with student
achievement level in most cases. We found significant association
between 3rd grade TerraNova scores and use of strategies for struggling
students (subscale 6). For OAT-R, student achievement level is
positively associated with implementation of formal procedures (subscale
10) and SBRR (subscale 12) in particular.

Table II-47.---Relationships between implementation and student
achievement level (N=51)

  ------------------------------- ----------- ----------- ----------- ----------
  Standard and scale              TerraNova   TerraNova   TerraNova   OAT-R
                                  grade 1     grade 2     grade 3     

  **Overall implementation**                                          

  **Three Program Standards**                                         

  Professional Learning (PL)                                          

  Comprehensive and Coherent                                          
  Program Design (CCPD)                                               

  Systematic and Explicit Reading                                     
  Instruction (SERI)                                                  

  **12 Subscales**                                                    

  1: Professional development                                         
  activities                                                          

  2: Assess-plan-teach                                                

  3: TRIO and literacy specialist                                     

  4: Teachers' perspective of                                         
  CCPD                                                                

  5: Five essential-frequency of                                      
  instruction                                                         

  6: Use of strategies for                                \+          
  struggling students                                                 

  7: Five essentials-increased                                        
  emphasis                                                            

  8: Principal's view of literacy                                     
  specialist                                                          

  9: Monitoring program                                               
  effectiveness                                                       

  10: Implementation of formal                                        \+
  procedures                                                          

  11: Resource and time                                               
  allocation                                                          

  12: SBRR                                                            \+
  ------------------------------- ----------- ----------- ----------- ----------

\+ suggests a significant positively relationship between the
implementation of the particular scale/standard and student achievement;
empty cells denote no significant relationship.

In examining the relationships, it is important to keep in mind that the
correlational nature of the analysis hinges on the variance of the data.
For example, if all schools have high implementation on a scale or a
dimension, it is unlikely that variation in implementation fidelity will
emerge as a significant factor in influencing student achievement.

Relationship Between ELLCO and Student Achievement

We examined the relationship between student achievement level and
school-level ELLCO scores using multiple regression models. The student
achievement results came from TerraNova and ORA from grades 1--3. To
examine the relationship, we regressed achievement level in 2007 on
spring ELLCO scores (for overall, two subtotals, and 14 areas),
controlling for cohort differences. The data drew from 104 RFO schools
from cohorts 1--3.

Table III-48 presents the summary of these regressions. In general,
ELLCO scores are excellent predictors for TerraNova achievement level,
especially for the 2nd and 3rd graders, but not for OAT-R. Specifically,

-   ELLCO aspects in approaches to book reading, children's writing, and
    assessment, as well as the subtotal for language, literacy, and
    curriculum, are positively associated with TerraNova achievement
    level from grades 1--3.

-   In addition, presence and use of technology, approaches to
    curriculum integration, recognizing diversity in the classroom, and
    facilitating home support for literary, as well as general classroom
    environment subtotal and total ELLCO scores, are positively related
    to TerraNova achievement level for the 2nd and 3rd graders.
    Organization of the classroom is positively correlated with
    TerraNova performance from grades 1 and 3; and 3rd graders'
    performance on TerraNova and OAT-R is positively associated with
    presence of books.

Table III-48.---Relationships between ELLCO observation and student
achievement level, by observed area (N=104)

  ------------------------- ------------ ------------ ------------ -----------
  Observed area             TerraNova\   TerraNova\   TerraNova\   OAT-R
                            grade 1      grade 2      grade 3      

  Organization of the       \+                        \+           
  classroom                                                        

  Contents of the classroom                           \+           

  Presence and use of                    \+           \+           
  technology                                                       

  Opportunities for child                \+                        
  choice and initiative                                            

  Classroom management                                             
  strategies                                                       

  Classroom climate                                   \+           

  **General Classroom                    \+           \+           
  Environment Subtotal**                                           

  Oral language                                                    
  facilitation                                                     

  Presence of books                                   \+           \+

  Approaches to book        \+           \+           \+           
  reading                                                          

  Approaches to children's  \+           \+           \+           
  writing                                                          

  Approaches to curriculum               \+           \+           
  integration                                                      

  Recognizing diversity in               \+           \+           
  the classroom                                                    

  Facilitating home support              \+           \+           
  for literacy                                                     

  Approaches to assessment  \+           \+           \+           

  **Language, Literacy and  \+           \+           \+           
  Curriculum Subtotal**                                            

  **Total**                              \+           \+           
  ------------------------- ------------ ------------ ------------ -----------

\+ Suggests a significant positively relationship between the
implementation of the particular scale/standard and student achievement;
empty cells denote no significant relationship.

We did not analyze the relationship between student achievement change
and ELLCO scores, parallel to what we did for Westat implementation
measures, because we only had 1 year of ELLCO data.

In summary, we found that in general, ELLCO scores are excellent
predictors for TerraNova achievement level, especially for 2nd and 3rd
graders, but not so for OAT-R achievement.

References

Borman, G.D., Hewes, G.M., Overman, L.T., and Brown, S. (2003).
Comprehensive school reform and achievement: A meta-analysis. *Review of
Educational Research,* 73 (2): 125-230.

Cohen, J. (1988). *Statistical power analysis for the behavioral
sciences.* Hillsdale, NJ: Erlbaum.

CTB McGraw-Hill (2003). *CAT technical report: TerraNova Second
Edition*. Monterey, CA: Author.

Good, R.H., Wallin, J.U., Simmons, D.C., Kame'enui, E.J., and Kaminski,
R.A. (2002). *System-wide percentile ranks for DIBELS benchmark
assessment.* Technical Report 9. Eugene, OR: University of Oregon.

Lipsey, M.W., and Wilson, D.B. (2001). *Practical meta-analysis*.
Thousand Oaks, CA: Sage Publications. (1993).

Section IV. Conclusion

This concluding section presents findings organized around research
questions developed collaboratively by ODE, the Reading First-Ohio
Center, Westat, and Learning Point Associates to frame efforts to bring
together ongoing evaluation and research studies to address state
Reading First priorities and literacy priorities generally more
effectively and efficiently. The set of questions was finalized after
our surveys and protocols were developed and after site visits had
begun; therefore, the fit between the year 4 case study findings and the
research questions is not precise. Nevertheless, the findings contribute
to our understanding of the impact of Reading First in the schools,
implementation issues, and sustainability.

Question 1: What is the valued added to students' reading achievement if
they are in an RFO school versus if they are not?

Our analyses address both the progress of schools participating in RFO
and how progress in these schools compares with similar schools not
receiving such services. The data suggest that reading achievement in
RFO schools is better in now than it had been at program initiation. The
strongest gains are found for year 1 of program participation, after
which growth levels off. Initial data comparing students in Reading
First and Title I programs suggest that students in Reading First
programs perform significantly better on achievement measures than
students in Title I programs. Further, this difference increases the
longer the RF program is in place.

**Question 2: What is the value added to teacher instructional practices
for teachers if they are in an RFO school versus if they are not?**

The case study indicates that features of the Reading First program are
now implemented in non-Reading First schools in the case study
districts. Comparison schools in the case study now use the same reading
series used in the RFO schools and do so in literacy blocks that are at
least 90 minutes long. Comparison school teachers and administrators now
refer to the five components of reading. Recently, comparison school
teachers have had more access to formative assessments and growing
requirements that these assessments be administered on a schedule.

Instructional practices still differ, however, between RFO and
comparison schools in the case study. In Reading First schools, there is
more consistency in instruction. Our data suggest this is due to the
guidance teachers have received from a literacy specialist and the
long-term and consistent professional development they have participated
in as a group. Our data also indicate that RFO teachers use assessments
to make instructional decisions more than non-RFO teachers. Related to
this, small groups were more frequently observed in RFO classrooms than
in non-RFO classrooms.

**Question 3: What is the quality of implementation of the RFO schools
and what explains the variance in levels of implementation?**

The data indicate the implementation of Ohio's model for RFO is strong
and has increased over time. In terms of the key features of the
program, professional development, instruction, and program support and
oversight each year have generally progressed in implementation.
Teachers are definitely focused on the five essential features, have
grown more comfortable using small group instructional and management
techniques, and have grown in the use of, and appreciation for, the
value of assessment data. ELLCO data provide confirming evidence that
implementation improves over time and by year 2 is strong.

That said, there is considerable variance in implementation by school
according to the survey data. Features that predict level of
implementation have not yet been identified. And some variables
hypothesized to predict implementation, such as stability of
staff---principals, teachers, and literacy specialists---fail to do so
with consistency.

**Question 4: What features of the RFO program are effective in terms of
value added to student achievement?**

While our answer to this question is incomplete at this time, , both the
survey and the case study data suggest the following to be critical
features:

-   Consistent and uniform use of curriculum and materials aligned with
    research on best practices;

-   The presence of a coach or literacy specialist to support teachers
    and provide individual guidance;

-   Use of assessments for instruction decisionmaking;

-   Ongoing, systematic professional development, designed to move
    teachers from providing the basics of SBRR to addressing individual
    school and teacher needs; and

-   Principal engagement and leadership that includes both clear
    expectations for instruction and ongoing program monitoring.

**Question 5: What is the quality of professional development offered
through the RFO program?**

Professional development for teachers has had a strong impact on the
Reading First schools, as indicated by teachers and other respondents in
the Reading First schools. Teachers stated that professional development
influenced literacy instruction in the school due to improving
consistency in teaching literacy; providing up-to-date, research-based
information on literacy and the five components; helping them develop
insights into their students' needs; giving them new and effective
instructional strategies; and improving collaboration among teachers.\
The role of literacy specialist in providing support is seen as very
important.

**Question 6: What happens to implementation of the RFO program in a
school after the school's Reading First funding has been discontinued?**

In nearly all of the case study districts and schools, plans were made
to maintain the Reading First instructional model: using the same
reading materials, emphasizing the five components, using data to inform
instruction, and forming small groups. However, most of the districts
and schools will not maintain a high level of instructional supports due
to fiscal constraints that all districts but one have encountered.

Only two schools plan to retain a full-time literacy specialist. Two
others hope to, but a fiscal crisis may make it impossible.

**Appendix A. Responses to the Teacher Survey**

Table A-1.---Teachers' assessments of various aspects of professional
development: 2005 and 2007

  --------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------
  Reading-related             2005                             2007                             Gamma
  professional development                                                                      
  experience                                                                                    

                              Strongly   Disagree   Strongly   Strongly   Disagree   Strongly   
                              agree/\               disagree   agree/\               disagree   
                              agree                            agree                            

  They have drawn on my own   75.6       21.7       2.8        84.7       12.4       2.9        0.266
  input                                                                                         

  They have provided me with  94.2       4.0        1.8        89.4       7.2        3.4        
  new and useful information                                                                    
  on the five essential areas                                                                   
  within the Ohio reading                                                                       
  standards framework                                                                           

  They have taken away time   35.6       54.7       9.8        31.1       51.5       17.5       
  better spent on planning                                                                      
  lessons                                                                                       

  They have taken away time   48.4       43.5       8.1        39.3       45.1       15.5       -0.203
  better spent on teaching                                                                      

  They have been effectively  81.3       16.9       1.8        85.9       11.7       2.4        
  integrated into a                                                                             
  comprehensive plan that my                                                                    
  school has for instruction                                                                    

  They have been clearly      95.0       5.0        0.0        95.6       3.4        1.0        
  linked to the Ohio English                                                                    
  Language Arts (ELA) content                                                                   
  standards for K--3                                                                            

  They have included          83.0       15.2       1.8        78.0       19.1       2.9        
  information about reading                                                                     
  theory that initiated                                                                         
  change in my reading                                                                          
  instruction                                                                                   

  They have provided          86.0       11.7       2.3        91.3       5.3        3.4        
  opportunities to practice                                                                     
  what I've learned                                                                             

  They have provided adequate 52.3       40.1       7.7        67.8       25.0       7.2        0.270
  time to plan instruction                                                                      

  They have repeated          71.9       27.2       0.9        68.6       25.6       5.8        
  information that I already                                                                    
  knew                                                                                          

  They have provided          51.6       40.0       8.4        52.2       38.6       9.2        
  individualized technical\                                                                     
  assistance                                                                                    

  They have been supported by 88.3       9.9        1.8        91.3       6.8        1.9        
  district support services                                                                     
  and material resources                                                                        

  They have been led by       93.3       4.9        1.8        94.7       4.4        1.0         
  knowledgeable and effective                                                                   
  instructors                                                                                   
  --------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

**Table A-2.---Teachers' assessments of how adequately reading-related
topics are addressed: 2005 and 2007**

  ----------------------- -------------- ------------ ----------- -------------- ------------ ----------- --------
  Topic                   2005                                    2007                                    Gamma

                          More than      Less than    Not         More than      Less than    Not         
                          adequately/\   adequately   addressed   adequately/\   adequately   addressed   
                          adequately                              adequately                              

  Reading foundations,    86.1           11.2         2.7         81.9           14.7         3.4         
  theory, and speech to                                                                                   
  print development                                                                                       

  Phonemic awareness      94.7           5.3          0.0         87.5           7.2          5.3         -0.444
  development                                                                                             

  Systematic and explicit 93.3           6.7          0.0         87.1           8.6          4.3         -0.360
  phonics instruction                                                                                     

  Fluency elements (e.g., 93.3           6.2          0.4         95.2           2.4          2.4         
  automaticity, accuracy,                                                                                 
  prosody)                                                                                                

  Vocabulary (e.g.,       91.9           7.6          0.4         94.7           3.3          1.9         
  direct and contextual                                                                                   
  methods and word                                                                                        
  selection)                                                                                              

  Reading comprehension   91.0           9.0          0.0         94.7           5.3          0.0         
  strategies (e.g.,                                                                                       
  organization for                                                                                        
  instruction,                                                                                            
  before-during-after)                                                                                    

  Composition (e.g.,      73.0           22.1         5.0         82.8           15.3         1.9         0.281
  including reading to                                                                                    
  support writing,                                                                                        
  writing to support                                                                                      
  reading)                                                                                                

  Understanding the       91.1           8.4          0.4         90.4           7.7          1.9         
  Assess-Plan-Teach                                                                                       
  method of teaching                                                                                      

  Understanding the       86.2           11.2         2.7         90.3           8.7          1.0         
  concept of                                                                                              
  differentiation in                                                                                      
  general                                                                                                 

  Teaching reading to     29.8           28.9         41.3        27.1           29.5         43.5        
  English language                                                                                        
  learner (ELL) students                                                                                  

  Teaching reading to     24.1           43.8         32.1        34.1           37.5         28.4        
  students with diagnosed                                                                                 
  learning disabilities                                                                                   

  Teaching reading to     64.3           31.7         4.0         77.4           19.2         3.4         0.291
  struggling readers                                                                                      

  Teaching reading to     61.2           33.0         5.8         70.3           23.9         5.7         
  above average readers                                                                                   

  Small group teaching    76.8           19.2         4.0         87.1           10.0         2.9         0.326
  methods                                                                                                 

  Small group classroom   55.6           37.7         6.7         71.8           19.6         8.6         0.273
  management techniques                                                                                   

  Scientifically based    93.3           5.8          0.9         91.9           5.7          2.4         
  reading research (SBRR)                                                                                 

  Standards-based lessons 87.1           10.2         2.7         92.3           5.8          1.9         

  Administering,          89.3           10.7         0.0         93.8           5.8          0.5         
  analyzing, and                                                                                          
  interpreting assessment                                                                                 
  results (e.g., TPRI,                                                                                    
  DIBELS, TerraNova)                                                                                      

  Assessing individual    89.8           9.8          0.4         91.9           7.7          0.5         
  student's progress                                                                                      
  through diagnostic                                                                                      
  assessments                                                                                             

  Selecting instructional 77.9           18.5         3.6         82.2           13.9         3.8         
  materials and tools                                                                                     
  based on SBRR                                                                                           

  Selecting instructional 76.0           20.4         3.6         81.8           14.8         3.3         
  materials for students                                                                                  
  based on individual                                                                                     
  reading levels                                                                                          

  Alignment of            85.8           11.1         3.1         88.0           8.6          3.3         
  state/district                                                                                          
  assessments with                                                                                        
  reading curriculum and                                                                                  
  instruction                                                                                             

  Alignment and use of    79.6           16.9         3.6         82.7           12.5         4.8          
  English Language Arts                                                                                   
  (ELA) academic content                                                                                  
  standards and reading                                                                                   
  research for lesson                                                                                     
  planning                                                                                                
  ----------------------- -------------- ------------ ----------- -------------- ------------ ----------- --------

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

**Table A-3.---Teachers' opinions regarding services and support
provided by the Trio: 2005 and 2007**

  -------------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------
  Opinion                          2005                             2007                             Gamma

                                   Strongly   Disagree   Strongly   Strongly   Disagree   Strongly   
                                   agree/\               disagree   agree/\               disagree   
                                   agree                            agree                            

  The building Trio has            86.0       11.7       2.3        88.5       9.6        1.9        
  strengthened the reading program                                                                   
  in our school                                                                                      

  The building Trio has provided   86.1       12.6       1.3        89.9       9.1        1.0        
  services that were not available                                                                   
  in our school before                                                                               

  Services provided by the         17.1       61.3       21.6       17.8       48.1       34.1       -0.164
  building Trio have been of                                                                         
  little value                                                                                       

  Services provided by the         86.0       13.5       0.5        88.0       10.5       1.4        
  building Trio were delivered in                                                                    
  a timely manner                                                                                    

  The building Trio data manager   80.8       16.5       2.7        86.1       11.5       2.4        
  has helped teachers to monitor                                                                     
  student progress                                                                                   

  The building Trio resource       88.3       8.6        3.2        92.3       6.7        1.0         
  coordinator has helped teachers                                                                    
  to secure and/or share materials                                                                   
  necessary to teach reading                                                                         
  -------------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

**Table A-4.---Teachers' opinions regarding their school's current
reading program: 2005 and 2007**

  --------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------
  Opinion                     2005                             2007                             Gamma

                              Strongly   Disagree   Strongly   Strongly   Disagree   Strongly   
                              agree/\               disagree   agree/\               disagree   
                              agree                            agree                            

  There is strong and clear   92.0       7.1        0.9        89.0       10.0       1.0        
  leadership for the school's                                                                   
  reading program                                                                               

  There is a school-wide      93.8       5.8        0.4        88.0       11.5       0.5        -0.339
  commitment to beginning                                                                       
  reading instruction                                                                           
  grounded in scientifically                                                                    
  based reading research                                                                        
  (SBRR)                                                                                        

  The school uses SBRR        96.4       3.6        0.0        96.6       2.9        0.5        
  reading materials                                                                             

  Teachers support the        95.5       4.5        0.0        90.9       8.7        0.5        
  direction of the school's                                                                     
  reading program                                                                               

  Our school has a shared     91.1       8.5        0.4        85.5       14.0       0.5        
  vision of the school's                                                                        
  reading program                                                                               
  --------------------------- ---------- ---------- ---------- ---------- ---------- ---------- --------

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

**Table A-5.---Teachers' opinions regarding their school's current
reading program: 2005 and 2007**

  -------------------------------- ---------- ---------- ---------- ---------- ---------- ----------
  Opinion                          2005                             2007                  

                                   Strongly   Disagree   Strongly   Strongly   Disagree   Strongly
                                   agree/\               disagree   agree/\               disagree
                                   agree                            agree                 

  The reading program is closely   94.1       5.0        0.9        92.3       7.7        0.0
  coordinated across teachers at                                                          
  the same grade level                                                                    

  The reading program is closely   92.4       6.7        0.9        89.9       9.6        0.5
  coordinated across instructional                                                        
  staff in the K--3 grades                                                                

  Our school is characterized by a 79.4       18.8       1.8        78.6       20.9       0.5
  high degree of collaboration                                                            
  among teachers in teaching                                                              
  reading                                                                                 

  The reading program is closely   97.8       2.2        0.0        97.6       2.4        0.0
  aligned with Ohio's English                                                             
  Language Arts content standards                                                         
  for grades K--3                                                                         

  Our intervention programs for    87.6       11.1       1.3        87.1       10.0       2.9
  struggling readers are                                                                  
  coordinated with the core                                                               
  reading program                                                                         
  -------------------------------- ---------- ---------- ---------- ---------- ---------- ----------

**Table A-6.---Frequency of teachers' use of various instructional
materials for struggling readers: 2005 and 2007**

<table style="width:100%;">
<colgroup>
<col style="width: 25%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 14%" />
</colgroup>
<tbody>
<tr class="odd">
<td rowspan="2">Instructional material</td>
<td colspan="4">2005</td>
<td colspan="4">2007</td>
<td rowspan="2">Gamma</td>
</tr>
<tr class="even">
<td><p>Always/</p>
<p>almost always</p></td>
<td>Often</td>
<td>Some-times</td>
<td><p>Rarely/</p>
<p>never</p></td>
<td><p>Always/</p>
<p>almost always</p></td>
<td>Often</td>
<td>Some-times</td>
<td><p>Rarely/</p>
<p>never</p></td>
</tr>
<tr class="odd">
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>Narrative textbook</td>
<td>51.1</td>
<td>31.1</td>
<td>9.1</td>
<td>8.7</td>
<td>42.0</td>
<td>40.5</td>
<td>11.7</td>
<td>5.9</td>
<td></td>
</tr>
<tr class="odd">
<td>Informational textbook</td>
<td>33.6</td>
<td>40.5</td>
<td>16.8</td>
<td>9.1</td>
<td>32.2</td>
<td>42.8</td>
<td>20.2</td>
<td>4.8</td>
<td></td>
</tr>
<tr class="even">
<td>Phonics workbooks</td>
<td>46.2</td>
<td>32.3</td>
<td>13.9</td>
<td>7.6</td>
<td>47.3</td>
<td>26.6</td>
<td>19.8</td>
<td>6.3</td>
<td></td>
</tr>
<tr class="odd">
<td>Narrative decodable or leveled books</td>
<td>66.1</td>
<td>25.4</td>
<td>6.7</td>
<td>1.8</td>
<td>65.1</td>
<td>29.2</td>
<td>5.3</td>
<td>0.5</td>
<td></td>
</tr>
<tr class="even">
<td>Children’s narrative trade books</td>
<td>38.7</td>
<td>39.2</td>
<td>15.3</td>
<td>6.8</td>
<td>40.3</td>
<td>41.3</td>
<td>15.0</td>
<td>3.4</td>
<td></td>
</tr>
<tr class="odd">
<td>Student writing</td>
<td>35.7</td>
<td>40.2</td>
<td>18.8</td>
<td>5.4</td>
<td>39.9</td>
<td>37.0</td>
<td>17.8</td>
<td>5.3</td>
<td></td>
</tr>
<tr class="even">
<td>Teacher-made materials</td>
<td>33.8</td>
<td>40.9</td>
<td>23.6</td>
<td>1.8</td>
<td>36.5</td>
<td>39.9</td>
<td>20.2</td>
<td>3.4</td>
<td></td>
</tr>
<tr class="odd">
<td>Computer reading programs/ reading games</td>
<td>31.8</td>
<td>31.4</td>
<td>13.9</td>
<td>22.9</td>
<td>45.5</td>
<td>28.7</td>
<td>14.4</td>
<td>11.5</td>
<td>0.247</td>
</tr>
<tr class="even">
<td>Overheads/pictures/other visuals</td>
<td>37.1</td>
<td>37.1</td>
<td>18.8</td>
<td>7.1</td>
<td>35.4</td>
<td>37.3</td>
<td>19.6</td>
<td>7.7</td>
<td></td>
</tr>
<tr class="odd">
<td>Audiotape cassettes</td>
<td>34.8</td>
<td>43.3</td>
<td>14.3</td>
<td>7.6</td>
<td>39.2</td>
<td>36.8</td>
<td>20.6</td>
<td>3.3</td>
<td></td>
</tr>
<tr class="even">
<td>At-home extensions/enrichment activities</td>
<td>23.0</td>
<td>37.4</td>
<td>28.8</td>
<td>10.8</td>
<td>21.2</td>
<td>35.6</td>
<td>29.8</td>
<td>13.5</td>
<td></td>
</tr>
</tbody>
</table>

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

Table A-7.---Frequency of teachers' use of various tools/sources to
assess student improvement: 2005 and 2007

<table>
<colgroup>
<col style="width: 26%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
<col style="width: 8%" />
</colgroup>
<tbody>
<tr class="odd">
<td rowspan="2">Tool/source</td>
<td colspan="4">2005</td>
<td colspan="4">2007</td>
<td rowspan="2">Gamma</td>
</tr>
<tr class="even">
<td><p>Always/</p>
<p>almost always</p></td>
<td>Often</td>
<td>Some-times</td>
<td><p>Rarely/</p>
<p>never</p></td>
<td><p>Always/</p>
<p>almost always</p></td>
<td>Often</td>
<td>Some-times</td>
<td><p>Rarely/</p>
<p>never</p></td>
</tr>
<tr class="odd">
<td>DIBELS assessments</td>
<td>37.8</td>
<td>41.8</td>
<td>16.9</td>
<td>3.6</td>
<td>44.7</td>
<td>44.2</td>
<td>9.1</td>
<td>1.9</td>
<td>0.180</td>
</tr>
<tr class="even">
<td>Other diagnostic assessments (e.g., Rigby benchmarking, running
records)</td>
<td>11.7</td>
<td>40.5</td>
<td>30.6</td>
<td>17.1</td>
<td>16.7</td>
<td>40.2</td>
<td>24.9</td>
<td>18.2</td>
<td></td>
</tr>
<tr class="odd">
<td>TPRI assessment</td>
<td>9.5</td>
<td>15.4</td>
<td>44.8</td>
<td>30.3</td>
<td>12.6</td>
<td>15.9</td>
<td>36.7</td>
<td>34.8</td>
<td></td>
</tr>
<tr class="even">
<td>TerraNova assessment</td>
<td>2.3</td>
<td>1.9</td>
<td>28.0</td>
<td>67.8</td>
<td>6.8</td>
<td>9.3</td>
<td>32.2</td>
<td>51.7</td>
<td>0.344</td>
</tr>
<tr class="odd">
<td>Assessments that are part of the core reading series/other classroom
tests</td>
<td>47.6</td>
<td>36.0</td>
<td>11.1</td>
<td>5.3</td>
<td>50.5</td>
<td>32.7</td>
<td>12.0</td>
<td>4.8</td>
<td></td>
</tr>
<tr class="even">
<td>Student work products</td>
<td>68.9</td>
<td>26.2</td>
<td>4.0</td>
<td>0.9</td>
<td>68.9</td>
<td>27.3</td>
<td>2.9</td>
<td>1.0</td>
<td></td>
</tr>
<tr class="odd">
<td>Student interest in reading and writing</td>
<td>55.1</td>
<td>32.0</td>
<td>11.1</td>
<td>1.8</td>
<td>53.1</td>
<td>33.5</td>
<td>10.5</td>
<td>2.9</td>
<td></td>
</tr>
</tbody>
</table>

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

Table A-8.---Teachers' assessments of the effect of sources on reading
instruction: 2005 and 2007

+----------------------+------+------+------+------+------+------+----+
| Factor               | 2005 |      |      | 2007 |      |      | G  |
|                      |      |      |      |      |      |      | am |
|                      |      |      |      |      |      |      | ma |
+----------------------+------+------+------+------+------+------+----+
|                      | Sub  | Some | No   | Sub  | Some | No   |    |
|                      | stan | ef   | ef   | stan | ef   | ef   |    |
|                      | tial | fect | fect | tial | fect | fect |    |
|                      |      |      |      |      |      |      |    |
|                      | ef   |      |      | ef   |      |      |    |
|                      | fect |      |      | fect |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Feedback received    | 19.1 | 55.1 | 25.8 | 24.0 | 50.5 | 25.5 |    |
| through the Early    |      |      |      |      |      |      |    |
| Language and         |      |      |      |      |      |      |    |
| Literacy Classroom   |      |      |      |      |      |      |    |
| Observation (ELLCO)  |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Feedback received    | 14.8 | 43.8 | 41.4 | 19.9 | 43.3 | 36.8 |    |
| through the Coaching |      |      |      |      |      |      |    |
| for Effective        |      |      |      |      |      |      |    |
| Classroom            |      |      |      |      |      |      |    |
| Instruction:         |      |      |      |      |      |      |    |
| Close-Ups of         |      |      |      |      |      |      |    |
| Teaching tool        |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Other school-based   | 42.8 | 47.3 | 9.9  | 47.6 | 39.9 | 12.5 |    |
| assistance from      |      |      |      |      |      |      |    |
| Reading First        |      |      |      |      |      |      |    |
| building Trio (e.g., |      |      |      |      |      |      |    |
| from literacy        |      |      |      |      |      |      |    |
| specialist, data     |      |      |      |      |      |      |    |
| manager, or resource |      |      |      |      |      |      |    |
| coordinator)         |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Assistance from      | 18.6 | 51.4 | 30.0 | 18.1 | 45.6 | 36.3 |    |
| non-Reading First    |      |      |      |      |      |      |    |
| consultants (i.e.,   |      |      |      |      |      |      |    |
| textbook             |      |      |      |      |      |      |    |
| consultants, guided  |      |      |      |      |      |      |    |
| reading experts)     |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| State-sponsored      | 28.6 | 33.2 | 38.2 | 23.4 | 33.8 | 42.8 |    |
| Summer Institutes    |      |      |      |      |      |      |    |
| for Reading          |      |      |      |      |      |      |    |
| Instruction (SIRI)   |      |      |      |      |      |      |    |
| (at a school or      |      |      |      |      |      |      |    |
| regional location)   |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Other formal         | 32.6 | 52.8 | 14.7 | 32.5 | 51.5 | 16.0 |    |
| coursework in        |      |      |      |      |      |      |    |
| reading              |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| State and/or         | 67.6 | 29.8 | 2.7  | 69.6 | 26.1 | 4.3  |    |
| district standards   |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| The Reading First    | 57.1 | 35.7 | 7.1  | 59.3 | 32.1 | 8.6  |    |
| professional         |      |      |      |      |      |      |    |
| development          |      |      |      |      |      |      |    |
| curriculum           |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| State curriculum     | 48.6 | 45.9 | 5.5  | 45.9 | 45.4 | 8.7  |    |
| framework or         |      |      |      |      |      |      |    |
| curriculum           |      |      |      |      |      |      |    |
| recommendations      |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| District curriculum  | 53.1 | 40.2 | 6.7  | 51.7 | 40.6 | 7.7  |    |
| framework or         |      |      |      |      |      |      |    |
| recommendations      |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Materials, books,    | 78.7 | 20.9 | 0.4  | 78.8 | 20.2 | 1.0  |    |
| and resources        |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Recommendations from | 21.6 | 46.8 | 31.5 | 25.6 | 40.6 | 33.8 |    |
| the school principal |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Recommendations from | 35.6 | 56.0 | 8.4  | 50.0 | 41.7 | 8.3  | 0  |
| other teachers       |      |      |      |      |      |      | .2 |
|                      |      |      |      |      |      |      | 30 |
+----------------------+------+------+------+------+------+------+----+
| TPRI, DIBELS,        | 62.2 | 33.3 | 4.4  | 63.0 | 32.7 | 4.3  |    |
| TerraNova student    |      |      |      |      |      |      |    |
| assessment data      |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Reading books,       | 40.0 | 52.9 | 7.1  | 37.8 | 48.8 | 13.4 |    |
| journals, reports,   |      |      |      |      |      |      |    |
| etc., about current  |      |      |      |      |      |      |    |
| reading research and |      |      |      |      |      |      |    |
| effective strategies |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+
| Conferences at which | 24.7 | 44.3 | 31.1 | 18.9 | 47.1 | 34.0 |    |
| reading research is  |      |      |      |      |      |      |    |
| presented or         |      |      |      |      |      |      |    |
| discussed (national, |      |      |      |      |      |      |    |
| state, local)        |      |      |      |      |      |      |    |
+----------------------+------+------+------+------+------+------+----+

NOTE: Shading indicates that the 2005--07 difference is significant at
the p \<.05 level. Percents may not add to 100 due to rounding.

**Figure A-1.---Frequency of cohort 2 teachers' use of phonics
workbooks, by grade level: 2007**

![](media/image37.wmf){width="5.9944444444444445in"
height="3.3208333333333333in"}

**Figure A-2.---Frequency of cohort 2 teachers' use of student writing,
by grade level: 2007**

![](media/image38.wmf){width="5.9944444444444445in"
height="3.3208333333333333in"}

NOTE: Percents may not add to 100 due to rounding.

**Figure A-3.---Frequency of cohort 2 teachers' use of computer reading
programs/reading games,\
by grade level: 2007**

![](media/image39.wmf){width="5.985416666666667in"
height="2.9930555555555554in"}

**Figure A-4.---Frequency of cohort 2 teachers' use of
overheads/pictures/other visuals, by grade level: 2007**

![](media/image40.wmf){width="5.985416666666667in"
height="3.0229166666666667in"}

**Figure A-5.---Frequency of cohort 2 teachers' use of at-home
extensions/enrichment activities,\
by grade level: 2007**

![](media/image41.wmf){width="5.985416666666667in"
height="3.0034722222222223in"}

NOTE: Percents may not add to 100 due to rounding.

**Figure A-6.---Frequency of cohort 2 teachers' use of assessing student
performance with reliable and valid measures, by grade level: 2007**

![](media/image42.wmf){width="6.495833333333334in"
height="3.261111111111111in"}

NOTE: Percents may not add to 100 due to rounding.

**Figure A-7.---Frequency of cohort 2 teachers' diagnosing students who
fall behind to pinpoint reading strengths/weaknesses, by grade level:
2007**

![](media/image43.wmf){width="6.495833333333334in"
height="3.251388888888889in"}

**Figure A-8.---Frequency of cohort 2 teachers' use of assessment data
to decide what scientifically based reading research (SBRR) content
needs to be taught, by grade level: 2007**

![](media/image44.wmf){width="6.495833333333334in"
height="2.7944444444444443in"}

NOTE: Percents may not add to 100 due to rounding.

**Figure A-9.---Frequency of cohort 2 teachers' use of monitoring
independent student performance of the task, by grade level: 2007**

![](media/image45.wmf){width="6.495833333333334in"
height="3.2715277777777776in"}

**Figure A-10.---Frequency of cohort 2 teachers' promoting
self-regulation, by grade level: 2007**

![](media/image46.wmf){width="6.495833333333334in"
height="3.2416666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-11.---Frequency of cohort 2 teachers' use of engaging
students in identifying phonemes, by grade level: 2007**

![](media/image47.wmf){width="6.495833333333334in" height="3.28125in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-12.---Frequency of cohort 2 teachers' use of engaging
students in categorizing phonemes, by grade level: 2007**

![](media/image48.wmf){width="6.495833333333334in" height="3.23125in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-13.---Frequency of cohort 2 teachers' use of engaging
students in blending phonemes to form words, by grade level: 2007**

![](media/image49.wmf){width="6.495833333333334in" height="3.28125in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-14.---Frequency of cohort 2 teachers' use of engaging
students in segmenting words into phonemes, by grade level: 2007**

![](media/image50.wmf){width="6.495833333333334in" height="3.23125in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-15.---Frequency of cohort 2 teachers' use of engaging
students in deleting or adding phonemes to form new words, by grade
level: 2007**

![](media/image51.wmf){width="6.495833333333334in"
height="3.261111111111111in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-16.---Frequency of cohort 2 teachers' use of engaging
students in substituting phonemes to make new words, by grade level:
2007**

![](media/image52.wmf){width="6.495833333333334in"
height="3.2416666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-17.---Frequency of cohort 2 teachers' use of engaging
students in breaking spoken words into sounds, by grade level: 2007**

![](media/image53.wmf){width="6.495833333333334in"
height="3.2416666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-18.---Frequency of cohort 2 teachers' use of engaging
students in blending sounds to form words, by grade level: 2007**

![](media/image54.wmf){width="6.495833333333334in"
height="3.251388888888889in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-19.---Frequency of cohort 2 teachers' use of engaging
students in applying knowledge of phonics as they read words, sentences,
and text, by grade level: 2007**

![](media/image55.wmf){width="6.495833333333334in"
height="3.2715277777777776in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-20.---Frequency of cohort 2 teachers' use of engaging
students in reading and re-reading passages orally as they receive
guidance, by grade level: 2007**

![](media/image56.wmf){width="6.495833333333334in"
height="3.251388888888889in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-21.---Frequency of cohort 2 teachers' use of engaging
students in actively engaging with words, by grade level: 2007**

![](media/image57.wmf){width="6.495833333333334in"
height="3.2416666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-22.---Frequency of cohort 2 teachers' use of engaging
students in participating in word analysis---compound words,
inflectional endings, by grade level: 2007**

![](media/image58.wmf){width="6.495833333333334in"
height="3.2416666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-23.---Frequency of cohort 2 teachers' use of engaging
students in using dictionaries and other reference aids to learn word
meanings, by grade level: 2007**

![](media/image59.wmf){width="6.495833333333334in"
height="3.2069444444444444in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-24.---Frequency of cohort 2 teachers' use of engaging
students in using graphic and semantic organizers to focus on concepts
and how they are related to other concepts, by grade level: 2007**

![](media/image60.wmf){width="6.495833333333334in"
height="3.216666666666667in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-25.---Frequency of cohort 2 teachers' use of engaging
students in recognizing story structure to identify and interpret story
events/narrative elements, by grade level: 2007**

![](media/image61.wmf){width="6.495833333333334in"
height="3.2069444444444444in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-26.---Frequency of cohort 2 teachers' use of engaging
students in participating in reciprocal teaching conversations, by grade
level: 2007**

![](media/image62.wmf){width="6.495833333333334in"
height="3.286111111111111in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-27.---Frequency of cohort 2 teachers' use of engaging
students in predicting what might occur in the text, by grade level:
2007**

![](media/image63.wmf){width="6.495833333333334in"
height="3.2069444444444444in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-28.---Frequency of cohort 2 teachers' putting more emphasis
on vocabulary instruction, by grade level: 2007**

![](media/image64.wmf){width="6.495833333333334in"
height="3.1618055555555555in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-29.---Frequency of cohort 2 teachers' putting more emphasis
on phonemic awareness instruction, by grade level: 2007**

![](media/image65.wmf){width="6.495833333333334in"
height="3.2069444444444444in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Figure A-30.---Frequency of cohort 2 teachers' putting more emphasis
on systematic phonics instruction, by grade level: 2007**

![](media/image66.wmf){width="6.495833333333334in"
height="3.1618055555555555in"}

NOTE: Percents may not add to 100 due to missing or non-ascertained data
or rounding.

**Appendix B. Responses to the Principal Survey**

**Table B-1.---Principals' assessments of the professional development
offered to teachers during the school year: 2005 and 2007**

  ----------------------------- ---------- ---------- ---------- ---------- ---------- ----------
  Statement                     2005                             2007                  

                                Strongly   Disagree   Strongly   Strongly   Disagree   Strongly
                                agree/\               disagree   agree/\               disagree
                                agree                            agree                 

  Kindergarten through third    50.0       31.3       18.8       62.5       25.0       12.5
  grade teachers select the                                                            
  specific professional                                                                
  development activities that                                                          
  they attend and/or                                                                   
  participate in                                                                       

  Reading-related professional  100.0      0.0        0.0        93.8       0.0        6.3
  development is readily                                                               
  available and accessible for                                                         
  K--3 teachers (e.g.,                                                                 
  convenient location and/or                                                           
  timing)                                                                              

  After initial reading-related 100.0      0.0        0.0        87.5       0.0        12.5
  professional development, our                                                        
  literacy specialist provides                                                         
  follow-up to teachers                                                                

  Our school's K--3 instruction 100.0      0.0        0.0        93.8       0.0        6.3
  is influenced by                                                                     
  reading-related professional                                                         
  development                                                                          

  Self-reflection is part of    100.0      0.0        0.0        93.8       0.0        6.3
  teachers' professional                                                               
  development plans                                                                    

  The assistance of outside     86.7       13.3       0.0        80.0       20.0       0.0
  experts (e.g., text-book or                                                          
  university consultants) is                                                           
  available and used as needed                                                         
  in professional development                                                          

  Professional development      81.3       18.8       0.0        80.0       20.0       0.0
  provides adequate time for                                                           
  teachers to plan                                                                     

  Professional development      73.3       26.7       0.0        81.3       12.5       6.3
  provides adequate time for                                                           
  teachers to practice what                                                            
  they've learned                                                                      

  Professional development is   100.0      0.0        0.0        87.5       6.3        6.3
  integrated into a long-range                                                         
  plan that has school-wide                                                            
  support                                                                              

  Professional development is   100.0      0.0        0.0        93.8       0.0        6.3
  connected to Ohio's English                                                          
  Language Arts (ELA) standards                                                        

  Professional development      93.8       6.3        0.0        75.0       18.8       6.3
  targets specific teacher                                                             
  needs                                                                                

  There are structured          93.8       6.3        0.0        93.8       0.0        6.3
  opportunities for Reading                                                            
  First staff and staff                                                                
  involved in other literacy                                                           
  initiatives to exchange ideas                                                        
  (i.e., through meetings or                                                           
  professional development                                                             
  sessions)                                                                            

  After January 1, 2005, The    100.0      0.0        0.0        90.9       0.0        9.1
  Reading First-Ohio                                                                   
  Professional Development                                                             
  Program Review Guide was used                                                        
  to evaluate all specific                                                             
  professional development                                                             
  programs in reading                                                                  
  ----------------------------- ---------- ---------- ---------- ---------- ---------- ----------

NOTE: Percents may not add to 100 due to rounding.

Table B-2.---Principals' assessments about the extent to which the
various professional development opportunities had an effect to support
literacy: 2005 and 2007

  --------------------------- ------------- -------- -------- ------------- -------- --------
  Professional development    2005                            2007                   
  opportunity                                                                        

                              Substantial   Some     No       Substantial   Some     No
                              effect        effect   effect   effect        effect   effect

  New Leadership Meeting      57.1          35.7     7.1      50.0          41.7     8.3
  (October)                                                                          

  Classroom Walk-Through      54.5          45.5     0.0      50.0          43.8     6.3
  (CWT) Training\                                                                    
  (1-1/2 days)                                                                       

  Advanced Classroom          50.0          50.0     0.0      46.2          46.2     7.7
  Walk-Through (CWT) Training                                                        
  (1 day)                                                                            

  Data summit I (December)    53.3          40.0     6.7      21.4          71.4     7.1

  Sustainability meeting with 46.2          38.5     15.4     27.3          54.5     18.2
  Mike Schmoker (December)                                                           

  Reading First-Ohio          28.6          57.1     14.3     62.5          31.3     6.3
  school-based professional                                                          
  development for teachers                                                           
  --------------------------- ------------- -------- -------- ------------- -------- --------

NOTE: Percents may not add to 100 due to rounding.

Table B-3.---Principals' opinions about the professional development
sessions relative to the CWT

  ----------------------------------------- ---------- ---------- ---------- -------
  Statement                                 Strongly   Disagree   Strongly   Don't
                                            agree/\               disagree   know
                                            agree                            

  The Classroom Walk-Through Model (i.e.,   100.0      0.0        0.0        0.0
  review curriculum, identify instructional                                  
  strategies, assess student engagement,                                     
  survey the learning environment, analyze                                   
  the data, and reflect with the teacher)                                    
  is easy for me to use                                                      

  It is feasible for me to review classroom 80.0       20.0       0.0        0.0
  data prior to or after conducting a CWT                                    

  The professional development sessions     93.3       0.0        0.0        6.7
  covered what to look for in a CWT                                          

  The timeframe associated with the         93.3       6.7        0.0        0.0
  classroom walk-through model (2--4                                         
  minutes) is adequate for recording the                                     
  "look-for's"                                                               

  The professional development sessions     93.3       6.7        0.0        0.0
  adequately covered how to develop a                                        
  reflective prompt for conversations with                                   
  teachers                                                                   

  I have been able to engage in reflective  80.0       20.0       0.0        0.0
  conversations with teachers post-CWT                                       

  The CWT is a good idea, but I have        76.9       23.1       0.0        0.0
  limited time to do it given my current                                     
  responsibilities                                                           

  The CWT professional development sessions 80.0       20.0       0.0        0.0
  strengthened my role as instructional                                      
  leader in the building                                                     
  ----------------------------------------- ---------- ---------- ---------- -------

**Table B-4.---Principals' opinions regarding procedures and/or
guidelines for coordination and integration of the school's reading
program**

  -------------------------- --------- ----------- ------- --------- ----------- -------
  Area                       2005                          2007                  

                             Fully/\   Partially   Not at  Fully/\   Partially   Not at
                             mostly                all     mostly                all

  Linking expectations for   93.8      6.3         0.0     87.5      12.5        0.0
  student learning to the                                                        
  Ohio English Language Arts                                                     
  (ELA) academic standards                                                       
  for grades K--3                                                                

  Linking expectations for   93.8      6.3         0.0     93.8      6.3         0.0
  student learning to the                                                        
  reading curriculum for                                                         
  grades K--3                                                                    

  Linking Reading First-Ohio 75.0      12.5        12.5    87.5      12.5        0.0
  to the overall school                                                          
  improvement plan                                                               

  Coordinating reading       35.7      14.3        50.0    25.0      18.8        56.3
  instruction with "feeder"                                                      
  preschool programs                                                             
  -------------------------- --------- ----------- ------- --------- ----------- -------

NOTE: Percents may not add to 100 due to rounding.

**Table B-5.---Principals' opinions about how to monitor the reading
program and to use the results**

  ---------------------------- ---------- ---------- ---------- ---------- ---------- ----------
  Statement                    2005                             2007                  

                               Strongly   Disagree   Strongly   Strongly   Disagree   Strongly
                               agree/\               disagree   agree/\               disagree
                               agree                            agree                 

  **Status and Use**                                                                  

  Procedures have been         100.0      0.0        0.0        100.0      0.0        0.0
  developed for collecting and                                                        
  using data                                                                          

  Procedures have been         100.0      0.0        0.0        100.0      0.0        0.0
  implemented for collecting                                                          
  and using data                                                                      

  There is one uniform         81.3       18.8       0.0        87.5       12.5       0.0
  reporting system used at                                                            
  both the district and school                                                        
  levels                                                                              

  Results from program         66.7       33.3       0.0        81.3       18.8       0.0
  monitoring help to evaluate                                                         
  teacher performance                                                                 

  Results from program         100.0      0.0        0.0        100.0      0.0        0.0
  monitoring help to evaluate                                                         
  student needs and inform                                                            
  interventions                                                                       

  Results from program         93.8       6.3        0.0        100.0      0.0        0.0
  monitoring help our school                                                          
  make better use of                                                                  
  instructional materials                                                             

  The monitoring and program   87.5       12.5       0.0        100.0      0.0        0.0
  improvements specific to our                                                        
  school's Reading First-Ohio                                                         
  program are linked to our                                                           
  overall school improvement                                                          
  plan                                                                                

                                                                                      

  **Quality of Information**                                                          

  Our school monitoring system 86.7       13.3       0.0        93.8       6.3        0.0
  yields information                                                                  
  sufficient to inform                                                                
  building-level                                                                      
  decisionmaking                                                                      

  Our school monitoring system 100.0      0.0        0.0        93.8       6.3        0.0
  yields information                                                                  
  sufficient to inform                                                                
  classroom-level or                                                                  
  instructional decisionmaking                                                        

  Our school monitoring system 100.0      0.0        0.0        93.8       6.3        0.0
  yields information                                                                  
  sufficient to document and                                                          
  demonstrate accountability                                                          
  for Reading First-Ohio state                                                        
  and federal requirements                                                            
  ---------------------------- ---------- ---------- ---------- ---------- ---------- ----------

NOTE: Percents may not add to 100 due to rounding.

**Table B-6.---Principals' responses about the frequency of use of
various program monitoring strategies**

  -------------------- ---------- ------- ------------ ---------- ---------- ------- ------------ ----------
  Strategy             2005                                       2007                            

                       Always/\   Often   Some-times   Rarely/\   Always/\   Often   Some-times   Rarely/\
                       almost                          never      almost                          never
                       always                                     always                          

  Student assessment   75.0       25.0    0.0          0.0        75.0       25.0    0.0          0.0
  data from                                                                                       
  standardized                                                                                    
  assessments\                                                                                    
  (e.g., DIBELS, TPRI,                                                                            
  TerraNova)                                                                                      

  Other student        13.3       26.7    26.7         33.3       43.8       37.5    12.5         6.3
  diagnostic data\                                                                                
  (e.g., Rigby                                                                                    
  benchmarking, other                                                                             
  reading tasks)                                                                                  

  Student assessment   31.3       50.0    6.3          12.5       43.8       50.0    0.0          6.3
  data from                                                                                       
  textbook/classroom                                                                              
  tests in reading                                                                                

  Teacher attendance   62.5       31.3    6.3          0.0        56.3       25.0    12.5         6.3
  at professional                                                                                 
  development sessions                                                                            

  Discussions with     18.8       31.3    43.8         6.3        12.5       50.0    31.3         6.3
  parents                                                                                         

  Discussions with     50.0       43.8    6.3          0.0        40.0       53.3    6.7          0.0
  individual teachers                                                                             

  Discussions with     62.5       37.5    0.0          0.0        56.3       37.5    6.3          0.0
  literacy specialists                                                                            

  Discussions at       56.3       25.0    12.5         6.3        50.0       31.3    12.5         6.3
  faculty or                                                                                      
  grade-level meetings                                                                            
  (e.g., as an agenda                                                                             
  item)                                                                                           

  Classroom            56.3       37.5    6.3          0.0        56.3       43.8    0.0          0.0
  observations                                                                                    
  (including                                                                                      
  walk-throughs)                                                                                  

  Teacher              31.3       43.8    18.8         6.3        31.3       31.3    31.3         6.3
  self-assessment of                                                                              
  effectiveness in                                                                                
  teaching the five                                                                               
  essential reading                                                                               
  components                                                                                      

  Teachers'            18.8       50.0    25.0         6.3        26.7       53.3    20.0         0.0
  assignments to                                                                                  
  students                                                                                        

  Student work         25.0       43.8    25.0         6.3        18.8       56.3    25.0         0.0
  products                                                                                        
  -------------------- ---------- ------- ------------ ---------- ---------- ------- ------------ ----------

NOTE: Percents may not add to 100 due to rounding.

**Appendix C**

**Reading First-Ohio Surveys**

**READING FIRST-OHIO**

**THE OHIO DEPARTMENT OF EDUCATION**

**SURVEY OF ELEMENTARY SCHOOL TEACHERS**

![](media/image67.wmf){width="1.961111111111111in"
height="2.410416666666667in"}

**SPRING 2007**

> Affix Label Here

Cohort 2

**SPRING 2007**

**READING FIRST-OHIO**

The ohio department of education (ODE)

Survey of GradE K--3 elementary school Teachers

+-----------------------------------+-----------------------------------+
| **PLEASE RETURN COMPLETED SURVEY  | **IF YOU HAVE ANY QUESTIONS       |
| IN THE PRE-ADDRESSED ENVELOPE TO  | CONCERNING THIS SURVEY\           |
| WESTAT**                          | PLEASE CALL OR E--MAIL:**         |
|                                   |                                   |
| > Westat\                         | > Dr. Amber Winkler               |
| > Reading First-Ohio Surveys      | >                                 |
| >                                 | > 1--800--937--8281, ext. 2285    |
| > 1650 Research Boulevard         | >                                 |
| >                                 | > <amberwinkler@westat.com>       |
| > TB 146F                         |                                   |
| >                                 |                                   |
| > Rockville, MD 20850             |                                   |
|                                   |                                   |
| **[OR]{.underline} FAX TO:**      |                                   |
|                                   |                                   |
| > 1--800--533--0239               |                                   |
|                                   |                                   |
| Attn: Reading First-Ohio Surveys  |                                   |
+-----------------------------------+-----------------------------------+

*Please respond by:* **April 27, 2007.**

All information that would permit identification of the individual
respondent will be held in strict confidence, will be used only by
persons engaged in and for the purposes of the survey, and will not be
disclosed or released to others for any purpose as required by law.

READing first-ohio

Survey of Grade K--3\
elementary school Teachers

***Thank you for taking the time to answer this survey about reading in
your school. We are interested in your feedback about classroom
instruction, professional development opportunities, and your school's
reading program. This survey is intended for all teachers in grades K--3
who teach reading and should take no longer than 45 minutes to complete.
Please be reminded that each of your responses is important to our
understanding of the Reading First-Ohio program; thus, it is critical
that you answer every question. Following up on missing survey responses
is time-consuming for all involved. If you have any questions about this
survey, please contact Amber Winkler toll-free at 1--800--937--8281,
ext. 2285 or at amberwinkler@westat.com.***

A. YOUR CLASSROOM INSTRUCTION IN READING

This brief section includes questions about the grades you teach and
time spent teaching reading/language arts.

**1. Do you teach reading in the following grades?** *(Please circle one
response for each item.)*

Yes No

a\. Kindergarten 1 2

b\. First grade 1 2

c\. Second grade 1 2

d\. Third grade 1 2

e\. Fourth grade 1 2

f\. Fifth grade 1 2

g\. Sixth grade 1 2

h\. Other *(please specify):* 1 2

**2. How would you describe your teaching role?**

Regular classroom teacher 1

Special needs teacher 2

Literacy specialist 3

Other (*please specify*): 4

**3. On average during this school year (2006--07), how many minutes PER
DAY do you spend teaching reading/language arts to your class?** *If you
teach more than one class, respond for the first class you teach.*

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Minutes PER DAY

**4. During this school year (2006--07), has the average number of
minutes you spend each day teaching reading/language arts to this class
decreased, remained the same, or increased from last year (2005--06)?**
*(Please circle one.)*

Decreased 1

Remained the same 2

Increased 3

This section includes questions (5 through 9) about each of the five
dimensions of reading, as well as related writing skills. The five
dimensions of reading are as follows:

1\) [Phonemic awareness]{.underline}: teaching children to focus on and
manipulate phonemes in spoken syllables and words.

2\) [Phonics]{.underline}: teaching children to use letter-sound
relations, stressing the acquisition of letter-sound correspondences and
their use in reading and spelling.

3\) [Oral reading fluency]{.underline}: teaching children to read orally
with speed, accuracy, and proper expression.

4\) [Vocabulary]{.underline}: teaching children the meaning and
pronunciation of words (both oral and print).

5\) [Reading comprehension]{.underline}: teaching children to derive
meaning from text by engaging in intentional, problem-solving thinking
processes.

Please keep in mind that questions in this section address instruction
across grade levels K--3; therefore, a "never" response may be
appropriate.

***5. During the past school year (2006--07), how often have you asked
your students to do the following PHONEMIC AWARENESS activities?**
(Please circle one response for each item.)*

  ------------------------------------- ------- --------- ------ --------------- -------
  Activity                              Daily   Several   Once a Occa-sionally   Never
                                                days a    week                   
                                                week                             

                                                                                 
  ------------------------------------- ------- --------- ------ --------------- -------

a\. Identify phonemes 1 2 3 4 5

b\. Categorize phonemes 1 2 3 4 5

c\. Blend phonemes to form words 1 2 3 4 5

d\. Segment words into phonemes 1 2 3 4 5

e\. Delete or add phonemes to form new words 1 2 3 4 5

f\. Substitute phonemes to make new words 1 2 3 4 5

***6. During the past school year (2006--07), how often have you asked
your students to do the following PHONICS activities?** (Please circle
one response for each item.)*

  ------------------------------------- ------- --------- ------ --------------- -------
  Activity                              Daily   Several   Once a Occa-sionally   Never
                                                days a    week                   
                                                week                             

                                                                                 
  ------------------------------------- ------- --------- ------ --------------- -------

a\. Break spoken words into sounds 1 2 3 4 5

b\. Blend sounds to form words 1 2 3 4 5

c\. Apply knowledge of phonics as they read words, sentences, and text 1
2 3 4 5

***7. During the past school year (2006--07), how often have you asked
your students to do the following ORAL READING FLUENCY activities?**
(Please circle one response for each item.)*

  ------------------------------------- ------- --------- ------ --------------- -------
  Activity                              Daily   Several   Once a Occa-sionally   Never
                                                days a    week                   
                                                week                             

                                                                                 
  ------------------------------------- ------- --------- ------ --------------- -------

a\. Read and re-read passages orally as they receive guidance 1 2 3 4 5

b\. Model fluent reading through use of audiotapes, choral reading,
tutors, peer guidance, reader's theatre, etc. 1 2 3 4 5

c\. Read books at their own independent level of reading ability to
complement classroom reading instruction 1 2 3 4 5

***8. During the past school year (2006--07), how often have you asked
your students to do the following VOCABULARY DEVELOPMENT activities?**
(Please circle one response for each item.)*

  ------------------------------------- ------- --------- ------ --------------- -------
  Activity                              Daily   Several   Once a Occa-sionally   Never
                                                days a    week                   
                                                week                             

                                                                                 
  ------------------------------------- ------- --------- ------ --------------- -------

a\. Learn specific words before reading 1 2 3 4 5

b\. Actively engage with words (e.g., make up sentences with words;
brainstorm examples in the respective word category such as types of
*occupations*) 1 2 3 4 5

c\. Participate in word analysis: compound words, inflectional endings
(-s, -ing, -ed) 1 2 3 4 5

d\. Use prior knowledge to try to figure out what a word means 1 2 3 4 5

e\. Use dictionaries and other reference aids to learn word meanings 1 2
3 4 5

f\. Use computers in vocabulary instruction 1 2 3 4 5

g\. Gain repeated exposure to words through hearing and seeing them in
various contexts (e.g., conversing with adults, being read to, reading
on their own) 1 2 3 4 5

h\. Restructure a task as necessary (e.g., substitute easy words for
more difficult words) 1 2 3 4 5

***9. During the past school year (2006--07), how often have you asked
your students to do the following READING COMPREHENSION activities?**
(Please circle* one response for each item.)

  ------------------------------------- ------- --------- ------ --------------- -------
  Activity                              Daily   Several   Once a Occa-sionally   Never
                                                days a    week                   
                                                week                             

                                                                                 
  ------------------------------------- ------- --------- ------ --------------- -------

a\. Use graphic and semantic organizers to focus on concepts and how
they are related to other concepts 1 2 3 4 5

b\. Answer questions 1 2 3 4 5

c\. Generate questions 1 2 3 4 5

d\. Recognize story structure to identify and interpret story
events/narrative elements 1 2 3 4 5

e\. Summarize text (e.g., identify key ideas, condense/generalize
information) 1 2 3 4 5

f\. Participate in reciprocal teaching conversations (e.g., lead
discussion by posing a question, discuss text meaning, clarify group
misunderstandings of text meaning, etc.) 1 2 3 4 5

g\. Predict what might occur next in the text 1 2 3 4 5

h\. Use mental imagery 1 2 3 4 5

B. CLASSROOM INSTRUCTION: TEACHING CYCLES

This section includes questions about the *Assess-Plan-Teach* method of
teaching. Please answer questions for this school year (2006--07).

***10. How often do you engage in the "assess" component of the
Assess-Plan-Teach method?** (Please circle one response for each item.)*

  ------------------------------------- ------- -------- --------- --------- -------
  *Assess* component                    Daily   Weekly   Monthly   Several   Never
                                                                   times a   
                                                                   year      

  a\. Assess student performance with   1       2        3         4         5
  reliable and valid measures                                                
  (screening, diagnostic, progress                                           
  monitoring, outcome measures)                                              

  b\. Interpret assessment results and  1       2        3         4         5
  apply to materials selection                                               

  c\. Interpret assessment results and  1       2        3         4         5
  apply to classroom instruction                                             

  d\. Using grade-level indicators,     1       2        3         4         5
  engage in informal observation of                                          
  student performance in academic                                            
  reading tasks                                                              

  e\. Diagnose students who fall behind 1       2        3         4         5
  to pinpoint reading                                                        
  strengths/weaknesses                                                       
  ------------------------------------- ------- -------- --------- --------- -------

***11. How often do you engage in the "plan" component of the
Assess-Plan-Teach method?*** (Please circle one response for each item.)

  ------------------------------------- ------- -------- --------- --------- -------
  *Plan* component                      Daily   Weekly   Monthly   Several   Never
                                                                   times a   
                                                                   year      

  a\. Based on assessment data, decide  1       2        3         4         5
  what scientifically based reading                                          
  research (SBRR) content needs to be                                        
  taught                                                                     

  b\. Define student learning           1       2        3         4         5
  objectives/goals                                                           

  c\. Assign students to groups based   1       2        3         4         5
  on student needs/reading level                                             

  d\. Plan interventions to be          1       2        3         4         5
  delivered by classroom aide or tutor                                       
  inside or outside of my class                                              

  e\. Plan interventions to be          1       2        3         4         5
  delivered by me inside or outside of                                       
  my class                                                                   

  f\. Identify best ways to assess      1       2        3         4         5
  student performance and to analyze                                         
  data                                                                       
  ------------------------------------- ------- -------- --------- --------- -------

***12. How often do you engage in the "teach" component of the
Assess-Plan-Teach method?*** (Please circle one response for each item.)

  ------------------------------------- ------- -------- --------- --------- -------
  *Teach* component                     Daily   Weekly   Monthly   Several   Never
                                                                   times a   
                                                                   year      

  a\. Build in deliberate supports      1       2        3         4         5
  (scaffolding) to provide assistance                                        
  until student can do task                                                  

  b\. Engage in joint problem solving   1       2        3         4         5

  c\. Make connections (e.g., come to   1       2        3         4         5
  shared understanding of goal, provide                                      
  verbal feedback)                                                           

  d\. Focus attention on the task at    1       2        3         4         5
  hand                                                                       

  e\. Explain the task                  1       2        3         4         5

  f\. Model the task                    1       2        3         4         5

  g\. Assist student performance of the 1       2        3         4         5
  task                                                                       

  h\. Provide for student practice of   1       2        3         4         5
  the task                                                                   

  i\. Monitor independent student       1       2        3         4         5
  performance of the task                                                    

  j\. Promote self-regulation (i.e.,    1       2        3         4         5
  allow child to regulate the joint                                          
  activity, relinquish control when                                          
  child can work independently)                                              
  ------------------------------------- ------- -------- --------- --------- -------

***13. How often do you use each of the following measures when
assessing student improvement in reading?** (Please circle one response
for each item.)*

  --------------------------------------------- ---------- ------- ------------ ----------
  Measure                                       Always/\   Often   Some-times   Rarely/\
                                                almost                          never
                                                always                          

  --------------------------------------------- ---------- ------- ------------ ----------

a\. DIBELS assessments 1 2 3 4

b\. Other diagnostic assessments (e.g., Rigby benchmarking, running
records) 1 2 3 4

c\. TPRI assessment 1 2 3 4

d\. TerraNova assessment 1 2 3 4

e\. Assessments that are part of the core reading series/other classroom
tests 1 2 3 4

f\. Student work products 1 2 3 4

g\. Student interest in reading and writing. 1 2 3 4

C. CLASSROOM INSTRUCTION: MATERIALS AND STRATEGIES

This section includes questions about instructional materials and
strategies you may have used during this school year (2006--07) in
teaching reading.

***14. How often have you used the following primary and/or
supplementary instructional materials during this school year for
struggling readers (2006--07)?** (Please circle one response for each*
item.)

  ----------------------------------------- --- ---------- --- ------- --- ------------ --- ----------
  Instructional material                        Always/\       Often       Some-times       Rarely/\
                                                almost                                      never
                                                always                                      

  a\. Narrative textbook                    1              2           3                4   

  b\. Informational textbook                1              2           3                4   

  c\. Phonics workbooks                     1              2           3                4   

  d\. Narrative decodable or leveled books  1              2           3                4   

  e\. Children's narrative trade books      1              2           3                4   

  f\. Student writing                       1              2           3                4   

  g\. Teacher-made materials                1              2           3                4   

  h\. Computer reading programs/reading     1              2           3                4   
  games                                                                                     

  i\. Overheads/pictures/other visuals      1              2           3                4   

  j\. Audiotape cassettes                   1              2           3                4   

  k\. At-home extensions/enrichment         1              2           3                4   
  activities                                                                                
  ----------------------------------------- --- ---------- --- ------- --- ------------ --- ----------

***15. During this school year (2006--07), how often have you used the
following strategies in your reading instruction for struggling
readers?** (Please circle one response for each item.)*

  ------------------------------------------- ---------- ------- ------------ ----------
  Strategy                                    Always/\   Often   Some-times   Rarely/\
                                              almost                          never
                                              always                          

  a\. Select materials for students that      1          2       3            4
  target individual needs                                                     

  b\. Develop individualized curriculum       1          2       3            4
  content based on student interests                                          

  c\. Scale back curriculum content based on  1          2       3            4
  student ability                                                             

  d\. Deliver individual or group             1          2       3            4
  interventions inside or outside my class                                    

  **e. Use trained paraprofessionals (e.g.,   1          2       3            4
  classroom aides, intervention specialists)                                  
  to deliver interventions inside or outside                                  
  of my class**                                                               

  **f. Use volunteer adult and student peer   1          2       3            4
  tutors to deliver interventions inside or                                   
  outside of my class**                                                       

  **g. Use literacy specialist to provide     1          2       3            4
  instructional feedback**                                                    
  ------------------------------------------- ---------- ------- ------------ ----------

***16. To what extent have the following sources affected your reading
and language arts classroom instruction this school year (2006--07)?**
(Please circle one response for each item.)*

  --------------------------------------------------- -------------- -------- --------
  Source                                              Sub-stantial   Some     No
                                                      effect         effect   effect

  a\. Feedback received through the *Early Language   1              2        3
  and Literacy Classroom Observation* (ELLCO)                                 

  b\. Feedback received through the *Coaching for     1              2        3
  Effective Classroom Instruction*: *Close-Ups of                             
  Teaching* tool                                                              

  c\. Other school-based assistance from              1              2        3
  *Reading-First* building Trio (e.g., from literacy                          
  specialist, data manager, or resource coordinator)                          

  d\. Assistance from non-Reading First consultants   1              3        4
  (i.e., textbook consultants, guided reading                                 
  experts)                                                                    

  e\. State-sponsored *Summer Institutes for Reading  1              2        3
  Instruction* (SIRI) (at a school or regional                                
  location)                                                                   

  f\. Other formal coursework in reading              1              2        3

  g\. State and/or district standards                 1              2        3

  h\. The *Reading First Professional Development*    1              2        3
  curriculum                                                                  

  i\. State curriculum framework or curriculum        1              2        3
  recommendations                                                             

  j\. District curriculum framework or                1              2        3
  recommendations                                                             

  k\. Materials, books, and resources                 1              2        3

  l\. Recommendations from the school principal       1              2        3

  m\. Recommendations from other teachers             1              2        3

  n\. TPRI, DIBELS, TerraNova student assessment data 1              2        3

  o\. Reading books, journals, reports, etc., about   1              2        3
  current reading research and effective strategies                           

  p\. Conferences at which reading research is        1              2        3
  presented or discussed (national, state, local)                             
  --------------------------------------------------- -------------- -------- --------

***17. To what extent do you agree or disagree with the following
statements regarding changes that you've made in your instructional
practices for all students during this school year (2006--07)?** (Please
circle one response for each item.)*

  ---------------------------------------------------------------------------------
  Statement                                Strongly   Agree   Disagree   Strongly
                                           agree                         disagree
  ---------------------------------------- ---------- ------- ---------- ----------
  a\. I have used a different core reading 1          2       3          4
  program to guide instruction                                           

  b\. I have put more emphasis on explicit 1          2       3          4
  skill and strategy instruction for                                     
  comprehension.                                                         

  c\. I have put more emphasis on          1          2       3          4
  vocabulary instruction                                                 

  d\. I have put more emphasis on          1          2       3          4
  instruction in phonemic awareness                                      

  e\. I have put more emphasis on helping  1          2       3          4
  students become fluent readers.                                        

  f\. I have put more emphasis on          1          2       3          4
  literature-based programs that emphasize                               
  reading and writing activities with                                    
  embedded phonics                                                       

  g\. I have put more emphasis on silent,  1          2       3          4
  independent reading                                                    

  h\. I have put more emphasis on          1          2       3          4
  systematic phonics instruction.                                        

  i\. I have involved parents differently  1          2       3          4
  in the reading program                                                 
  ---------------------------------------------------------------------------------

**18. Overall, to what extent do you feel these changes have resulted in
improvements in reading achievement for the grade(s) you teach?**
*(Please circle one.)*

No improvement 1

Little improvement 2

Moderate improvement 3

Substantial improvement 4

**D. PROFESSIONAL DEVELOPMENT IN READING**

This section includes questions about your participation in professional
development, the content of professional development activities, and
supports related to those activities.

**19. On average, how much professional development in reading (in
minutes) have you attended PER MONTH this school year (2006--07)?**
*(Please circle one.)*

0 1 *(Go to Question 21.)*

1 to 60 minutes 2

61 to 90 minutes 3

91 to 120 minutes 4

121 to 150 minutes 5

151 to 180 minutes 6

More than 180 minutes 7

**20. Approximately what percentage of professional development time
have you spent a) in a group with other K--3 teachers, b) with other
grade-level teachers, and c) one on one with the school literacy
specialist?**

a\. Percent of time with other K--3 teachers \_\_\_\_\_\_%

b\. Percent of time with other grade-level teachers \_\_\_\_\_\_%

c\. Percent of time one on one with school literacy specialist
\_\_\_\_\_\_%

TOTAL 100%

**How much of this total professional development time was spent using
online resources supported by the Reading First-Ohio Center (i.e.,
Teachscape and/or e-Read Ohio online modules on the five essential
components)?**

0 percent 1

1--25 percent 2

26--50 percent 3

51--75 percent 4

76--100 percent 5

***21. To what extent do you agree or disagree with the following
statements about the professional development experiences in which
you've participated this school year (2006--07)?** (Please circle one
response for each item.)*

  ---------------------------------------- ---------- ------- ---------- ----------
  Statement                                Strongly   Agree   Disagree   Strongly
                                           agree                         disagree

  a\. They have drawn on my own input      1          2       3          4

  b\. They have provided me with new and   1          2       3          4
  useful information on the five essential                               
  areas within the Ohio reading standards                                
  framework                                                              

  c\. They have taken away time better     1          2       3          4
  spent on planning lessons                                              

  d\. They have taken away time better     1          2       3          4
  spent on teaching                                                      

  e\. They have been effectively           1          2       3          4
  integrated into a comprehensive plan                                   
  that my school has for instruction                                     

  f\. They have been clearly linked to the 1          2       3          4
  Ohio English Language Arts (ELA) content                               
  standards for K--3                                                     

  g\. They have included information about 1          2       3          4
  reading theory that initiated change in                                
  my reading instruction                                                 

  h\. They have provided opportunities to  1          2       3          4
  practice what I've learned                                             

  i\. They have provided adequate time to  1          2       3          4
  plan instruction                                                       

  j\. They have repeated information that  1          2       3          4
  I already knew                                                         

  k\. They have provided individualized    1          2       3          4
  technical assistance                                                   

  l\. They have been supported by district 1          2       3          4
  support services and material resources                                

  m\. They have been led by knowledgeable  1          2       3          4
  and effective instructors                                              
  ---------------------------------------- ---------- ------- ---------- ----------

***22. How adequately have the following topics been addressed in
reading-related professional development programs up to this point in
the current school year (including professional development taken during
summer 2006)?*** (Please circle one response for each item.)

+---------------------------------------+------+------+------+------+
| Reading-related topic                 | More | Ad   | Less | Not  |
|                                       | than | equa | than | a    |
|                                       | ad   | tely | ad   | ddre |
|                                       | equa |      | equa | ssed |
|                                       | tely |      | tely |      |
+---------------------------------------+------+------+------+------+
| a.  Reading foundations, theory, and  | 1    | 2    | 3    | 4    |
|     speech to print development       |      |      |      |      |
+---------------------------------------+------+------+------+------+
| b.  Phonemic awareness development    | 1    | 2    | 3    | 4    |
+---------------------------------------+------+------+------+------+
| c.  Systematic and explicit phonics   | 1    | 2    | 3    | 4    |
|     instruction                       |      |      |      |      |
+---------------------------------------+------+------+------+------+
| d.  Fluency elements (e.g.,           | 1    | 2    | 3    | 4    |
|     automaticity, accuracy, prosody)  |      |      |      |      |
+---------------------------------------+------+------+------+------+
| e.  Vocabulary (e.g., direct and      | 1    | 2    | 3    | 4    |
|     contextual methods and word       |      |      |      |      |
|     selection)                        |      |      |      |      |
+---------------------------------------+------+------+------+------+
| f.  Reading comprehension strategies  | 1    | 2    | 3    | 4    |
|     (e.g., organization for           |      |      |      |      |
|     instruction, before-during-after) |      |      |      |      |
+---------------------------------------+------+------+------+------+
| g.  Composition (e.g., including      | 1    | 2    | 3    | 4    |
|     reading to support writing,       |      |      |      |      |
|     writing to support reading)       |      |      |      |      |
+---------------------------------------+------+------+------+------+
| h.  Understanding the                 | 1    | 2    | 3    | 4    |
|     Assess-Plan-Teach method of       |      |      |      |      |
|     teaching                          |      |      |      |      |
+---------------------------------------+------+------+------+------+
| i.  Understanding the concept of      | 1    | 2    | 3    | 4    |
|     differentiation in general        |      |      |      |      |
+---------------------------------------+------+------+------+------+
| j.  Teaching reading to English       | 1    | 2    | 3    | 4    |
|     language learner (ELL) students   |      |      |      |      |
+---------------------------------------+------+------+------+------+
| k.  Teaching reading to students with | 1    | 2    | 3    | 4    |
|     diagnosed learning disabilities   |      |      |      |      |
+---------------------------------------+------+------+------+------+
| l.  Teaching reading to struggling    | 1    | 2    | 3    | 4    |
|     readers                           |      |      |      |      |
+---------------------------------------+------+------+------+------+
| m.  Teaching reading to above average | 1    | 2    | 3    | 4    |
|     readers                           |      |      |      |      |
+---------------------------------------+------+------+------+------+
| n.  Small group teaching methods      | 1    | 2    | 3    | 4    |
+---------------------------------------+------+------+------+------+
| o.  Small group classroom management  | 1    | 2    | 3    | 4    |
|     techniques                        |      |      |      |      |
+---------------------------------------+------+------+------+------+
| p.  Scientifically based reading      | 1    | 2    | 3    | 4    |
|     research (SBRR)                   |      |      |      |      |
+---------------------------------------+------+------+------+------+
| q.  Standards-based lessons           | 1    | 2    | 3    | 4    |
+---------------------------------------+------+------+------+------+
| r.  Administering, analyzing, and     | 1    | 2    | 3    | 4    |
|     interpreting assessment results   |      |      |      |      |
|     (e.g., TPRI, DIBELS, TerraNova)   |      |      |      |      |
+---------------------------------------+------+------+------+------+
| s.  Assessing individual student's    | 1    | 2    | 3    | 4    |
|     progress through diagnostic       |      |      |      |      |
|     assessments                       |      |      |      |      |
+---------------------------------------+------+------+------+------+
| t.  Selecting instructional materials | 1    | 2    | 3    | 4    |
|     and tools based on SBRR           |      |      |      |      |
+---------------------------------------+------+------+------+------+
| u.  Selecting instructional materials | 1    | 2    | 3    | 4    |
|     for students based on individual  |      |      |      |      |
|     reading levels                    |      |      |      |      |
+---------------------------------------+------+------+------+------+
| v.  Alignment of state/district       | 1    | 2    | 3    | 4    |
|     assessments with reading          |      |      |      |      |
|     curriculum and instruction        |      |      |      |      |
+---------------------------------------+------+------+------+------+
| w.  Alignment and use of English      | 1    | 2    | 3    | 4    |
|     Language Arts (ELA) academic      |      |      |      |      |
|     content standards and reading     |      |      |      |      |
|     research for lesson planning      |      |      |      |      |
+---------------------------------------+------+------+------+------+

***23. In addition to the formal professional development activities,
the literacy specialist at your school works with select teachers in a
variety of ways. In which of the following ways have you received help
from the literacy specialist?*** (Please circle one response for each
item.)

  ------------------------------------------------------------ ----- -----
  Type of help                                                 Yes   No

  **Instructional Supports**                                         

  a\. Help in understanding and applying the essential reading 1     2
  components                                                         

  b\. Help in selecting and implementing research-based        1     2
  instructional strategies                                           

  c\. Help in obtaining intervention materials                 1     2

  d\. Help in selecting intervention materials                 1     2

  e\. Help in using intervention materials                     1     2

  f\. Help in interpreting assessment data                     1     2

  g\. Help in planning, coordinating, and implementing the     1     2
  core reading program                                               

  h\. Help in identifying students with severe literacy        1     2
  learning problems                                                  

  i\. Help reflecting on my lesson plans and ways to improve   1     2
  them                                                               

  j\. Help reflecting on my teaching and ways to improve my    1     2
  instruction                                                        

  k\. Help in arranging my classroom or setting up learning    1     2
  centers                                                            

  l\. Help in using computer reading programs                  1     2

  m\. Help with technology (e.g., training on computers, palm  1     2
  pilots, projectors, other media)                                   

  n\. Learning about literacy instruction through having       1     2
  lessons modeled for me (e.g., demonstrations)                      

  o\. Learning about literacy instruction through having       1     2
  one-on-one coaching or conference sessions                         

  p\. Learning about literacy instruction through having       1     2
  formal or informal observations\                                   
  of my classroom (including Close-Ups)                              

  **Collegial Supports**                                             

  q\. Help working with instructional aides or other           1     2
  paraprofessionals who assist inside or outside of the              
  classroom                                                          

  r\. Help working with tutors or volunteers who assist        1     2
  outside the classroom                                              

  s\. Help working with supplemental service providers (i.e.,  1     2
  entities that contract with the school for services)               

  t\. Help working with parents in parental involvement        1     2
  programs                                                           
  ------------------------------------------------------------ ----- -----

E. YOUR SCHOOL'S READING PROGRAM

This section consists of questions about your school's reading program,
specifically about the Reading First-Ohio building Trio, supplemental
programs, and potential problems with program implementation.

***24. To what extent do you agree or disagree with the following
statements regarding services and support provided by the Reading
First-Ohio Trio (i.e., literacy specialist, data manager, resource
coordinator)?** (Please circle one response for each item.)*

+---+--------------------------------------+------+------+------+------+
| S |                                      | Stro | A    | Disa | Stro |
| t |                                      | ngly | gree | gree | ngly |
| a |                                      | a    |      |      | disa |
| t |                                      | gree |      |      | gree |
| e |                                      |      |      |      |      |
| m |                                      |      |      |      |      |
| e |                                      |      |      |      |      |
| n |                                      |      |      |      |      |
| t |                                      |      |      |      |      |
+===+======================================+======+======+======+======+
| a | The building Trio has strengthened   | 1    | 2    | 3    | 4    |
| . | the reading program in our school    |      |      |      |      |
|   |                                      |      |      |      |      |
|   |                                      |      |      |      |      |
+---+--------------------------------------+------+------+------+------+
| b | The building Trio has provided       | 1    | 2    | 3    | 4    |
| . | services that were not available in  |      |      |      |      |
|   | our school before .                  |      |      |      |      |
|   |                                      |      |      |      |      |
+---+--------------------------------------+------+------+------+------+
| c | Services provided by the building    | 1    | 2    | 3    | 4    |
| . | Trio have been of little value.      |      |      |      |      |
|   |                                      |      |      |      |      |
|   |                                      |      |      |      |      |
+---+--------------------------------------+------+------+------+------+
| d | Services provided by the building    | 1    | 2    | 3    | 4    |
| . | Trio were delivered in a timely      |      |      |      |      |
|   | manner                               |      |      |      |      |
|   |                                      |      |      |      |      |
+---+--------------------------------------+------+------+------+------+
| e | The building Trio data manager has   | 1    | 2    | 3    | 4    |
| . | helped teachers to monitor student   |      |      |      |      |
|   | progress                             |      |      |      |      |
|   |                                      |      |      |      |      |
| e |                                      |      |      |      |      |
+---+--------------------------------------+------+------+------+------+
| f | The building Trio resource           | 1    | 2    | 3    | 4    |
| . | coordinator has helped teachers to   |      |      |      |      |
|   | secure and/or share materials        |      |      |      |      |
|   | necessary to teach reading           |      |      |      |      |
+---+--------------------------------------+------+------+------+------+

***25. To what extent do you agree or disagree with the following
statements about your school's current reading program?** (Please circle
one response for each item.)*

  ----------- --------------------------------------- ---------- ------- ---------- ----------
  Statement                                           Strongly   Agree   Disagree   Strongly
                                                      agree                         disagree

  a\.         There is strong and clear leadership    1          2       3          4
              for the school's reading program                                      

  b\.         There is a school-wide commitment to    1          2       3          4
              beginning reading instruction grounded                                
              in scientifically based reading                                       
              research (SBRR) .                                                     

  c\.         The school uses SBRR reading materials  1          2       3          4

  d\.         Teachers support the direction of the   1          2       3          4
              school's reading program.                                             

  e\.         The reading program is closely          1          2       3          4
              coordinated across teachers at the same                               
              grade level                                                           

  f\.         The reading program is closely          1          2       3          4
              coordinated across instructional staff                                
              in the K--3 grades                                                    

  g\.         There is a core curriculum in reading   1          2       3          4
              that all teachers follow                                              

  h\.         There is a common set of assessments in 1          2       3          4
              reading that all teachers use, other                                  
              than the TPRI, DIBELS, and TerraNova                                  

  i\.         Our school is characterized by a high   1          2       3          4
              degree of collaboration among teachers                                
              in teaching reading                                                   

  j\.         Our school communicates regularly with  1          2       3          4
              parents and families about our reading                                
              program                                                               

  k\.         Diagnostic assessment guides            1          2       3          4
              instructional planning                                                

  l\.         Our school has a shared vision of the   1          2       3          4
              school's reading program                                              

  m\.         The reading program is closely aligned  1          2       3          4
              with Ohio's English Language Arts                                     
              content standards for grades K--3.                                    

  n\.         Our intervention programs for           1          2       3          4
              struggling readers are coordinated with                               
              the core reading program                                              

  o\.         Our school has adequate resources to    1          2       3          4
              effectively implement our reading                                     
              curriculum                                                            
  ----------- --------------------------------------- ---------- ------- ---------- ----------

**26. Which of the following statements describe how you use the core
reading program?** *(Please circle one response for each item.)*

  ------------------------------------------------------------ ----- -----
  Statement                                                    Yes   No

  ------------------------------------------------------------ ----- -----

a\. I use the core reading program as a framework for a scope and
sequence of reading content and skills 1 2

b\. I use the core reading program as a resource for a few key lesson
plans 1 2

c\. I use the core reading program as my primary tool for instruction,
utilizing *most* of the key lesson plans on a fairly uniform schedule 1
2

d\. I use the core reading program as a daily script with a sequence of
questions to teach specific skills 1 2

***27. What type of impact has each of the following factors had on your
school's implementation of your core reading program?*** (Please circle
one response for each item.)

  ---------------------------------------------- ---------- -------- ----------
  Factor                                         Positive   No\      Negative
                                                 impact     impact   impact

  ---------------------------------------------- ---------- -------- ----------

**Informational or Resource-Driven Characteristics**

a\. Availability of appropriate instructional materials or other
resources 1 2 3

b\. Availability of effective supplemental services (including tutoring)
1 2 3

c\. Availability of technical assistance when I encounter program
problems 1 2 3

d\. Degree of parental involvement 1 2 3

**Individual or Personal Characteristics**

e\. Degree of clarity about my role in the program and/or what is
expected of me 1 2 3

f\. Personal expertise in scientifically based reading instruction 1 2 3

g\. Issues regarding the longevity of the program 1 2 3

h\. Degree of alignment between program philosophy and my own teaching
philosophy 1 2 3

  ------------------------------------------ ---------- -------- ---------- ------------
                                             Positive   No\      Negative   Not
                                             impact     impact   impact     applicable

  ------------------------------------------ ---------- -------- ---------- ------------

**Organizational or Programmatic Characteristics**

i\. Influence of district leadership 1 2 3

j\. Influence of school leadership 1 2 3

k\. Classroom size 1 2 3

l\. Degree of collaboration among teachers 1 2 3

m\. Amount of time allotted for instruction 1 2 3

n\. Amount of time allotted for teacher planning 1 2 3

o\. Staff mobility and turnover 1 2 3

p\. Reading initiatives other than Reading First 1 2 3 4

q\. Non-reading initiatives (such as math or science\
programs) 1 2 3 4

F. PARENTAL INVOLVEMENT AND OVERALL IMPACT OF PROGRAM

This section includes questions about parental involvement and the
overall impact of the Reading First-Ohio program at your school.

***28. What literacy-related strategies is your school currently
undertaking to involve parents? Of the strategies your school has
implemented, how effective have they been?*** (Please circle one
response for each item in Column A. If you circle "Yes" in Column A,
please also circle one response in Column B.)

  ----------------------------------- ---------- ------ ----------- ------------ -----------
  Strategy                            **Column          **Column                 
                                      A**               B**                      

                                      Yes        No     Extremely   Moderately   Not
                                                        effective   effective    effective

  a\. Publishing a parent newsletter  1          2      1           2            3
  with literacy-related                                                          
  news/activities                                                                

  b\. Discussing literacy at parent   1          2      1           2            3
  open houses and/or parent                                                      
  conferences                                                                    

  c\. Giving parents an opportunity   1          2      1           2            3
  to volunteer in reading classrooms                                             

  d\. Holding Family Literacy Nights  1          2      1           2            3
  or similar opportunities for                                                   
  parents and children to celebrate                                              
  reading                                                                        

  e\. Providing ideas for parents to  1          2      1           2            3
  encourage reading at home                                                      

  f\. Providing books and other       1          2      1           2            3
  materials for parents to read with                                             
  their children                                                                 

  g\. Informing parents of their      1          2      1           2            3
  child's reading progress                                                       

  h\. Offering Adult Literacy classes 1          2      1           2            3

  i\. Inviting parents into the       1          2      1           2            3
  reading classroom to observe                                                   
  strategies that will help them work                                            
  with their children at home                                                    

  j\. Acquainting parents with the    1          2      1           2            3
  services of the school and/or                                                  
  public library                                                                 

  k\. Acquainting parents with how to 1          2      1           2            3
  use computer technology to enhance                                             
  literacy                                                                       

  l\. Including parents in            1          2      1           2            3
  school-wide decisions about reading                                            
  instruction                                                                    
  ----------------------------------- ---------- ------ ----------- ------------ -----------

***29. To what extent do you agree or disagree about the impact of
parental involvement at your school?*** (Please circle one response for
each item.)

  ------------------------------------------ ---------- ------- ---------- ----------
  Impact                                     Strongly   Agree   Disagree   Strongly
                                             agree                         disagree

  ------------------------------------------ ---------- ------- ---------- ----------

a\. Participation in Reading First-Ohio has had no impact on parental
involvement at our school 1 2 3 4

b\. Parental involvement is easier to encourage in reading than it is in
other subject areas 1 2 3 4

c\. Parental involvement (or lack of) is a product of parent background
and education, not necessarily what the school does to enhance it 1 2 3
4

d\. Our current reading program includes activities and strategies to
enhance parental involvement in literacy 1 2 3 4

e\. I have seen some improvement in parental involvement during the last
year (2006--07) as compared to the previous year (2005--06) 1 2 3 4

f\. The Reading First-Ohio building Trio has positively impacted
parental involvement 1 2 3 4

***30. The Reading First-Ohio program has provided a variety of supports
to enhance the instructional process in reading. To what extent have the
following contributed to the relative success of the program at your
school?*** (Circle one on each line.)

  --------------------------------------------- -------------- -------------- --------------
                                                No             Minor          Major
                                                contribution   contribution   contribution

  a\. Teacher professional development provided 1              2              3
  by the Reading First-Ohio literacy specialist                               

  b\. Support provided by the Reading           1              2              3
  First-Ohio data manager                                                     

  c\. Program-aligned materials                 1              2              3

  d\. The Reading First-Ohio coaching model     1              2              3

  e\. Teachers' use of assessment data for      1              2              3
  instructional decision-making                                               

  f\. Emphasis on the essential components of   1              2              3
  reading                                                                     

  g\. District support                          1              2              3
  --------------------------------------------- -------------- -------------- --------------

**31. Overall, how successful has Reading First-Ohio been in your
school?** *(Circle one.)*

Not successful 1

Slightly successful 2

Somewhat successful 3

Very successful 4

**32. What type of impact has Reading First-Ohio had on student academic
achievement in reading in your school over the last 3 years (or since
your participation)?** *(Circle one.)*

No impact 1

A little impact 2

Some impact 3

Extensive impact 4

G. BACKGROUND

This section includes questions about your background and experience.

**33. What is your gender?** *(Please circle one.)*

Female 1

Male 2

**34. What is the highest degree you've received?** *(Please circle
one.)*

Bachelor's degree 1

Bachelor's degree plus additional graduate course credit(s) 2

Master's degree 3

Master's degree plus additional graduate course credit(s) 4

Doctorate 5

Other *(please specify):* 6

**35. Which license do you hold?** *(Please circle one.)*

Provisional teaching license 1

Professional teaching license 2

Temporary teaching license 3

Substitute teaching license 4

Conditional teaching permit (i.e., intervention specialist) 5

Alternative educator license 6

Other *(please specify):* 7

**36. In what areas are you certified to teach?** *(Please circle all
that apply.)*

a\. Elementary education 1

b\. Early childhood education 2

c\. Reading 3

d\. Special education 4

e\. Bilingual education 5

f\. Other *(please specify):* 6

**37. Including this year, how long have you been teaching?** *(Please
provide one response for each item.)*

a\. Number of years teaching at any school, grade, or subject Years

b\. Number of years teaching reading or language arts Years

c\. Number of years teaching at this school Years

d\. Number of years teaching current grade Years

**38. Please complete the following information so that we can contact
you if we need clarification of any responses.**

Name:

Phone number:

E-mail address:

**39. If there is anything else that you would like to tell us, please
do so here.**

**THANK YOU!**

**READING FIRST-OHIO**

**THE OHIO DEPARTMENT OF EDUCATION**

**SURVEY OF ELEMENTARY SCHOOL PRINCIPALS**

![](media/image68.wmf){width="0.8673611111111111in"
height="1.6645833333333333in"}

**SPRING 2007**

> Affix Label Here

Cohort 2

**SPRING 2007**

**READING FIRST-OHIO**

The ohio department of education (ODE)

Survey of principals

+-----------------------------------+-----------------------------------+
| **PLEASE RETURN COMPLETED SURVEY  | **IF YOU HAVE ANY QUESTIONS       |
| IN THE ENCLOSED PRE-ADDRESSED     | CONCERNING THIS SURVEY\           |
| ENVELOPE TO WESTAT**              | PLEASE CALL OR E-MAIL:**          |
|                                   |                                   |
| > Westat\                         | > Dr. Amber Winkler               |
| > Reading First-Ohio Surveys      | >                                 |
| >                                 | > 1--800--937--8281, ext. 2285    |
| > 1650 Research Boulevard         | >                                 |
| >                                 | > <amberwinkler@westat.com>       |
| > TB 146F                         |                                   |
| >                                 |                                   |
| > Rockville, MD 20850             |                                   |
|                                   |                                   |
| **[OR]{.underline} FAX TO:**      |                                   |
|                                   |                                   |
| > 1--800--533--0239               |                                   |
|                                   |                                   |
| Attn: Reading First-Ohio Surveys  |                                   |
+-----------------------------------+-----------------------------------+

**Please respond by April 27, 2007.**

All information that would permit identification of the individual
respondent will be held in strict confidence, will be used only by
persons engaged in and for the purposes of the survey, and will not be
disclosed or released to others for any purpose as required by law.

**READING FIRST-OHIO**

**PRINCIPAL SURVEY**

***Thank you for taking the time to answer this survey about reading in
your school. We are interested in your feedback about professional
development opportunities and your school's reading program. The survey
should take no longer than 45 minutes to complete. Please be aware that
some questions may require the input of your school's literacy
specialist or other staff. While we do not discourage this practice, we
do ask at the end of the survey that you inform us of those staff
members who contributed to the completion of the instrument. Please
remember that each response is important to our understanding of the
Reading First-Ohio (RFO) program; thus, it is critical that every
question be answered. Following up on missing survey responses is
time-consuming for all involved. If you have any questions about this
survey, please contact Amber Winkler toll-free at 1-800-937-8281,
extension 2285, or amberwinkler@westat.com.***

**A. PROFESSIONAL DEVELOPMENT IN READING**

This section includes questions about time, content, participation, and
services related to professional development in reading for K--3
faculty/staff at your school.

**1. How much time is allocated for K--3 teacher professional
development in *reading*, PER MONTH?** *(Please circle one.)*

None 1

1 to 60 minutes 2

61 to 90 minutes 3

91 to 120 minutes 4

121 to 150 minutes 5

151 to 180 minutes 6

More than 180 minutes 7

**Of this time, what percentage occurs after school hours?**

0 percent 1

1 to 25 percent 2

26 to 50 percent 3

51 to 75 percent 4

76 to 100 percent 5

**2. How much time is allocated during the school day for K--3 teacher
professional development in *all areas* (including reading) PER MONTH?**

None 1

1 to 60 minutes 2

61 to 90 minutes 3

91 to 120 minutes 4

121 to 150 minutes 5

151 to 180 minutes 6

More than 180 minutes 7

**3. In your best estimation, what percentage of K--3 teachers attend
READING FIRST-OHIO professional development programs and activities in
reading?***(Please circle one.)*

Less than 60 percent 1

60 to 69 percent 2

70 to 79 percent 3

80 to 89 percent 4

90 to 100 percent 5

**4. In your best estimation, what percentage of K--3 teachers attend
OTHER professional development programs and activities in reading?**
*(Please circle one.)*

0 percent 1

1 to 25 percent 2

26 to 50 percent 3

51 to 75 percent 4

76 to 100 percent 5

**5. Approximately, what percentage of professional development in
reading has been delivered under the auspices of Reading First-Ohio?**

\_\_\_\_\_\_\_\_\_\_\_\_ %

**6. During this school year (including summer 2006), what areas were
covered in reading-related professional development activities offered
to teachers in your school? *(Please circle one response for each
item.)***

+----------------------------------------------------------+-----+-----+
| Area of reading-related professional development         | >   | >   |
|                                                          | Yes |  No |
+----------------------------------------------------------+-----+-----+
| a\. Reading foundations and theory                       | > 1 | > 2 |
+----------------------------------------------------------+-----+-----+
| b\. Review of scientifically based reading research      | > 1 | > 2 |
| (SBRR) related to the five essential components of       |     |     |
| reading (i.e., phonemic awareness, phonics, fluency,     |     |     |
| vocabulary, and comprehension)                           |     |     |
+----------------------------------------------------------+-----+-----+
| c\. Using research to inform and implement instructional | > 1 | > 2 |
| strategies                                               |     |     |
+----------------------------------------------------------+-----+-----+
| d\. SBRR modifications and interventions for struggling  | > 1 | > 2 |
| readers                                                  |     |     |
+----------------------------------------------------------+-----+-----+
| e\. Reviewing and selecting materials/resources for      | > 1 | > 2 |
| reading based on SBRR                                    |     |     |
+----------------------------------------------------------+-----+-----+
| f\. Using your state/district curriculum frameworks and  | > 1 | > 2 |
| English Language Arts (ELA) content standards in         |     |     |
| curriculum planning                                      |     |     |
+----------------------------------------------------------+-----+-----+
| g\. Constructing standards-based lessons                 | > 1 | > 2 |
+----------------------------------------------------------+-----+-----+
| h\. Using assessment data to inform classroom            | > 1 | > 2 |
| instruction                                              |     |     |
+----------------------------------------------------------+-----+-----+

***7. To what extent do you agree or disagree with each of the following
statements regarding reading-related professional development activities
in which staff participated during the current school year (including
summer 2006)?** (Please circle one response for each item.)*

  --------------------------------- ---------- ------- ---------- ---------- -------
  Statement                         Strongly   Agree   Disagree   Strongly   Don't
                                    agree                         disagree   know

  a\. Kindergarten through 3rd      1          2       3          4          5
  grade teachers select the                                                  
  specific professional development                                          
  activities that they attend                                                
  and/or participate in                                                      

  b\. Reading-related professional  1          2       3          4          5
  development is readily available                                           
  and accessible for K--3 teachers                                           
  (e.g., convenient location and/or                                          
  timing)                                                                    

  c\. After initial reading-related 1          2       3          4          5
  professional development, our                                              
  literacy specialist provides                                               
  follow-up to teachers                                                      

  d\. Our school's K--3 instruction 1          2       3          4          5
  is influenced by reading-related                                           
  professional development                                                   

  e\. Self-reflection is part of    1          2       3          4          5
  teachers' professional                                                     
  development plans                                                          

  f\. The assistance of outside     1          2       3          4          5
  experts (e.g., text-book or                                                
  university consultants) is                                                 
  available and used as needed in                                            
  professional development                                                   

  g\. Professional development      1          2       3          4          5
  provides adequate time for                                                 
  teachers to plan                                                           

  h\. Professional development      1          2       3          4          5
  provides adequate time for                                                 
  teachers to practice what they've                                          
  learned                                                                    

  i\. Professional development is   1          2       3          4          5
  integrated into a long-range plan                                          
  that has school-wide support                                               

  j\. Professional development is   1          2       3          4          5
  connected to Ohio's English                                                
  Language Arts (ELA) standards                                              

  k\. Professional development      1          2       3          4          5
  targets specific teacher needs                                             

  l\. There are structured          1          2       3          4          5
  opportunities for Reading First                                            
  staff and staff involved in other                                          
  literacy initiatives to exchange                                           
  ideas (i.e., through meetings or                                           
  professional development                                                   
  sessions)                                                                  

  m\. After January 1, 2005, *The   1          2       3          4          5
  Reading First-Ohio Professional                                            
  Development* *Program Review                                               
  Guide* was used to evaluate all                                            
  specific professional development                                          
  programs in reading                                                        
  --------------------------------- ---------- ------- ---------- ---------- -------

**8. In your opinion, to what extent have the professional development
efforts associated with Reading First-Ohio changed teachers'
instructional practices?** *(Please circle one.)*

No change 1

Little change 2

Moderate change 3

Substantial change 4

***9. How frequently do you use information from the following sources
to monitor your school's professional development program in reading?**
(Please circle one response for each item.)*

  ---------------------------------------- ---------- ------- ----------- ----------
  Source of information                    Always/\   Often   Sometimes   Rarely/\
                                           almost                         never
                                           always                         

  a\. Teacher-reported satisfaction with   1          2       3           4
  the quality of professional development                                 

  b\. Observations of professional         1          2       3           4
  development sessions                                                    

  c\. Teacher attendance at professional   1          2       3           4
  development sessions                                                    
  ---------------------------------------- ---------- ------- ----------- ----------

***10. How important are each of the following services provided by your
school's building Trio (i.e., literacy specialist, data manager,
resource coordinator)?** (Please circle one response for each item.)*

  --------------------------------- ----------- ----------- ----------- ----------- ----------
  Service                           Very        Important   Somewhat    Not at all  Not
                                    important               important   important   provided

  a\. Lesson modeling               1           2           3           4           5

  b\. Assistance with selecting and 1           2           3           4           5
  using scientifically based                                                        
  reading research (SBRR)                                                           
  materials, strategies, and                                                        
  interventions                                                                     

  c\. Assistance to teachers in     1           2           3           4           5
  creating lesson plans                                                             

  d\. Assistance in completing      1           2           3           4           5
  required federal/state/local                                                      
  reports and documents                                                             

  e\. Instruction in the five       1           2           3           4           5
  essential reading components                                                      
  (phonemic awareness, phonics,                                                     
  fluency, vocabulary,                                                              
  comprehension)                                                                    

  f\. Instruction in proven         1           2           3           4           5
  instructional strategies                                                          

  g\. One-on-one coaching with      1           2           3           4           5
  teachers                                                                          

  h\. Help in monitoring student    1           2           3           4           5
  progress                                                                          

  i\. Help using assessment data to 1           2           3           4           5
  inform classroom planning and                                                     
  instruction                                                                       

  j\. Help in diagnoses and         1           2           3           4           5
  interventions for struggling                                                      
  readers                                                                           

  k\. Help working with             1           2           3           4           5
  instructional aides or other                                                      
  paraprofessionals                                                                 

  l\. Observation of teachers'      1           2           3           4           5
  instructional practices and                                                       
  provision of feedback                                                             
  --------------------------------- ----------- ----------- ----------- ----------- ----------

***11. How would you rate the collective knowledge base of your school's
K--3 teachers in the following reading-related areas?** (Please circle
one response for each item.)*

+------------------------------------------------+------+------+------+
| Area                                           | ##   | Adeq | Lim  |
|                                                | # St | uate | ited |
|                                                | rong |      |      |
+================================================+======+======+======+
| a\. Foundational topics (e.g., psychology of   | 1    | 2    | 3    |
| reading, structure of the English language)    |      |      |      |
+------------------------------------------------+------+------+------+
| b\. State and district curriculum frameworks   | 1    | 2    | 3    |
| and English Language Arts (ELA) content        |      |      |      |
| standards                                      |      |      |      |
+------------------------------------------------+------+------+------+
| c\. Current scientifically based reading       | 1    | 2    | 3    |
| research (SBRR)                                |      |      |      |
+------------------------------------------------+------+------+------+
| d\. Strategies for teaching the five essential | 1    | 2    | 3    |
| components of reading (phonemic awareness,     |      |      |      |
| phonics, fluency, vocabulary, comprehension)   |      |      |      |
+------------------------------------------------+------+------+------+
| e\. Strategies for addressing the needs of     | 1    | 2    | 3    |
| struggling readers in K--3                     |      |      |      |
+------------------------------------------------+------+------+------+
| f\. Processes for aligning curriculum and      | 1    | 2    | 3    |
| instruction with state ELA standards and SBRR  |      |      |      |
| recommendations                                |      |      |      |
+------------------------------------------------+------+------+------+
| g\. Selecting materials based on SBRR          | 1    | 2    | 3    |
+------------------------------------------------+------+------+------+
| h\. Developing relevant classroom assessments  | 1    | 2    | 3    |
+------------------------------------------------+------+------+------+
| i\. Administering student assessments properly | 1    | 2    | 3    |
+------------------------------------------------+------+------+------+
| j\. Using assessment data to inform classroom  | 1    | 2    | 3    |
| instruction                                    |      |      |      |
+------------------------------------------------+------+------+------+

***12. What effect have the following professional development
opportunities that occurred during the past school year (including the
summer of 2006) had on your ability to support literacy in your
school?** (Please circle one response for each item.)*

+-----------------------------------------+-------+-----+------+------+
| Professional development opportunity    | ### S | S   | No   | Did  |
|                                         | ubsta | ome | ef   | not  |
|                                         | ntial | eff | fect | at   |
|                                         |       | ect |      | tend |
|                                         | ### e |     |      |      |
|                                         | ffect |     |      |      |
+=========================================+=======+=====+======+======+
| a\. New Leadership Meeting (October)    | 1     | 2   | 3    | 4    |
+-----------------------------------------+-------+-----+------+------+
| b\. Classroom Walk-Through (CWT)        | 1     | 2   | 3    | 4    |
| Training (1-1/2 days)                   |       |     |      |      |
+-----------------------------------------+-------+-----+------+------+
| c\. Advanced Classroom Walk-Through     | 1     | 2   | 3    | 4    |
| (CWT) Training (1 day)                  |       |     |      |      |
+-----------------------------------------+-------+-----+------+------+
| d\. Data Summit I (December)            | 1     | 2   | 3    | 4    |
+-----------------------------------------+-------+-----+------+------+
| e\. Sustainability Meeting w/ Mike      | 1     | 2   | 3    | 4    |
| Schmoker (December)                     |       |     |      |      |
+-----------------------------------------+-------+-----+------+------+
| f\. Reading First-Ohio school-based     | 1     | 2   | 3    | 4    |
| professional development for teachers   |       |     |      |      |
+-----------------------------------------+-------+-----+------+------+

B. YOUR SCHOOL'S READING PROGRAM

This section includes questions about the reading program serving K--3
students, including its selection, related interventions, and materials
focused on reading.

**13. What is the name of the reading program, model, or textbook series
that you identify as your school's *core* reading program?** *(Please
write in.)*

**14.** **To what extent do you agree or disagree with the following
statements regarding your school's reading program in grades K--3?**
(*Please circle one response for each item.)*

  ----------------------------------------------------------------------------------
  Statement                                 Strongly   Agree   Disagree   Strongly
                                            agree                         disagree
  ----------------------------------------- ---------- ------- ---------- ----------
  a\. There is a student assessment system  1          2       3          4
  in place that uses reliable and valid                                   
  measurement tools to measure growth in                                  
  reading                                                                 

  b\. Diagnostic assessment information is  1          2       3          4
  used to plan instruction for individual                                 
  students                                                                

  c\. Professional development supports the 1          2       3          4
  implementation of our school's reading                                  
  program                                                                 

  d\. Pressure is felt by some staff who    1          2       3          4
  feel that state and/or reading program                                  
  standards conflict with their own                                       
  standards of best practice                                              

  e\. Our reading program is inclusive of   1          2       3          4
  all grade-level indicators in Ohio's                                    
  academic content standards                                              

  f\. Evaluation criteria guide the         1          2       3          4
  selection of research-based materials                                   
  that support instruction                                                

  g\. Our core reading program includes     1          2       3          4
  guidance and materials that are relevant                                
  for a wide range of students, including                                 
  gifted and special needs students                                       

  h\. Teachers use SBRR instructional       1          2       3          4
  materials that include strategies in the                                
  five dimensions of reading                                              

  i\. Teachers have discretion over         1          2       3          4
  materials they use                                                      
  ----------------------------------------------------------------------------------

***15. How often have you (the school principal) participated or
assisted in the following reading-related activities during this school
year (2006--07)?** (Please circle one response for each item.)*

  --------------------------------------------- ------ ------- -------- -------
  Activity                                      Once a Once a  Seldom   Never
                                                week   month            

  --------------------------------------------- ------ ------- -------- -------

a\. Helping teachers use reading strategies (e.g., through
demonstrations) 1 2 3 4

b\. Helping teachers use instructional materials in reading 1 2 3 4

c\. Helping teachers use assessment data to inform their instruction 1 2
3 4

d\. Visiting classrooms during the language arts block 1 2 3 4

e\. Participating in professional development with teachers 1 2 3 4

f\. Reading books to children or allowing children to read to you 1 2 3
4

g\. Meeting (or conferring) with the literacy specialist\
(e.g., planning the reading curriculum, discussing student progress,
supporting individual teachers) 1 2 3 4

h\. Meeting (or conferring) with the resource coordinator\
(e.g., best methods to secure and utilize necessary materials) 1 2 3 4

i\. Attending professional meetings about the teaching of reading 1 2 3
4

j\. Sharing literature about reading with teachers 1 2 3 4

k\. Celebrating student accomplishments in reading 1 2 3 4

l\. Recognizing teachers who have excelled in literacy instruction 1 2 3
4

m\. Communicating with parents about reading (e.g., through newsletters,
parent-teacher conferences, invitations to observe classrooms
informally) 1 2 3 4

n\. Meeting with teachers to discuss the school's literacy program 1 2 3
4

***16. To what extent has your school implemented formal procedures
and/or guidelines for coordinating and integrating your core reading
program in the following areas?** (Please circle one response for each
item.)*

  ------------------------------------------ -------------- -------- ----------- -------
  Area                                       Degree                              
                                             implemented?                        

                                             Fully          Mostly   Partially   Not\
                                                                                 at all

  a\. Developing clear definitions of what   1              2        3           4
  students need to know and ensuring                                             
  teachers are aware of such definitions                                         

  b\. Linking expectations for student       1              2        3           4
  learning to the Ohio English Language Arts                                     
  (ELA) academic standards for grades K--3                                       

  c\. Linking expectations for student       1              2        3           4
  learning to the reading curriculum for                                         
  grades K--3                                                                    

  d\. Articulating a process for aligning    1              2        3           4
  the curriculum across the grades                                               

  e\. Articulating the relationship between  1              2        3           4
  student assessment data and classroom                                          
  instruction                                                                    

  f\. Linking Reading First-Ohio to the      1              2        3           4
  overall school improvement plan                                                

  g\. Using diagnosis and referral           1              2        3           4
  procedures to determine eligibility of                                         
  K--3 struggling readers for services                                           
  (including determination of when services                                      
  are no longer needed)                                                          

  h\. Addressing the needs of struggling     1              2        3           4
  readers in K--3 along a continuum of                                           
  intervention                                                                   

  i\. Alignment of intervention programs and 1              2        3           4
  materials with the core reading program                                        

  j\. Coordinating educational technology    1              2        3           4
  with SBRR instructional materials,                                             
  interventions, and supplemental programs                                       

  k\. Coordinating reading instruction with  1              2        3           4
  "feeder" preschool programs                                                    

  l\. Establishing home-school partnerships  1              2        3           4
  that support children's reading progress                                       
  ------------------------------------------ -------------- -------- ----------- -------

***17. What type of impact has each of the following factors had on your
school's implementation of your core reading program in grades K--3?**
(Please circle one response for each ite*m.)

  --------------------------------------------------- ---------- -------- ----------
  Factor                                              Positive   No       Negative
                                                      impact     impact   impact

  **Organizational Factors**                                              

  a\. Allocation and amount of Reading First-Ohio     1          2        3
  resources, including instructional materials and                        
  supplemental services                                                   

  b\. Physical infrastructure to put scientifically   1          2        3
  based reading research program in place (e.g.,                          
  facilities/space for instruction/tutoring; computer                     
  technology/software for data)                                           

  c\. Alignment of standards, assessment, and reading 1          2        3
  program                                                                 

  d\. Multiple literacy-based reform initiatives      1          2        3
  currently in place                                                      

  e\. Other non-literacy reform initiatives currently 1          2        3
  in place (such as math or science programs)                             

  f\. Amount of time allotted for teacher planning    1          2        3

  g\. Amount of time allotted for professional        1          2        3
  development (including time with literacy                               
  specialist)                                                             

  h\. Amount of time scheduled for reading            1          2        3
  instruction                                                             

  **Personnel-Related Factors**                                           

  i\. Experience and expertise of classroom teachers  1          2        3
  (including knowledge about how to use data to                           
  inform classroom instruction)                                           

  j\. Experience and expertise of literacy specialist 1          2        3

  k\. Staff mobility and turnover                     1          2        3

  l\. Degree of collaboration among teachers          1          2        3

  **External Factors**                                                    

  m\. Degree of parental involvement in and outside   1          2        3
  the classroom                                                           

  n\. Communication or direction from the *district*  1          2        3
  about Reading First-Ohio                                                

  o\. Communication or direction from the *state*     1          2        3
  about Reading First-Ohio                                                
  --------------------------------------------------- ---------- -------- ----------

***18. As a result of participating in Reading First-Ohio, has your
school implemented any of the following CHANGES in grades K--3?**
(Please circle one response for each item.*)

  -----------------------------------------------------------------------
  Change                                                      Yes   No
  ----------------------------------------------------------- ----- -----
  a\. Added or enhanced school-day programs for struggling    1     2
  readers in grades K--3 (including computer-assisted               
  programs)                                                         

  b\. Implemented additional before- and/or after-school      1     2
  intervention programs for struggling readers in grades K--3       
  (including volunteer programs)                                    

  c\. Adopted new instructional materials for grades K--3     1     2

  d\. Adopted new *district* assessments in reading for       1     2
  grades K--3                                                       

  e\. Reassigned existing teachers to accommodate individual  1     2
  strengths and weaknesses                                          

  f\. Provided common planning time for teachers to           1     2
  collaborate with literacy specialist and/or colleagues            

  g\. Focused instruction on one or more of the five          1     2
  essential components of reading (phonemic awareness,              
  phonics, fluency, vocabulary, comprehension)                      

  h\. Changed professional development offerings in reading   1     2

  i\. Decreased time devoted to subjects other than English   1     2
  Language Arts                                                     

  j\. Made programmatic decisions based on student assessment 1     2
  data in reading                                                   
  -----------------------------------------------------------------------

C. PROGRAM COHERENCE AND MONITORING

This section includes questions about how your reading program is
monitored and the results are used.

***19. How frequently do you use information from the following to
monitor the effectiveness of your school's K--3 reading program?**
(Please circle one response for each item.)*

  --------------------------------------------- ---------- ------- ------------ ----------
  Source of information                         Always/\   Often   Some-times   Rarely/\
                                                almost                          never
                                                always                          

  --------------------------------------------- ---------- ------- ------------ ----------

a\. Student assessment data from standardized assessments (e.g., DIBELS,
TPRI, TerraNova) 1 2 3 4

b\. Other student diagnostic data (e.g., Rigby benchmarking, other
reading tasks) 1 2 3 4

c\. Student assessment data from textbook/classroom tests in reading 1 2
3 4

d\. Teacher attendance at professional development sessions 1 2 3 4

e\. Discussions with parents 1 2 3 4

f\. Discussions with individual teachers 1 2 3 4

g\. Discussions with literacy specialists 1 2 3 4

h\. Discussions at faculty or grade-level meetings (e.g., as an agenda
item) 1 2 3 4

i\. Classroom observations (including walk-throughs) 1 2 3 4

j\. Teacher self-assessment of effectiveness in teaching the five
essential reading components 1 2 3 4

k\. Teachers' assignments to students 1 2 3 4

l\. Student work products 1 2 3 4

***20. To what extent do you agree with the following statements about
how the reading program is monitored and the results are used at your
school for grades K--3?** (Please circle one response for each item.*)

  ---------------------------------------------------------------------------------
  Statement                                Strongly   Agree   Disagree   Strongly
                                           agree                         disagree
  ---------------------------------------- ---------- ------- ---------- ----------
  **Status and Use**                                                     

  a\. Procedures *have been developed* for 1          2       3          4
  collecting and using data                                              

  b\. Procedures *have been implemented*   1          2       3          4
  for collecting and using data                                          

  c\. There is one uniform reporting       1          2       3          4
  system used at both the district and                                   
  school levels                                                          

  d\. Results from program monitoring help 1          2       3          4
  to evaluate teacher performance                                        

  e\. Results from program monitoring help 1          2       3          4
  to evaluate student needs and inform                                   
  interventions                                                          

  f\. Results from program monitoring help 1          2       3          4
  our school make better use of                                          
  instructional materials                                                

  g\. The monitoring and program           1          2       3          4
  improvements specific to our school's                                  
  Reading First-Ohio program are linked to                               
  our overall school improvement plan                                    

  **Quality of Information**                                             

  h\. Our school monitoring system yields  1          2       3          4
  information sufficient to inform                                       
  building-level decisionmaking\                                         
  (i.e., personnel, scheduling, resource                                 
  decisions)                                                             

  i\. Our school monitoring system yields  1          2       3          4
  information sufficient to inform                                       
  classroom-level or instructional                                       
  decisionmaking                                                         

  j\. Our school monitoring system yields  1          2       3          4
  information sufficient to document and                                 
  demonstrate accountability for Reading                                 
  First-Ohio state and federal                                           
  requirements                                                           

  **Role of Data Manager**                                               

  k\. The data manager in our Reading      1          2       3          4
  First-Ohio building Trio satisfactorily                                
  maintains the school's data reporting                                  
  system (e.g., validates data submitted                                 
  by school staff)                                                       

  l\. The data manager in our Reading      1          2       3          4
  First-Ohio building Trio satisfactorily                                
  performs various types of data analyses                                

  m\. The data manager in our Reading      1          2       3          4
  First-Ohio building Trio satisfactorily                                
  assists school staff in monitoring                                     
  student progress                                                       
  ---------------------------------------------------------------------------------

**D. CLASSROOM WALK-THROUGH**

This section includes questions about the Reading First-Ohio classroom
walk-through.

**21. The Reading First-Ohio classroom walk-through (CWT) was covered in
the fall professional development workshops. Did you attend one or more
of these workshops?**

Yes 1

No 2 (*Skip to Question 23.*)

***22. To what extent do you agree or disagree with the following
statements about the professional development sessions relative to the
CWT and your use of CWT?*** (Please circle one response for each item.)

  ------------------------------------ ---------- ------- ---------- ---------- -------
  Statement                            Strongly   Agree   Disagree   Strongly   Don't
                                       agree                         disagree   know

  a\. The classroom walk-through model 1          2       3          4          5
  (i.e., review curriculum, identify                                            
  instructional strategies, assess                                              
  student engagement, survey the                                                
  learning environment, analyze the                                             
  data, and reflect with the teacher)                                           
  is easy for me to use.                                                        

  b\. It is feasible for me to review  1          2       3          4          5
  classroom data prior to or after                                              
  conducting a CWT                                                              

  c\. The professional development     1          2       3          4          5
  sessions covered what to look for in                                          
  a CWT                                                                         

  d\. The timeframe associated with    1          2       3          4          5
  the classroom walk-through model                                              
  (2--4 minutes) is adequate for                                                
  recording the "look-for's"                                                    

  e\. The professional development     1          2       3          4          5
  sessions adequately covered how to                                            
  develop a reflective prompt for                                               
  conversations with teachers                                                   

  f\. I have been able to engage in    1          2       3          4          5
  reflective conversations with                                                 
  teachers post-CWT                                                             

  g\. The CWT is a good idea, but I    1          2       3          4          5
  have limited time to do it given my                                           
  current responsibilities                                                      

  h\. The CWT professional development 1          2       3          4          5
  sessions strengthened my role as                                              
  instructional leader in the building                                          
  ------------------------------------ ---------- ------- ---------- ---------- -------

E. PARENTAL INVOLVEMENT AND OVERALL IMPACT OF PROGRAM

This section includes questions about parental involvement and the
overall impact of the Reading First-Ohio program at your school.

***23. What literacy-related strategies is your school currently
undertaking to involve parents? Of the strategies your school has
implemented, how effective have they been?*** (Please circle one
response for each item in Column A. If you circle "Yes" in Column A,
please also circle one response in Column B.)

  ----------------------------------- ---------- ------ ----------- ------------ -----------
  Strategy                            **Column          **Column                 
                                      A**               B**                      

                                      Yes        No     Extremely   Moderately   Not
                                                        effective   effective    effective

  a\. Publishing a parent newsletter  1          2      1           2            3
  with literacy-related                                                          
  news/activities                                                                

  b\. Discussing literacy at parent   1          2      1           2            3
  open houses and/or parent                                                      
  conferences                                                                    

  c\. Giving parents an opportunity   1          2      1           2            3
  to volunteer in reading classrooms                                             

  d\. Holding Family Literacy Nights  1          2      1           2            3
  or similar opportunities for                                                   
  parents and children to celebrate                                              
  reading                                                                        

  e\. Providing ideas for parents to  1          2      1           2            3
  encourage reading at home                                                      

  f\. Providing books and other       1          2      1           2            3
  materials for parents to read with                                             
  their children                                                                 

  g\. Informing parents of their      1          2      1           2            3
  child's reading progress                                                       

  h\. Offering Adult Literacy classes 1          2      1           2            3

  i\. Inviting parents into the       1          2      1           2            3
  reading classroom to observe                                                   
  strategies that will help them work                                            
  with their children at home                                                    

  j\. Acquainting parents with the    1          2      1           2            3
  services of the school and/or                                                  
  public library                                                                 

  k\. Acquainting parents with how to 1          2      1           2            3
  use computer technology to enhance                                             
  literacy                                                                       

  l\. Including parents in            1          2      1           2            3
  school-wide decisions about reading                                            
  instruction                                                                    
  ----------------------------------- ---------- ------ ----------- ------------ -----------

***24. To what extent do you agree or disagree with the following
statements about the impact of parental involvement at your school?***
(Please circle one response for each item.)

  ------------------------------------------ ---------- ------- ---------- ----------
  Statement                                  Strongly   Agree   Disagree   Strongly
                                             agree                         disagree

  ------------------------------------------ ---------- ------- ---------- ----------

a\. Participation in Reading First-Ohio has had no impact on parental
involvement at our school 1 2 3 4

b\. Parental involvement is easier to encourage in reading than it is in
other subject areas 1 2 3 4

c\. Parental involvement (or lack of) is a product of parent background
and education, not necessarily what the school does to enhance it 1 2 3
4

d\. Our current reading program includes activities and strategies to
enhance parental involvement in literacy 1 2 3 4

e\. I have seen some improvement in parental involvement during the last
year (2006--07) as compared to the previous year (2005--06) 1 2 3 4

f\. The Reading First-Ohio building Trio has positively impacted
parental involvement 1 2 3 4

***25. The Reading First-Ohio program has provided a variety of supports
to enhance the instructional process in reading. To what extent have the
following contributed to the relative success of the program at your
school?*** (Circle one on each line.)

  --------------------------------------------- -------------- -------------- --------------
  Type of Support                               No             Minor          Major
                                                contribution   contribution   contribution

  a\. Teacher professional development provided 1              2              3
  by the Reading First-Ohio literacy specialist                               

  b\. Support provided by the Reading           1              2              3
  First-Ohio data manager                                                     

  c\. Program-aligned materials                 1              2              3

  d\. The Reading First-Ohio coaching model     1              2              3

  e\. Teachers' use of assessment data for      1              2              3
  instructional decision-making                                               

  f\. Emphasis on the essential components of   1              2              3
  reading                                                                     

  g\. District support                          1              2              3

  h\. The Reading First-Ohio Program Monitoring 1              2              3
  Tool                                                                        

  i\. The professional development provided by  1              2              3
  the Reading First-Ohio Center                                               
  --------------------------------------------- -------------- -------------- --------------

**26. How successful has Reading First-Ohio been in your school?**
*(Circle one.)*

Not successful 1

Slightly successful 2

Somewhat successful 3

Very successful 4

**27. What type of impact has Reading First-Ohio had on student academic
achievement in reading in your school over the last 3 years (or since
your participation)?** *(Circle one.)*

No impact 1

A little impact 2

Some impact 3

Extensive impact 4

F. BACKGROUND INFORMATION AND SCHOOL CHARACTERISTICS

This section includes questions about your background and experience and
questions about general school characteristics.

**28. How many years have you been principal at THIS SCHOOL?** *(Count
part of a year as 1 year.)*

\_\_\_\_\_ Years

**29. How many TOTAL years have you been a principal?** *(Count part of
a year as 1 year.)*

\_\_\_\_\_ Total years

**30. What grade levels does your school serve?** *(Please circle one.)*

PreK and/or K through grade 2 1

PreK and/or K through grade 3 2

PreK and/or K through grade 4 3

PreK and/or K through grade 5 4

PreK and/or K through grade 6 5

PreK and/or K through grade 8 6

***31. Please indicate the number of children currently enrolled in
grades K--3 and the number of teachers assigned to each grade.***

  ------ ------------------------------------------------ ---------- ----------
                                                          Number of  Number of
                                                          children   teachers

  ------ ------------------------------------------------ ---------- ----------

a\. Kindergarten

b\. First grade

c\. Second grade

d\. Third grade

**32. Approximately, what are the percentages of male and female
students in your school?**

a\. Male students %

b\. Female students %

> Total 100%

**33. Below, please indicate the following information about students in
your school for both last year (2005--06) and the current year
(2006--07):** *(Please use NA for Not Applicable and DK for Don't
Know).*

-   Total enrollment in your school

-   The percentage of students in your school who are economically
    disadvantaged

-   The percentage of students in your school who are limited English
    proficient (LEP)

-   The percentage of students in your school with disabilities

  ----------- -------------- --------------- -------------- --------------
  Year        Total          Percent         Percent LEP    Percent with
              enrollment     economically                   disabilities
                             disadvantaged                  

  a\.                        \%              \%             \%
  2005--06                                                  

                                                            

                                                            

  b\.                        \%              \%             \%
  2006--07                                                  
  ----------- -------------- --------------- -------------- --------------

**34. Approximately what percentage of students in your school
represents each of the following racial groups?** *(Please answer each
item with a number, which can be 0, and verify that numbers equal 100
percent.)*

a\. African American %

b\. American Indian/Native Alaskan %

c\. Asian/Pacific Islander %

d\. Hispanic %

e\. Multiracial %

f\. White %

> Total 100%

**35. How many classroom teachers teach *reading* in grades K--3?**

\_\_\_\_\_ Teachers

**Of those, how many are new to the school this year, and how many
taught a different grade level last year (grade 4 or above)?**

a\. \_\_\_\_\_ Teachers new to the school this year

b\. \_\_\_\_\_ Teachers who taught grade 4 or above last year

**36. How many reading support personnel (e.g., instructional aides,
intervention specialists) are assigned to grades K--3 (both those funded
by Reading-First Ohio and those that are not)?**

\_\_\_\_\_ Reading support personnel

**37. How many *new* professional or instructional staff positions (full
or part time) have been created in grades K--3 since your school
received Reading First-Ohio funding, including Reading First-Ohio
building Trio positions, classroom teachers, and reading support
staff?** *(Please fill in one number for each item. If no positions were
created, write 0.)*

a\. \_\_\_\_\_ Reading First-Ohio Trio positions

b\. \_\_\_\_\_ Classroom teacher positions

c\. \_\_\_\_\_ Reading support staff positions (such as instructional
aides or intervention specialists)

**38. Please indicate the approximate percentage of current teaching
staff for grades K--3 who are:**

a\. Veteran teachers with more than 10 years of teaching experience %

b\. Experienced teachers with 4 to 10 years of teaching experience %

c\. New teachers with 1 to 3 years of teaching experience %

> Total 100%

**39. Please indicate the approximate percentage of current teaching
staff for grades K--3 who are:**

a\. Certified teachers with elementary, reading, early childhood, or
other related state certification %

b\. Probationary teachers with a temporary teaching certification %

c\. Emergency teachers without teaching certification %

> Total 100%

**40. Please fill out the following information for each individual who
helped to complete the survey in the event we need clarification of any
responses.**

> a\) Name:
>
> Position:
>
> Phone number:
>
> E-mail address:
>
> b\) Name:
>
> Position:
>
> Phone number:
>
> E-mail address:

**41. Please estimate the percentage of the survey completed by each
individual below.**

a\. Principal %

b\. Assistant or vice principal %

c\. Literacy specialist %

d\. Reading First-Ohio district coordinator %

e\. Reading teacher %

f\. Other (*please specify*)*:* %

> Total 100%

**42. If there is anything else that you would like to tell us, please
do so here.**

**THANK YOU!**

[^1]: The report also points out places where year 3 data for cohort 1
    and cohort 2 are significantly different. These data should be
    interpreted with caution as the characteristics of the schools
    participating in the two cohorts are dissimilar in a number of ways.

[^2]: The questions guide all research and evaluation efforts
    surrounding RFO, not just the Westat/LPA evaluation.

[^3]: This is a new question and may be refined in the future.

[^4]: The report also points out places where year 3 data for cohort 1
    and cohort 2 are significantly different. These data should be
    interpreted with caution as the characteristics of the schools
    participating in the two cohorts are dissimilar in a number of ways.

[^5]: Some schools surveyed in 2005 were no longer part of the RFO
    program in 2007. Reasons for this reduction in schools included that
    two were eliminated from the program because of inadequate progress
    at the end of their second year, and one was eliminated because the
    school was closed. The survey data presented in the year 1 annual
    report for 2005 and the survey data presented in this report for
    2007 will therefore be different because of the change in schools
    included in the 2005 statistics. The result was that 18 schools
    remained.

[^6]: The cohort 2 teachers are different from the cohort 1 teachers in
    both years of experience and years teaching reading, with cohort 2
    teachers having less experience. In addition, fewer cohort 2
    teachers had a professional license than cohort 1 teachers.

[^7]: It is important to note that all the percentages reported here
    regarding students and teachers are based on the average of
    principal estimates rather than individual student- and
    teacher-level data.

[^8]: See Table 11 in *Analysis of the Relationship Between Students'
    Outcome Data and Implementation Fidelity*, submitted to ODE by
    Westat on December 7, 2006.

[^9]: Depending on when the professional development is delivered,
    attendance may be mandated or voluntary. Teachers participating in
    Reading First are expected to volunteer and are paid for their time
    when the training occurs outside of regular working hours.

[^10]: In comparisons across years, the first number refers to data from
    2007 and the second, to data from 2005.

[^11]: However, it should be noted that more teachers report receiving
    various kinds of individual assistance from the literacy specialist
    (see page 19).

[^12]: Cohort 2 teachers also rated coverage of phonemic awareness and
    systematic phonics instruction significantly lower than their cohort
    1 counterparts. However, ratings of these areas remained high.

[^13]: As noted earlier, comparing principal responses across the 2
    years is difficult because of both the small sample size and the
    fact that the composition of the sample has changed dramatically.

[^14]: This rating is significantly lower than the 91.5 percent rating
    from cohort 1 teachers.

[^15]: There is also a drop in the longitudinal data, but this drop is
    not significant.

[^16]: Comparison of findings from cohort 1 and cohort 2 indicate cohort
    2 generally rating effectiveness lower, with 5 out of the 12 areas
    examined showing significant declines.

[^17]: Only 23 percent of teachers responded that they had used a
    different core reading program during 2006--07 compared to 62.0
    percent in 2005, a significant decrease indicating that more
    widespread adoption of a new core curriculum occurred 2 years ago.

[^18]: A high percentage of principals (93.8) did indicate in the
    principal survey that they added or enhanced programs for struggling
    readers in grades K--3, including computer-assisted programs.

[^19]: It is interesting to note that the longitudinal sample reveals
    significant increases in the same areas reflected in Figure II-9,
    with the exception of using the literacy specialist to provide
    instructional feedback. The increase in this area is not
    significant.

[^20]: There was still a slight, albeit significant, increase in the
    numbers of teachers who reported using the TerraNova assessment
    results always/almost always to assess student improvement in 2007
    (6.8 percent) compared to 2005 (2.3 percent).

[^21]: There is also a decrease in the cross-sectional sample, though it
    is not statistically significant.

[^22]: In another question that asked teachers to rank a variety of
    features with regard to their role in enhancing the instructional
    process in reading, the features receiving the highest rankings for
    making a major contribution were emphasis on the five essential
    components of reading, professional development provided by the RFO
    literacy specialist, teachers' use of assessment for instructional
    decisionmaking, and program-aligned materials.

[^23]: See the Westat/LPA Year 3 Annual Report for details on program
    implementation for cohort 1.

[^24]: Last year, we excluded scale 6 in the analyses because of its
    negative correlations with other scales. After further consultation
    with experts, we decided to include the scale 6 in the
    implementation index. This slightly changed the results for cohort 1
    as compared to last year.

[^25]: Because cohort 2 surveys are slightly different from cohort 1, we
    empirically adjusted the factor loadings. However, the scale
    structure remains the same.

[^26]: Other factors that might be thought to influence
    implementation---such as district or principal leadership, extent of
    professional development, etc.---are currently part of our program
    implementation measure and cannot, in these analyses, be used as
    predictors.

[^27]: The stability index was calculated based on average amount of
    test times students remained in the same district divided by the
    total number of actual test administrations possible for that
    respective district.

[^28]: There are considerable differences among three cohorts not only
    in demographic and economic characteristics, but also in student
    achievement. To accurately reflect these differences and to provide
    annual feedback to ODE, we examined the results by cohort.

[^29]: Because of the hierarchical linear modeling (HLM) analysis, the
    subgroup included has to have more than 10 schools represented.

[^30]: According to Cohen (1988), effect size estimates below 0.20 are
    considered small. However, Cohen also suggests that relatively small
    effects (i.e., around d=0.20) are to be expected for fields closely
    aligned with education. Similarly, Lipsey and Wilson's (2001) more
    recent compendium of meta-analyses concluded that psychological,
    educational, and behavioral treatment effects of modest values of
    even d=0.10 to 0.20 should not be interpreted as trivial. This is
    especially true regarding effects of large-scale comprehensive
    school reforms (Borman et al., 2003).

[^31]: We did not report data for racial/ethnic groups other than
    African American, Hispanic, and white because the sample sizes by
    grade were too small to be useful.

[^32]: Cohort 2 did not have gaps in performance on the OAT-R to begin
    with.

[^33]: We did not conduct similar comparison by LEP because there were
    only about five LEP students for each grade level in cohort 2.

[^34]: The significant levels were slightly different partly because of
    the smaller sample sizes.

[^35]: Comparison group data were obtained from TerraNova test records
    that did not collect information on economic status of students.

[^36]: This is different from our previous approach in creating a proxy
    for proportion of items correct.

[^37]: For all TI schools, we regressed the post-test scores on the
    pretest scores. The adjusted post-test score is the difference
    between observed post-test score and RTM effect (i.e., the residual
    of the regression).

[^38]: We do not have data on how long a school has been in the Title I
    program. However, generally, schools do not vary much from year to
    year in their participation in Title I.

[^39]: We also looked at the percentages of multiple disabilities and
    emotional disturbances, but the sample size for school that can
    produce valid data are too small.

[^40]: For both groups, the percentage of students with mental
    retardation increased over the past 4 years while those with
    learning disabilities declined.

[^41]: We can only work with the subsample because ODE did not have
    unique IDs for individual students prior to 2003.

[^42]: The adjustment will be made where adjusted prescore=observed
    prescore-reliability of test\*(observed prescore-average population
    score).