Alabama Reading First

Outside Evaluation -- 2006-7 School Year

*Prepared by*

Edward Moscovitch

March, 2008

# Analysis of Reading First Data

## The Three Reading First Questions

Reading First requires participating states to report statistical data
on three key questions:

1.  Are Reading First schools reducing the number of grade 1-3 students
    reading below grade level?

2.  Are Reading First schools increasing the number of grade 1-3
    students reading at or above grade level?

3.  How does the progress of students at Reading First schools compare
    with gains at comparable schools across the state?

## Using Weighted Non-Proficiency

### Definition

This report uses the concept of *weighted non-proficiency* to measure
the progress of Alabama Reading First (ARFI) schools[^1] and to answer
the three questions posed by the U.S. Department of Education. Weighted
non-proficiency (WNP) is defined as follows:

-   A student who achieves the proficiency standard for his or her grade
    is given a score of zero -- that student is fully proficient (and
    not at all non-proficient).

-   A student who fails to meet the proficiency standard but comes
    reasonably close is given a score of 50% - that student is
    considered half-proficient -- and, therefore, half non-proficient.

-   A student whose performance is well below the benchmark is
    considered fully non-proficient -- and given a score of 100%

Weighted non-proficiency can be averaged across all students at a school
(or any subset, such as minority students) to get a school-wide score.
The lower WNP the better; when WNP equals zero, all students are
proficient.

### Setting the Benchmarks

In the following analysis, student progress at the end of 1^st^ and
2^nd^ grades is measured by the DIBELS score in oral reading
fluency.[^2] The DIBELS benchmarks are used to define full and partial
proficiency -- students who are at "low risk" as defined by the DIBELS
benchmarks are fully proficient and have WNP equal to 0. Students who
are at "high risk" and "some risk" have WNP of 100% and 50%,
respectively.

For students in 3^rd^ grade, proficiency is measured by the Stanford 10
assessment; students in the 5^th^ stanine and above are proficient;
students in the lowest 3 stanines have WNP=100% and students in stanine
4 are partially proficient.[^3]

### Why Partial Proficiency?

An evaluation system that measures only whether a student is proficient
or non-proficient creates an unhealthy incentive for schools -- to
concentrate on those students whose performance falls just below the
proficiency benchmark, and to pay less attention to students with very
low reading scores. The use of partial credit for moving students from
high risk recognizes the efforts teachers make to move their
lowest-performing students -- even if those students move only part-way
toward the proficiency goal.

The use of WNP in this evaluation is consistent with the measure of
proficiency used to determine Adequate Yearly Progress (AYP) for Alabama
(and several other states) -- a measure which also gives half-credit as
students move from high risk to some risk.

### Why Non-Proficiency?

Concentrating on non-proficiency instead of proficiency focuses
attention on the task remaining for each school. This is exactly the
same approach taken in the Safe Harbor measure in calculating AYP. The
idea there is that any school that moves at least 10% of the way toward
the goal in a given year -- that is, that reduces its non-proficiency by
10% - has made adequate progress. Similarly, WNP is a measure of how far
a school has yet to go.

As a practical matter, once partial credit is given for students at some
risk, measures of proficiency are misleadingly high and gains are
understated. Consider a school where 100 students in a given grade are
tested; 60 are at benchmark, 20 are half-proficient, and 20 are at high
risk. This is a disappointing performance, but the weighted proficiency
measure is 70% (60% plus half of 20%), which feels high. If the school
moves 10 students in the lowest group to partial proficiency and 10 in
the middle category to full proficiency (it would then be 70 -- 20 --
10) the partial proficiency measure rises from 70% to 80% - which
doesn't seem like such a big gain. But if we look at weighted
non-proficiency, we see that WNP falls from 30% to 20% - a 33.3%
reduction! Put another way, the school used to be 30% short of the goal
(20 students fully non-proficient and 20 only half-proficient) and has
closed a third of the gap.

## Comparison Schools

To answer the third Reading First question we need to determine which
schools outside Reading First are comparable to the ARFI schools. To do
this, we've calculated for every Alabama school that had 3^rd^ grade
students the combined percent of minority and low-income students. A
student who is both poor and minority is counted twice; thus, the
maximum percent for any school is 200%.[^4] For ARFI schools, the 10^th^
percentile is a combined poverty/minority ratio of 114% - that is, 10%
of all ARFI schools have a combined percentage higher than 114%.

The control group chosen for these calculations, then, consists of
non-ARFI schools with a combined poverty/minority ratio of 114% or
better. This definition of the control group was chosen since there are
some ARFI schools with a ratio of 114% or better. This is an imperfect
comparison, however, since the ARFI schools are concentrated at the very
highest poverty-minority ratios. The typical (median) ARFI school has a
combined ratio of 186%; this compares to 172% for the control group
schools. This is illustrated graphically in Chart 1 later in the
evaluation.

## Following Individual Students

The results reported here are obtained by following individual students
from one year to the next. This is a fairer measure of the work of a
school's faculty than comparing two separate cohorts in two different
years. That is, instead of comparing this year's 3^rd^ graders to last
year's 3^rd^ graders, this report will look at how last year's 2^nd^
graders did this year as 3^rd^ graders. The calculations reported here
are for those students for whom scores are known both last year and
this; we are thus tracking the progress of a particular group of
students over time.

When we approach the data in this way we are recognizing that average
scores alone do not measure the effectiveness of a faculty.
Socio-economic status has a huge impact on student performance; the high
scores of a suburban school filled with students from prosperous homes
whose parents read regularly to them when they were young isn't
necessarily doing a better job than an inner-city or Black Belt school
with lower scores where students enter school with very limited language
and vocabulary skills.

Regardless of where students start, however, it's entirely appropriate
to ask whether students are moving from one year to the next --
specifically, whether a reasonable proportion of the students who were
partially or fully non-proficient last year have moved up this year.

## Reading the Data

The following 3 tables show the one-year gain in student performance, by
grade and by school group, from the spring of 2006 to the spring of
2007. To explain the data in these tables, let's use the following
excerpt from the larger tables as an example:

![](media/image1.wmf){width="5.999305555555556in"
height="1.1069444444444445in"}

In the spring of 2007, ARFI schools tested 5,640 students finishing
their kindergarten year. (The test used at the end of kindergarten is
the DIBELS nonsense word fluency test).[^5] Of these, we are able to
find match scores (in this case, scores from the preceding fall) for
5,226 students, or 93%. In kindergarten, we look at performance the
previous fall (the test used for students entering kindergarten is
DIBELS letter naming fluency); in other grades we use the previous
spring. In the spring of 2007, ARFI schools tested 5,997 first grade
students; we have scores at the end of kindergarten (in the spring of
2006) for 4,542 of these students, or 76%.[^6]

Of the 4,542 first-grade students where we have matched scores, 93% are
in the same school as they'd attended the year before, 76% were
minorities, and 85% were low income.[^7] Continuing across the row, we
see that in the spring of 2006 -- as kindergarten students - 4% of these
students had been at level 1 (DIBELS high risk), 6% were at level 2
(DIBELS some risk), and 91% were at level 3 (DIBELS low risk, or
benchmark). This translates to WNP of 7% (all of the students at level 1
and half of those at level 2). A year later, at the end of 1^st^ grade,
8% were at level 1, 14% at level 2, and 78% at level 3 -- which
translates to WNP of 15%. Although this is still a very strong
performance (the DIBELS authors set the goal of 80% of students at
benchmark), fewer of these students were proficient at the end of first
grade than had met the goal a year earlier; WNP actually rose by 129%,
as shown in the first column.[^8]

The kindergarten results are shown in the first row. Only 56% of
students were at benchmark as they entered kindergarten; 15% were at
some risk. WNP was therefore 36%. At the end of kindergarten year, there
was a dramatic gain -- 89% of students were at benchmark and WNP had
fallen to only 8% - a 78% gain, as shown in the first column.

## Why Comparative Analysis doesn't fit Alabama

The tables below show student progress for ARFI schools, for control
group schools, and for other schools, as requested by Reading First. The
tables show that overall results for ARFI schools, taken as a whole, do
not in fact differ substantially from the control group schools. The
implied comparison between ARFI and control group schools is highly
misleading. ***All primary-grade schools in Alabama are included within
the Alabama Reading Initiative and receive the same kind of assistance,
using the same Reading First pedagogy and the same coaching staff, as
ARFI schools.*** Since the control group schools are also receiving
essentially the same kind of assistance as ARFI schools, it would be a
mistake to conclude that Alabama Reading First is having no impact if
its schools' performance is similar to that of the control group.

Given the (very positive) role played across Alabama by the ARI, a more
reasonable question is this:

-   *Do ARFI schools (and comparable schools) improve the reading
    performance of their students from year to year? Specifically, when
    we follow individual students from one year to the next, do we see
    reductions in WNP?*

## Looking at Data - One-Year Change

Complete data for ARFI schools, for comparison group schools (that is,
schools not in ARFI with combined poverty/minority ratios of 114% or
higher), and for other schools[^9] are shown in the tables on the next 3
pages.

### ARFI Results --Primary Grades

The ARFI schools are doing very well at addressing literacy scores of
kindergarten students. While 28% of students are at high risk and 15% at
some risk as they enter kindergarten, by the end of the year only 5%
remain at high risk and only 7% at some risk. This represents a 78%
reduction in WNP -- a very impressive result.

WNP rises somewhat in first grade, but the overall results at the end of
first grade are still good -- WNP is 15%, with 78% of students at
benchmark.

During the course of 2nd grade, students are losing ground (not just at
ARFI schools, but throughout the state). At the end of 1st grade, 79%
were at benchmark (WNP = 14%); a year later only 65% are at benchmark in
oral reading fluency (WNP = 27%).

### ARFI Results -- Upper Elementary

Results for grades 3 through 5, as measured by DIBELS and ARMT, are
shown in the upper half of the table; results as measured by the
Stanford 10 (SAT) are shown in the lower half. For all three measures,
the year-earlier results are oral reading fluency at the end of 2^nd^
grade. Reading First funding is limited to kindergarten through 3^rd^
grade, but the underlying purpose is to improve reading comprehension
not only in 3^rd^ grade but in the following grades as well. As Reading
First succeeds, we'd expect to see that the gains in 3^rd^ grade are
carried on in following years. For this reason, we've included grades 4
and 5 in these tables.

For the SAT and ARMT[^10] assessments (we use the SAT reading
comprehension score and ARMT reading score), there are four levels
instead of 3; the fourth represents advanced performance. For the ARMT,
levels 1 through 4 here reflect the actual ARMT score; for the SAT,
level 1 is stanines 1-3, level 2 is stanine 4, level 3 is stanines 5 and
6, and level 4 is stanines 7 and higher. The 4^th^ level is not used in
calculating WNP for the SAT assessment; it is used for the ARMT.[^11]

Looking only at oral reading fluency, there is little change during the
course of 3^rd^ grade; from a year-earlier score of 21%, WNP at the end
of third grade was virtually unchanged at 23%. If we look at the two
broader measures of reading comprehension, students lost ground during
3^rd^ grade, finishing at 30% using the ARMT or 35% using the SAT.

During the course of 4^th^ and 5^th^ grades, reading comprehension
scores show only minor change, whether measured by SAT or by ARMT.

![](media/image2.wmf){width="8.79513888888889in"
height="3.8833333333333333in"}

![](media/image3.wmf){width="8.863194444444444in"
height="3.5597222222222222in"}

For the reasons cited above in the text, the results for the control
group schools are very similar to those for ARFI schools.

![](media/image4.wmf){width="8.863194444444444in"
height="3.6152777777777776in"}

The final table shows results for lower poverty schools. While
kindergarten results for these schools are virtually the same as for
ARFI and for control group schools, and while these schools also show an
erosion of reading progress in 1^st^ and 2^nd^ grades, the proportion of
students falling below benchmark is less. While WNP in 3^rd^ grade and
beyond for ARFI and control group schools rose to 30% (ARMT) or 35%
(SAT), WNP in these schools with lower poverty and minority enrollment
stayed at 20% or less from 3^rd^ through 5^th^ grades.

To facilitate comparisons, Table 4 below shows the WNP results
side-by-side across the three groups of schools.

![](media/image5.wmf){width="5.883333333333334in"
height="3.640277777777778in"}

Although the scores of ARFI and control group schools are close, the
ARFI schools are doing slightly better with students in kindergarten,
first, and second grades. In 3^rd^ grade and beyond, however, there is
virtually no difference. This table somewhat understates the performance
of ARFI schools since the typical ARFI school in fact has a somewhat
higher proportion of low income and minority students than a typical
control-group school. A more accurate statement, then, would be that
ARFI school scores are slightly higher than control group schools,
despite the fact that they are working with a somewhat needier group of
students.

The side-by-side comparison emphasizes the difference demographics make.
Basic decoding skills, as measured by nonsense word fluency at the end
of kindergarten, are essentially the same for the high-poverty,
high-minority schools in ARFI and the control group schools as for the
lower poverty schools. But in first, second, and third grades, the
reading scores in the lower poverty, lower minority schools move ahead
of those in the ARFI and control group schools. This finding is not
surprising; children can be taught mechanical decoding skills regardless
of the language and vocabulary background they bring to school; as the
required fluency rises to 90 words per minute in 2^nd^ grade, and as the
focus shifts to comprehension of longer passages in 3^rd^ grade and
beyond, the limited language exposure of children in ARFI and other
high-poverty, high-minority schools translates into lower performance in
these higher grades.

## Impact of Poverty, Ethnicity on WNP Scores

The impact of poverty and ethnicity on student performance is shown
graphically in chart 1 below. Each circle represents one Alabama
elementary school. The size of the circles reflects the number of
students tested in 3^rd^ and 4^th^ grade. The horizontal position of the
circle represents the combined poverty/low income ratio of the school
and the vertical position represents the combined WNP for 3^rd^ and
4^th^ grades, from the SAT reading comprehension score.

![](media/image6.wmf){width="5.957638888888889in" height="3.4625in"}

There's a purple circle for each of the 777 Alabama schools that tested
more than 10 students in these two grades. The size of the circle
reflects how many students were tested. The green circles show the ARFI
schools; the solid blue line running from lower left to upper right is
the trend line showing the average relationship between demographics and
student scores. The chart underscores the extent to which ARFI has
concentrated its attention on the very neediest schools in Alabama; half
of all ARFI schools have a combined poverty-minority ratio of 186% or
higher. This is far higher than the statewide median (81%) and somewhat
higher than the median for the control group schools (172%).

Schools shown above the line have actual WNP higher than expected; these
are low performing schools. Schools below the line are high-performing;
their actual WNP is less than what we'd expect from demographics alone.

Looking at the data in this way, we can see how ARFI schools compare
with other schools with similar demographics (that is, those at the same
horizontal position). In general, ARFI school performance is similar to
that of comparable schools -- except for schools that are virtually all
poor and all-minority (combined ratio close to 200%). Two ARFI schools
in this range outperform all similar schools, and none of the ARFI
schools have WNP higher than 57%.

## Student Progress by Ethnicity

The analysis following individual students from one year to the next can
be done by ethnicity; the results are summarized in Chart 2 below:

![](media/image7.wmf){width="5.957638888888889in" height="3.4625in"}

The solid lines show results for minority students (Blacks, Hispanics,
and Native-Americans); the dotted lines are for Whites and Asians. The
lines are color-coded -- green for ARFI schools, blue for the control
group, and red for schools with lower poverty and minority ratios. As
we've seen, great gains are made in kindergarten; by 3^rd^ grade
proficiency of minority students is roughly where it was when they
entered kindergarten.

There are some interesting differences in this pattern:

-   The achievement gap between minority and non-minority students is
    relatively small when measuring basic reading skills; when we look
    at comprehension in 3^rd^ grade, we see that white and Asian
    students -- in all three school groups - make gains that are not
    matched by Black or Hispanic students.

-   In first and second grades, the achievement gap is smallest in ARFI
    schools; they do a better job of helping minority students keep pace
    in oral reading fluency than schools outside Reading First

-   Students in schools with lower poverty/minority ratios generally
    outperform students with similar ethnicity. (That is, Black students
    in places like Hoover will generally do better than Black students
    in cities like Bessemer)

-   Contrary to the normal pattern, Black and Hispanic students in ARFI
    schools actually perform better in nonsense word and oral reading
    fluency than do Black and Hispanic students in the schools with
    lower minority/poverty ratios.

-   Although the differences are small, students at ARFI schools
    outperform their demographic counterparts in control-group schools.
    This is true for both demographic groups and at all grade levels,
    with the exception of minority students' 3^rd^ grade reading
    comprehension, where the control group schools have a very slight
    edge.

## Looking at Data - 4-Year Comparison

The Reading First goal is to have students reading with good
comprehension by the end of 3^rd^ grade -- and to concentrate on
students who arrive in kindergarten without strong language and
vocabulary skills. The previous tables look at 1-year change; in the
long run, we are interested in how successful ARFI schools are helping
students who enter kindergarten with major deficiencies in language
development become successful readers at the end of 3^rd^ grade.

For just over 22,000 Alabama students who did not change schools between
kindergarten and third grade, we can take a direct measure of how ARFI
schools -- and all Alabama schools -- are doing over these four critical
years of students' development. For these students, we have DIBELS
results in the fall of 2003, as they entered the school system as
freshmen. We can observe their performance on the SAT 10 assessment at
the end of 3^rd^ grade in the spring of 2007 and compare it to
kindergarten scores 4 school years earlier.[^12]

An excerpt from the table displaying the results of this analysis is
shown below and helps walk us through the results.

![](media/image8.wmf){width="2.9930555555555554in"
height="0.6736111111111112in"}The excerpt shows the results for all
students at ARFI schools and for students at ARFI schools who entered
kindergarten at benchmark in letter-naming fluency. (This row is labeled
"G" for Green; not shown here are rows "Y" -- yellow -- some risk and
"R" -- red -- high risk). The columns of the table separate students by
their performance on the 3^rd^ grade SAT 10 assessment of reading
comprehension. Looking at the top row (labeled "ARFI", we see that 58%
of all ARFI students finished 3^rd^ grade at benchmark in reading
comprehension. [^13] Nineteen percent of the 3^rd^ graders at ARFI
schools were at some risk in reading comprehension, and 22% were at high
risk. The final column simply reminds us that this is the total row for
ARFI, and includes all ARFI students.

If we start at the Total column of the next row, we see that 51% of all
of the ARFI students in the study group entered kindergarten at
benchmark. Of this 51%, 66% also finished 3^rd^ grade at benchmark, 18%
at some risk, and 16% at high risk.

We can now turn to the complete results, shown in Table 5.

![](media/image9.wmf){width="3.1666666666666665in"
height="2.652083333333333in"}Looking first at ARFI schools, we see from
the total column that 51% of all students entering ARFI schools were at
benchmark in the fall of kindergarten year. This is lower than for the
control group or low poverty schools, which tells us that ARFI has in
fact selected needy schools.

Looking at the top ARFI row (in the Green column), we see that four
years later 58% of these same students finished 3^rd^ grade at
benchmark. One measure of ARFI success, then, is that over four school
years, the percent of students at benchmark rose from 51% to 58%, while
the percent at high risk fell from 28% to 22%.

Taking a closer look, we see that 66% of ARFI students who entered
school at benchmark finished 3^rd^ grade at benchmark, as did 55% of
those who entered at some risk and 46% of those who entered at high
risk. This indicates substantial upward mobility -- over half of the
students who entered at some risk and almost half of those who entered
at high risk were reading with good comprehension after four years in
school.

Students who enter kindergarten at some or high risk in ARFI schools (in
the Yellow and Red rows) are slightly more likely to be at benchmark in
reading comprehension 4 school years later than are students with
similar pre-K language skills who enter control group schools (55% and
46% for yellow and red entry at ARFI schools, compared with 52% and 43%
for yellow and red entry at control-group schools. Students at both ARFI
and control group schools are less likely to finish 3^rd^ grade at
benchmark than students who enter lower-poverty schools with similar
entry-level language skills (at the lower poverty schools, 87% of those
entering at benchmark finish at benchmark, as against only 66% at ARFI
schools).

Table 5 also shows us that 18% of ARFI students who enter at benchmark
finish 3^rd^ grade at some risk, while 16% finish 3^rd^ grade at high
risk. Thus, there's downward mobility as well as upward mobility. On
balance, more students move up than down, since the overall percent at
benchmark rises over the 4 school years. The likelihood of downward
mobility is almost identical at ARFI and control group schools, but
children who enter low-poverty schools at benchmark are less likely to
fall behind than their counterparts either at ARFI or control group
schools.

## Data Summary -- The Three Questions

Pulling together the findings above, we can answer the three Reading
First questions:

1.  **[Are ARFI schools moving struggling readers to
    proficiency?]{.underline}**[^14] Yes, particularly in kindergarten
    and first grade. The percentage of ARFI students who are proficient
    in basic decoding skills in the early grades is very high -- 89% of
    students meet the DIBELS benchmark in nonsense word fluency at the
    end of kindergarten (up from only 56% who are at benchmark when they
    enter kindergarten at the beginning of the school year) and 78% are
    proficient in oral reading fluency at the end of 1^st^ grade. In
    these early years, ARFI schools have all but eliminated the racial
    achievement gap -- Weighted non-proficiency for Blacks and Hispanics
    at the end of kindergarten is 8.2%, vs. 6.7% for Whites and Asians.
    At the end of first grade the scores are 15.7% and 13.7%.

2.  **[Are students already at benchmark in ARFI schools staying
    there?]{.underline}** There is substantial slippage from the end of
    1^st^ grade to the end of 3^rd^ grade. Some of this occurs as the
    oral reading fluency benchmark rises from 40 words per minute at the
    end of 1^st^ grade to 90 wpm at the end of 2^nd^; most of the
    reduction occurs when the focus switches from oral reading fluency
    to comprehension of longer passages, as measured by the SAT 10 and
    the ARMT. The reduction from fluency proficiency to comprehension
    fluency occurs primarily amongst minority students. On balance, the
    percent of students at benchmark in reading comprehension at the end
    of 3^rd^ grade is slightly higher than the percent at benchmark when
    they enter kindergarten, and the percent at high risk is slightly
    lower.

3.  **[Are ARFI schools performing better than comparable
    schools?]{.underline}** This really isn't an appropriate question
    for Alabama, since there is no true control group. *All* Alabama
    schools receive assistance in literacy strategies from the Alabama
    Reading Initiative, and the two programs share a common philosophy,
    common leadership, common training, and common coaching staffs. It
    is not surprising, then, that ARFI school performance is very
    similar to that of control-group schools. When comparisons are made
    by ethnic group, students at ARFI schools outperform those at
    control group schools -- by a small margin -- and do so for both
    major ethnic groups and at almost all grade levels. Minority
    students at ARFI schools actually do slightly better at basic
    decoding skills in grades K through 2 than minority students at the
    lower poverty schools that are not in ARFI and not in the control
    group.

This picture -- great strength in basic decoding skills and weakness in
comprehension and vocabulary skills -- is not new, and the ARI/ARFI
staff is changing its coaching and training programs to put greater
emphasis on vocabulary, language development, and comprehension.

This represents a natural evolution; any comprehensive literacy program
would begin with basic decoding skills. It's easier to get good data on
fluency and blending than on vocabulary and comprehension, and the use
of this data is central not only to student improvement but also to
achieving teacher buy-in. And, of course, children cannot read with
comprehension if they can't do basic decoding.

Once these basic skills are mastered, the program needs to switch its
emphasis, as ARI/ARFI is doing.

# Outlier Analysis

Six ARFI schools were chosen for outlier analysis - two each (one high
performing, one low performing) in three of the state's largest school
districts (Mobile and Montgomery Counties and Birmingham City). Six
coaches were interviewed; in the course of the interviews the coaches
also discussed other high and low performing schools that they are
currently working with.

None of the findings are surprising; here are some of the key points to
emerge from these interviews:

## Use of Data and Analysis to Drive Instruction

Coaches working with successful schools report that the principal and
the reading coaches engage in extensive discussion with teachers about
individual students -- including not only their progress, but their
specific needs and how to address them. Teachers are then more focused
on instruction to address individual needs. In particular, data meetings
at these schools are focused on what particular students need. One coach
reported being impressed with the quality of faculty discussion about
student needs and how to meet them.

In less successful schools, data meetings are also held, but they only
review the data, omitting the key next step, which is to look at
miscues, discuss why children are struggling, and what remedies they can
use to address student needs.

## Role of the Principal

It comes as no surprise that schools do best when they have a principal
truly committed to helping every student learn. It's possible that a
knowledgeable and well respected reading coach, in a school with a truly
committed faculty, can use data and change instruction even without a
strong principal. In most cases, however, it takes the drive and
encouragement of a strong principal to bring about this change.

### Data Driving Instruction

In one of the successful schools, the principal has arranged for common
planning time for each grade level once each week. Every four weeks the
principal meets with the team to discuss individual student progress;
the reading coach also meets with the team every four weeks (two weeks
after/before the principal) for this purpose. The principal at this
school plans the school schedule before the school year begins to make
sure that every student that needs intervention gets it and identifies
exactly which staff member will be providing that intervention and when.
She follows up during the school year to make sure-- everyday, no matter
what -- that the planned intervention actually occurs.

At another of the successful schools, principal runs the data meetings
("like you've never seen before," according to the coach). This
principal is herself very well informed about reading pedagogy and
reading research. She knows the data and understands its implications
for instruction; she discusses this and the relevant research
extensively with her teachers.

### Less Successful Schools

Although one occasionally finds principals who are actively hostile to
school turnaround and resistant to any outside advice, that wasn't the
case with principals in the less successful schools studied for this
project. In some cases, the school was a K-8 school, and the principal's
attention was focused on the middle-school grades. Birmingham City has
ordered elementary school principals to pay particular attention to
grades 3, 4, and 5, since they are the grades tested to determine
whether or not the school makes AYP.[^15]

Generally, the coaches I talked to characterized the less successful
principals as being good, hard-working people who genuinely wanted to
improve their schools and who were not defensive about suggested
improvements. One difficulty is that the principals want to be liked,
and don't know how to insist on change when teachers are resistant.
Paradoxically, the successful principals who set high standards and
insisted on improvement ***are*** very well liked and respected by their
faculties. Not surprisingly, leadership is an art form; a good principal
needs to have the self-confidence to put herself (or himself) on the
line insisting on change, knowing that most teachers will respond and,
whatever the short-term resistance, teachers will respect her for it.

At these less successful schools, the coaches reported that while
principals and school-based coaches were receptive to suggestions when
the outside coach was actually at the school, there was no follow
through -- suggested changes were dropped when the outside coach wasn't
there. Similarly, the principals don't consistently follow through to
make sure that teachers actually implement suggested improvements.

## Intervention

A key difference between successful and unsuccessful schools is the
amount and quality of intervention. At one of the successful schools,
the principal insists that "any warm body" be available for
intervention. This includes not only special education and Title 1
teachers, but para-professionals and even custodians. Every one who is
pressed into service for intervention is given specific training. At
successful schools, interventionists are active participants at data
meetings.

In the less successful schools, the amount of intervention is limited.
Coaches report that special education and Title 1 teachers, and
para-professionals, are not doing intervention. Intervention is left to
the classroom teacher and, in the afternoon, the reading coaches will
pull struggling students. Intervention is usually the first thing to be
sacrificed if the teacher falls behind or if the coaches are called on
to perform other roles or work with students in higher grades. The
reliance these schools place on coaches to do intervention limits the
time available for coaches to work with teachers.

## The Roadmap to Success

The principals and faculties at the less successful schools don't appear
to have a clear understanding of how all of the elements of a strong
literacy program fit together. This would include a failure to
understand:

-   That the key role of the building based coach is to help improve the
    instruction of classroom teachers

-   That improved instruction for everyone -- faithful implementation of
    the core curriculum -- is essential in helping on-grade and
    above-grade students improve, and in bringing down to a manageable
    level the number of students who need intervention

-   That classroom teachers have to take ownership of all their
    students, but that grade-level teams and interventionists have to
    work closely together to make sure that struggling readers get the
    help they need.

-   That the principal needs to set high expectations and follow
    through, and that when she does this she makes the reading coach's
    job easier.

-   That DIBELS and other data is there to help identify individual
    student needs, and that discussion of the results and their
    implication for instruction are central to any improvement.

The coaches also reported that the faculties at these schools had not
really bought into reform and were therefore not always receptive to
outside help. While teachers at successful schools looked forward to
visits from ARFI coaches as an opportunity to get help addressing needs
of still-struggling students, teachers at less successful schools looked
at these visits as just another interruption to their routine.

[^1]: Alabama's Reading First program is known as the Alabama Reading
    First Initiative (ARFI), a name which underscores the close ties
    between Alabama Reading First and the Alabama Reading Initiative
    (ARI), which is working with almost 1,000 Alabama schools to improve
    their literacy instruction.

[^2]: ARFI schools administer the SAT reading comprehension test in
    1^st^ and 2^nd^ grade, but other schools do not. All elementary
    schools in Alabama give the DIBELS in grades 1 and 2 (and
    kindergarten). Thus, any analysis that compares ARFI schools with
    other schools has to use the DIBELS for these grades.

[^3]: The standard for proficiency used in 3^rd^ grade -- stanine 5 or
    above on the SAT instead of level 3 or 4 on the Alabama Reading and
    Math Test (ARMT) -- reflects the higher standard ARI and ARFI have
    set. Statewide, 65% of students meet the ARI/ARFI definition of
    reading proficiency, versus 81% who score 3 or 4 in reading on the
    ARMT.

[^4]: In making these calculations, students are considered minority if
    they are Black, Hispanic, or Native-American. Statewide,
    Asian-American student scores are close to (or higher than) scores
    for non-Hispanic White students, so Asian-American students are not
    counted as minorities. While this is a reasonable way to proceed
    (particularly for children of highly-educated Indian or Chinese
    parents who work or study at high-tech centers like Birmingham and
    Huntsville), it may understate the achievement of the (relatively
    few) schools where the Asian students are of Cambodian or Laotian
    descent, whose parents are not usually so well educated. Since no
    data is available on students by particular Asian nationality, and
    since average scores for all Asian-American students in Alabama are
    so high, they are grouped with whites in this analysis.

[^5]: The selection of nonsense word and letter naming fluency for
    assessing kindergarten performance is based on an in-depth study of
    1,000 students who were followed from kindergarten through 3^rd^
    grade comprehension; these tests were most closely correlated with
    eventual reading comprehension, as measured by the SAT 10 at the end
    of 3^rd^ grade.

[^6]: Because of the way DIBELS data is stored by the University of
    Oregon, there is no easy way to follow students from one district to
    another. For this reason, the match percentage is relatively low;
    when we get to 4^th^ and 5^th^ grade, we're able to find
    year-earlier scores for over 90% of students tested. Also, since the
    system allows for a student to be at only one school during any
    given school year, the percent of kindergarten students at the same
    school is necessarily 100%.

[^7]: Data on poverty percentages requires complex matches with a
    separate state database; these were not available for kindergarten
    and 2^nd^ grade.

[^8]: A large percentage increase, but on a very low starting point, so
    the absolute increase is not as large as this high percentage
    implies.

[^9]: The other schools are those with combined poverty ratios less than
    114%. They are therefore referred to here as lower poverty schools.

[^10]: Alabama Reading and Math Test -- a modified version of the
    Stanford 10 given by Alabama and used to determine AYP.

[^11]: Because the ARMT cut scores are somewhat lower than on the SAT,
    the ARI has adopted a goal of level 4 performance on the ARMT. WNP
    for the ARMT is therefore calculated as 100% for students at level
    1, 67% for students at level 2, 33% for students at level 3, and 0
    for students at level 4. Statewide, 65% of students meet the
    ARI/ARFI definition of reading proficiency, versus 81% who score 3
    or 4 in reading on the ARMT.

[^12]: This analysis is limited to Cohort 1 ARFI schools, since cohort 2
    schools did not start with ARFI until these students were in second
    grade.

[^13]: As indicated earlier, benchmark on the SAT 10 in reading
    comprehension is defined as stanine 5 and above; high risk is
    stanines 1 through 3.

[^14]: The actual wording of the first question reads "Are Reading First
    schools reducing the number of grade 1-3 students reading below
    grade level?" This logically divides into two questions -- "Are they
    reducing the number below grade level?" and "Are those on grade
    level staying there?" -- as the answers are laid out here.

[^15]: Coaches funded from Reading First are only allowed to work in
    grades K-3. The ARFI central office has made this point to the
    Birmingham School District, which has assured them that the coaches
    working in upper elementary grades are funded locally or through the
    Alabama Reading Initiative, but not from Alabama Reading First.