**EVALUATION OF KANSAS READING FIRST GRANT**

**THREE-YEAR REPORT OF ACHIEVEMENT RESULTS**

**Prepared for:**

**Kansas State Department of Education**

**120 S.E. 10^th^ Avenue**

**Topeka, Kansas 66612-1182**

**Judy Pfannenstiel**

**Dianne Seltzer, Ed. D.**

**Vicki Yarnell**

**Research & Training Associates, Inc.**

**11030 Oakmont, Suite 200**

**Overland Park, KS 66210-1100**

**2008**

**KANSAS READING FIRST PROGRAM Achievement RESULTS**

**For Four Years: 2004-2007**

All students in grades 1 to 3 in Reading First schools were administered
tests that are used for both evaluation and accountability purposes. The
Stanford 10 was administered to obtain measures of four of the five
required components---phonemic awareness, phonics, vocabulary
development, and comprehension. The DIBELS was administered to assess
oral reading fluency. Beginning in spring 2006, the newly developed
State Reading Assessment was used for evaluation purposes in lieu of the
Stanford 10 for third graders. Third graders continued to be assessed on
oral fluency with the DIBELS.

Metrics provided by the Stanford 10 include a three-category description
of proficiency levels that correspond to the DIBELS proficiency levels:
*needs substantial intervention* (scoring at or below the 20^th^
percentile), *needs additional instruction* (scoring above the 20^th^ or
at or below the 40^th^ percentile), and *at grade level* (scoring above
the 40^th^ percentile). The SAT 10 provides an additional Normal Curve
Equivalent (NCE) metric that is composed of all multiple-choice items
contained on the four components.

The Kansas State Reading Assessment was administered for the first time
in spring 2006 and replaced the SAT 10 as the outcome measure for
Reading First programs. Thus, comparisons with third grade measures in
prior years must acknowledge a change in instrument. The state defined
the performance level categories using new cut scores in 2006. The
lowest category is *Warning*, followed by *Approaching Standard*, *Meets
Standard, Exceeds Standard* and *Exemplary*. Students who scored in the
categories *meets standard*, *exceeds standard*, and *exemplary* were
coded as *at grade level---meeting proficiency*.

##### 

##### STUDENT BACKGROUND CHARACTERISTICS

Reading First schools provided measures of student background
characteristics, including gender, eligibility for free or reduced-price
lunch, racial/ethnic status, English Language Learner (ELL) status, and
existence of an Individual Education Plan (IEP) for 2006-07. Reading
First schools are balanced in terms of gender (see Table 1). Almost 80%
of students in Reading First schools in grades 1 to 3 are eligible for
free or reduced-price lunch, indicating a high degree of economic
disadvantagement in Reading First schools.

Approximately 60% of the students in Reading First schools belong to a
racial/ethnic minority group. African-American students comprise
slightly less than 20% of students in Reading First schools and Hispanic
students comprise approximately one-third of students.

Almost one-fourth of students at each grade level in Reading First
schools are identified as English Language Learners with limited English
proficiency. Between 11-15% of students in grades 1 to 3 in Reading
First schools had IEPs, an increase over the percentage reported in
2005-06.

Table 1. Background Characteristics of Tested Reading First Students

for School Years 2005-06 and 2006-07

<table style="width:100%;">
<colgroup>
<col style="width: 14%" />
<col style="width: 4%" />
<col style="width: 8%" />
<col style="width: 5%" />
<col style="width: 9%" />
<col style="width: 5%" />
<col style="width: 7%" />
<col style="width: 5%" />
<col style="width: 9%" />
<col style="width: 5%" />
<col style="width: 8%" />
<col style="width: 5%" />
<col style="width: 9%" />
</colgroup>
<tbody>
<tr class="odd">
<td></td>
<td colspan="6"><h2 id="spring-2006">Spring 2006</h2></td>
<td colspan="6"><strong>Spring 2007</strong></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"><strong>1<sup>st</sup></strong></td>
<td colspan="2"><strong>2<sup>nd</sup></strong></td>
<td colspan="2"><strong>3<sup>rd</sup></strong></td>
<td colspan="2"><strong>1<sup>st</sup></strong></td>
<td colspan="2"><strong>2<sup>nd</sup></strong></td>
<td colspan="2"><strong>3<sup>rd</sup></strong></td>
</tr>
<tr class="odd">
<td>% Male</td>
<td>50</td>
<td>(1627)</td>
<td>49</td>
<td>(1518)</td>
<td>49</td>
<td>(1503)</td>
<td>51</td>
<td>(1956)</td>
<td>51</td>
<td>(1954)</td>
<td>50</td>
<td>(2064)</td>
</tr>
<tr class="even">
<td>% Free Lunch</td>
<td>77</td>
<td>(1639)</td>
<td>78</td>
<td>(1547)</td>
<td>74</td>
<td>(1521)</td>
<td>79</td>
<td>(1944)</td>
<td>80</td>
<td>(1935)</td>
<td>78</td>
<td>(2064)</td>
</tr>
<tr class="odd">
<td>% Minority</td>
<td>53</td>
<td>(1644)</td>
<td>53</td>
<td>(1545)</td>
<td>54</td>
<td>(1351)</td>
<td>61</td>
<td>(1932)</td>
<td>61</td>
<td>(1928)</td>
<td>62</td>
<td>(2041)</td>
</tr>
<tr class="even">
<td>% African American</td>
<td>22</td>
<td>(1644)</td>
<td>23</td>
<td>(1545)</td>
<td>21</td>
<td>(1351)</td>
<td>17</td>
<td>(1932)</td>
<td>17</td>
<td>(1928)</td>
<td>19</td>
<td>(2041)</td>
</tr>
<tr class="odd">
<td>% Hispanic</td>
<td>35</td>
<td>(1644)</td>
<td>34</td>
<td>(1545)</td>
<td>31</td>
<td>(1351)</td>
<td>33</td>
<td>(1932)</td>
<td>32</td>
<td>(1928)</td>
<td>32</td>
<td>(2041)</td>
</tr>
<tr class="even">
<td>% ELL</td>
<td>22</td>
<td>(1671)</td>
<td>22</td>
<td>(1573)</td>
<td>18</td>
<td>(1540)</td>
<td>25</td>
<td>(1942)</td>
<td>22</td>
<td>(1933)</td>
<td>24</td>
<td>(2064)</td>
</tr>
<tr class="odd">
<td>% with IEPs</td>
<td>11</td>
<td>(1671)</td>
<td>11</td>
<td>(1573)</td>
<td>9</td>
<td>(1537)</td>
<td>11</td>
<td>(1941)</td>
<td>12</td>
<td>(1933)</td>
<td>15</td>
<td>(2064)</td>
</tr>
</tbody>
</table>

**STUDENT ACHIEVEMENT**

Normal Curve Equivalent[^1] (NCE) scores obtained from multiple-choice
items spanning the four SAT 10 components indicate the performance of
Reading First students relative to a nationally-normed group. These
results are provided in Table 2 and indicate that end of 1^st^ grade
Reading First students scored lower than their national peers in both
2004 and 2005, scoring at the 47^th^ NCE in 2004 and at the 45^th^ NCE
in 2005. By the third and fourth years of implementation, 1^st^ graders
at Reading First schools scored at the national average of the 50^th^
NCE.

Reading First 2^nd^ graders scored lower than their national peers in
2004-2006. By the fourth year of implementation, 2^nd^ graders in
Reading First schools scored at the national average. While 3^rd^
graders at Reading First schools also scored below the national average
in 2004 and 2005, comparable data is not available for 2006 and 2007
when the State Reading Assessment was administered to 3^rd^ graders.

Table 2. Mean NCE Score for Reading First Students by Grade and Year

  ------- -------- ---------- -------- ---------- -------- ---------- -------- ----------
          2004                2005                2006                2007     

          Mean     N          Mean     N          Mean     N          Mean     N

  1st     47.4     \(1069\)   44.9     \(1587\)   49.5     \(1443\)   50.8     \(1860\)

  2^nd^   46.8     \(997\)    45.8     \(1398\)   47.9     \(1454\)   49.7     \(1846\)

  3^rd^   47.3     \(1038\)   48.4     \(1362\)                                
  ------- -------- ---------- -------- ---------- -------- ---------- -------- ----------

For a second analysis of Reading First effectiveness over time, the NCE
metric was then coded into categories that mirrored those for the
five-component analysis---scores at less than the 20th percentile were
coded 1's, scores at the 20^th^ to 39^th^ percentile were coded 2's, and
scores greater than the 40th percentile were coded 3's. This metric
forms the *All Multiple Choice* scale contained in subsequent tables.

Findings for performance levels are similar to findings for the NCE
scores and are depicted in Figure 1. Significantly fewer 1^st^ grade
Reading First students scored *at or above grade level* in both 2004 and
2005. By the third and fourth years of implementation, significantly
more Reading First 1^st^ graders (56% and 57%, respectively) scored *at
or above grade level*.

Figure 1. Percentage of Kansas Reading First and Comparison Students

Scoring at Highest Level---*At Grade Level*---on the Stanford 10 by
Grade and Year

![](media/image1.wmf)

The percentage of Reading First 2^nd^ graders scoring *at or above grade
level* did not significantly differ in 2004 and 2005 (53%), but 55% in
2006 and 57% in 2007 scored *at or above grade level.* The percentage of
3^rd^ grade Reading First students scoring *at or above grade level*
increased from 52% in 2005 to 56% in 2006. A change to the State
Assessment evidenced higher percentages of students scoring at the
proficient or above level---64% in 2006 and 66% in 2007.

A third analysis of Reading First effectiveness over time is provided in
Figure 2. Percentages of 1^st^ grade students who scored at the lowest
level (*needs substantial intervention*) and the highest level (*at or
above grade leve*l) for four years are provided for SAT 10 All Multiple
Choice Items and for the important comprehension subscale. The
expectation is that the percentage of students scoring at the lowest
level will decrease and the percentage scoring at the highest level will
increase over time.

Figure 2. Changes in Kansas Reading First Student Performance in First
Grade at

Lowest Level---*Needs Substantial Intervention*---and Highest
Level---*At Grade Level*---

On SAT 10 All Multiple Choice Items and Comprehension Subscale

![](media/image2.wmf)

By the end of the third and fourth years of the grant, significantly
*fewer* 1^st^ graders scored at the *needs substantial intervention*
level than did so in the first year of the grant (21% and 22% compared
to 27%) on the SAT 10 All Multiple Choice performance levels.
Additionally, significantly *more* 1^st^ graders scored *at or above
grade level* in the third and fourth years of the grant than did in the
first year of the grant (56% and 57% compared to 51%). Thus,
expectations are met for Reading First 1^st^ graders.

This pattern of achievement was repeated for the important comprehension
component for 1^st^ graders. By the third and fourth years of Reading
First, 18% and 16% of 1^st^ graders scored at the *needs substantial
intervention* level compared to 24% after the first year. The percentage
of 1^st^ graders who scored *at or above grade level* in the third and
fourth years of the grant in comprehension (67% and 68%, respectively)
was substantially higher than after the first year (57%), an increase of
more than 10%.

A similar analysis was conducted for 2^nd^ graders over time and is
displayed in Figure 3. The pattern of Reading First effectiveness that
was demonstrated for 1^st^ graders does not recur for 2^nd^ graders. The
percentages of students scoring in the lowest and highest levels are
fairly static over time, particularly in the important comprehension
component.

Figure 3. Changes in Kansas Reading First Student Performance in Second
Grade at Lowest Level---*Needs Substantial Intervention*---and Highest
Level---*At Grade Level*

On SAT 10 Multiple Choice Items and Comprehension Subtest

![](media/image3.wmf)

This analysis was also repeated with a different instrument for
assessing Reading First effectiveness---the DIBELS measure of oral
fluency. The results are presented in Figure 4 and indicate similar
results of effectiveness for 1^st^ graders---the percentage of students
scoring at the lowest level significantly decreased and the percentage
scoring *at or above grade level* significantly increased for oral
fluency.

The percentage of 2^nd^ graders who scored at the lowest level on the
DIBELS oral fluency measure was not significantly reduced in the first
three years of the grant (ranging from 36 to 38%); the percentage was
slightly lower in 2007 (34%). However, the percentage of 2^nd^ graders
who scored at the highest level increased from approximately 40% in 2004
and 2005 to 44% in 2006 and 47% in 2007. The percentages of 3^rd^
graders scoring at the lowest level on the DIBELS oral fluency measure
did not vary much over time (about 30%). The percentage of students
scoring *at or above grade level* slightly increased to 40% in 2007.

Figure 4. Changes in Kansas Reading First Student Performance in Grades
1-3 at Lowest Level---*Needs Substantial* Intervention---and Highest
Level---*At Grade Level*---

DIBELS Oral Fluency

![](media/image4.wmf)A fourth analysis of Reading First effectiveness
compares same cohorts from 1^st^ to 2^nd^ grade since the inception of
Reading First. As demonstrated in Figure 5, somewhat more than 25% of
first graders in 2004-2006 were *in need of substantial intervention* at
the end of the 1^st^ grade and a similar percentage were in *need of
substantial intervention* at the end of 2^nd^ grade. This percentage
decreased to 21% for 1^st^ graders in 2006 and was maintained at the end
of the 2^nd^ grade.

**Figure 5. Percentage of Kansas Reading First Students Scoring at
Lowest Level--*Needs Substantial Intervention---* on SAT 10 All Multiple
Choice Items for 1^st^ to 2^nd^ Grade Cohorts**

![](media/image5.wmf)

Figure 6 provides same cohort percentages of students scoring *At Grade
Level* from end of 1^st^ to end of 2^nd^ grade. Only the cohort entering
school in the 2004-05 school year demonstrated meaningful increases in
the percentage of students scoring at or above grade level.

Figure 6. Percentage of Kansas Reading First Students Scoring at Highest
Level---*At Grade Level---*on SAT 10 All Multiple Choice Items for 1^st^
to 2^nd^ Grade Cohorts

![](media/image6.wmf)

**Student Achievement by** **Reading Component**

Reading First schools are required to assess student achievement in each
of five components---phonemic awareness, phonics, vocabulary,
comprehension, and oral fluency. Table 3 provides this data by grade
level for the spring 2006 and 2007 test administrations.

Slightly more than three-fourths of 1^st^ graders in both years scored
*at grade level* in phonemic awareness. First graders scored lowest on
the phonics component; approximately 35% scored *at grade level*. In
both years, 1^st^ graders scored higher on the comprehension component
than they did on the phonics, vocabulary or oral fluency components.
Two-thirds of Reading First 1^st^ graders scored *at grade level* on the
important comprehension component.

Virtually all 2^nd^ graders had mastered phonemic awareness, and this
component produced a somewhat misleading Total Score for the All
Multiple Choice measure. Approximately 40% of Reading First 2^nd^
graders scored *at or above grade level* in phonics, comprehension and
oral fluency in spring 2006, significantly fewer than the 55% who scored
*at or above grade level* on the All Multiple Choice total score.
Approximately 50% of 2^nd^ graders score *at or above grade level* in
vocabulary for both years. Second graders component percentages were
stable across both years except for the phonics component, where the
percentage of students who scored *at or above grade level* increased
from 41% in spring 2006 to 48% in 2007.

**Table 3. Percentage Distribution of** **Reading First Student
Performance Levels by Component in Spring 2006 and Spring 2007**

+-----------------+-----+-----+-----+------+-----+-----+-----+------+
|                 | ### |     |     |      | ##  |     |     |      |
|                 | ### |     |     |      |  Sp |     |     |      |
|                 |   S |     |     |      | rin |     |     |      |
|                 | pri |     |     |      | g 2 |     |     |      |
|                 | ng  |     |     |      | 007 |     |     |      |
|                 | 200 |     |     |      |     |     |     |      |
|                 | 6 { |     |     |      |     |     |     |      |
|                 | #sp |     |     |      |     |     |     |      |
|                 | rin |     |     |      |     |     |     |      |
|                 | g-2 |     |     |      |     |     |     |      |
|                 | 006 |     |     |      |     |     |     |      |
|                 | -1} |     |     |      |     |     |     |      |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
|                 | > * | > * | > * | **(  | > * | > * | > * | **(  |
|                 | *Ne | *Ne | *At | N)** | *Ne | *Ne | *At | N)** |
|                 | eds | eds | >   |      | eds | eds | >   |      |
|                 | >   | > A |  Gr |      | >   | > A |  Gr |      |
|                 |  Su | ddi | ade |      |  Su | ddi | ade |      |
|                 | bst | tio | > L |      | bst | tio | > L |      |
|                 | ant | nal | eve |      | ant | nal | eve |      |
|                 | ial | > I | l** |      | ial | > I | l** |      |
|                 | >   | nst |     |      | >   | nst |     |      |
|                 |  In | ruc |     |      |  In | ruc |     |      |
|                 | ter | tio |     |      | ter | tio |     |      |
|                 | ven | n** |     |      | ven | n** |     |      |
|                 | tio |     |     |      | tio |     |     |      |
|                 | n** |     |     |      | n** |     |     |      |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
|                 | *   |     |     |      | #   |     |     |      |
|                 | *Gr |     |     |      |   G |     |     |      |
|                 | ade |     |     |      | rad |     |     |      |
|                 | 1** |     |     |      | e 1 |     |     |      |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| All Multiple    | 21  | 23  | 56  | \(14 | 22  | 20  | 57  | \(19 |
| Choice          |     |     |     | 02\) |     |     |     | 57\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Phonemic        | 7   | 14  | 78  | \(14 | 8   | 16  | 76  | \(18 |
| Awareness       |     |     |     | 93\) |     |     |     | 60\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Phonics         | 28  | 37  | 35  | \(14 | 24  | 38  | 37  | \(18 |
|                 |     |     |     | 93\) |     |     |     | 60\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Vocabulary      | 25  | 19  | 56  | \(14 | 27  | 19  | 55  | \(18 |
|                 |     |     |     | 93\) |     |     |     | 60\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Comprehension   | 18  | 15  | 67  | \(14 | 16  | 16  | 68  | \(19 |
|                 |     |     |     | 93\) |     |     |     | 59\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Oral Fluency    | 18  | 27  | 55  | \(15 | 19  | 25  | 55  | \(19 |
|                 |     |     |     | 67\) |     |     |     | 90\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| ##              | ##  |     |     |      | ##  |     |     |      |
|                 |  Gr |     |     |      |  Gr |     |     |      |
|                 | ade |     |     |      | ade |     |     |      |
|                 |  2  |     |     |      |  2  |     |     |      |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| All Multiple    | 26  | 19  | 55  | \(13 | 24  | 20  | 57  | \(19 |
| Choice          |     |     |     | 67\) |     |     |     | 55\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Phonemic        | 2   | 5   | 93  | \(14 | 2   | 5   | 93  | \(18 |
| Awareness       |     |     |     | 47\) |     |     |     | 60\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Phonics         | 46  | 13  | 41  | \(14 | 40  | 12  | 48  | \(18 |
|                 |     |     |     | 47\) |     |     |     | 61\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Vocabulary      | 33  | 18  | 50  | \(14 | 32  | 19  | 49  | \(18 |
|                 |     |     |     | 47\) |     |     |     | 61\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Comprehension   | 37  | 23  | 40  | \(14 | 36  | 24  | 40  | \(19 |
|                 |     |     |     | 47\) |     |     |     | 56\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Oral Fluency    | 36  | 21  | 44  | \(14 | 34  | 19  | 47  | \(19 |
|                 |     |     |     | 35\) |     |     |     | 88\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
|                 | ##  |     |     |      | *   |     |     |      |
|                 |  Gr |     |     |      | *Gr |     |     |      |
|                 | ade |     |     |      | ade |     |     |      |
|                 |  3  |     |     |      | 3** |     |     |      |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+
| Oral Fluency    | 31  | 33  | 37  | \(13 | 29  | 31  | 40  | \(18 |
|                 |     |     |     | 78\) |     |     |     | 27\) |
+-----------------+-----+-----+-----+------+-----+-----+-----+------+

Since 3^rd^ graders took the State Assessment instead of the SAT 10 in
2006 and 2007, the only component on which they were additionally
assessed was oral fluency. In 2006, 37% and in 2007 40% of 3^rd^ graders
scored *at or above grade level* on oral fluency.

Student achievement on four components for 1^st^ to 2^nd^ grade cohorts
is displayed in Figures 7-10.

Figures 7 and 8 show that 1^st^ grade cohorts in each year demonstrated
considerable growth by the end of the 2^nd^ grade in phonemic awareness
and phonics components. Figure 9 shows that 1^st^ grade cohorts did not
change appreciably from 1^st^ to 2^nd^ grades in the vocabulary
component. Figure 10 demonstrates that the comprehension component as
measured in the 1^st^ grade significantly declined by the end of the
2^nd^ grade.

Figure 7. Percentage of Kansas Reading First Students Scoring *At Grade
Level*\
in Phonemic Awareness

![](media/image7.wmf)

**Figure 8. Percentage of Kansas Reading First Students Scoring *At
Grade Level***

**in Phonics**

![](media/image8.wmf)

**Figure 9. Percentage of Kansas Reading First Students Scoring *At
Grade ***![](media/image9.wmf)***Level***

**in Vocabulary**

**Figure 10.** **Percentage of Kansas Reading First Students Scoring *At
Grade Level***

![](media/image10.wmf)**in Comprehension**

**Achievement Results for Disaggregated Groups**

Student performance at the end of the fourth year of the Reading First
program was examined by characteristics of students and required
categories of disaggregation. These include gender, eligibility for free
or reduced price lunch, racial/ethnic minority status, Individualized
Education Plan (IEP) status, and English Language Learner (ELL) status.

**[Gender]{.underline}**

Consistent with prior research findings, where gender differences exist,
they are consistently in favor of higher performance for females (see
Figure 11).

**Figure 11. Percentage of Kansas Reading First Students Scoring *At or
Above Grade Level* on the SAT 10 All Multiple Choice Items and Third
Grade State Assessment**

![](media/image11.wmf)**by Grade, Year and Gender**

Significantly more females in both 1^st^ and 2^nd^ grades score *at or
above grade level* on the SAT 10 All Multiple Choice items each year of
Reading First implementation. On the State Reading Assessment, 3^rd^
grade females scored significantly higher than their male counterparts.
Sixty-eight percent of females and 64% of males scored *at or above
grade level* in spring 2007.

Table 4 provides achievement results for the five components by grade
and gender for Reading First students in spring 2006 and 2007. Except
for phonemic awareness, there were no statistically significant
differences between female and male 1^st^ graders in 2006. For the
cohort of 1^st^ graders in 2007, females scored significantly higher on
each component (p \< .001). Approximately 10% more females than males
scored *at or above grade level* on each component.

**Table 4. Percentage of Reading First Students Scoring *At Grade
Level***

by Component, Grade, and Gender[^2] in Spring 2006 and 2007

  --------------- ----------- ------- ----------- ------- ----------- ------- ----------- ------- ----------- -------- ----------- -------
                  **2006**                                                    **2007**                                             

                  **1^st^**           **2^nd^**           **3^rd^**           **1^st^**           **2^nd^**            **3^rd^**   

                  **M**       **F**   **M**       **F**   **M**       **F**   **M**       **F**   **M**       **F**    **M**       **F**

  All Multiple    53          58 ^ns^ 52          58^3^   61          69^4^   52          63^1^   54          59^4^    64          68 ^ns^
  Choice                                                                                                                           

  Phonemic        74          81^3^   91          94 ^ns^                     71          82^1^   92          94^ns^               
  Awareness                                                                                                                        

  Phonics         33          37 ^ns^ 38          46^3^                       35          41^1^   46          50 ^ns^              

  Vocabulary      57          54 ^ns^ 49          51^4^                       50          59^2^   49          50 ^ns^              
  Development                                                                                                                      

  Comprehension   64          69 ^ns^ 39          44^3^                       64          72^1^   37          43^3^                

  DIBELS Oral     53          57 ^ns^ 40          47^3^   32          40^3^   50          61^1^   44          50^3^    38          42 ^ns^
  Fluency                                                                                                                          
  --------------- ----------- ------- ----------- ------- ----------- ------- ----------- ------- ----------- -------- ----------- -------

Reading First 2^nd^ grade females scored significantly higher than males
in both 2006 and 2007 in comprehension and oral fluency. Approximately
6% more female than male 2^nd^ graders scored *at or above grade level*.

[Eligibility for Free/Reduced Price Lunch]{.underline}

Eligibility for free or reduced price lunch is frequently used as an
indicator for household poverty. Students who were ineligible for free
or reduced-price lunch in 2007 (fewer than 25% of the sample)
significantly and largely outscored students who are eligible at every
grade level on the SAT 10 All Multiple Choice score and the State
Assessment for 3^rd^ graders (see Figure 12). The achievement gaps are
large at each grade level, with an approximate 25% difference in the
percentage of students who score *at or above grade level*.
Approximately 50% of eligible 1^st^ and 2^nd^ graders and more than 70%
of ineligible 1^st^ and 2^nd^ graders in these high-poverty Reading
First schools scored *at or above grade level*.

Figure 12. Percentage of Kansas Reading First Students Scoring *At Grade
Level*

on SAT 10 All Multiple Choice Items and State Assessment by Eligibility
for Free/Reduced-Price Lunch in Spring 2007

![](media/image12.wmf)

Figure 13 provides the percentage of students scoring *at or above grade
level* by grade and year for students eligible for free or reduced price
lunch. Fewer than half of eligible 1^st^ and 2^nd^ graders scored *at or
above grade level* in 2005 but 52% did so in 2006 and 2007. Caution must
be exercised in interpreting the increase for eligible 3^rd^ graders
from 52% in 2005 to 63% in 2007 because of the change in test instrument
initiated in 2006.

**Figure 13. Percentage of Kansas Reading First Students Scoring *At
Grade Level***

**on SAT 10 All Multiple Choice and Third Grade State Assessment**

![](media/image13.wmf)**by Grade and Year for Students Eligible for
Free/Reduced-Price Lunch**

Table 5 provides achievement results for the five components by
eligibility for free/reduced price lunch in spring 2006 and 2007. In
both years, students who were eligible for free/reduced price lunch
scored significantly lower than ineligible students on every component.
Differences are quite large. For the vocabulary, comprehension, and
reading fluency components in 2007, the percentage of students who
scored *at or above grade level* was at least 20% lower for students who
were eligible for free/reduced price lunch.

Table 5. Percentage of Reading First and Comparison Students Scoring *At
Grade Level*

by Component, Grade, and Eligibility for Free/Reduced-Price Lunch

Spring 2006 and 2007

  --------------- ----------- -------- ----------- -------- ----------- -------- ----------- -------- ----------- -------- ----------- --------
                  **Spring                                                       **Spring                                              
                  2006**                                                         2007**                                                

                  **1^st^**            **2^nd^**            **3^rd^**            **1^st^**            **2^nd^**            **3^rd^**   

                  **Yes**     **No**   **Yes**     **No**   **Yes**     **No**   **Yes**     **No**   **Yes**     **No**   **Yes**     **No**

  All Multiple    51          72^1^    50          74^1^    60          80^1^    52          79^1^    52          74^1^    63          77^1^
  Choice                                                                                                                               

  Phonemic        74          91^1^    91          97^1^                         74          86^1^    92          96^4^                
  Awareness                                                                                                                            

  Phonics         30          54^1^    38          54^1^                         34          52^1^    45          61^1^                

  Vocabulary      50          74^1^    44          72^1^                         49          76^1^    44          70^1^                
  Development                                                                                                                          

  Comprehension   63          81^1^    36          59^1^                         64          84^1^    35          59^1^                

  DIBELS Oral     51          67^1^    39          59^1^    31          51^1^    51          73^1^    43          63^1^    35          57^1^
  Fluency                                                                                                                              
  --------------- ----------- -------- ----------- -------- ----------- -------- ----------- -------- ----------- -------- ----------- --------

**Racial/Ethnic Status**

Non-minority Reading First students significantly and largely outscored
minority students at every grade level on the SAT 10 All Multiple Choice
and State Reading Assessment (see Figure 14). The achievement gaps are
large at each grade level, with a 18-24% difference in the percentage of
students who scored *at or above grade level*.

**Figure 14. Percentage of Kansas Reading First Students Scoring *At
Grade Level* on SAT 10 All Multiple Choice Items and Third Grade State
Assessment by Minority Status**

![](media/image14.wmf)Table 6 provides achievement results for the five
components by minority status for spring 2006 and 2007. In both years,
non-minority Reading First students scored significantly and
meaningfully higher than minority students on each component---phonics,
phonemic awareness, vocabulary, comprehension, and reading fluency. The
greatest discrepancy for minority 1^st^ graders compared with
non-minority 1^st^ graders was on the vocabulary component. Second grade
minority students also widely differed from their non-minority
counterparts on the vocabulary and comprehension components.

Table 6. Percentage of Reading First and Comparison Students Scoring at
Grade Level

by Component, Grade Level, and Minority Status in Spring 2006 and 2007

  --------------- ----------- --------- ----------- --------- ----------- --------- ----------- --------- ----------- --------- ----------- ---------
                  **Spring                                                          **Spring                                                
                  2006**                                                            2007**                                                  

                  **1^st^**             **2^nd^**             **3^rd^**             **1^st^**             **2^nd^**             **3^rd^**   

                  **Min**     **Non**   **Min**     **Non**   **Min**     **Non**   **Min**     **Non**   **Min**     **Non**   **Min**     **Non**

  All SAT 10      46          66^1^     45          66^1^     57          73^1^     51          69^1^     47          71^1^     58          79^1^
  Multiple Choice                                                                                                                           

  Phonemic        71          85^1^     91          94^4^                           72          83^1^     91          96^1^                 
  Awareness                                                                                                                                 

  Phonics         28          43^1^     36          47^1^                           33          44^1^     41          58^1^                 

  Vocabulary      43          69^1^     38          63^1^                           46          68^1^     38          67^1^                 
  Development                                                                                                                               

  Comprehension   58          76^1^     31          52^1^                           63          76^1^     31          53^1^                 

  DIBELS Oral     48          62^1^     38          49^1^     29          43        50          64^1^     42          54^1^     35          49^1^
  Fluency                                                                                                                                   
  --------------- ----------- --------- ----------- --------- ----------- --------- ----------- --------- ----------- --------- ----------- ---------

[Special Education Status]{.underline}

As would be expected, students with Individual Education Plans (IEPs)
for special education scored much lower on both the SAT 10 All Multiple
Choice measure and the State Reading Assessment at 3^rd^ grade (see
Figure 15).

![](media/image15.wmf)Figure 15. Percentage of Reading First Students
Scoring *At Grade Level* on the

SAT 10 All Multiple Choice Items and State Reading Assessment

for Special Education Students

The percentage of 1^st^ grade students with IEPs who scored *at or above
grade* *level* varied widely for the three years, increasing from 29% in
2005 to 38% in 2006 and back to 29% in 2007. A similar pattern was
demonstrated for 2^nd^ graders with IEPs. A more consistent pattern of
increases in the percentage of students scoring *at or above grade*
*level* was demonstrated for 3^rd^ graders with IEPs, with 34% in 2005,
43% in 2006, and 57% in 2007 scoring *at or above grade* *level.*
(Recall, however, that 2005 results are for the SAT 10 and 2006 and 2007
results are for the State Reading Assessment).

Table 7 provides achievement results for the five components for Reading
First students with IEPs in spring 2006 and 2007. The percentage of
students with IEPs who scored *at or above grade level* for each
component in 2007 was similar to the 2006 percentages for both 1^st^ and
2^nd^ graders in phonemic awareness, phonics, and oral fluency. The 2007
percentages were lower for 1^st^ graders for the vocabulary component
and for 2^nd^ graders in comprehension. Significantly more 3^rd^ graders
with IEPs scored higher on the State Reading Assessment in 2007, but
significantly fewer scored *at or above grade level* on oral fluency.

Table 7. Percentage of Reading First Students Scoring *At Grade Level*
by Component and Grade for Students with IEPs and English Language
Learners

Spring 2006 and 2007

<table>
<colgroup>
<col style="width: 22%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 5%" />
<col style="width: 6%" />
<col style="width: 5%" />
<col style="width: 6%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 5%" />
<col style="width: 7%" />
</colgroup>
<tbody>
<tr class="odd">
<td></td>
<td colspan="6"><strong>Students with IEPs</strong></td>
<td colspan="6"><h1 id="english-language-learners">English Language
Learners</h1></td>
</tr>
<tr class="even">
<td></td>
<td colspan="2"><strong>1<sup>st</sup></strong></td>
<td colspan="2"><strong>2<sup>nd</sup></strong></td>
<td colspan="2"><strong>3<sup>rd</sup></strong></td>
<td colspan="2"><strong>1<sup>st</sup></strong></td>
<td colspan="2"><strong>2<sup>nd</sup></strong></td>
<td colspan="2"><strong>3<sup>rd</sup></strong></td>
</tr>
<tr class="odd">
<td></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
<td><strong>06</strong></td>
<td><strong>07</strong></td>
</tr>
<tr class="even">
<td>All Multiple Choice</td>
<td>38</td>
<td>29</td>
<td>31</td>
<td>24</td>
<td>43</td>
<td>57</td>
<td>38</td>
<td>46</td>
<td>38</td>
<td>43</td>
<td>48</td>
<td>53</td>
</tr>
<tr class="odd">
<td>Phonemic Awareness</td>
<td>61</td>
<td>56</td>
<td>76</td>
<td>77</td>
<td></td>
<td></td>
<td>68</td>
<td>66</td>
<td>90</td>
<td>91</td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>Phonics</td>
<td>27</td>
<td>21</td>
<td>19</td>
<td>24</td>
<td></td>
<td></td>
<td>32</td>
<td>33</td>
<td>21</td>
<td>41</td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td>Vocabulary Development</td>
<td>35</td>
<td>26</td>
<td>27</td>
<td>24</td>
<td></td>
<td></td>
<td>35</td>
<td>35</td>
<td>28</td>
<td>31</td>
<td></td>
<td></td>
</tr>
<tr class="even">
<td>Comprehension</td>
<td>44</td>
<td>44</td>
<td>25</td>
<td>13</td>
<td></td>
<td></td>
<td>52</td>
<td>63</td>
<td>26</td>
<td>25</td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td>DIBELS Oral Fluency</td>
<td>35</td>
<td>30</td>
<td>22</td>
<td>23</td>
<td>23</td>
<td>11</td>
<td>42</td>
<td>48</td>
<td>35</td>
<td>43</td>
<td>24</td>
<td>30</td>
</tr>
</tbody>
</table>

**[English Language Learner Status]{.underline}**

More than one-third of 1^st^ and 2^nd^ grade students identified as
English Language Learners (ELL) with limited English proficiency scored
*at or above grade level* in 2005 and 2006 (see Figure 16). In 2007, 46%
of 1^st^ grade ELL students and 43% of 2^nd^ grade ELL students scored
*at or above grade level*. Almost half of 3^rd^ grade ELL students
scored *at or above grade level* on the State Assessment in 2006 and
more than half did so in 2007.

Figure 16. Percentage of Kansas Reading First English Language Learners
Students

![](media/image16.wmf)Scoring at Grade Level on Total SAT 10 Reading
Score and Third Grade State Assessment

Table 7 provides achievement results for the five components for Reading
First ELL students in spring 2006 and 2007. The percentage of ELL
students who scored *at or above grade level* in phonemic awareness was
similar for 1^st^ graders in both years at approximately 66% and for
2^nd^ graders in both years at approximately 90%. One-third of 1^st^
grade ELL students scored *at or above grade level* in phonics in both
years. Twenty-one percent of 2^nd^ grade ELL students scored *at or
above grade level* on the phonics component in 2006, but almost twice as
many (41%) did so in 2007. Approximately one-third of 1^st^ and 2^nd^
grade ELL students scored *at or above grade level* in vocabulary
development in both years. First grade ELL students scored well on the
comprehension component, with 52% in 2006 and 63% in 2007 scoring *at or
above grade level.*

The generally lower comprehension scores for 2^nd^ graders was also
evident in the results for ELL students. Only 25% of 2^nd^ grade ELL
students scored *at or above grade level* in 2006 and 2007.

Oral fluency scores increased from 2006 to 2007 at each grade level.

**Highest Performing Districts and Schools: Unmatched Grade Level
Cohorts**

A final Reading First national reporting requirement is to identify the
highest performing Reading First districts and schools for the 2006-07
school year. The SAT 10 All Multiple Choice NCE score was used for
purposes of identifying the highest performing Reading First districts
and schools using annual data by grade level.  The NCE metric was coded
into categories that mirrored those for the five component
analysis---students scoring at less than the 20th percentile were coded
1s, the 20^th^ or greater but less than 40^th^ percentile were coded 2s,
and greater than the 40th percentile were coded 3s. Fisher Exact
non-parametric tests of significance (testing the hypothesis that no
change occurred) were conducted for students who were in 1st grade in
spring 2006 and students who were in 2nd grade in spring 2007.[^3] These
tests were conducted to identify the largest significant gains between
2006 and 2007 for this Reading First cohort in each district and then
for buildings within districts. A second method of identifying highest
performing schools was to examine consistency in a high percentage of
students scoring at grade level for cohorts over time.[^4]

When data was aggregated to the district level, two districts---Garden
City and Pittsburg---demonstrated statistically significant positive
increases for the 1^st^ to 2^nd^ grade cohort from 2006 to 2007 (gains
are significant at p \<.001). This is a decrease from the prior year
when four districts--- Arkansas City, Parsons, Topeka, and Wichita---had
demonstrated significant district-level increases.

In general, significant changes occurred primarily at the building
level. The district-level increase for the Garden City School District
was largely due to performance at Victor Ornelas Elementary, which
demonstrated an increase from 31% to 53% of the cohort scoring on grade
level in 2007. Both Reading First schools in the Pittsburg School
District demonstrated significant increases for the 1^st^ to 2^nd^ grade
cohort. At George Nettels Elementary, 53% of 1^st^ graders scored at
grade level in 2006 and 78% of 2^nd^ graders scored at grade level in
2007. For Westside Elementary, the cohort increased from 53% at grade
level in 2006 to 83% in 2007

Additional buildings in the Arkansas City, and Topeka demonstrated high
performance. Jefferson Elementary in the Arkansas City School District
had demonstrated significant increases in past years. While the 2006-07
comparison did not yield a statistically significant increase from 1^st^
to 2^nd^ grade, the school was successful in maintaining a very high
percentage (90%) of 2^nd^ graders who scored at or above grade level in
2007. Similarly, Garfield Elementary in the Parsons School District had
demonstrated significant increases in the past and maintained those high
percentages of students scoring at or above grade level in 2007. In
2006, 69% of 1^st^ graders at Garfield Elementary scored at grade level
and in 2007 84% of 2^nd^ graders scored at or above grade level.

None of the schools In the Topeka school district demonstrated
significant increases in 2006-07. However, Linn Elementary demonstrated
significant increases in prior years and in 2007, 96% of 2^nd^ graders
scored at or above grade level. (Over the past four years, end of 1^st^
grade performance has differed widely at this school, ranging from about
40% to about 70% scoring at or above grade level, indicating a lack of
consistency of effects for 1^st^ graders).

**Matched Achievement Results---Grade 1 to Grade 2**

Reading First national reporting for evaluation and accountability is
based on annual grade level cohorts. The advantage of this methodology
is that all students are represented in the annual data reporting. The
disadvantage is that high poverty schools often experience high
mobility, thereby rendering annual comparisons by grade level less valid
for purposes of identifying effectiveness.

School year 2004 provided the first opportunity for then 1^st^ graders
to participate in the Reading First program. The achievement results for
these 1^st^ graders was matched to their achievement in the subsequent
year as 2^nd^ graders; approximately 55% of students who had been tested
as 1^st^ graders were matched to the 2^nd^ grade test data.[^5]
Similarly, Reading First 1^st^ graders in 2005 were matched to 2^nd^
grade results in 2006; almost 60% of students who had been tested as
1^st^ graders in 2005 were matched to the 2^nd^ grade test data in 2006.
Finally Reading First 1^st^ graders in 2006 were matched to the 2^nd^
grade test data in 2007; 74% of students who had been tested as 1^st^
graders in 2006 were matched to the 2^nd^ grade test data in 2007.
Figure 17 provides the results for three matched cohorts as they
progressed from 1^st^ to 2^nd^ grade in terms of the percentages who
scored at the lowest level---*needs substantial intervention*---and the
percentage who scored at the highest level---*at or above grade level*.

Figure 17. First-Second Grade Matched Comparisons of Percentage of
Reading First Students Scoring at Lowest Level---*Needs Substantial
Intervention*---and Highest Level---*At Grade Level* on SAT 10 All
Multiple Choice Items

![](media/image17.wmf)

The expectation is that the percentage of students scoring at the lowest
level declines and the percentage scoring at the highest level
increases. In the 2004-05 early implementation years, the percentage of
students scoring at the lowest level did not decrease and the percentage
scoring at the highest level showed a modest increase from 54 to 58%.
For the second matched cohort in 2005-06, the percentage of students
scoring at the lowest level showed a modest decrease from 29 to 26% and
the percentage at the highest level increased from 51 to 56%. For the
2006-07 cohort, the percentage of students in the lowest category did
not decrease from 1^st^ to 2^nd^ grade, remaining at 20%, but this
percentage was lower than the percentages for the prior two cohorts.

The 59% of students scoring in the highest category as 1^st^ graders in
2007 was larger than in prior years and increased to 61% in the 2^nd^
grade.

Figure 18 provides the percentages of 1^st^ graders in 2006 that
increased, decreased, or maintained their level at the end of 2^nd^
grade. Of the 59% of 1^st^ graders who scored *at or above grade level*
(Level 3) in 2006, 48% maintained their Level 3 performance at the end
of 2^nd^ grade. Fifteen percent of 1^st^ graders decreased their
performance at the end of 2^nd^ grade. Of the 20% of 1^st^ graders who
scored at Level 1 in 2006, 13% maintained that level of performance and
6% increased to Level 2 at the end of 2^nd^ grade. Of the 21% of 1^st^
graders who scored at Level 2 in 2006, 7% maintained that level of
performance, 4% decreased their performance, and 10% increased to Level
3 (*at or above grade level*).

The matched data indicate that evaluation and accountability models that
focus on only three categories of performance allow little opportunity
to demonstrate change over time. Almost one-half of Reading First
students perform at the ceiling (level 3) in 1^st^ grade and remain
there. Thus, models that focus on change alone are insufficient for
demonstrating the effectiveness of the Reading First program.

Figure 18. Longitudinal Grade 1 (2006) and Grade 2 (2007) Performance

![](media/image18.wmf)

# School Level Results

In a final effort to examine evidence for the effectiveness of the
Reading First program, matched 1^st^ and 2^nd^ grade performance on the
SAT 10 All Multiple Choice contained in Figure 17 above are provided for
each Reading First building in Table 8 for the years that schools
participated in Reading First. The expectation is that the percentage of
students scoring at the lowest level declines and the percentage scoring
at the highest level increases.

Table 8. First-Second Grade Matched Comparisons of Percentage of Reading
First Students Scoring at Lowest Level---*Needs Substantial
Intervention*---and Highest Level---*At Grade Level* on SAT 10 All
Multiple Choice Items by District and School

+-------------------------+--------+--------+--------+--------+--------+
|                         | **%    | **%    | **%    | **%    | **N**  |
|                         | Level  | Level  | Level  | Level  |        |
|                         | 1**    | 1**    | 3**    | 3**    |        |
|                         |        |        |        |        |        |
|                         | **     | **     | **     | **     |        |
|                         | 2006** | 2007** | 2006** | 2007** |        |
+=========================+========+========+========+========+========+
| **Arkansas City:**      | 10     | 10     | 76     | 72     | 29     |
| Adams                   |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Arkansas City:**      | 9      | 0      | 82     | 94     | 34     |
| Jefferson               |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Derby:** Paul B.      | 15     | 18     | 52     | 42     | 33     |
| Cooper                  |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Derby:** Oaklawn      | 4      | 28     | 52     | 40     | 25     |
+-------------------------+--------+--------+--------+--------+--------+
| **Emporia:** W. A.      | 27     | 21     | 58     | 64     | 33     |
| White                   |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Garden City:**Buffalo | 56     | 36     | 20     | 38     | 50     |
| Jones                   |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Garden City:**Victor  | 28     | 20     | 39     | 56     | 75     |
| Ornelas                 |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Garden City:**        | 27     | 23     | 54     | 46     | 48     |
| Garfield                |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Kansas City:**        | 37     | 30     | 27     | 23     | 28     |
| Caruthers               |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Kansas City:** Grant  | 41     | 38     | 38     | 47     | 34     |
+-------------------------+--------+--------+--------+--------+--------+
| **Lawrence:** Kennedy   | 21     | 15     | 54     | 67     | 39     |
+-------------------------+--------+--------+--------+--------+--------+
| **Lawrence:** New York  | 0      | 13     | 100    | 75     | 16     |
+-------------------------+--------+--------+--------+--------+--------+
| **Lawrence:** Prairie   | 9      | 11     | 77     | 73     | 44     |
| Park                    |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Lawrence:** Woodlawn  | 21     | 18     | 64     | 68     | 28     |
+-------------------------+--------+--------+--------+--------+--------+
| **Parsons:** Garfield   | 3      | 3      | 72     | 90     | 29     |
+-------------------------+--------+--------+--------+--------+--------+
| **Parsons:** Guthridge  | 12     | 15     | 79     | 67     | 33     |
+-------------------------+--------+--------+--------+--------+--------+
| **Parsons:** Lincoln    | 0      | 0      | 77     | 81     | 31     |
+-------------------------+--------+--------+--------+--------+--------+
| **Pittsburg:**George    | 41     | 12     | 54     | 78     | 41     |
| Nettels                 |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Pittsburg:**Westside  | 44     | 0      | 56     | 81     | 36     |
+-------------------------+--------+--------+--------+--------+--------+
| **Salina:** Oakdale     | 9      | 5      | 53     | 72     | 57     |
+-------------------------+--------+--------+--------+--------+--------+
| **Salina:** Sunset      | 2      | 5      | 82     | 82     | 55     |
+-------------------------+--------+--------+--------+--------+--------+
| **Shawnee Mission:**    | 22     | 22     | 60     | 67     | 45     |
| Comanche                |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Shawnee Mission:**    | 4      | 9      | 91     | 91     | 23     |
| South Park              |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Avondale    | 40     | 47     | 33     | 33     | 15     |
| East                    |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Avondale    | 23     | 23     | 53     | 50     | 30     |
| West                    |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Linn        | 12     | 0      | 76     | 94     | 17     |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Lundgren    | 19     | 27     | 54     | 54     | 26     |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Quincy      | 39     | 35     | 43     | 39     | 23     |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Quinton     | 36     | 45     | 50     | 45     | 22     |
| Heights                 |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Scott       |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Topeka:** Highland    | 21     | 31     | 56     | 59     | 39     |
| Park Central            |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Wichita:** Cessna     | 18     | 24     | 58     | 61     | 38     |
+-------------------------+--------+--------+--------+--------+--------+
| **Wichita:** Franklin   | 0      | 11     | 74     | 71     | 35     |
+-------------------------+--------+--------+--------+--------+--------+
| **Wichita:** Gardiner   | 27     | 37     | 51     | 45     | 49     |
+-------------------------+--------+--------+--------+--------+--------+
| **Wichita:** Pleasant   | 31     | 20     | 47     | 44     | 45     |
| Valley                  |        |        |        |        |        |
+-------------------------+--------+--------+--------+--------+--------+
| **Wichita:** Clark      | 54     | 46     | 25     | 21     | 28     |
+-------------------------+--------+--------+--------+--------+--------+

In order to determine whether the changes in Level 1 or Level 3 are
statistically significant for the matched 1^st^ to 2^nd^ grade cohort,
the Fisher's Exact Test was used and is provided in Table 9. Three
cohorts of 1^st^ to 2^nd^ grade tests of significance---2004 to 2005,
2005 to 2006, and 2006 to 2007---are provided. Tests of significance for
3 x 3 data (i.e., 3 levels for each year) are not always valid when the
cell size is less than 5. This frequently happens when schools have
large percentages of their students *at or above grade level*, a desired
outcome. Table 9 also contains a t-test of significance conducted on
average NCE scores for 1^st^ to 2^nd^ grade performance in 2006 to 2007.
Both methods for examining effectiveness at the building level pose
advantages and disadvantages. The Exact Test is limited by frequency
distributions at the building level that often do not yield sufficient
Ns in many cells. Alternatively, using the average as the basis for
assessing improved or declining performance yields little information on
whether gains/losses represent the lowest performing students.

Five schools (14%) demonstrated consistent significant improvements for
both types of significance testing---Buffalo Jones and Victor Ornelas in
Garden City, Kennedy in Lawrence, Oakdale in Salina, and Linn in Topeka.
Six schools (17%) consistently demonstrated no significant changes.
Twenty-five percent of schools demonstrated either a significant decline
or non-significant changes when two tests of significance are
considered. Only one school demonstrated consistent declining
performance. For 42% of Reading First schools, one test indicated
significant improvement and the other indicated non-significant changes.

Examining patterns of effectiveness across three years of matched 1^st^
to 2^nd^ grade performance in Table 9 reveals much variation in
conclusions that can be drawn about Reading First effectiveness or
ineffectiveness at the building level.

[^1]: NCE scores range from 1 to 99 and match percentile scores at the
    1^st^, 50^th^, and 99^th^ NCE.

[^2]: ^\*^ 1 = \< .0001 2 = \< .001 3 = \< .01 4 = \< .05 ns = not
    significant

[^3]: Because all third graders were tested on the new State Reading
    Assessment starting in spring 2006, this test replaced the SAT 10
    for evaluation and accountability purposes. Significance testing was
    not conducted on these different measures for the 2005 2^nd^ grade
    and 2006 3^rd^ grade cohort and for cohorts in subsequent years.

[^4]: When high percentages of students score at one level, the numbers
    of students remaining in other cells is often less than 5. Such
    small numbers in cells indicate that tests of significance may not
    be appropriate for identifying high performance.

[^5]: The lower-than-desired match rate is due to a number of factors
    aside from mobility and include inconsistent student identification
    numbers used in the early years of the grant. By 2006 and 2007, the
    statewide student identification numbering system was in effect and
    the percentage of matched data improved.