**New York State Growth Model Pilot:**

**Additional Information Request**

**Principle 1: Universal Proficiency-Confidence Interval**

*Please provide further justification for the use of a confidence
interval in conjunction with a growth model.*

New York proposes to use a confidence interval in conjunction with its
integrated model (status and growth data combined in the school
Performance Index). The rationale for this is based on two main
points: 1) confidence intervals are appropriate means for reducing
misclassification of schools as not having made AYP when in fact they
may have, due to sampling error; and 2) the proposed integrated
"proficiency plus" design avoids some of the possible misclassification
of labeling schools as having met AYP when in fact they have not that
might occur if "additional ways to meet AYP" were added, such as
providing a separate growth decision in addition to status.

That confidence intervals are appropriate for controlling possible
misclassification due to sampling error of schools as not having made
AYP when in fact they may have was discussed in the main proposal. A
citation for this view is provided in *Generalizability Analysis for
Performance Assessments of Student Achievement or School Effectiveness,*
where Cronbach, Linn, Brennan, & Haertel (1997) argue for the use of a
confidence interval for accountability systems stating:

> Restricting inference to the historical statement would defeat the
> purpose of many school-accountability uses of assessment results,
> where inferences reach beyond students recently taught. Note, for
> example, that analysis of one year\'s data, developing the SE for the
> finite student body, cannot support such actions as rewarding a school
> for satisfactory performance, or imposing sanctions on a school that
> did poorly; the finite-population analysis provides no basis for
> assessing the uncertainty in the school mean that arises from random
> variation in student intake. Nor would the analysis support the
> inference that the program in school A is better than that in school
> B, even if they draw on equivalent populations. Nor, thirdly, does the
> finite-population SE provide a basis for arguing that a school mean
> higher in Year 2 than in Year 1 implies improved instruction, rather
> than a fortuitously superior intake of pupils. (p. 21)

New York is not applying a separate confidence interval to a decision
based on growth separate from status, as most states have proposed.
Under New York's "proficiency plus" proposal, the same confidence
interval (.90) that is applied to AYP status determinations is now
applied to the determination of whether students are proficient or on
track towards proficiency. According to Peer Review Guidance, "the
justification for employing confidence intervals around the AYP status
target is based largely on reducing the impact of score volatility due
to changes in the cohorts being assessed from one year to another, and
thus reducing the potential for inappropriately concluding that the
effectiveness of the school is improving or declining." Under New York's
"proficiency plus" model, this remains the rationale for use of a
confidence interval. Unlike models where status and growth are
calculated separately and therefore only matched scores are within the
growth model, New York's "proficiency plus" model includes the
performance of **all** students in the current year with valid scores,
regardless of whether a matched score for that student is available to
measure growth. We believe that our "proficiency plus" model results in
a more reliable judgment and avoids the weaknesses involving match rates
that are of concern when status and growth are calculated separately. It
is true that repeated ways to "meet AYP"---such as through status,
status with confidence interval, safe harbor, safe harbor with
confidence interval, growth, and growth with confidence interval---would
be subject to cumulative possible misclassification error of saying that
a school did meet AYP when in fact it did not. Many people have pointed
out that this cumulative error over multiple ways to meet a criterion is
a characteristic of the conjunctive rule of the many ways a school may
*fail to meet* AYP, but that is not the issue here. New York **does
not** propose a separate way to meet AYP through a growth decision, but
rather proposes having only **one** decision.

Lastly, contrary to the fears of some when the Growth Model Pilot was
first proposed by USED, that using growth data would "let many schools
off the hook" and water-down NCLB accountability provisions, the
modeling for New York indicates that, like almost every state approved
thus far, the impact on schools making AYP will be modest to negligible.

While New York believes that its proposal has several advantages, we are
willing to consider modifying the use of student growth data to create a
separate growth decision for whether a school meets AYP, without the use
of a confidence interval, to be used in conjunction with its existing
status determination, which includes a confidence interval.

**Principle 1: Universal Proficiency-Five Year Proficiency Projection**

*Please provide a rationale for why limited English proficient students,
students with disabilities, and "students who enter high school far
below standards" are given an extra year to make adequate growth.*

In its Final Title I rulemaking, the USED permits states to propose to
the Secretary of Education how it will make AYP determinations using an
extended-year rate that gives credit for students who take longer than
four years to graduate with a regular diploma. At present, New York is
transitioning from a system in which students who score between 55-64 on
a Regents exam may graduate with a regular diploma to one in which
students must achieve a score of at least 65 in order to be able to meet
graduation requirements. Currently, a score of 55-64 represents the
achievement of "basic proficiency" and a score of 65 or higher
represents the achievement of "proficiency."

Under growth model, certain limited English proficient students,
students with disabilities, and students who enter high school far below
standards who meet the current standards for graduation with a regular
diploma within four years would be considered "on track" towards meeting
the proficiency standard (65 or higher) using a five year extended
cohort rate. New York's rationale is the same one applied by the USED in
adoption of its final regulations: there are groups of students for whom
it is appropriate to give schools and districts additional credit
towards making AYP if such students are able to achieve State standards
within an extended period of time.

**Principle 4: Inclusion of All Students-Alternate Assessment**

*Please provide information related to how students who take the
alternate assessment based on alternate academic achievement standards
are included in the performance index and growth model calculations.*

As described in the main proposal, New York proposes to include students
who take the alternate assessment based on alternate academic
achievement standards through the status portion, but not the growth
portion of its proposed integrated system. The New York State Alternate
Assessment (NYSAA) was developed to meet both requirements of the
Individuals with Disabilities education Act (IDEA 1997) and the No Child
Left Behind Act (NCLB). The NYSAA was developed to meet the requirements
of these federal mandates. The NYSSA measures the achievement of
students with severe cognitive disabilities relative to NYS Learning
Standards using alternate achievement levels based on a datafolio
approach. The content area matter assessed by NYSAA is linked to grade
level content, though the content is reduced in scope and complexity.
The NYSAA does not lend itself to the transformed scale score
methodology that New York uses for other students in the Grades 3-8
English Language Arts and Mathematics Testing Program. As per USED
requirements, students who take NYSAA are factored into the State's
accountability system. Like the Grades 3-8 English Language Arts,
Mathematics and Science and the high school Regents Testing programs,
the State has four specific performance levels for students taking this
assessment. NYSAA performance levels are counted the same as general
assessment performance levels when determining Performance Indices (PI)
for English, mathematics, and science. NCLB regulations allow a maximum
of one percent of scores in calculating the PI for each accountability
measure for a district to be based on proficient and advanced proficient
scores on the NYSAA.

New York has a fully approved State Accountability system; in section
2.1 of the *Accountability Peer Review Workbook* (Revised August 1,
2008), New York clearly articulates the state statutes and policies that
ensure that students who take the Alternate Assessment are included on
accountability indicators. No student is excluded from the
accountability system regardless of their classification.

*Please provide greater detail regarding whether and how all students
including those currently proficient, are included in the growth model.*

**All** students are included in the State's proficiency plus model. The
Department will be making AYP growth determinations for non-proficient
students only in the proposed growth model. If a student were to be
proficient in one academic year and in the following year they regress
and are not proficient, this change in their status for accountability
is taken into account by the State's Performance Indexing system so the
school and district will be held accountable for the roll back in their
performance.

The Board of Regents has directed the State Education Department (SED)
to develop a "growth for all" methodology that will create consequences
for schools and districts based upon the growth of all students,
including those who are proficient. SED is currently researching
approaches to include all students in the growth model for purposes of
identifying High Performing/Rapidly Improving schools, but these
determinations are not related to AYP status as specified in the
Department's Accountability Workbook. Rather the "growth for all" model
will inform the methodology by which high performing and rapidly
improving schools and districts, as described in Section 1.6 of approved
*Accountability Peer Review Workbook* (Revised August 1, 2008), are
identified and the types of rewards that such schools and districts may
receive as a consequence of such recognition.

**Principle 5: State Assessment System & Methodology-Reporting**

*Please expand on how the results of the growth model will be reported
to parents and the public at large. If possible, please provide report
card templates.*

The State Education Department intends to post to its website "growth
tables" that would allow any interested party to enter the assessment,
the grade, the length of time a student has been enrolled in a school or
district, and the student's prior year scale score and receive the scale
score that the student must have received in the current year in order
for the school or district to receive credit for the student being on
track for proficiency. Each school and district report card will provide
a link to this website. State report cards will be revised to show the
percentage of students by grade and by disaggregated group for which a
school or district has been credited with being on track towards
proficiency (See Appendix A: Prospective Report Card Template). The
Department provides Individual Student Reports to parents each year;
these reports show a student's progression through the Grades 3-8
Testing Program so parents are able to view how their children are
performing, if they are meeting State Learning Standards and, and the
distance they need to improve to meet State Learning Standards if they
are not proficient.

**Principle 5: State Assessment System & Methodology-High School**

*Please clarify how high schools are included in the growth model,
including examples, as necessary. What is sufficient growth for high
school students?*

*How does the model ensure that all students will actually reach
proficiency?*

New York proposes employing a value table methodology in its high school
growth model. New York's currently approved high school status model is
based upon a Performance Index in which schools and district receive
"partial credit" for students whose highest score within four years of
first entry into grade 9 is 55-64 and "full credit" for students who
score 65 or higher within four years of first entry into grade 9. Under
New York's high school growth proposal, a value table will be utilized
under which all of New York's currently approved methodology for making
high school determinations will remain the same except for the following
two modifications:

-   Students who score at Level 1 or low-Level 2 on the Grade 8 ELA or
    math exam and who score at, or above, 55 within three years of first
    entry into grade 9 will be considered "on track towards" proficiency
    and a school or district will receive "full credit" for the
    student's performance.

-   Certain students with disabilities, late-arriving English language
    learners, and students who score at low Level 1 on the Grade 8 ELA
    or math exam who score at or above 55 within four years of first
    entry into grade 9 will be considered "on track towards" proficiency
    on an extended high school cohort and a school or district will
    receive "full credit" for the student's performance.

All other elements of New York's high school Performance Index will
remain unchanged. New York will first have sufficient data to implement
the proposed high school growth model in 2008, when it will be possible
to match students who take a high school Regents exam with their scores
from the New York state assessments in English language arts and
mathematics taken in grade 8 in spring of 2006.

Because the data were not yet available, no modeling was done on the
proposed high school growth model. However, the z-score equivalents
represent more growth than proposed for the grades 3-8 system. For
example, in 2007, 5.9% of grade 8 students scored Level 1 on the ELA
assessment (a z-score of about -1.65). About 17.6% of high school
students scored 55 or above (z-score of about -.93). Thus, to get credit
for "enough growth," a student who was at Level 1 in grade 8 would need
to grow about 0.7 of a standard deviation in two years, or about 0.35 of
a standard deviation per year---which is more growth than required in
general for students to become proficient within four years in grades
3-8. The score of 65 or higher was obtained by about 69% of the students
by grade 12 on the ELA Regents Exam in 2003; a score of 65 would
therefore be at the 31^st^ percentile, or a z-score of about -.50. A
student who gained 0.7 of a standard deviation in two years between
grades 8 and 10 has a high likelihood of gaining another 0.4 of a
standard deviation in the next two years before grade 12. The proposed
high school growth system does not guarantee that a student must obtain
proficiency, but it only gives credit for significant growth that is in
line with being on track to becoming proficient. Because a student must
pass the designated Regents Exams in mathematics and English language
arts with at least a score of 65 in order to get a Regents Diploma, high
school students have incentives to raise their scores to proficient.

**Principle 6: Tracking student progress-Match Rates**

*Please provide additional evidence of the match rate for all students
and for subgroups for two or more years.*

The main proposal demonstrated the state's ability to track and match
students overall. The main proposal also discussed how missing student
data would not reduce or bias the overall accountability decision
because proposal integrates status and growth information.

The following figures show that over 90% of the students are matched for
possible growth calculations. The subgroups generally maintain their
representative proportion in the growth data as they do in the overall
population, with the exception of students with limited English
proficiency, whose relative percentage in growth is lower. The table
provides the frequencies (numbers of students) and percentage of all
students for each of the NCLB subgroups (race/ethnicity), economic
disadvantaged, students with disabilities, and student with limited
English proficiency. The "Integrated Dataset" consists of all students
with a valid score in 2007, including students who also had sufficient
data from 2006 to be able to calculate a growth score. The Growth
Dataset consists only of students with sufficient data to calculate a
growth score in 2007, i.e., they had valid scores for 2006 and 2007, as
described in the main proposal. The table is for data from the ELA
assessment. The math results are similar.

As shown in the table, there were around 1.18 million students in grades
3-8 with at least a valid score in 2007. There were around 0.9 million
students with growth scores, or approximately 280,000 students without
growth scores. Of these, almost 200,000 are students in grade 3 in 2007
that do not have a second year of data for growth. Thus, the percentage
of matched students of those who could possibly be matched is over 90%.
The percentage of each subgroup in the growth dataset is within 3%
(absolute) of its percentage in the total population/integrated dataset,
with the exception of students with limited English proficiency, which
is over 4% less absolute; about 20% of the LEP students had sufficient
data to be included for growth. This is probably due largely to the fact
that LEP students who took the English Language Proficiency assessment
were not included for growth. Other subgroups changed 15% or less of
their base percentage (e.g., Hispanic students represented 19.7% of the
total population with Status scores, and 17.1% of the population with
growth scores; the difference is 2.6% (absolute), which is 13% of the
19.7%).

  -------------------------------------------------------------------------------------------------------------------------------------------
                **Integrated                                                   **Growth Dataset                               
                Dataset (ELA,                                                  (ELA, 2007)**                                  
                2007)**                                                                                                       
  ------------- ---------------- ------------- --------------- --------------- ---------------- ------------- --------------- ---------------
  **eth_07**    **Fre-quency**   **Percent**   **Cumulative\   **Cumulative\   **Fre-quency**   **Percent**   **Cumulative\   **Cumulative\
                                               Frequency**     Percent**                                      Frequency**     Percent**

  **Asian**     82218            6.96          82218           6.96            58666            6.53          58666           6.53

  **AA**        222916           18.86         305134          25.82           168119           18.71         226785          25.24

  **Hisp**      232167           19.65         537301          45.47           153406           17.07         380191          42.31

  **Nat. Am**   5483             0.46          542784          45.93           4149             0.46          384340          42.77

  **Multi**     453              0.04          543237          45.97           279              0.03          384619          42.80

  **P**         382              0.03          543619          46.00           245              0.03          384864          42.83

  **White**     638038           54.00         1181657         100.00          513756           57.17         898620          100.00

  **pov_07**    566327           47.93         1181657         []{dir="rtl"}   403521           44.90         898620          []{dir="rtl"}

  **spec_07**   161767           13.69         1181657         []{dir="rtl"}   116534           12.97         898620          []{dir="rtl"}

  **lep_07**    67481            5.71          1181657         []{dir="rtl"}   14071            1.57          898620          []{dir="rtl"}
  -------------------------------------------------------------------------------------------------------------------------------------------

**Appendix A: Prospective Report Card Web-Based Template**

  --------------- --------------------- ------------------ ---------------
  **English                                                
  Language Arts**                                          

                                                           

                  \% On track to        \% Proficient      Total Tested
                  Proficiency                              

  Grade 3         9                     54                 133

  Grade 4         7                     60                 115

  Grade 5         7                     59                 117

  Grade 6         6                     57                 143

  Grade 7         6                     48                 120

  Grade 8         6                     49                 139
  --------------- --------------------- ------------------ ---------------

![](media/image1.wmf){width="6.427083333333333in"
height="4.145833333333333in"}

  ----------------- --------------------- ------------------ ---------------
  **Mathematics**                                            

                                                             

                    \% On track to        \% Proficient      Total Tested
                    Proficiency                              

  Grade 3           7                     79                 134

  Grade 4           5                     70                 119

  Grade 5           6                     62                 116

  Grade 6           5                     65                 145

  Grade 7           6                     50                 123

  Grade 8           5                     53                 139
  ----------------- --------------------- ------------------ ---------------

![](media/image2.wmf){width="6.427083333333333in"
height="4.145833333333333in"}

  ---------------------------------- -------- ----------- ----- ----- -------------
  **Results in Grade 3 English                                        
  Language Arts**                                                     

                                                                      

  2008 - 09 School Year                                               

                                                                      

                                              \% scoring              
                                              at                      
                                              level(s):               

                                     Total    2 - 4       3 - 4 4     On track to
                                     Tested                           Proficiency

  All Students                       133      87%         54%   8%    19%

  Female                             77       90%         57%   10%   21%

  Male                               56       84%         50%   4%    16%

  American Indian or Alaska Native                                     

  Black or African American          14       79%         57%   0%    18%

  Hispanic or Latino                 25       84%         32%   4%    15%

  Asian or Native Hawaiian/Other     5        80%         60%   0%    20%
  Pacific Islander                                                    

  White                              89       90%         60%   10%   19%

  Multiracial                                                          

  Small Target Groups                                                  

  General-Education Students         114      92%         63%   9%    20%

  Students with Disabilities         19       58%         0%    0%    11%

  English Proficient                 130      \-          \-    \-    \-

  Limited English Proficient         3        \-          \-    \-    \-

  Economically Disadvantaged         60       88%         43%   8%    13%

  Not Disadvantaged                  73       86%         63%   7%    19%

  Migrant                                                              

  Not Migrant                        133      87%         54%   8%    18%
  ---------------------------------- -------- ----------- ----- ----- -------------