### U.S. DEPARTMENT OF EDUCATION

Growth Model Proposal Peer Recommendations

for the

[NCLB Growth Model Pilot Applications]{.smallcaps}

### September 23, 2008

### Introduction

During the course of their deliberations on the growth model proposals
over the past several years, the peer review panels returned repeatedly
to several issues prevalent in multiple proposals. In many cases, there
was a lack of clarity regarding a State's proposal or whether a State's
assessment, accountability, or data system possessed the necessary
technical quality to support the State's growth model proposal. In an
effort to synthesize the cross-cutting issues and to clarify what a
State growth model proposal should include for the peers to fully
evaluate that proposal, two peers created the following document. This
document, along with the Department's guidance and the May 2006
cross-cutting document from the first peer review panel, provide useful
information for any State developing a growth model proposal.

Please note that this document is a product of the peers. The opinions
and views expressed herein do not necessarily reflect the position of
the Department.

**[Section A: SUMMARY OF CONTENT AND HISTORY OF PILOT
PROGRAM]{.underline}**

1.  The recommendations in this memo are structured to encourage States
    to submit high-quality and innovative growth-model proposals that
    will follow the bright-line principles of No Child Left Behind
    (NCLB) and that will facilitate adequate student progress over time
    towards the goal of 100% proficiency by the year 2013-14.

2.  States with the capability, resources, and data requirements
    necessary to fully carry out and implement the proposed growth model
    (including a fully approved assessment system that can support the
    state's proposed growth model and a mature student identification
    system for establishing match rates by race/ethnicity and for all
    student subgroups) are encouraged to apply.

3.  A brief history is provided below that summarizes context for the
    growth model pilot program. Section B contains the core NCLB
    principles outlined by the U.S. Department of Education (the
    Department) that must be preserved in the proposed growth-based
    accountability system. Section C provides suggested structure for
    elements and components of the growth model proposal and inclusion
    of supporting materials. Careful attention to the information
    contained in Sections B and C should help States prepare and submit
    a successful proposal.

4.  In May 2006, the peer-review process led to the development of a
    cross-cutting document describing peer concerns resulting from the
    first round of review of the growth-model proposals. Their
    recommendations are included in these recommendations, but States
    are encouraged to review that original document (available online at
    <http://www.ed.gov/admins/lead/account/growthmodel/cc.doc>).

5.  States are advised to maintain a 30-page limit to their proposals,
    which should present a complete and concise description of how the
    proposed growth model meets the core principles. In the proposal,
    evidential data should be provided to show how each core principle
    will be or has been met.

6.  If computational or statistical details regarding the growth model
    or examples validating the model for multiple scenarios of student
    attendance and variations in growth patterns are not included in the
    content of the 30-page proposal, States might also consider
    submitting an additional 10-page technical summary.

7.  Evidence submitted in support of the proposal should be clearly
    labeled and easily accessible. For example, peer reviewers
    anticipate that a State would extract or specifically identify
    material from the technical manual of its assessment system, and
    explain why that material is relevant evidence for a specific aspect
    of its proposal linked to one of the core principles, rather than
    merely appending the technical manual.

8.  If appendices are necessary to support the proposal, they can be
    submitted electronically or via CD-ROM. Examples of appendices may
    include technical reports on vertical scaling for the assessment
    used by States; standard setting technical reports; or examples of
    reports explaining student or school progress and the growth model
    to parents.

9.  During the review process, the peers evaluate how all the elements
    of a State's proposal will work together to create a coherent
    accountability system. States are strongly urged to submit proposals
    that completely address each of the core principles. The sections
    below are designed to assist States in submitting a comprehensive
    growth model proposal.

[History and Context of the Growth Model Pilot Program]{.underline}

In early 2005, the Department indicated it would take a commonsense
approach to implementing the *No Child Left Behind Act of 2001* (NCLB).
This approach, crafted as an equation called *Raising Achievement: A New
Path for No Child Left Behind*, established that, when States show
results and follow the tenets of NCLB, they will be eligible for new
tools to help them meet the law\'s goal of getting every child to grade
level proficiency by 2013-14. Since that time, the Department, through a
letter from Secretary of Education Margaret Spellings (November 21,
2005), invited States to participate in a pilot project where growth
models would determine whether schools made adequate yearly progress
(AYP) under ESEA Title I, Part A. That letter provided details about
seven Core Principles that would be used to evaluate the state growth
model proposals, an overview of how the models would be reviewed, and
the inaugural application deadline. Prior to that deadline, Department
staff conducted three Question and Answer sessions with State education
officials via teleconference. Since the initiation of the Growth Model
Pilot Program, two additional requests for growth model pilot proposals
have been released. As of September 2008, eleven States have had their
proposals fully approved by the Department. On August 18, 2008,
Secretary Spellings announced a continuation of the Pilot Program with a
deadline for proposals set for October 15, 2008.

Growth models track individual student achievement from one year to the
next, giving schools credit for student improvement over time. The pilot
is designed to serve two purposes in its support of NCLB. First,
rigorous evaluation of the approved growth models will provide important
information to the Department and to States about effective ways in
which schools can meet NCLB proficiency goals, thus closing the
achievement gap for all students. Second, reliable identification of
student growth is predicated on the use of high-quality data and
assessment systems; thus, the results of the program will continue to
inform all stakeholders about exceptional accountability models that
assure student attainment of proficiency by 2014.

The States that have had their growth models approved by the Department
proposed innovative and high-quality methodologies for measuring student
progress, raising student achievement, and enhancing school
accountability. Following rigorous internal and external peer review,
the proposed models were determined to meet the seven core principles
outlined in the Secretary's original letter of invitation to the pilot.
These approved growth models include projection models, the use of
transition value tables, and the setting of empirically derived growth
targets for individual students. Each State's model is unique and
well-grounded within its NCLB accountability system. The Department
continues to be committed to supporting States' ingenuity in the
development and evaluation of additional technically sound methodologies
for measuring and promoting student improvement that are consistent with
the seven core principles.

**[Section B: CORE PRINCIPLES]{.underline}**

The Department requires proposals submitted under the Growth Model Pilot
to include specific information and evidence that addresses how the
proposed growth-based accountability system will meet the bright-line
principles of NCLB. The remaining section of this memo describes the
technical elements and data-driven evidence that a growth model proposal
is recommended to contain in order to be consistent with these core
principles, but leaves each State free to propose its own specific
growth model. In addition, these core principles may serve as an
evaluation guide for peer review of submitted proposals.

1.  Ensure that all students are proficient by 2014 and set annual goals
    to ensure that the achievement gap is closing for all groups of
    students identified in the statute.

2.  Set expectations for annual achievement based on meeting grade-level
    proficiency, not based on student background or characteristics.

3.  Hold schools accountable for student achievement in reading/language
    arts and mathematics separately.

4.  Ensure that all students in tested grades are included in the
    assessment and accountability system, hold schools and districts
    accountable for the performance of each student subgroup, and
    include all schools and districts.

5.  Include assessments that produce comparable results from grade to
    grade and year to year in grades three through eight and high school
    in both reading/language arts and mathematics and that have been
    operational for more than one year. The full assessment system, as
    administered in the school year, must be *Fully Approved* or *Fully
    Approved with Recommendations* before the State determines AYP based
    on the assessments administered in that school year (i.e., the
    assessments administered in 2008-09 must be fully approved before
    the AYP determinations that include the 2008-09 assessment results
    in order for the State to include its growth model in its
    accountability system).

6.  Track student progress as part of the State data system.

7.  Include student participation rates and student achievement on a
    separate academic indicator in the State accountability system.

**[Section C: SUGGESTED PROPOSAL STRUCTURE.]{.underline}**

***C1 Abstract***. An abstract (approximately 350 words or less) should
succinctly identify the method used to establish adequate yearly growth
under the growth model, and briefly describe how the inclusion of growth
is to be embedded within the State's accountability system (e.g., state
how the proposed model might include a combination of status, safe
harbor, and growth; growth alone; or other scenarios). The abstract
should not include details of the growth model, reserving technical
details and examples for the proposal or the 10-page technical summary.
The abstract should provide a straight-forward snapshot of the
growth-based accountability system and how it will be used by the State
for AYP decisions.

***C2 State's Capacity regarding its Data, Assessment, and
Accountability Systems***. Effective growth models -- that is, models
that accurately target and capture changes in individual student
achievement -- are only possible if States are employing reliable and
valid assessment systems that successfully track students over time and
across schools, such as by assigning unique identification numbers
beginning with their entry into the State system. Before describing the
technical elements of their growth model, States should first describe
evidence of their capability to fulfill the requirements of a
growth-based accountability system. Information to emphasize in this
section includes, but should not be limited to, the following:

-   State data infrastructure

    -   The state should describe its capability and experience with the
        data warehouse system for entering, storing, retrieving, and
        analyzing the large number of records that will be accumulated
        over time.

    -   The state should describe its experience in analyzing
        longitudinal data on student performance.

-   Match rates and analyses demonstrating lack of bias in unmatched
    students.

    -   Evidence of clear and consistent matching capacity should be
        demonstrated by the State, with match rates provided in a table
        for all students, race/ethnicity groups, and all relevant
        subgroups (including *all* NCLB subgroups) of students. The
        numerator and denominator used in the match rate calculations
        should also be included in this table and clearly identified, as
        well as the overall number of students assessed each year.

    -   The peer reviewers would expect to see the data tables that
        support the capacity of the State to adequately match students
        from year to year. The State should be able to account for all
        students, both those matched and not matched (i.e., the State
        should be able to indicate why there is no prior or successive
        assessment score for unmatched students).

    -   The identification and tracking system should be mature enough,
        and contain sufficient demographic information on students
        (e.g., ethnic/race category, disability status, participation in
        free/reduced price lunch), to enable statistical analyses that
        support lack of bias in unmatched students across two or more
        years.

    -   If match rates show bias against one or more subgroups, the
        State should provide an explanation as to why that is occurring
        and what, if any, steps are being taken to reduce this bias.

    -   If growth targets for some students are set three (or more)
        years into the future, match rates should be provided for three
        (or more) years of data and appropriate analyses undertaken to
        ensure that, in particular, low performing students are not
        being excluded from the growth model and accountability system.

    -   If match rates are declining significantly over three or more
        years, States should describe the expected impact on
        accountability criteria and identify corrections to the
        identification and/or data matching system to address this
        issue.

    -   States should describe their ongoing adaptations to the
        identification and tracking system to improve match rates.

-   Current accountability system

    -   States should describe their existing accountability system for
        AYP determinations in order to demonstrate how growth
        calculations will contribute to the existing system. States
        should include their minimum group size; whether or not they use
        confidence intervals; describe any averaging processes such as
        uniform averaging. In their descriptions of the growth model,
        States will need to consider how these AYP definitions will
        continue or change once growth is incorporated into AYP
        decisions.

    -   States should indicate that student participation rates are
        included as one of the criteria in the Statewide accountability
        system, as well as describe how the additional academic
        indicators are incorporated into accountability decisions.

-   Student participation rates

    -   (Also noted above) In determining AYP, a State should include,
        in addition to academic achievement, (1) participation rates on
        the State's assessment, and (2) "at least one other academic
        indicator, as determined by the State for all public elementary
        school students" and graduation rate for public high schools,
        and may include other academic indicators such as "decreases in
        grade-to-grade retention rates." For purposes of developing a
        growth model, these requirements should be addressed in a
        State's proposal.

-   Growth-based accountability

    -   The State should describe its accountability plan in terms of
        evaluation of AMOs for school performance and how these plans
        relate to evidence from the growth model.

    -   States should outline the consequences that will be applied to
        schools that fail to meet their AMOs and describe how these
        consequences meaningfully reflect the results of the student
        growth evaluations.

    -   The proposed interventions should comply with the Section 1116
        requirements for public school choice, supplemental educational
        services, etc.

    -   If appropriate, the State should explain how it plans to focus
        its school intervention efforts by incorporating the results
        from the growth model. For example, a State should be prepared
        to explain how a school that does not meet either traditional
        AYP goals or growth-based accountability goals might be subject
        to more rigorous intervention efforts than schools not making
        AYP on only one accountability measure.

    -   In describing the model and its use for determination of AYP
        using growth, States should ensure that the growth or
        proficiency of high-performing students does not compensate for
        performance of other students.

-   Assessment quality, data systems, and growth

    -   Scores on testing instruments for which measurement of student
        growth (or status) is to be determined must be reliable and
        valid. States should provide a description of their approved
        assessment system (including data on the reliability and
        validity of the assessments being used in the growth model),
        indicating that students have been annually assessed in both
        reading/language arts and mathematics in grades 3-8 and one
        additional high-school grade.

    -   States should be able to demonstrate that their assessment and
        data systems support the requirements necessitated by use of a
        growth model. Evidence for assessment quality supporting a
        growth model application can be found in:

        -   Detailed description of how the assessment system provides
            for comparable information as students move from one grade
            to the next (e.g., vertically-equated for longitudinal
            connections). This should also include information related
            to the stability of the assessment system over the past few
            years and into the future.

-   Alternate assessments

    -   The proposal should include a full description of alternate
        assessments based on grade-level, alternate, or modified
        academic achievement standards. States should provide
        information on how students taking the alternate assessments are
        to be included in the proposed growth model.

    -   The proposal should also include examples and explanation to
        demonstrate how students who have taken the alternate assessment
        (or a native language assessment) in year 1 are accommodated in
        the growth calculations if they take the general assessment in
        year 2 or any subsequent years (or vice versa).

***C3 Model Description***

A comprehensive description of the proposed growth model and how it will
be used in conjunction with or as a replacement to current practice for
the State's accountability decisions should be provided. Establishment
and justification of growth targets, projections, supporting tables and
charts, and examples of *how* the growth model is enacted in practice
and under varying circumstances should be included to support the
model's validity. If necessary, a technical appendix may be used to
supply additional information on the statistical details and validity
calculations for the growth model. Overall, States should demonstrate
how their growth model is operationalized for *all* students in relation
to the complete accountability system of which the growth model may be
only one component.

-   States should provide a justification for their decisions on how
    adequate yearly growth (AYG) is to be determined based on their
    growth model.

-   States should describe how the proposed use of the growth model fits
    into their accountability system, specifically for targeting
    resources to individual students identified through growth
    calculations as in need of these services.

-   States should demonstrate that their system will get 100 percent of
    students to proficiency by 2014.

-   Confidence intervals around growth projections are discouraged, but
    if proposed they should be fully justified and not exceed 1 standard
    error (68%) (see the peers' *cross-cutting* document (link above)
    for additional information)

[Validity Demonstrations]{.underline}

-   The expectation of the Growth Model Pilot is that AYG be assessed
    for *all* students in the system. Thus, the model description
    section of the proposal should contain data-driven validity
    demonstrations, with examples that extend more than one-year out
    through the model. All States proposing growth models should already
    have the data to implement these validity calculations given that
    they have their statewide assessment and accountability systems in
    place.

-   Formulas to assess growth should be clearly stated and multiple
    examples for hypothetical children with different outcomes should be
    provided. Ideally, the examples that States provide to support their
    proposed growth model should extend outward for at least three
    years.

-   Resetting of growth targets is rarely, if ever, justifiable in a
    growth-based accountability system. Details on problems with this
    approach can be found in the peers' *cross-cutting* document (link
    above).

-   Based on data already available in the system, States should provide
    data tables for the current year to see how many students and
    schools would make AYP by adding growth to the original AYP
    accountability system.

-   Full academic year

    -   The existence and impact of schools with high mobility, both
        within district and out-of-district, should be addressed in the
        proposal. States should fully explain their decisions
        surrounding "full academic year" (FAY) rules and the impact this
        has on match rates, on the presence of missing data, and on
        growth calculations. Modifications to the FAY rules, if
        proposed, should be fully explained and justified.

```{=html}
<!-- -->
```
-   Additional data tables should be included in order to support the
    validity of inferences from the growth-model. These include examples
    that would demonstrate how the model works for students under
    various circumstances. States should clarify the impact on the
    inferences that will be drawn from AYP determinations under these
    circumstances. The list below is not all inclusive, but assuming
    data are available for at least three years in the system, examples
    should be provided under these circumstances for:

    -   A child who changes schools

    -   A child who changes LEAs

    -   A child who is retained

    -   A child who is new to the State

    -   Schools with varying configurations and non-aligned time-frames
        for administration of the assessments (i.e., K-5 with testing in
        Spring; 6-9 with testing in Fall)

    -   A child who makes progress but then falls behind

    -   A child who is proficient at all times but steadily declining

    -   A child who changes assessments (with possible alternate
        achievement standards) during the growth calculation period
        (e.g., a regular assessment, a native-language assessment, or an
        alternate assessment aligned with grade-level achievement
        standards)

    -   Calculations for growth on all students, not just non-proficient
        students