### U.S. DEPARTMENT OF EDUCATION

SUMMARY BY THE PEER REVIEW TEAM OF JUNE 2008

REVIEW OF DIFFERENTIATED ACCOUNTABILITY PROPOSALS

### Introduction

On June 13 and 14, 2008, a team of peer reviewers external to the U.S.
Department of Education (Department) met to review and evaluate the
proposals that seventeen states had submitted for potential
participation in the Department's differentiated accountability pilot,
which is designed to provide states with an opportunity to explore ways
to better match the interventions to improve student achievement to the
academic reasons that led to a school's identification for improvement
and to help states to target resources and interventions to those
schools most in need of intensive interventions and significant reform.

The Department would like to take this opportunity to thank publicly the
group of peers who thoughtfully read, debated, and considered the state
proposals for this pilot and issues of student achievement and
accountability. These peers -- Dr. Margaret (Macke) Raymond, Dr. Chris
Cobitz, Sheila Evans-Tranumn, Jacqueline Jackson, Dr. Pete Goldschmidt,
Raul Gonzalez, Gary Huggins, Jim Lanich, Dr. Maggie McLaughlin, Peter
McWalters, Jim Peyser, Dianne Pichè, Dr. Rachel Quenemoen, John Winn,
and Dr. Martin West -- represent a diverse group of individuals, all of
whom are committed to education and student achievement. We would
especially like to thank them for producing the memorandum set forth
below, which the Department believes will have enormous utility, and
which was their idea and effort entirely. The Secretary, senior
leadership, and all of those involved with the differentiated
accountability pilot appreciate the peers' service and dedication to
this project and, most importantly, to students.

The Department would also like to thank the state leaders and their
staff for their efforts to develop proposals for this pilot project.
Although not all proposals were approved, each proposal that was
submitted represented many hours of work and much dedication to ensuring
that all students achieve to high levels.

During the course of their deliberations on the differentiated
accountability proposals, the peer reviewers returned repeatedly to
several issues prevalent in multiple proposals. These overarching
questions guided the discussion of the peers throughout their review and
helped them recognize the strengths, shortcomings, and pitfalls of
differentiated accountability models in general and for the seventeen
state proposals in particular. Consequently, in addition to the report
for each state's proposal, the peer review panel created the following
document to further explain the cross-cutting issues the members felt
pertained to most of the seventeen reviewed proposals and that would
need to be addressed by any future differentiated accountability
proposals considered by the Department. While the peers expressed mixed
views on the complex topics discussed, the points raised in this
document should help guide states as they develop differentiated
accountability models in the future.

Please note that the remainder of this document is a product of the peer
review panel, members of which are external to and independent of the
Department. The Department, its officers, and its staff had no influence
over the contents of this document. The opinions and views expressed in
the remainder of this document do not necessarily reflect those of the
Department

**CROSS-CUTTING ISSUES**

**from the**

**PEER PANEL**

**on**

**DIFFERENTIATED ACCOUNTABILITY**

Differentiated Accountability represents a significant new development
in accountability policy and practice. The initial proposals submitted
by the first round of states varied considerably in their approaches to
distinguishing among schools that did not meet Adequate Yearly Progress
(AYP) and in their targeted supports and interventions. During its
deliberations, the Peer Panel on Differentiated Accountability
recognized the need, even before any pilots are implemented, to build a
record of insights and concerns to support knowledge creation and
dissemination. This memorandum presents a set of generalized issues that
the Peer Panel identified from the set of proposals it reviewed.

**Differentiation Methods**

While several methods were proposed to differentiate among schools that
missed AYP, a few important issues became apparent quickly. None of the
methods proposed were free of significant limitations. The Peer Panel
expressed strong interest in framing these issues to ensure that they
receive further consideration and that they are made clear for the state
agencies that will be implementing them or those states contemplating
using similar approaches in the future.

**The methods appeared largely to be based on methods of convenience
rather than a focus on the underlying causes of schools inability to
meet AYP.**

Across the proposals, there was a general lack of qualitative
description supporting the validity of the differentiation. In assessing
the differentiation methods, the Peers sought methods that could be
validated qualitatively based not on AYP targets, but rather on the
schools' needs to become effective for the students who are not
proficient. In many cases, the differentiation made sense at the school
summary or subgroup level but did not take into account underlying
realities. One such example is the possibility that a subgroup meeting
AYP in a given year (through secondary measures) might actually have the
lowest proficiency rate in the school.

Many proposed differentiation methods appeared to have been designed to
respond to common complaints such as schools missing AYP due to a
failure to meet only one proficiency target or schools that were only a
few percentage points away from their proficiency target. While these
are common concerns, a deeper understanding of the situation in the
school that caused the AYP results is probably more important when
developing a differentiation method leading to interventions designed to
assist the school in meeting the needs of all students.

For example, some states proposed using the average distance from the
proficiency target for each subgroup to separate the pool of schools in
need of improvement. There is a lack of evidence that under this
approach, schools in the differentiated categories have more
instructional issues in common than they do with schools from any other
category. The state proposals presented no analysis suggesting that the
differentiation produced groups whose schools in one group are similar
to each other and are different from schools in other groups. A guiding
question is: "Does it make sense in terms of the interventions being
proposed to differentiate on the proposed basis?" A more concerted
effort to link differentiation method to underlying sources of need
would produce more alignment with intervention strategies.

**Some methods created unintended and undesirable consequences.**

The Peer Panel was concerned that the methods proposed by states were
not fully explored and potentially could have unintended undesirable
consequences. While the use of safe harbor to determine a school's AYP
status is allowable, it is not clear that for schools missing AYP --
after all the permissible secondary determinations -- perpetuating the
secondary determination methods for the purposes of differentiating on
the basis of need is useful. Clearly, a school that misses only one
subgroup's proficiency AMO but meets another 10 by safe-harbor cannot be
comparable to a school that misses one target but meets all the others
outright.

Other proposals used the percentage of AYP targets missed, an approach
that creates different results based on the heterogeneity of the school.
Specifically, if one subgroup misses its target, it is more likely that
a school with few subgroups will be categorized for more intensive
interventions than a school with a many subgroups, since the method
stratifies levels of need by aggregating percentages of targets missed
into discreet ranges. Very few of the proposals included a backstop to
ensure that schools in milder levels of intervention did not allow any
single group to persistently under-perform.

Finally, some proposals differentiated schools based primarily on
relative performance levels rather than against an absolute standard.
While such an approach could make sense, were it to reflect limitations
in state capacity to provide meaningful, intense interventions to a
large number of schools, evidence of capacity constraints was often
missing from the proposals. Using a norm reference without substantial
justification did not speak well for the ability to address the needs of
non-proficient students, since some schools could continue to miss AYP
for years and never appear on the list of the lowest 10% in the proposed
ratings.

**The proposals failed to revisit the differentiation, once made.**

Frequently, there was no explicit mention of how these differentiation
methods would support transitioning schools from one tier of
intervention to another through subsequent years. In some proposals, a
school would qualify for a less intense tier of intervention if it met a
certain criteria for the "all students" subgroup even though that was
neither the subgroup that had been struggling nor tied to effective
interventions for the non-proficient students. Once a division of the
schools was made by the differentiation approach, that tool was not used
to refine interventions or develop explicit targets for intervention
plans.

**Some proposals let schools persistently miss AYP without entering
restructuring.**

In some proposals, schools could be delayed or prevented altogether from
progressing toward restructuring as a result of being placed in a less
severe intervention category. In other words, the differentiated
accountability model did not ensure that no schools will be held
indefinitely at a given level of intervention. This is particularly
relevant when considering how the proposed systems treated schools that
meet proficiency targets for the all-students subgroup but persistently
miss them for specific subgroups. While information on the nature of the
performance problems in such schools could be used to determine the
nature of the interventions to which they are subjected, they should
ultimately be subjected at least to the restructuring options specified
in NCLB.

**Supplemental Educational Services and Public School Choice**

The timing and quality of supplemental education services consumed a
considerable amount of the Peers' attention. While no-one disapproved of
a state accelerating the requirement to offer supplemental education
services to support students enrolled in schools in need of improvement,
many peers voiced concern that, absent a strong commitment to improve
both access *and quality*, re-ordering the requirements to offer Public
School Choice and Supplemental Education Services did not result in an
equivalent or greater level of support and remediation for students
compared to what exists today. The concern arises from the view of a
subset of peers that public school choice is a right extended to Title I
beneficiaries and not necessarily as a treatment for low-performing
schools, so that delaying the introduction of Public School Choice might
be impermissible. (The remaining Peers were agnostic on this point.)

The Peers also expressed significant concerns about the possibility of
districts in improvement being allowed to act as supplemental
educational services providers.  Therefore, it was decided that as a
condition of approval, all proposals that allowed this possibility
should be amended to insure that districts in improvement were not
allowed to directly provide SES services.  Proposals that were vague on
this point were asked to clarify that districts in improvement would not
be allowed to be direct providers of services. The condition is
consistent with the requirements that districts wishing to offer
Supplemental Education Services under current waivers from USDE must
demonstrate sufficient capacity to support such activities.

**Intervention Strategies**

The call for proposals encouraged states to advance plans for innovative
ideas to intervene with schools that have been identified as needing
improvement, particularly with schools that have progressed to the
Restructuring phase of the NCLB accountability framework. The Request
for Proposals sought leadership in advancing new approaches or
theoretical structures that could effectively turn around schools in
greatest need for improvement. Among the proposals, some states were
keenly in tune with forward thinkers who focused their interventions to
the areas of need in schools, and in the restructuring phase of NCLB.
The proposals from Ohio, Illinois, Georgia and Maryland aligned their
intervention strategies with the specific attributes that caused schools
to persistently miss their targets. Their approaches employed a
combination of local influence and state-guided expertise.

None the less, the majority of the proposals were neither as assertive
nor as creative as the Peers were hoping to see. The pivotal role of
Intervention Services in the larger accountability framework might have
led states to different ways of tying results of differentiation methods
to actual choice of intervention services. The suggested strategies --
professional development, leadership training, instructional specialists
and the like -- seemed to be incremental approaches to improvement when
more global and transformative intervention would have been expected.
Given the strategies that were advanced, the Peers were keen to ensure
that states make serious attempts to evaluate the effectiveness of their
proposed interventions, as discussed in greater detail below.

**Exercise of State Authority**

In the review of the applications from the 17 states that applied for
alternative accountability-differentiation status, it was observed that
only a few of the states (Illinois, Louisiana, and Ohio) referenced one
of the key underlying issues regarding student performance, school
improvement, and the closing of achievement gaps. That issue is the
extent to which State Education Agencies (SEAs) work directly with
school districts or Local Education Agencies (LEAs) to build district
capacity.

As the US Department of Education (USED) makes final decisions regarding
these applications, the Peers strongly recommend that ED take into
consideration the critical nature of the SEA-LEA interdependency
regarding a number of issues, notably:

-   Building district capacity

-   State support vs. state intervention under NCLB authority

-   Using federal requirements and funds as a lever to aid in aggressive
    SEA intervention in schools in "restructuring" status under NCLB

If SEAs are to be held accountable for ensuring that all students have
access to adequate standards-based programs, removing barriers that
prevent students from reaching proficiency, and supporting system-wide
capacity-building, the SEAs must be positioned to work directly with the
LEAs on such issues as school choice, teacher assignment, governance,
the direction of state resources, access to time for professional
development and extended learning, and the long-term development of a
cohort of highly qualified educators. Only then will states (and
districts) develop the capacity to achieve our most challenging goal:
bringing all students to proficiency.

The Peers strongly encourage the Secretary and the US Department of
Education to explore the alignment and the interdependency of federal,
state, and district education agencies in formulating the decisions
regarding the approval of state applications for alternative
accountability-differentiation status.

**Evaluation Planning**

The Peer Panel was acutely aware of the need for states to go beyond
simple program monitoring and arrange for state-of-the-art evaluation of
the implementation and impact of their Differentiated Accountability
model. Not only will each state need to have real-time feedback as it
rolls out its design, but the US Department of Education and other
states need to build their understanding of how these models work in
practice over time.

Recognizing that there was little time to develop the proposals, the
Peers none the less found that most states did not give sufficient
attention to evaluation of their new efforts. Considering the intended
impact of differentiated accountability and its importance to the
overall performance of K-12 public education, the Peers were unanimous
in the view that robust evaluation is critical to furthering our
collective knowledge of how to remedy underperforming schools.

The evaluation plan for each state, then, should include early
identification of the data requirements from the State Education Agency
and the schools involved in the pilot so that a robust base of evidence
is developed. The evaluation plan should include an analysis of the
degree to which the proposed method of differentiation identified
schools needing the greatest interventions. Specifically, research
should determine if the computations correctly identify the schools that
are known by other means to be the most seriously in distress and
exclude schools that are recognized not to need the most intense levels
of services. The evaluation should also discuss potential adjustments to
the differentiation method that are found to produce greater accuracy.

Evaluation is also needed to measure the efficacy of the intervention
strategies employed by states to remedy school performance. The Peers
recognized that multiple interventions and differences in local context
would be difficult issues to manage in an evaluation, but urged states
to pursue the most rigorous impact evaluations as conditions would
support.