**Nevada's Proposal to Add a Growth Component to its AYP System**

The State of Nevada through its Department of Education respectfully
requests to modify its general system of determining school level and
district level Adequate Yearly Progress (AYP) by adding a longitudinal
growth component. The State requests to be able to begin implementation
of this addition to the system immediately. Growth comparisons would
first affect school and district AYP decisions following the 2006-07
test administrations.

There is broad-based support within Nevada's educational community to
make this change. Since the inception of NCLB, there have been repeated
overtures by local educators to the state and by the state to the
federal government to allow the recognition of growth when determining
AYP. The federal announcement for flexibility in this arena was met with
strong support. In particular, the Nevada Association of School
Superintendents (NASS) expressed an early interest in pursuing this
flexibility. This was consistent with Department intent as evidenced by
the position taken by Nevada's Superintendent of Public Instruction, Dr.
Keith Rheault several years ago. Dr Rheault was one of 14 education
commissioners to petition the federal government to consider amendments
to NCLB that would allow the measurement of growth as part of AYP.

Moreover, anecdotal accounts suggest that the most commonly held
perception of academic success is the increase of the level of knowledge
and skill attainment among all students, regardless of their starting
point. This perception has often been expressed by many Nevada educators
and parents, as well as by the media. An accountability system that is
sensitive to student change will rightfully recognize the educational
aims of the majority. A growth model (component) that measures such
student change is critical to the ongoing credibility of the State's
system of accountability.

Consistency in aim and measurement has additional benefits. Only by
measuring growth can the AYP system carry out the spirit of the NCLB Act
which is to attend to all students, leaving no child behind. The growth
component forces attention on all students. Both high performers and low
performers are expected to grow, regardless of current status. Moreover,
focusing on growth will also result in a more balanced recognition of
school success. Schools that are effective in achieving their growth
targets should be recognized and rewarded. Another related and more
technical outcome will be a closer alignment of the accountability
system with the primary goals of NCLB. The growth component will better
stimulate the primary intended consequence of the accountability system,
which is attention to all students regardless of current status, and
especially to those students traditionally underserved. Additionally,
the growth component will dissuade unintended consequences such as an
over-focus on "bubble" students: those students very close to the
proficiency cut point. As a result, system validity will be supported,
with both the State Department and School Districts having more
confidence when making annual determinations.

[A Collaborative Effort]{.underline}

The State Department meets monthly with the Nevada Association of School
Superintendents (NASS) to discuss many issues. Among these,
accountability is often at the forefront of those discussions. NASS
formed a longstanding subcommittee which has met to address a multitude
of AYP related issues over the years. One of the most earnestly
discussed has been the introduction of a meaningful growth component to
the State's AYP system.

In developing its proposal, the Department consulted with its Technical
Advisory Committee (TAC). The TAC is comprised of six national experts
in measurement in addition to contractor and department staff
representatives. The inescapable connection between testing and
accountability has prompted ongoing discussions regarding accountability
as part of the Department's quarterly TAC meetings.

[Foundation]{.underline}

To develop the Nevada proposal, the Department has closely attended to
and adhered to the federal guidelines provided to states regarding
growth models. Justifiably, the federal government is asking states to
address a multitude of issues as they consider the measurement of
growth. Nevada's proposal attempts to address in detail the seven
principles outlined in the federal guidelines (directives)
(requirements).

Ultimately the peer review of proposals will likely focus on both
technical and practical details. There do, however, appear to be some
critical "gatekeeper" issues implied by the federal guidlines. The
Department believes that these general foundational issues have been
scrupulously addressed by the State. First, as discussed above, there is
broad-based support among Nevada's educational community and the public
for this adjustment to the system. This includes recognition that the
addition of growth adds complexity to an already complex AYP
determination process. Therefore there will need to be an ongoing
commitment to training both educators and the public on the
interpretation of growth, including its measurement application. It is
recognized that adding a growth component to the system is no panacea.
On the contrary, it is recognized that growth expectations will be very
challenging. (See the appendix for NASS presentation.)

Second, Nevada has implemented a system of student identification that
assigns unique identifiers to students throughout the state in an
automated and centralized fashion. The advent of this system allows for
the efficient tracking of students as they move between schools and
districts within the state. It can be used efficiently to identify
students who leave the state for an extended period and who later return
to the state. This is clearly a cornerstone to efficient student-level
longitudinal analyses. (See appendix for documentation regarding
Nevada's unique identification system.)

Third, and perhaps most important, Nevada's system of standards-based
assessment has matured to a level that allows for meaningful
longitudinal tracking of student academic achievement in mathematics and
reading. Nevada's system includes reading and math tests in grades 3
through 8 and in high school. (Tests in grades 4, 6, and 7 were
implemented live in 2005-2006 to complete the system.) It is important
to add that reading and math tests have been administered since 2004 and
that census data is now available for the 2005 and 2006 administration
cycles. The grade 3 through 8 assessments benefit from a vertical
articulation of achievement standards closely aligned to Nevada's
content standards.

The Department has historically collaborated closely with the U.S.
Department of Education. The State submitted its final assessment system
to the federal government for review in August of 2005. Upon an informal
request from the federal government, the Department provided
supplementary information in November of 2005. The State did receive
feedback in June of 2006 indicating Nevada's *Approval Pending*
classification. In July, the Department responded by submitting a plan
to complete the final assessment system that was subsequently approved
by the federal government. Nevada has since shared a bi-monthly update
on progress and is complying closely with the outlined plan.

Three additional contextual issues are important to share. First, at
this time Nevada only requests the addition of the growth component to
the annual judgments of elementary and middle schools. Due to various
technical issues, there is no initial intent to add a growth component
at the high school level.

Second, the State feels strongly that the measurement of growth is a key
to a credible and well-purposed accountability system. However, it also
feels strongly that absolute measures of achievement are critical to the
system. Therefore, the State proposes to add the growth component to the
system while continuing to measure achievement status and Safe Harbor as
it has done since the inception of the NCLB Act. The State does plan to
carefully evaluate the impact of both Safe Harbor and the growth
component, with the ultimate intention of replacing Safe Harbor with the
growth component not prior to 2008.

Third, throughout the proposal, primary attention is given to the annual
judgments of schools. It is the intent of the State to hold districts
accountable for student growth just as it holds schools accountable for
student growth. Therefore, references to schools are equally references
to school districts. Where differences between school level and
district-level analyses are relevant, clear distinctions will be made.

As noted, it is strongly held that the increased flexibility afforded by
a growth component will bring credibility to a system of accountability
that has been under attack since its inception. More importantly, this
flexibility provides the best chance for federal legislation to meet its
expressed intention: meeting the needs of all students and moving them
toward the ultimate expectation that all students will be proficient by
the 2013-14 school year.

The federal flexibility on growth comes at a propitious time for Nevada.
Nevada has been steadily implementing systems that will support the
growth model. Since NCLB was passed, the State has worked diligently to
implement its requirements. As evidence, the 2003 State Legislature
passed legislation substantively amending key statutes affecting public
education. For example, it significantly revised the State assessment
system and theState's school accountability designation system. Both had
been built on a foundation of norm-referenced testing which has been
replaced with standards-based (criterion-referenced) assessments. The
legislature significantly revised district and state responses to school
inadequacy, requiring that corrective actions be taken for all schools,
including Title I schools if identified as inadequate. In addition, all
schools, districts, and the state are required to develop annual
improvement plans regardless of AYP outcomes. The 2003 legislation also
revamped the State's centralized data collection and reporting systems.
This revision has resulted in the development of the State's unique
identification system and web-based annual report card system. The
legislation encourages longitudinal tracking of student achievement and
sets up provisions allowing the tracking of students as they matriculate
to secondary education institutions. In short, bills passed in 2003
brought the state into full compliance with NCLB and in fact added rigor
beyond NCLB's minimum requirements.

Upon the initiative of Nevada's Governor, the Legislature passed
statutes in 2005 furthering educational reform. In particular, this
legislation established the Commission on Excellence, and provided
approximately 100 million dollars for a competitive grant program for
schools and school districts. The funding was earmarked for the
implementation of school improvement plans. A strong evaluation program
is tied to the grant money to ensure the identification of successful
practices. The Commission, in fulfilling its oversight responsibilities,
has required that, to the extent possible, the evaluation of student
achievement be based on a growth model.

Moreover, 2005 legislation included funding for school districts to
implement teacher incentive programs. And perhaps most importantly,
legislation was passed commissioning an independent study of adequacy of
per pupil expenditures.

The State fully appreciates this opportunity, and, in advance, thanks
the peer reviewers and federal government for their consideration. The
implementation of a growth model is an important step for Nevada and its
educators in keeping focused on educational reform and positive outcomes
for all students.

The remainder of the State's proposal is built around the seven
principles outlined in recently issued federal guidance. Nevada
originally submitted a proposal to add a growth component in February of
2006. It did receive feedback from the federal government regarding its
proposal in September of 2006. As part of that response, some specific
feedback regarding several principles was offered (principles 1, 2, 4,
5, and 6). Nevada's growth proposal addresses the specific feedback in
the following sections.

[Principle 1---The 100% Proficiency Goal]{.underline}

The NCLB Act is very clear on the expectation that 100% of students,
regardless of student group status, will be proficient by the 2013-14
school year. This absolute achievement expectation is operationalized
through annual status comparisons. The Safe Harbor provision of the NCLB
Act builds from the absolute status expectation, recognizing that all
students may not be able to make the transition from non-proficiency to
proficiency in a single year. If a school can demonstrate a significant
reduction in the percentage of non-proficient students (e.g., 10%
reduction), that is judged to indicate that the school has made
significant improvement toward the absolute status expectation.

The Act also is equally clear on the requirements for grade-by-grade
standards-based assessments in grades 3 through 8 in Reading and
Mathematics and for those to be fully implemented by the end of the
2005-06 school year. The fully implemented system enables better and
more precise measurement of the progress schools are making toward the
100% proficiency requirement. A longitudinal measure of growth can
answer a different question than does Safe Harbor (e.g. effectiveness
vs. improvement). But in principle, by retaining the status comparisons
as primary in making AYP determinations, the growth component enables a
better estimate of the degree to which schools are making progress
toward the 2013-14 100% proficiency goal than does Safe Harbor.
Moreover, it ensures needed attention to students who are already
scoring at or above proficiency so that those students are more closely
supported in reaching their academic potential to exceed
(minimum)expectations.

In keeping with the spirit of the governing legislation, Nevada proposes
to add a growth component to the AYP determination process, while
continuing to measure school status and improvement annually. The
concept of growth is built on the assumption that students, regardless
of where they currently stand, are expected to demonstrate growth
annually. The student achievement spectrum is variable, and so we know
it is likely there will always be a range of student performance. For
example, we might expect recent immigrants or students new to the
educational system to be lower on that spectrum compared to students who
have been in the system for a longer period of time.

However, it is reasonable to expect that the sensitivity of such a
system over time should inform a more effective educational system. That
means it is reasonable to assume that the distribution of school-level
performance in 2013-14 should be very different than it is at baseline
(2005-06), with far fewer students categorized in the lowest levels of
achievement. The establishment of the 2013-14 growth goal is built on
this assumption and, more specifically, assumes that the majority of
students will be maintaining performance at or above proficiency or
clearly making progress toward the proficiency goal.

[Measuring Growth]{.underline}

Student performance will be tracked longitudinally to identify annually
the extent of individual student movement relative to Nevada's
achievement standards (e.g. Emergent/Developing, Approaches Standard,
Meets Standard, Exceeds Standard)[^1]. Points will be awarded to
students based on the extent of individual change. For example, movement
from just below proficient to just above proficient will be awarded a
point value. This value may be greater than that of a student who
maintains performance just above the level of proficiency, which may in
turn be greater than the point value awarded to students maintaining
below proficient performance. Figure 1 provides a hypothetical value
table that can be used to express the measurement of growth in this
manner.

Figure 1. Basic Value Table

+-----------+-----------+-----------+-----------+-----------+-----------+
|           |           | Current   |           |           |           |
|           |           | Year      |           |           |           |
|           |           | Pe        |           |           |           |
|           |           | rformance |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           |           | Emergent  | Ap        | Meets     | Exceeds   |
|           |           |           | proaching |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
| Previous  | Emergent  | 0         | 50        | 75        | 100       |
|           |           |           |           |           |           |
| Year      |           |           |           |           |           |
| Pe        |           |           |           |           |           |
| rformance |           |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | Ap        | 0         | 25        | 60        | 85        |
|           | proaching |           |           |           |           |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | Meets     | 0         | 0         | 50        | 75        |
+-----------+-----------+-----------+-----------+-----------+-----------+
|           | Exceeds   | 0         | 0         | 50        | 75        |
+-----------+-----------+-----------+-----------+-----------+-----------+

Student level information will be aggregated and averaged to compute an
index score representing growth in order to make judgments about schools
and their various student groups. These aggregate growth scores will
then be compared against annual measurable objectives that define
expected growth. The aggregation process used to estimate school growth
will not differ in principle from the current aggregation process that
is used to estimate school status and Safe Harbor performance.
Ultimately, a growth index will be calculated for the school as a whole,
as well as for each identifiable student group with respect to
ethnic/race distinctions, IEP, LEP, and low socio-economic statuses.

[Growth Expectations]{.underline}

In keeping with the methodologies required by NCLB for status and Safe
Harbor comparisons, the annual judgments regarding growth will be based
on a comparison between school progress and State-determined annual
measurable objectives of growth (AMOs).

Three separate but interrelated steps must be taken prior to
establishing the annual growth targets. The establishment of the growth
AMOs will borrow from the methodology outlined in NCLB for establishing
status AMOs. In short, annual growth objectives will be equidistant
across time. This will be derived by subtracting baseline growth
performance, (established through use of the NCLB 20^th^ percentile
school enrollment method) from the 2013-14 growth target and then
dividing this difference by the number of years bridging the baseline of
2005-06 and 2013-14 (eight years). Figure 2 presents a hypothetical
trajectory to aid in a conceptual understanding of the AMOs.

Figure 2. Growth AMOs

![](media/image1.pct){width="5.999305555555556in"
height="3.7618055555555556in"}

In this hypothetical example, the final target is an index score of 125,
and the estimated state baseline is 70. Annually, we then expect a
change in index of 6.875 to

demonstrate adequate growth.

In a sense, establishing the trajectory is the easy part. The difficult
parts are 1) establishing the value table that defines the point totals
awarded to students as they demonstrate growth across the achievement
level spectrum, 2) defining the 2013-14 goal built on the aforementioned
assumptions, and 3) establishing baseline values. Each step will be
outlined in turn.

**Establishing the Value Table**. The term "value" in value table has
two interrelated meanings. Obviously one of the meanings is the point
total awarded to movement, or lack thereof. The other meaning is the
social value associated with the point total. In the above example, it
was pointed out that movement across the proficient cut point might
receive a greater point value than maintenance of proficiency, and that
maintenance of proficiency might receive more points than maintenance of
performance below proficiency. These are relative value judgments. There
is no a priori, matter of fact value table. There is no right or wrong
value.

To a great extent, the establishment of these values is a policy matter,
not all that different from the setting of cut scores on a test to
define different achievement levels. It is the intent of the Nevada
Department of Education to approach the definition of the value tables
by borrowing from a model used for standard setting.

First a group of educators[^2] will be brought together to learn about
the use of the growth component in the accountability system. Their
focus will be on the possible changes that could be observed among
students and on discussing the amount of educational progress that would
necessitate the various changes. Decisions of these sorts will be
informed by study of the content standards and of test content across
grades. Impact data, to include a preliminary value table, will be
shared to provide panelists with a sense of the probability of movement
across the various achievement levels and how differences in change may
be valued. However, as in traditional standard setting approaches, the
values associated with growth should be influenced more by absolute and
relative content knowledge and by skill attainment expectations than by
impact information. The goal for this group will be to derive a
recommended value table.

Once the recommended value table is in place, it will be shared with the
Nevada Association of School Superintendents (NASS) and with the State
Board of Education. The NASS group will have an opportunity to suggest
changes prior to taking the final recommended table to the State Board
for adoption.

There is a complicating factor in implementing the use of a value table
that requires attention. As noted, the growth model allows us to stretch
the current definition of progress from change at the level of
proficiency to change across the entire achievement scale. However,
Nevada currently only identifies four distinct achievement levels. Using
the value table approach, this does not allow for much "stretch". As
part of the measurement of growth, Nevada proposes to further refine
these disctinctions by splitting each of the two lowest achievement
levels (Emergent/Developing & Approaches Standard) into two distinct
categories (Low & High). Some consideration will be given to making an
additional split above the level of proficiency. At a minimum, the
result will be six achievement scale distinctions for purposes of
measuring growth. More importantly, this further refinement will add to
the sensitivity of the scale and will aid in the interpretation of
growth[^3]. Figure 3 depicts a hypothetical value table including these
proposed distinctions.

Figure 3. Expanded Value Table (Hypothetical Values)

+----+---------+--------+--------+---------+---------+-----+-------+
|    |         | C      |        |         |         |     |       |
|    |         | urrent |        |         |         |     |       |
|    |         | Year   |        |         |         |     |       |
|    |         | Perfo  |        |         |         |     |       |
|    |         | rmance |        |         |         |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
|    |         | Em     | Em     | App     | App     | Me  | Ex    |
|    |         | ergent | ergent | roaches | roaches | ets | ceeds |
|    |         |        |        |         |         |     |       |
|    |         | Low    | High   | Low     | High    |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
| P  | E       | 0      | 50     | 100     | 120     | 140 | 160   |
| re | mergent |        |        |         |         |     |       |
| v. |         |        |        |         |         |     |       |
|    | Low     |        |        |         |         |     |       |
| Ye |         |        |        |         |         |     |       |
| ar |         |        |        |         |         |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
|    | E       | 0      | 0      | 90      | 110     | 130 | 150   |
|    | mergent |        |        |         |         |     |       |
|    |         |        |        |         |         |     |       |
|    | High    |        |        |         |         |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
|    | App     | 0      | 0      | 50      | 100     | 120 | 140   |
|    | roaches |        |        |         |         |     |       |
|    |         |        |        |         |         |     |       |
|    | Low     |        |        |         |         |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
|    | App     | 0      | 0      | 25      | 75      | 110 | 130   |
|    | roaches |        |        |         |         |     |       |
|    |         |        |        |         |         |     |       |
|    | High    |        |        |         |         |     |       |
+----+---------+--------+--------+---------+---------+-----+-------+
|    | Meets   | 0      | 0      | 0       | 0       | 100 | 120   |
+----+---------+--------+--------+---------+---------+-----+-------+
|    | Exceeds | 0      | 0      | 0       | 0       | 100 | 120   |
+----+---------+--------+--------+---------+---------+-----+-------+

As noted, the establishment of the value table is the critical first
step. Once accomplished, other critical steps can be taken. The
hypothetical tables that have been presented as part of this proposal
were derived from work by Hill et al (2005) and Betebenner (2005). The
Hill and Betebenner studies provide strong rationale for the steps that
need to be taken to establish value tables and to establish reasonable
growth expectations.[^4][^5]

**Establishing the 2013-14 Target**. As in establishing the value table,
a similar but truncated process will be used to establish the 2013-2014
target distribution and, ultimately, the target index score.. The NASS
AYP subcommittee, made up of State department and district staff
representing the State's school districts, will discuss the matter and
define the 2013-2014 expectations. This definition of expectations will
be taken to the NASS group as a whole, and in turn to the State Board of
Education for revision and adoption.

The 2013-2014 target distribution will be consistent with current status
and Safe Harbor expectations. Of course, the status expectation will
remain 100% proficiency, while Safe Harbor will continue to be the 10%
reduction in non-proficient students. Taken to its extreme, the Safe
Harbor expectation across the original 12-year trajectory allows for a
2013-14 proficiency rate below 100% proficiency. In fact, depending on a
school's or student group's starting point, the 2013-2014 proficiency
level could be markedly lower than 100%[^6].

While the status comparisons in 2013-14 will require 100% proficiency,
it will be possible for the 2013-14 growth target to be met without all
students scoring at or above proficiency. This effect is very similar in
nature to the current Safe Harbor proficiency that allows a school to
meet AYP even if a subpopulation has a percentage of proficient students
that is lower than the annual goal. The growth target for 2013-14 will
expect that the great majority of students are maintaining or improving
performance at or above the Meets Standard level and that the remaining
students are making significant progress toward the Meets Standard goal.

This allowance for a relative judgment to be considered in addition to
status in 2013-14 is consistent with the law (e.g., Safe harbor) and is
appropriate. Nevada is a fast- growing state, and there is no
expectation that the rate of growth will abate. It is therefore
reasonable to assume that in each school year including 2013-14, Nevada
will begin to educate many students across the grade span for the first
time. There is no guarantee that the students who come to the state will
score at or above proficiency on Nevada's academic measures. However,
regardless of a student's level of proficiency in 2012-13, Nevada
schools and districts will be held accountable for the growth of all
students. In short, the growth expectation for 2013-14 will be a more
rigorous accountability expectation than the current Safe Harbor
expectation.

**Establishing Baseline**. Once the value table is set, a baseline will
be derived by evaluating the change in student performance between the
2004-2005 and the 2005-2006 school years. Consistent with the NCLB
defacto procedure for setting of baseline for status comparisons, the
State will array schools from lowest to highest in terms of their 2005
to 2006 growth index scores. School enrollments will be arrayed as well,
so that the school's index score associated with the 20^th^ percentile
of state total enrollment can be identified.

Because grades 4, 6, and 7 tests were administered "live" for the first
time in Spring 2006, growth from 2005 to 2006 will be based on both
actual change and estimated change. Change from grades 3 to 4 and grades
5 to 6 will be based on actual change as tests in grades 3 and 5 were
"live" in 2005, with all additional tests "live" in 2006. Change from
grades 4 to 5, 6 to 7, and 7 to 8 will be estimated.

It is important to note that the estimation procedure is designed to add
additional objective information for establishing baseline. However, it
is not necessary to estimate growth for purposes of establishing
baseline. Actual change from grade 3 to 4 and from grade 5 to 6 can be
augmented with historical information pertaining to change from 3 to 5
and from 5 to 8 in order to establish baseline. These procedures, taken
together, provide far better information than was available to the State
for purposes of setting the original NCLB status baseline based on
2001-2002 performance.

> To estimate change, Nevada's test contractor, Measured Progress, will
> derive

estimates of student performance based on field test performance in
Spring 2005 among 4^th^, 6^th^, and 7^th^ grade students. The field
administration in Spring 2005 was a census administration. All students
took short forms of the test, and, based on item performance, the 2006
test forms were constructed. Student performance on field-tested items
can be used to estimate what their performance would have been on the
full 2006 test form. This estimated performance can then be used just as
the grade 3 and 5 "live" 2005 performance data was used to calculate
student-level change scores.

Currently, two procedures are being considered for estimating growth.
These procedures were shared with the Nevada TAC on February 22, 2006.
The preference was to wait until 2006 data was available in order for
2005 performance to be placed on the scale established from the first
"live" administration. The appendix includes the contractor proposal for
the estimation procedures.

As discussed above, once the value table is set, the 2013-2014 target is
established and baseline is derived, the State will produce the
equidistant trajectory that will define annual growth expectations.
Later in this proposal, the vertical articulation of standards from
grades 3 through 8 is discussed in greater detail. Because of the
consistency across grades, it is anticipated that the same growth
trajectories will be in place for all schools serving grades 3 through
8. The separate measurement of growth in reading and math will also be
discussed in greater detail. It is important to note that while it is
anticipated that the value table for both subjects will be identical
just as will the target distribution of student performance, AMOs
bridging baseline and 2013-2014 will likely differ between reading and
math because of baseline differences between subject areas,.

Because of the high stakes associated with AYP decisions and because of
the subjectivity involved in defining growth expectations, it is not
possible to jump start this process by providing a finalized value table
and 2013-14 expectations. The process of setting up the value table and
finalizing growth expecations is akin to setting achievement levels and
defining proficiency based on test score performance; this is an
activity that requires the voice of many stakeholders. Toward this end,
Nevada anticipates running the panelist meetings to derive
recommendations for value tables and 2013-14 target expectations in
mid-January, 2007. It is projected that these recommendations will be
shared and adopted by policy groups during the month of February, 2007.
As part of the process of adoption and deciding to formally include
growth in the determination of AYP, the Department proposes to share the
policy materials with the federal government no later than March 1,
2006.

[Making Annual Determinations]{.underline}

Once the trajectory of growth is established, an index score for each
school and each of its identifiable student groups will be derived
annually based on the degree of change in the respective group's
distribution of performance across the achievement spectrum. Difference
scores between the current year index and the previous year index will
be computed,. It is these difference scores that will be compared
against the growth AMO[^7]. Building on our hypothetical trajectory (see
Figure 2), a graph plotting a set of whole-school comparisons is
provided to aid in conceptual understanding (see Figure 4).

Using the hypothetical example, to make the growth AMO, the school index
score would have to change by at least 6.875 each school year and, in
this example, from 2006 to 2007. Schools that grow at a lesser rate will
have failed to meet the requirements of the growth objective, and
schools that grow more rapidly than the targeted amount would to be
considered for special recognition. In essence, a comparison between the
slope of the state trajectory and the school trajectory is made.

This presentation oversimplifies the process for illustrative reasons.
As for status comparisons, the same rules apply for all AYP
subpopulations. In other words, the index score or slope for each
identifiable student group will be compared against the State level
expectations. The subgroup (growth) comparisons, however, will only
affect the actual AYP determination when a subgroup fails to meet the
status goal. As noted below, the analysis will still be performed for
every group to inform the school improvement planning process and to aid
in the targeting of interventions and resources.

In making whole school and student group comparisons, and to remain
consistent with Nevada's current model, observed performance will be
bound by a confidence interval prior to comparing it to the State
trajectory.[^8] It is anticipated that the standard error of difference
between means will be used to establish the confidence intervals.

As indicated, a growth index will be computed and plotted for every
school and student group regardless of status and Safe Harbor
performance. During this period of time, the growth component will be
used in much the way Safe Harbor is used. If a school or student group
fails to meet the status and Safe Harbor AMOs, the school can still make
AYP if the school or relevant student group meets the growth AMO and the
additional academic indicator AMO (e.g. average student daily
attendance). Presented below is a graphic display of the various
comparisons depicting the relevance of the growth component to the AYP
decision for a hypothetical school (see Figure 5). Plus signs indicate
that a comparison has been met and minus signs indicate that a
comparison was failed. Empty cells indicate too few students to make a
comparison.

It is important to elaborate on this latter issue. Nevada currently uses
an n-size of 25 students for making status comparisons. Because of
increased restrictions associated with longitudinally tracking students
and because of the use of formal statistical tests, no minimum n-size
requirement for growth comparisons is being contemplated at this time.
The exception may be a cap based on n-size limitations associated with
the statistical methodology itself (e.g., n = 10).

Figure 4. Hypothetical Whole School Comparisons

![](media/image2.pct){width="5.99375in" height="3.85in"}

In the tabled example, although growth is calculated for many student
groups in reading and math, it only affects the school's AYP outcome
when considering math performance among Hispanic and LEP students. The
demonstrated growth among these groups allows the school to make AYP if
these groups also met the other indicator requirement.

Following this logic, there is no weighted judgment being contemplated
during the initial implementation of the growth component. Moreover, as
already noted, for at least the first two years of implementation, the
growth component will be used as an additional factor to consider before
making a final AYP determination. During that time period, its influence
based on current use will be evaluated with an eye toward replacing the
current Safe Harbor comparisons. Other potential uses of the growth
component will be considered. During this evaluation period,
consideration will be given to developing a weighted compensatory system
of inclusion combining status and growth into a single unified judgment.
It is unlikely that a weighted judgment would be used to replace the
sequence of comparisons that are used to determine AYP. However, a
weighted composite might prove effective for school improvement
planning.

Figure 5. Hypothetical School Profile

![](media/image3.png){width="9.0in" height="5.0625in"}

Part = Participation SH = Safe Harbor Other Indicator = Average Student
daily Attendance

[Consequences, Interventions, and Recognition]{.underline}

Nevada would like to use the growth component in multiple ways in
addition to influencing the AYP decision. The State would like to use
the component to identify schools that are making significant progress
with traditionally "hard to serve" student groups.

Currently the State uses status and Safe Harbor information to recognize
schools for exemplary and high achievement. Just as with current
negative AYP outcomes, the State has found that within-school
homogeneity and small enrollment numbers have undue influence on the
special recognitions. The State through regulation has defined a
category of schools labeled "most improved" that is independent of other
AYP outcomes. Schools can be recognized for moving substantial
percentages of students across the cut point distinguishing
Emergent/Developing from Approaches Standard and across the Meets
Standard to Exceeds Standard cut point. To date, the immaturity of the
State testing program has been an obstacle to awarding that recognition.
It is hoped that the fully-implemented testing system and the proposed
growth component will make this special recognition, and perhaps others,
more feasible.

The State will also attempt to take advantage of the power of the growth
comparisons to more effectively target school improvement efforts and to
direct interventions whether planned by the school or required by the
state or school district as part of corrective actions. One difficulty
with the current AYP system is that it is difficult to judge the
magnitude of AYP failure. One can only add up the number of failures by
student group within a subject area to get a sense of magnitude.
Additionally, Nevada has developed a set of metric analyses that attempt
to quantify the average "distance" (negative difference) from the State
status and Safe Harbor AMOs, but even these values suffer from the
static nature of the model and the cross-sectional nature of the annual
comparisons.

With the advent of the growth component, the information provided by the
metric values relative to status can be augmented with more precise
information regarding where progress is actually being made. This will
better enable schools to specify their annual improvement goals, and
will aid districts and the State in making intervention decisions.

Additionally, Nevada proposes to use the growth component to
differentiate consequences. A school that can demonstrate that it is
making significant progress with student groups, meaning that it is
moving those groups toward proficiency and beyond, should benefit from
that progress even when that school may continue to struggle with one or
another student group. By looking at the growth of all groups regardless
of status and Safe Harbor comparisons (where possible), this
differentiation is possible. Corrective actions including school choice
and supplemental services would not likely be affected by this
differentiation; these actions would still be included as corrective
actions.

As an example, Figure 6 provides a graphical representation of how the
information from growth might be used to differentiate action. In this
situation, assuming that the school has demonstrated repetitive failure
in math, consequences might only accrue relevant to students in the
Hispanic and LEP student groups. In the hypothetical example, the
school's failure is in mathematics, while the school as a whole, African
American, White and Low SES students have all demonstrated significant
progress. This is also true of their performance in English Language
Arts.

Additionally, however, suppose that in the examplar school the growth
performance of Hispanic and LEP students is lacking not only in
mathematics, but also in reading. As a result, the school in its annual
school improvement plan would have better information at the program
level that could be used to target reading interventions for these
student groups. Under the current model of AYP analysis with the absence
of the growth column, a school might assume that no intervention or
special attention is required for these students since AYP has been met
in this subject. While the State makes every attempt to encourage
schools and districts to use fine-grained information to inform the
school improvement process, the reality is that AYP results and the
profile information provided by the state are the main sources of
information used extensively in that process.

But as a consequence of using the more refined growth component, the
resultant AYP profile would support State and district efforts to
require schools to intervene on behalf of these same students still
needing support in the ELA area, even though ELA AYP has been met.

To close the discussion on Principle One, the State believes strongly
that the addition of the growth component and the continuation of status
comparisons will better meet the intent and spirit of NCLB than the
current system can. The additional gains to the credibility of the AYP
system will also insure that better information and attention will be
given to and used by schools in order to meet the needs of all students.

The "value table" approach is a means for introducing a growth component
to an assessment system presently absent a built-in vertical scale.
Advantages of the value table approach include its ease of
comprehensibility for stakeholders, its relative ease of calculation,
its potential for alignment with standards, and the potential use of
stakeholder input in the development of categorical value judgments, all
of which can positively affect buy-in.

Figure 6. Differentiating Action

![](media/image4.png){width="9.0in" height="5.0625in"}

[Principle 2 -- Establishing Appropriate Growth Measures for
Students]{.underline}

The State's trajectory of growth will bridge the 2006 administration to
the 2013-2014 target year. The method for determining growth AMOs
borrows from the NCLB Acts prescriptions regarding the setting of
baseline for status and the requirement for equal interval increments in
annual expectations. As discussed above, the establishment of the
2013-2014 target boils down to a series of standards-based policy
decisions. These include 1) the establishment of grade level cut points
on tests distinguishing the various achievement levels (completed
August, 2006), 2) the establishment of a value table for weighting
student level change (to be completed January 2007), and 3) the
establishment of the 2013-2014 expected school level distribution of
student performance (to be completed January 2007).

By focusing on Nevada content and achievement standards and test
content, the target distribution of performance should not be biased by
group characteristics or normative growth expectations. It is important
to emphasize that all schools and student groups within schools will be
held to the same measurable growth objectives. Growth objectives will in
no way be moderated by student group characteristics. In other words,
just like the current status comparisons, a single statewide growth AMO
will be established against which whole school and student group
performance will be compared.

Once the trajectory is established, there is no expectation that the
annual growth targets will be changed. If a significant change were to
occur with respect to the State's testing program, this could prompt the
need for a reanalysis of baseline and hence, a reanalysis of the annual
requirements leading to the 2013-2014 target distribution.

There is no expectation that such a change to the testing program will
occur. The assessment system has been in place since 2001-02 and has
reached a significant level of maturity as the grade 4, 6, and 7 tests
were introduced. The State has a long-term contractual relationship with
its test developer and anticipates that this relationship will continue.

[Principle 3 -- Accountability for English Language Arts and
Mathematics]{.underline}

As highlighted in Figure 6, decisions regarding growth will be made
separately by content area (English Language Arts and Mathematics) and
will be based solely on those tests that cover the complete range
between grades 3 and 8 (Reading and Math) . Although it is anticipated
that the value table and 2013-14 target distribution of scores will be
identical for both content areas, it is likely that baseline values in
reading and mathematics will differ. This being the case, the growth
AMOs will differ by content area.

In Nevada, reading and writing tests are used to estimate proficiency in
English Language Arts. While the reading tests, like the math tests, are
administered in grades 3 through 8, the writing assessments are
administered in grades 5 and 8 only. The gap in coverage prior to grade
5 and between grades 5 and 8 make it extremely difficult to estimate
growth in writing as a subset of English Language Arts. Because of this,
the growth model will only include information pertaining to reading and
mathematics. However, writing test performance will continue to be
included in determining AYP. It will impact both English Language Arts
status and Safe Harbor calculations.

[Principle 4 -- Inclusion of All Students]{.underline}

Nevada's proposed growth model is in keeping with the prescriptions of
the NCLB Act regarding the measurement of academic achievement. The
growth component encompasses performance of students who have been in a
school during the current school year for a full academic year[^9] and
for whom previous assessment scores are available, so long as those
scores are from the Nevada standards-based reading and math tests. A
school is responsible for effectively educating all students regardless
of whether they tested within that school in the previous year or not.
This means that students who transfer from school to school within a
Nevada school district or even between school districts within Nevada
will be included in the growth analysis for schools.

Demographic characteristics of students accompany assessment performance
information. As a result, each student is included when considering the
school as a whole. Additionally, each student receives a race/ethnicity
classification (e.g. American Indian, Asian/Pacific Islander, Hispanic,
African American, White), and if relevant, is classified among several
special student groups (e.g. students with disabilities (IEP), students
with limited English proficiency (LEP), and students receiving
free/reduced priced lunch (Low SES)). As a result, any given student can
impact the analysis of up to five student groups. In Nevada, a
significant percentage of students do in fact contribute to the analysis
of four student groups (e.g. Whole School, Hispanic, LEP, Low SES)[^10].

Like status and Safe Harbor analyses, growth analyses will be conducted
for all student groups within a school and separately by reading and
math. As noted above, these analyses are further constrained to students
enrolled for a full academic year and for whom tracked assessment
records are available.

Growth analyses will encompass students in grades 3 through 8. Grade 3
performance is critical to the analysis of growth, but only as it
relates to grade 4 performance. This is the case because Nevada does not
administer standards-based reading and math tests prior to grade 3.
Similarly, grade 8 performance is included in the growth analysis but
only as an outcome and not as a predictor. There are no grade 9 tests.

An example of the application of growth analyses overtime as applied to
Nevada's elementary schools will show its cumulative and stable use as
predictive of trends. There are two prominent elementary school grade
configurations in Nevada public schools. This includes schools serving a
K-5 student population and schools serving a K-6 student population. As
a consequence, in the first year in which the growth component will
affect AYP determinations, the growth measures for the school as a whole
and for each of its student groups will be aggregated as follows:

(2006-07 Index) - (2005-06 Index) = Basis of growth comparison

> 2006-07 Index = Summation of student level change from grades 3, 4, &
> (5)
>
> (2005-06 performance) to grades 4, 5, & (6) (2006-07
>
> performance).

2005-06 Index = Summation of student level change from grades 3, 4, &
(5)

> (2004-05 performance) to grades 4, 5, & (6) (2005-06
>
> performance) (e.g. baseline).

It is important to highlight the effect of the growth component upon
student transfers between schools, whether within a given school
district or between Nevada school districts within this context. In
practice, this means that middle schools are responsible for the growth
of their students even during the first full academic year within which
they are enrolled. In other words, there is no lag between grades 5 and
6 or between grades 6 and 7 (depending on middle school configuration)
in the estimation of growth so long as the student new to the middle
school has been enrolled since count day (e.g., for a full-academic
year).

Therefore, the proposed model makes schools responsible for the growth
of students even when the school was not the responsible entity for the
previous year's student assessment period. Because of this
responsibility and the varying grade configurations within the state, at
this juncture it is not anticipated that separate growth trajectories
for reading and math will be established for elementary and middle
schools.

As implied by the above discussion, the growth component analyses at
implementation will only consider change in adjacent school years (e.g.
change from 2006 to 2007). The growth trajectory depicts the current
anticipation that annual growth comparisons through 2013-2014 will be
based on the current year compared to the previous year. Growth
expectations will continue after 2013-2014; although, based on an
analysis of performance between 2006 and 2013-2014, some change to
growth expectations is likely to occur.

All middle and elementary schools encompassing grades included in the
analysis of growth will be included in the growth component of the AYP
determination process. Similarly, all Nevada districts will be included
within the growth component of the model since all seventeen school
districts serve elementary and middle schools. As previously noted,
there is no current plan to implement a growth component at the high
school level.

Under the growth model, districts will be treated just the same as
schools within the system with few exceptions. The primary difference is
in the level of aggregation. School districts will be held to the growth
expectations that are defined through the establishment of the 2013-2014
target and baseline based on school aggregate performance. The growth
AMOs used for schools will apply equally to school districts.

At the district level, student achievement evaluation will be similar to
the school-level analysis. Any student who is enrolled in the school
district on a continuous basis from count day to testing window is
included in all achievement analyses including growth analyses at the
school district aggregate. This includes students who transfer among
schools within the school district within the current school year as
well as students who transfer from other school districts within the
state prior to count day.

It is important to note that the performance of students in grades 3
through 8 is included in the complementary status and Safe Harbor
comparisons, regardless of an ability to track an assessment history.
This is true for all students who have been enrolled for a full academic
year within the school. This covers students who are new to the state,
students enrolled in third grade, and students who participate in
Nevada's alternate assessment program (NASAA).

.

[Principle 5 -- Annual State Assessment System and
Methodology]{.underline}

Assumptions about the validity of state assessment systems and
methodologies are being made by states such as Nevada in proposing
growth models. Their assessment systems must be able to reliably support
such analyses and the interpretations made from them. The analyses must
be defensible and must hold utility for accountability and school
improvement efforts (e.g. must be valid). Nevada's testing system and
accountability system are mature and produce valid information. An
examination of the evolution of Nevada's assessment systems and its AYP
determination process supports this assumption. The best place to start
this discussion is to provide a thumbnail sketch of the Nevada
assessments that are and have been included in the AYP determination
process (See Figure 7) since the outset of NCLB.

Nevada administers tests in grades 3 through 8 and in grades 10 through
12. The cornerstone of the assessment system rests upon
criterion-referenced tests in reading and in math administered at each
of those grade levels. Standards-based science tests are also included
in the program in grades 5 and 8. A high school level science assessment
is currently being developed and will be implemented during the 2007-08
school year. Writing performance assessments are administered in grades
5 and 8, where they are scored analytically, as well as in the high
school grade span where they are scored holistically. These
standards-based assessments are administered to all students, and
results from each are used in the AYP model. The standards-based high
school tests must also be passed by students seeking standard Nevada
high school diplomas.

In addition to the standards-based assessments, Nevada administers
norm-referenced tests in grades 4, 7, and 10. Nevada uses the ITBS in
grades 4 and 7 and the ITED in grade 10 and requires tests in English
Language Arts, Mathematics, and Science. These tests serve as critical
evaluation/validation tools

At each grade level tested, students with significant cognitive
disabilities may be eligible to participate using an alternate
assessment (NASAA). The NASAA assessment does yield achievement level
scores that are similar in nature to the achievement level scores
yielded by the CRT and performance tests. For purposes of measuring
status and Safe Harbor, achievement level scores for NASAA are treated
just the same as achievement level scores from the state's traditional
tests. However, as these alternate assessments are built from
specialized achievement standards, results from these alternate
assessments cannot be included in the currently planned analyses of
growth.

Not reflected in Figure 7 are additional specialized or local
assessments used in Nevada. Among these are the Title III Language
Acquisition Assessment used at each grade level, the NAEP assessments
administered in grades 4, 8, and 12, and a variety of local assessments
that are administered by school districts. These assessments are not
currently used in determining AYP, and there is no current plan to
include them in the measurement of growth. As indicated in Figure 7,
only the grade 3 through 8 reading and math tests administered statewide
are to be included in the analysis of growth.

Figure 7. The Nevada Assessment System

![](media/image1.pct)

CRT=Criterion-Referenced Test NRT=Norm-Referenced Test

Perf=Performance Test HSPE (CRT)=High School Proficiency Examination

Each of the shaded tests, grey and green, is used in determining
Adequate Yearly Progress (AYP). ^G^ indicates inclusion in growth

component.

Where ELA is indicated, a combination of reading and writing measures
are used.

\* The HSPE science test is being field tested in 2005-06 with full
implementation slated for 2007-08.

Consistent with the expectations for accountability under NCLB, there
has been a significant shift in recent years in Nevada in the nature and
application of its assessment system. Prior to the advent of NCLB,
school accountability in Nevada was predicated on NRT performance.
Initially NRT performance was used in establishing baseline AMOs used in
the first year of NCLB-related AYP analyses, Since then, Nevada has
increased its reliance on the several criterion referenced tests which
are closely aligned with State content standards.

Changes over the years and the evolution of the full implementation of
the grade 3 through 8 tests are evidenced in Figure 7. Not depicted in
the table is the 2004-2005 full census field administration of the grade
4, 6, and 7 reading and math tests, nor is the focused field test in
2003-2004 included.

In both intent and execution, Nevada's assessment system meets the
specifications of the NCLB act. Nevada has fully complied with the
timing and expectations for rigor in test implementation. Additionally,
significant effort has been devoted to technical assessment issues
ensuring such critical factors as the use of multiple testing formats
and the content alignment of Nevada's tests.

As noted ealier, Nevada is in the process of providing the final pieces
of evidence to comply with the final assessment expectations needed for
approval under NCLB. Nevada received a Pending Approval classification
in June, 2006 and will submit its final evidence to be reviewed in
January of 2007.

[Tests Used in Growth Analysis]{.underline}

As stated previously, only the grade 3 through 8 reading and math tests
are to be included in the growth analyses. This will include grade 3
through 8 performance in which student participation was accommodated.
Accommodated test administrations must be consistent with Individualized
Education Plans for students with disabilities and with individualized
programs of study for students with limited English proficiency.
Additionally, the accommodations must be approved by the Nevada
Department of Education prior to use. Annually Nevada publishes lists of
approved accommodations, rules for their use, and a process for
requesting non-listed accommodations. Modified test administrations,
however, lead to the invalidation of student scores. Invalidated student
scores are classified in the lowest possible level of achievement (e.g.,
emergent low). This classification is also assigned to students who are
expected to test, but who fail to participate.

All standards-based assessments in grades 3 through 8 and at high school
(e.g. CRTs, HSPEs, Perf, Alternate) are included in determining AYP and,
more specifically, in the analysis of status and Safe Harbor. As noted
above, while grade 3 performance is critical over time as a dependent
factor in grade 4 performance, grade 3 performance in any "current"
school year will not affect the growth indices for schools during the
"current" year. Third grade performance will, however, be included in
the calculation of the status and safe harbor indicators each and every
year.

Like grade 3 performance in a current school year, the writing
assessment in grades 5 and 8, all tests in the high school grade span,
and performance from alternate assessments will not be included as part
of the growth component in its initial implementation. Results from
those tests will still be included when calculating status and Safe
Harbor indicators.

[Test Comparability]{.underline}

The appendix includes two excerpted chapters from Nevada's 2005
technical bulletin for its CRT and HSPE examinations in Reading, Math,
and Science.[^11] These chapters provide overviews for the test
development process and the equating process used in Nevada. The test
development steps ensure the alignment between each item and Nevada
content indicators, the minimization of test and item bias, and the
pulling of forms that ensure content comparability across forms. The
complementary equating process also affects test content and is the
primary mechanism used to ensure comparability across forms of the test
in terms of test difficulty. Nevada uses an embedded field test approach
for its live examinations. For administration, this involves a spiraling
of multiple form versions. However, operational items impacting
students' proficiency estimates do not change between spiraled versions.
Hence, the equating process ensures comparability across years.

Nevada uses a horizontal achievement level scale in grades 3 through 8.
Each of its subject area tests in grades 3 through 8 are placed on a
standard scale ranging from 100 to 500 score points. For each test, a
score of 300 scaled points is given a "Meets Standard" classification
(e.g. proficient) and a scaled score of 200 is given an "Approaches
Standard" classification. Any score less than 200 is given a
classification of "Emergent/Developing". To distinguish between "Meets
Standard" and "Exceeds Standard" a horizontal cut point is not possible.
Hence, there are differences used in scaled score requirements by
subject and grade in order to make this distinction. The relationship of
these cut points is depicted graphically below (Figure 8)..

More important in establishing the vertical articulation of the
achievement cut points is the relative consistency in the percentages of
students being classified into the various levels. This is a critical
issue for the growth component and for the status and Safe Harbor
indicators as well. Since the AYP determinations are school-level, this
requires that student performance be aggregated to that level. In most
instances, the students making up that school aggregation are nested
within multiple grades. Without consistency across grade levels and
years, annual changes in performance would be difficult to interpret.

Although the point values making up the horizontal scale of achievement
levels are more or less arbitrary, the difficulty and balance of the
assessments is not. The grade 3, 5, and 8 tests were being developed
prior to NCLB in compliance with the previous reauthorization of the
Federal Elementary and Secondary Education Act. Thus it was known in
advance that results from grades 3 and 5 would likely be combined for
school accountability, but grade 8 would likely stand alone.

For these reasons as well as the timing of implementation, standards or
cut points were set on the grades 3 and 5 tests prior to those for grade
8. In both instances a bookmark procedure was used to set standards. In
short, this involved extensive review of Nevada's content standards and
achievement standards prior to thorough content reviews of the test
content itself. Participants made two rounds of cut point
recommendations prior to being shown impact data.

Figure 8. Horizontal Reading Scale

![](media/image2.wmf){width="6.0in" height="3.85625in"}

Figure 9. Horizontal Math Scale

![](media/image3.wmf){width="5.997916666666667in"
height="3.751388888888889in"}

In the case of grades 3 and 5, the final recommendations from the
standard setting committees were taken to Nevada's State Board of
Education for adoption. The committee recommendations were subsequently
shared with the State Board within a broader policy context. This
resulted in statistical adjustments to the recommended cut points in
order to account for measurement error. It also involved the sharing of
impact data from the Nevada tests as well as from recent NAEP
administrations and from performance on a nationally referenced
examination used in Nevada. Two overarching goals drove the process.
First, the cut points had to be based on a thorough content review
(standards driven). Second, across grades there needed to be consistency
because of the intended accountability use of the tests.

Although NCLB was implemented before standards were set on the grade 8
tests, a replicable and effective process had already been put in place.
Standard setting for grade 8 followed a near identical path as that
taken with grades 3 and 5. The State Board also had access to impact
data from grades 3 and 5 when it adopted cut points for grade 8.

Presented below is a series of graphs that demonstrate the resultant
consistency in impact across grades, years, and subjects. Where
available, data from the 2003, 2004, and 2005 school years is presented.

Figure 10. Reading Consistency Across Grades

![](media/image4.wmf){width="6.002083333333333in"
height="3.8756944444444446in"}

Figure 11. Statistical Comparison of Reading Achievement Levels (2005)

![](media/image5.wmf){width="5.999305555555556in" height="4.1in"}

Figure 12. Statistical Comparison of Reading Achievement Levels (2004)

![](media/image6.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 13. Statistical Comparison of Reading Achievement Levels (2003)

![](media/image7.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 14. Achievement Level Consistency in Reading Across Years (Grade
3)

![](media/image8.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 15. Achievement Level Consistency in Reading Across Years (Grade
5)

![](media/image9.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 16. Achievement Level Consistency in Reading Across Years (Grade
8)

![](media/image10.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 17. Math Consistency Across Grades

![](media/image11.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 18. Statistical Comparison of Math Achievement Levels (2005)

![](media/image12.wmf){width="5.999305555555556in" height="4.1in"}

Figure 19. Statistical Comparison of Math Achievement Levels (2004)

![](media/image13.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 20. Statistical Comparison of Math Achievement Levels (2003)

![](media/image14.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 21. Achievement Level Consistency in Math Across Years (Grade 3)

![](media/image15.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 22. Achievement Level Consistency in Math Across Years (Grade 5)

![](media/image16.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

Figure 23. Achievement Level Consistency in Math Across Years (Grade 8)

![](media/image17.wmf){width="5.995833333333334in"
height="4.094444444444444in"}

The figures depict consistency in terms of the percentages of students
falling into the various achievement levels across grades and school
years, as well as consistency within grade levels across years. This is
true for both reading and math performance. Where shown, error bands are
estimated using the average grade size by school (e.g. average size of
grade 3 enrollment within elementary schools).

The analyses are constrained to grades 3, 5 and 8 for obvious reasons.
The stacked bar charts (figures 10 and 17) provide place holders for
grades 4, 6, and 7. It is anticipated that with the full implementation
of the grade 4, 6, and 7 tests, the accountability system will adapt to
their inclusion. Their inclusion will have demonstrable affects on the
status, Safe Harbor, and the proposed growth indicators.

Because of the established cut points in grades 3, 5, and 8, their
relative consistency over time, and the confidence the State has in
them, standards (cut points) were established for grades 4, 6, and 7
that maintain this consistency while carefully attending to the
alignment between test content and Nevada's content and achievement
standards. To accomplish this, the Nevada Department of Education worked
with its test vendor to conduct a standards validation procedure. The
validation exercise occurred during the last week of May, 2006.
Committee recommendations were presented to Nevada's State Board of
Education during the first week in August, 2006 and were adopted by that
committee. The technical report for the standards validation exercise is
included in the appendix.

In short, the validation process relied on an adapted bookmark
procedure, on recommendations springing from a diverse representation of
educators and other stakeholders, and on a thorough review of test
content, achievement level descriptions, and impact data. Panelists were
provided starting cut points as a referent to their standards-setting
activities. These starting cut points had been developed to ensure grade
3 through 8 consistency. Statistical smoothing procedures were then used
to develop the starting points. These procedures are described in more
detail in the technical documentation.

The cut points for grades 4, 6, and 7 become critical to the measurement
of change across grades and years and are instrumental in developing the
2006 growth baseline. Of course, it is this baseline in conjunction with
the established 2013-2014 growth target that determines the growth
trajectory bridging 2006 with 2013-2014.

The point of this discussion is to reiterate that significant changes to
the assessment system can prompt changes to the AYP determination
process. As stated previously, no change in the grade 3 through 8
program is anticipated at this time and the state/contractor
relationship is a stable one. Therefore, after establishing the growth
trajectory during winter of 2007, no revision to the growth trajectory
is anticipated.

The establishment of the growth trajectory in winter 2007 is purposeful.
Although there is broad-based educational support for the inclusion of
the growth component, it adds a complexity and the field of stakeholders
will have to receive information and training to better understand and
use the information.

Toward this end, the Department plans to produce school-level reports
disaggregated by student groups depicting school growth from 2005 to
2006. In Nevada, field test data is typically not reported, but in rare
circumstances such as this one, it has been reported at the school
level.

The school-level reports will demonstrate to schools what their AYP
classification would have been if a judgment had been made based on 2006
performance. They will also clearly lay out the expectations for 2007.
Although additional information will be included, the graphic display of
data is likely to resemble the presentation of data above in Figure 4.

The State is also planning to augment the current set of reports
provided by its test vendor to include student-level growth reports.
These reports will be shared during the spring of 2007 with school
district staff, principals, and the public at large for input. These
augmented vendor reports will be conceptually similar to the
school-level reports. However, student-level reports will demonstrate
scaled score performance (vertical axis) across a two year period
(horizontal axis) for reading and math separately. The vertical access
will also highlight achievement level cut points so that students and
their parents or guardians will be aware of both scaled score changes
and achievement level changes. Figure 24. depicts a mock-up of the
graphic display of student performance.

In this hypothetical example, the student earned a scaled score value of
290 in the 2006 school year and a scaled score value of 310 in the 2007
school year. As a result, the student depicted moved from the
categorical achievement level of Approaches Standard to Meets Standard.
Additionally, as on all its reports, Nevada will provide textual
information to support score interpretation. That text is will most
likely be placed on the back of the report.

Figure 24. Mock-Up of Student Growth

![](media/image18.wmf){width="6.0in" height="3.058333333333333in"}

The current accountability model will be strengthened by the addition of
the growth component. Together, they will make more explicit two
expectations for individual students. First, the model expects that all
students will be proficient (status expectation) and second, each
student, regardless of starting point, will be expected to make progress
(growth component). The growth component for all students is made
explicit by the relative values placed on movement across the
achievement spectrum. Moreover, the accountability system, based on the
value table and the intended use of the growth component, will gain in
sensitivity to change at the school and student level. It will now
recognize that in any given school year, all students may not be
proficient but so long as students are making significant progress
toward that goal and beyond, they will be recognized and rewarded. This
does not, however, alter the expectation that schools will be
accountable for progress towards their status target AMOs.

Currently, until a student reaches the level of Meets Standard
(proficiency), he or she is expected to make more progress than what
reasonably would be considered a single year's growth. Under the growth
model, even for those students who have met the Meets Standard level of
achievement, there is model incentive for them to continue to grow and
to "Exceed" standards. This change would again constitute making more
than a year's growth.

It is important to emphasize that the growth component of the model must
not only reflect the shared definition of educational success (to move
all students forward regardless of starting point), but be sensitive to
measuring it and to motivating behavior relative to it. There has to be
a communicated expectation that students who meet standards can exceed
standards, and that schools need to be recognized for this success. To
elaborate further, the growth component is not the status component. It
is designed to measure another aspect of academic performance
complementary to absolute achievement expectations. We now use
achievement levels to assist in the valuing of growth and to weight
growth at points in the scale differently. The implementation of the
growth component will recognize that it is just as important to move
proficient students to mastery as it is to move emergent students to
approaching academic expectations. The referent for the growth component
is student growth, not absolute proficiency.

The present model carries the assumption that stasis from year to year
is akin to a year's growth. However, for students that are not at the
level of Meets Standard, little or no value will be given for stasis
below the level of Meets Standard.

No exact quantified assumption is being made regarding the number of
academic years a student in the Emergent/Developing level of achievement
lags behind relative to a student who is at Meets Standard. However, it
is assumed that a student at the lowest level of achievement must make
more progress to attain the Meets Standard achievement level than a
student whose level of achievement is just below the Meets Standard cut
point. Again, because of this difference, the model places value on
movement between adjacent achievement levels but gives greater value to
students who make more progress than those who simply change between two
adjacent levels of achievement.

As noted, student and school characteristics do not affect the growth
AMOs. The AMOs are established by defining the target index values
irrespective of student characteristics, establishing baseline indices
irrespective of student characteristics, and then developing an equal
interval trajectory bridging baseline to the 2013-2014 target.
Similarly, student-level expectations are not moderated by any
student-level characteristic other than the student's level of
achievement in the previous school year. Students at lower levels of
achievement must grow more than students at higher levels of
achievement.

**Principle 6 -- Tracking Student Progress**

A prerequisite condition for the introduction of a valid and robust
longitudinal system is the ability to track units of analysis
effectively and efficiently. Nevada's proposed system requires the State
to track district progress, school progress, and, most importantly,
student progress. While the ability to track individual students is the
key issue, because AYP decisions are being made at the aggregate level,
attention will also be paid to school level and district level tracking.

For many years, Nevada has maintained unique identifiers for its schools
and districts (see appendix) and has shared this information with the
public. Nevada encompasses 17 school districts, contiguous with county
boundaries, and some 600 schools and programs are embedded within those
districts. Based on statutory guidelines and other tracking concerns,
specialized groupings of schools within districts have lead to sub level
district codes providing for the necessary distinctions in identifying
students (e.g. Clark County Charter Schools).

Given the numbers of school districts in the state, a two-digit code is
used to identify school districts[^12]. A five-digit code is used to
identify schools, which includes the two-digit district code and three
additional numbers that uniquely identify the school within the
district. This system for coding schools and districts is longstanding,
and there is no anticipation that the coding scheme will be changed.
Annually new schools are added and at times older schools leave the
system. Given Nevada's size, management of the system is not difficult.

Nevada's student population approximates 400,000. Although on a relative
basis Nevada is a small state, the management of that number of records
over time is challenging. To meet this ongoing challenge, the State
implemented a unique student identification system during the 2004-2005
school year. This automated process is one of several components built
onto the State's student information system (SSIS). That system collects
information on a daily basis from each of Nevada's school districts, and
thus, centralizes data collection from each of its schools.

With the initial implementation of the system, unique numbers were
assigned to students and their existent records within the SSIS. The
process allowed for assignment of more than 95% of the available student
records (in excess of 400,000 student records).

Some revision has occurred and now the system has assigned numbers to
more than 98% of the students in the system. It is important to note
that this level of match has been secured without any individual case
management.

To validate the matching of cases from the SSIS, the Department has
undertaken 5 match studies that use independent criteria to establish a
match; however, the same matching variables are used but the algorithm
or match sequence differs. The match involved a two-year match based on
test results from 2005 and 2006. For each of those studies the match
rate after controlling for students that leave the system and students
that are new to the system exceeded 98%.

Of course, there is still a need to account for the approximate 2% of
missing cases. To do this, the web-based system includes automated case
management steps.

For example, as new students arrive, the system uses required
characteristic fields to be entered locally at the school site for
purposes of student identification. Using sophisticated algorithms, the
automated process checks the presumed new student against all existing
records. If no match is made, a 10-digit unique identifier built from
the characteristic information is assigned. If either a match or
possible matches are identified, this information is fed back to the
school electronically for validation. It is important to note that the
previously identified 2% of unmatched cases require a similar sort of
remediation at the school level. Those cases are being resolved on an
ongoing basis at this time.

The system of automated number assignment effectively tracks students as
they move between schools and school districts within Nevada. The
sophisticated algorithms and storage of longitudinal records in the
State data warehouse enable the state to efficiently reassign student
identifiers for students who leave the state for some period of time and
who return to the system at a later date. (See appendix for technical
information regarding this system component.)

The matching criteria include fields such as student names, date of
birth, demographic characteristics (gender, ethnicity), social security
numbers where available, and other key items. It is important to
indicate that school districts, through their local student information
systems, also employ unique student identifiers. These identifiers are
less robust in tracking students as they move between districts but they
are useful to the state as secondary match criterion. These unique
identifying factors and others enable the state to keep track of
students as they move between schools and districts.

The automated system of assignment and case remediation is in effect a
system of quality assurance. Unique identifiers provided to presumably
new students are assigned only after an exhaustive matching process is
employed. That process utilizes many of the described factors including
phonemic indicators to aid in the search. When clear matches or
confirmation that a student is new to the system cannot be made, the
automated feedback system requires case management to occur at the local
site.

For purposes of accountability and, more specifically, state testing,
the State uses a student pre-identification system. This system at
designated dates pulls population files from the SSIS. This information
is provided to test vendors so that student labels can be produced and
subsequently affixed to student score sheets to aid in the test
administration process. The initial population pull extracts only a
handful of student variables. This includes information pertaining to
student identification and location. An additional "slug" key is
assigned to each student in the pre-id population pull to aid in
subsequent matching tasks.

At a later date, typically during the middle of the week of test
administration, a second pull of data occurs and is shared with the test
vendor. This second pull includes all the relevant demographic
information available in local systems to assist in the coding and
analysis stages for both testing and accountability (e.g. years in
school, ethnicity, special population status, etc.). The lag between
data pulls provides a reasonable opportunity to schools and districts to
remediate student information prior to testing.

On a year-round basis, the state SSIS uses an automated data validation
system. As data are collected daily from school districts, data are
staged and evaluated in a staging server. If problems with the collected
information are identified, feedback reports are issued to local sites
cueing the need for data remediation. For example, out-of-range or
missing values will cue a validation report. This system can not,
however, distinguish the degree of accuracy of coded information when it
is present and within range (e.g. the accuracy of the assigned ethnic
code).

At the time the population pull is shared with test vendors, the system
produces preliminary pre-id files that are shared with the school
districts and schools. During the four- to six-week window bridging the
population pull and the test window, schools and districts can remediate
problems within the preliminary pre-id files so that the information
shared with the test vendors during the week of testing are as accurate
as possible. Again, these files contain all the unique student
identification and demographic information required for AYP analysis in
addition to other variables. Beginning in 2006-2007, the pre-ID files
sent to the test vendor will be augmented to include assessment history
information. (e.g. For the grade 4 population, necessary information
regarding student performance on state tests, such as, scaled scores and
achievement levels by subject area, from grade 3 will be included.). The
sharing of this information with the test vendor will efficiently
streamline the matching process and will allow the test vendor to
produce longitudinal student-level and aggregate reports as described
above.

The weakness of this system is that students who arrive new to a school
just prior to the population pull or between that pull and the test
window are not provided with pre-identification labels. For these
students the schools must complete all the required codes directly on
student answer documents. For students brand new to the state, or to a
school within a state during the month before a testing window,
participation in testing is required, but they are excluded from
achievement level analyses as they will not have been enrolled for a
full-academic year.

Once the test vendors receive the comprehensive sets of student answer
documents, they engage in their own set of quality assurance procedures
that involve extensive data review. A primary concern at this stage is
the identification and removal of duplicate records. Before removal of
records from analysis, this information is remediated with the state,
local school districts, and schools. This process primarily eliminates
instances where students complete different subject area tests on
different score sheets. (Nevada uses comprehensive answer documents in
which multiple tests can be completed using a single answer document.)
This can occur, for instance, when a student is absent for several days
and is asked to make up a missed examination period. In this type of
instance, the records are merged to achieve a complete single record for
the student in question.

Through this process, a clean file is produced by the test vendor which
contains all the critical information required for annual AYP analyses.
This relatively complete file is used to inform preliminary school and
district-level AYP classifications.

As a result of the various steps taken along the way, relatively little
cleanup is required by the state prior to sharing preliminary findings
at the local level. However, several steps are taken during the period
between the receipt of data from the test contractor and the production
and release of preliminary AYP findings. For example, some auditing does
occur prior to this point. For example, remediation will be attempted
for students who show up as having been in the school for more than a
single year but for whom no assessment history is available. The State
can take advantage of its own archived assessment information to fill
these gaps and/or remediate these sorts of problems with schools prior
to sharing preliminary findings.

Additionally, and in keeping with provisions in the NCLB Act, there is a
minimum thirty-day appeal window following the issuance of preliminary
findings in which schools and school districts can dispute state
findings. Once again, this provides opportunities to remediate miscoded
information. Miscoded information has traditionally been the primary
rationale for appeals. It is anticipated that the additional data
remediation steps that are coming to fruition will dramatically minimize
the numbers of appeals.

Given the thorough process described above, very few students who should
be matched will remain unmatched. Infrequently, a student is coded as
being present within a school or school district for more than a school
year when no assessment history is available. Given the sourcing of this
information and the many opportunities for remediation, it is most
likely, even in this instance, that the inaccurate code is the years in
school code and not the lack of assessment history.

Nonetheless, there will be rare instances of this mismatch.
Additionally, there will be rare instances of students who are matched
accurately but for whom an assessment record is missing or, more
accurately, for whom the assessment record indicates non- participation
in a particular year.

For students who fail to participate as required or for students whose
performance is invalidated, an imputation scoring process will be used.
Consistent with current status comparisons, for purposes of
accountability analyses, those students will receive the lowest possible
achievement level categorization and the associated value table score
which will be used in the school level aggregation. No imputation based
on student demographic characteristics is being contemplated.

[Evaluation of the System]{.underline}

As noted above, there is no plan to alter the growth trajectories once
established. However, the tracking system allows for a corpus of
information to be built overtime that is critical to the ongoing
evaluation of performance. First, this information will be used in the
initial several years to compare the relative impact of status, Safe
Harbor, and growth to final AYP determinations. Of primary interest is
the relative contribution of Safe Harbor and growth to AYP
determinations. (i.e. Do they identify different schools?) The
expectation is that with time the growth component can replace Safe
Harbor altogether.

Second, from the corpus of information the state will be able to easily
identify schools that are "beating the odds" and sustaining this
excellence over time. We will also be able to identify schools that are
struggling based on a comparison of progress against the growth targets
but also based on comparisons to (other) schools on a normative basis.

Third, with time the State, school districts, and schools will be better
positioned to estimate the amount of time it will take for students to
achieve proficiency and above, or the amount of academic growth that is
required to become proficient in a set amount of time. These latter uses
will become more important as the state closes in on the 2013-2014
school year and as new students arrive and are included in the system
over time. Additionally, the presumed sensitivity of the growth
component to local efforts will better enable schools to use growth
information, in conjunction with "assessments for learning" (or
formative assessments as opposed to summative accountability measures)
to differentiate instruction and to evaluate the efficacy of
instructional adjustments.

Fourth, in addition to other aspects of the model, the rich corpus of
longitudinal data will enhance the State's ability to assess the impact
of a variety of technical aspects of the system. For example, studies
involving the use of different error estimates can be employed. An
evaluation of the impact of missing data, or conversely imputed values
on accountability decisions should occur. An evaluation of differences
in growth based on student longevity in the system will be conducted.

Finally, the choice of the value table approach was made in part based
on its apparent transparency. It is a complex system and will require
training and support but seems superior in usability and in its
possibilities for stakeholder interpretation to its regression and
hierarchical linear modeling (HLM) methodological counterparts. The
State does intend to evaluate findings from the value added system
through the use of a research-based HLM approach.

[Principle 7 -- Participation rates and the Additional Academic
Indicator]{.underline}

The state proposes to continue to calculate rates of participation as it
has done since the inception of the NCLB AYP process. This calculation
is done separately by student group and by content area. The criteria
for judging participation will not change with the advent of the growth
component.

Like participation, the use of the additional academic indicators
(student average daily attendance and graduation rates) will not change
with the advent of the growth component. As noted previously, the use of
the growth component will be similar to the Safe Harbor calculation,
although growth will be calculated for every student group annually
where data is available. A school must make the additional indicator in
addition to the growth component, assuming the status criterion for the
relevant subject and student group has not been met. The state is not
planning to implement any additional academic indicators at this time.

Conclusion

After the passage of the NCLB Act, USED conducted peer reviews of state
proposed accountability systems. At that time Nevada proposed a system
that included improvement measures as a precursor to an eventual
effectiveness model to be added after the full assessment system was
implemented. But the improvement aspect of the proposed system was not
allowed. Another important aspect of Nevada's original workbook plan,
which was reviewed favorably by the federal government, was the
recognition that Nevada's accountability system was a dynamic system and
not a finished system. The plan addressed how changes to the assessment
system could lead to modifications to the accountability design. It also
laid out ongoing evaluation activities that could derive information to
prompt change.

After implementation of Nevada's accountability system under NCLB,
Nevada continued to express its interest in a more sensitive
accountability model. This was done as part of several national
presentations regarding the implementation of Nevada's system, and more
importantly, by the State Superintendent as he, with many colleagues,
called for amendments to the federal legislation that would allow for
growth.

While expressing these opinions, the State respectfully complied with
the federal legislation. This is evidenced by the fact that as of the
2006 AYP designation process, more than 50% of Nevada's schools had been
identified as failing AYP for at least one year. As predicted in the
original proposal, nearly 50% of Nevada schools were identified as in
need of improvement following the 2006 designations and close to 100
schools designated as being in their third full year of improvement.

The identification of schools can be a positive factor if it stimulates
positive change. The accountability system must be able to support these
positive outcomes over time in order to sustain change. Because of the
sheer numbers of schools being identified and the insensitivity of the
current model in recognizing growth, Nevada's NCLB accountability system
is under growing attack from the field. The credibility of the system is
in peril.

Adding a growth component accomplishes positive goals. First, it allows
for a broader system and one that is more commensurate with the goal of
local educators. This change should increase local buy-in and enhance
the credibility of the system. Second, the growth model promises nothing
in terms of changing the current status of schools or in terms of
minimizing the numbers of schools that will be identified in the future.
But it provides a reason for optimism for schools that have at this
juncture been demoralized by an absolute system they feel incapable of
measuring up to. Third, and most importantly, the growth component
forces attention to be given to all students. Several benefits accrue
from this. The state has received many criticisms from the educational
community regarding the inability to make such drastic changes with
students that are performing at very low levels. By contrast, many
complaints have risen regarding the presumed neglect of gifted and
talented children because the system does not properly recognize
performance above the level of proficiency. The State is also learning
anecdotally of unintended consequences of the current model that could
be overcome by the addition of a growth component. In particular, an
overfocus on "bubble" kids and the use of test-driven practices not
informed by State content standards should be minimized by this
addition.

Nevada's assessment system complies with the provisions of the NCLB Act
and has matured to a point where it can provide greater utility to the
field and to the general reform movement. This system can now be used
for more effective accountability decisions and can provide better
support for schools and districts in evaluating program effectiveness,
professional development needs, and various curricular and instructional
issues.

Appendix

NASS Power Point Presentation

Unique ID Software Specifications

Unique ID Users Manual

Unique ID Administration Manual

Measured Progress Growth Estimation Proposal

Nevada Technical Bulletin--- Chapter 3 Test Development Process

Chapter 9 Equating and Scaling

Measured Progress Standards Validation Technical Report

Nevada's School and School District Numbering System

[^1]: In Nevada, Meets Standard is equivalent to Proficient.

[^2]: The Department intends to request school districts to nominate
    representative educators to complete this process. It is expected
    that elementary and middle school teachers and district staff with
    content expertise will be the primary participants. Teachers and
    district staff with a strong understanding of the State assessment
    system will also be included. Every attempt will be made to
    represent the diversity of Nevada's student population among the
    panelists.

[^3]: There is a trade-off involved in stretching the achievement level
    scale. Reliable classification of students into the various
    achievement levels must be maintained.

[^4]: Hill, R., Gong, B, Marion, S, DePascale, C., Dunn, J., & Simpson,
    M. (2005). Using Value Tables to Explicitly Value Student Growth.
    Paper Presented at the Conference on Longitudinal Modeling of
    Student Achievement; Betebenner, D.W. (2005). Performance Standards
    in Measures of Educational Effectiveness. Working Paper
    (Damian.Betebenner@bc.edu)

[^5]: The cut points between emergent low and emergent high, and between
    approaches low and approaches high, will be determined using two
    pieces of statistical information. First, the cuts will be
    established in a way that maximizes the distribution of students in
    the various levels between grade levels. The cuts will also be
    determined by considering conditional standard errors. In no case
    will the cut fall along a point in the scale that is within a
    standard error of the approaches cut point and the meets standard
    cut point.

[^6]: A student group that began the NCLB cycle with a PAC rate of 10%
    and that subsequently reduced the percentage of non-proficient
    students by 10% every year would have an approximate PAC rate of 74%
    in 2013-14.

[^7]: The index score is derived from the distribution of students in
    the value table. A student's cell location determines their score.
    Student scores are summed and divided by the number of students in
    the given aggregate to establish the aggregates index score (e.g.
    school index, or LEP index).

[^8]: Because students act as their own controls in a longitudinal
    model, we can expect greater score stability and less volatility as
    a consequence of cohort differences. However, the longitudinal
    cohorts themselves differ annually and so taking into account
    sampling error is appropriate and, arguably, necessary to defend
    decisions. Nevada would suggest use of a 95% limit but would
    consider lowering that limit to a single standard error if required
    by the federal government.

[^9]: In Nevada, a full academic year is defined by continuous
    enrollment from count day (early fall) through the testing window
    (spring).

[^10]: Combining grades 3, 4, and 5, more than 21,500 students (22.3% of
    the population) were classified as LEP. Of these students, over 89%
    were classified as Hispanic, and of this subset more than 80% are
    classified as having low SES.

[^11]: Nevada can provide the full technical bulletin from 2005 if
    desirable. The 2006 technical document will be available by December
    15, 2006.

[^12]: Sub district codes include a key that identifies the primary
    school district.