**TO:** Raymond J. Simon

Assistant Secretary for Elementary and Secondary Education

/s/

**FROM:** Richard J. Dowd

Regional Inspector General for Audit

**SUBJECT:** Best Practices for Management Controls Over Scoring of the
State Assessments Required Under the *No Child Left Behind* Act of 2001

This Management Information Report presents information that the U.S.
Department of Education (Department) may find beneficial in helping
states to ensure data quality and reliability. Our review objectives
were to (1) identify any current management controls required by the
Department over scoring of the state assessments under the *No Child
Left Behind* Act of 2001, and (2) provide the Department information on
the types of management controls over scoring used for the National
Assessment of Educational Progress (NAEP) and the benefits of having
standards for a minimum level of controls.

# REVIEW RESULTS

The Department should consider developing and issuing best practices for
management controls over scoring of state assessments required under the
*No Child Left Behind* Act of 2001, such as those used for NAEP, by
states that it has identified as having cost effective controls, or the
*Model Contractor Standards and State Responsibilities for State Testing
Programs*. While the Department has issued *Information Quality
Guidelines*, it has not issued specific guidance regarding the need for
management controls over scoring of state assessments. The *No Child
Left Behind* Act of 2001, Public Law 107-110, enacted January 8, 2002,
places more emphasis on the accountability of results and defines
consequences for schools that do not make adequate yearly progress.
Interviews with officials from the Office of Elementary and Secondary
Education (OESE), NCS Pearson (NCS), Educational Testing Service (ETS),
our NAEP audit work, and review of prior audit reports indicate a need
for the Department to consider developing and issuing best practices for
management controls over scoring of state assessments.

The *No Child Left Behind* Act of 2001, Public Law 107-110, Section
1111(b)(3)(C)(iii), indicates state assessments shall be used for
purposes for which such assessments are valid and reliable, and be
consistent with relevant, nationally recognized professional and
technical standards. In our opinion, for the assessment to be reliable,
the data should be accurate.

Sound business practices require a minimum level of management controls
over scoring the assessments conducted in response to the *No Child Left
Behind* Act of 2001. Management controls over scoring should increase
data quality and data reliability, which are important aspects of the
Government Performance and Results Act. The design of each state's
assessment should include some minimum level of management controls over
scoring to ensure data quality and reliability. The types of controls
would vary by the type of assessment and scoring process (electronic or
manual) and type of question (multiple-choice and/or constructed
response). These controls should cover the receipt and control, data
quality, scoring, and analysis processes. The receipt and control
process would cover the front end of the scoring process. For example,
were all the assessments administered accounted for and available for
scoring? The data quality process would cover (1) quality assurance
steps taken to ensure all assessment booklets were properly entered into
the scoring process and the questions were tied to the correct answer
key, (2) monitoring during the scoring process, and (3) data quality
assurance steps during and after scoring. The scoring process for
constructed response type questions would consider controls such as
scorer qualifications or interrater reliability (a measure of scorer
consistency). The analysis process would cover quality assurance steps
designed to ensure that the data analysis was performed properly and
that data abnormalities were identified and resolved. For example,
plausibility checks could be performed to compare data to expectations,
historical precedent, and data obtained through other analysis methods
to make sure the results make sense.

**The Department and States Could Benefit From Developing and Issuing
Best Practices for Management Controls Over Scoring State Assessments**

Officials from OESE informed us that the Department does not require any
management control standards over scoring of state assessments because
it lacks the statutory and regulatory authority to require standards.
All states have gone through on-site compliance reviews at least once
based on the 1994 reauthorization standards. The Department is currently
updating its Peer Reviewer Guidance for Evaluating Evidence of Final
Assessments Under Title I of the Elementary and Secondary Education Act
to incorporate aspects for the *No Child Left Behind* Act of 2001. The
purpose of this guidance is (1) to inform states of useful evidence that
complies with Title I of the Elementary and Secondary Education Act of
1965, as amended (Title I), final assessment requirements; and (2) to
guide teams of peer reviewers who will examine evidence submitted by the
states and advise the Department on whether a state has met Title I
requirements. The intent of these requirements is to help states develop
comprehensive assessment systems that provide accurate and valid
information for holding districts and schools accountable for student
performance against state standards. While the guidance requires peer
reviewers to examine the evidence that states' administration, scoring,
analysis, and reporting procedures meet high standards, it leaves this
qualitative judgment to the reviewer. Developing and issuing best
practices for management controls over scoring would assist the reviewer
and help the states develop assessment systems that provide accurate and
valid information. The Department issued the *Information Quality
Guidelines* to the states in February 2003 to provide policy and
procedural guidance for maximizing the quality, objectivity, utility,
and integrity of the information that it disseminates. Our review of the
guidelines disclosed that they addressed high-level standards. The
guidelines did not require management controls over scoring of state
assessments. In general, the guidelines sought to have the states take
steps to ensure that the data submitted to the Department was accurate
and reliable. The method used to ensure data accuracy and reliability
was left to the states to determine. Similar to the *Information Quality
Guidelines*, the Department has the authority to issue guidelines for
management controls over scoring.

Management Controls Over Scoring State Assessments Would Address Known
and Potential Problems

Our NAEP audit work and prior audit work performed by the Office of
Inspector General (OIG), General Accounting Office (GAO), and Texas
State Auditor's Office have identified the need for states to improve
their data quality and reliability controls. We performed audit work to
determine whether management controls over scoring of the NAEP 2000
assessment were in place and adequate to provide reasonable assurance
that the assessment results could be relied on during the period October
1, 1999, through September 30, 2000. We determined that the management
controls over scoring of the NAEP 2000 assessment were adequate and
generally working as intended.

However, discussions with ETS and NCS officials, as part of that audit,
indicated management controls over scoring state assessments varied from
state to state and there were no uniform minimum levels of management
controls. States apply fewer and less complex management controls over
scoring compared to NAEP. State funding, timelines, and the complexity
of the assessments affected each state's use of management controls over
scoring. State assessments generally required fewer management controls
over scoring than NAEP because they tended to be less complicated and
easier to score. States used more multiple-choice questions to lower the
cost, the time required, and the risk of error. While the scope of the
NAEP assessment is different than the scope of state assessments, many
of the NAEP management controls could be used as a model of best
practices for controls over scoring of the state assessments. (See the
Appendix to this report for additional information on NAEP 2000 audit
work.)

Prior audits by OIG and GAO identified the need for the Department to
(1) improve its management controls over school improvement data to
ensure the data are reliable, valid, and timely; (2) monitor state
scoring of assessments; and (3) facilitate the sharing of information on
experiences to reduce state expenses. Prior audit work performed by the
OIG and GAO disclosed that the monitoring methods states use to ensure
the accuracy of test contractors' scoring and reporting do not always
provide adequate assurance of complete and accurate scoring results.
Most states hire a contractor to score Title I assessments. About
one-third of these states did not monitor the scoring performed by the
contractor. Some states that hire contractors have found errors in the
contractors' scoring, and in some cases, these errors have had serious
negative consequences for schools and students. Several states reported
problems with scoring and calculation errors, regardless of whether they
had monitoring procedures in place. These states reported that local
district officials, parents, and state agency staff discovered the
errors. (See the Appendix to this report for additional information on
review of prior audit reports.)

Scoring contractors' errors can result in serious negative consequences
for schools and students and lead to costly litigation and negative
public opinion. Scoring errors could potentially jeopardize the
successful implementation of the *No Child Left Behind* Act of 2001.
Reports that include scoring errors would erroneously tell parents,
communities, educators, and boards of education that schools are or are
not doing well. Scoring errors may also have an effect on adequate
yearly progress reported to the Department by the states. The Department
uses adequate yearly progress in its decision making process, which
could affect funding decisions made by states related to needed
improvement, corrective action, or restructuring measures.

School districts and schools that fail to make adequate yearly progress
toward statewide proficiency goals will, over time, be subject to
improvement, corrective action, and restructuring measures aimed at
getting them back on course to meet state standards. Schools that meet
or exceed adequate yearly progress objectives or close achievement gaps
will be eligible for State Academic Achievement Awards. (See Background
for additional information on school improvement, corrective action, and
restructuring measures.)

The Education Leaders Council and Accountability Works have assisted
participating states in developing *Model Contractor Standards and State
Responsibilities for State Testing Programs* (Model) to communicate
expectations for the development and administration of state testing
programs. The Model indicates that with high school diplomas, monetary
awards for schools and school systems dependent on test results, it is
imperative that state assessments be of high quality, meet professional
standards for best practice, be delivered in a timely manner, and be
scored accurately. With increasingly tight budgets, it is similarly
imperative that assessment programs be developed and implemented in an
efficient and cost effective manner without sacrificing quality.

## Suggestion for Enhancing Management Controls over Scoring of State Assessments

We suggest that OESE consider developing and issuing best practices for
management controls over scoring of state assessments required under the
*No Child Left Behind* Act of 2001, such as those used for NAEP, states
that OESE has identified as having cost effective controls, or the
*Model Contractor Standards and State Responsibilities* *for State
Testing Programs*.

# BACKGROUND

## No Child Left Behind Act of 2001

On January 8, 2002, President Bush signed into law the *No Child Left
Behind* Act of 2001. The Act contains the President's four basic
education reform principles: stronger accountability for results,
increased flexibility and local control, expanded options for parents,
and an emphasis on teaching methods that have been proven to work. An
"accountable" education system involves several critical steps. These
steps generally require states to (1) create standards, (2) test
students' progress toward the standards, and (3) hold schools,
districts, and themselves accountable for making adequate yearly
progress toward meeting the state's standards. They also provide that
performance will be publicly reported in district and state report
cards, and that there will be consequences for districts and schools
that continually fail to make adequate yearly progress toward the
standards.

School districts and schools that fail to make adequate yearly progress
toward statewide proficiency goals will, over time, be subject to
improvement, corrective action, and restructuring measures aimed at
getting them back on course to meet State standards. Schools that meet
or exceed adequate yearly progress objectives or close achievement gaps
will be eligible for State Academic Achievement Awards. The *No Child
Left Behind* Act of 2001 significantly strengthens the school
improvement provisions under section 1116 of Title I.

[School Improvement (schools that fail to make adequate yearly progress
for]{.underline} [two consecutive years are subject to improvement
measures)]{.underline}

The *No Child Left Behind* Act of 2001, section 1116(b) establishes the
School Improvement requirements.

-   Requires schools identified for improvement to develop two-year
    improvement plans incorporating strategies from scientifically-based
    research on how to strengthen the core academic subjects and address
    the specific issues that caused the school to be identified for
    improvement.

```{=html}
<!-- -->
```
-   Requires schools identified for improvement to reserve annually at
    least 10 percent of their Title I, Part A, funds for professional
    development that directly addresses the problems that led to
    identification for improvement.

```{=html}
<!-- -->
```
-   Requires local education agencies (LEA) to immediately provide
    students attending schools identified for improvement the option of
    attending another public school, including a public charter school,
    that is not identified for improvement. LEAs must provide or pay for
    transportation to the new school, within certain limits (see 20
    percent requirement below).

```{=html}
<!-- -->
```
-   Permits students attending schools in the second year of school
    improvement (failure to make adequate yearly progress for three
    consecutive years) to use Title I funds to obtain supplemental
    educational services from a State approved public- or private-sector
    provider of their choice. The school improvement provisions cap the
    per-child cost of such services at the lesser of the LEA per-child
    Title I, Part A, allocation or the cost of services.

```{=html}
<!-- -->
```
-   Requires LEAs to "promptly" notify parents of eligible students
    attending schools identified for improvement, corrective action, or
    restructuring of their option to transfer their child to a better
    public school or to obtain supplemental educational services.

```{=html}
<!-- -->
```
-   Requires LEAs to give priority to low-achieving students from
    low-income families in making available choice and supplemental
    educational services. Only low-income children are eligible for
    supplemental services.

```{=html}
<!-- -->
```
-   Requires LEAs to use an amount equal to 20 percent of their Title I,
    Part A, allocations to pay for transportation of students exercising
    a choice option or obtaining supplemental educational services for
    eligible students. If an LEA reserves such funds from its Title I,
    Part A, allocation, the LEA may not reduce allocations to schools
    identified for corrective action or restructuring by more than 15
    percent.

```{=html}
<!-- -->
```
-   Permits a student who transferred to another school under these
    provisions to remain in that school through its highest grade, but
    the LEA is required to provide transportation to the new school only
    as long as the student's original school is subject to school
    improvement, corrective action, or restructuring.

[Corrective Action]{.underline}

The *No Child Left Behind* Act of 2001, section 1116(b)(7) establishes
the Corrective Action requirements.

-   Strengthens corrective action (required after four years of failing
    adequate yearly progress) to include actions more likely to bring
    about meaningful change at the school, such as replacing school
    staff responsible for the continued failure to make adequate yearly
    progress, comprehensive implementation of a new curriculum
    (including professional development), and reorganizing the school
    internally. Corrective action schools also must continue to provide
    choice and supplemental services options to their students.

[Restructuring]{.underline}

The *No Child Left Behind* Act of 2001, section 1116(b)(8) establishes
the Restructuring requirements.

-   Adds a new restructuring requirement for schools that fail to
    respond to corrective actions. If a school fails to make adequate
    yearly progress after one year of corrective action, it must begin
    planning for restructuring, which involves fundamental changes such
    as reopening the school as a public charter school, replacing all or
    most of the school's staff, or turning operation of the school over
    to a private management company with a demonstrated record of
    effectiveness, and implement its restructuring plan the following
    year. Schools identified for restructuring also must continue to
    provide choice and supplemental services options to their students.

## Government Performance and Results Act

This review falls under the context of the Government Performance and
Results Act (GPRA), specifically data quality and reliability. The
Department's Strategic Plan for 2002 -- 2007, goals one and six, are
related to our review objectives and GPRA requirements.

Goal One, Create a Culture of Achievement, is related as it pertains to
linking federal education funding to accountability for results. The *No
Child Left Behind* Act of 2001 will strengthen Title I accountability by
requiring states to implement statewide accountability systems covering
all public schools and students. These systems must be based on
challenging state standards in reading and mathematics, annual testing
for all students in grades 3 through 8, and once in grades 10 through
12, and annual statewide progress objectives ensuring that all students
reach proficiency within 12 years. Assessment results and state progress
objectives must be broken out by poverty, race, ethnicity, disability,
and limited English proficiency to ensure that no group is left behind.
School districts and schools that fail to make adequate yearly progress
toward statewide proficiency goals will, over time, be subject to
improvement, corrective action, and restructuring measures aimed at
getting them back on course to meet state standards. Schools that meet
or exceed adequate yearly progress objectives or close achievement gaps
will be eligible for State Academic Achievement Awards.

Goal Six, Establish Management Excellence, is related in two ways.
First, developing and maintaining management controls, in this case,
controls over scoring state assessments, establish excellence. Second,
achieving budget and performance integration by linking funding
decisions to results, establishes excellence. As indicated previously,
schools that fail to make adequate yearly progress will be subject to
improvement, corrective action, and restructuring measures, which could
affect the state's funding decisions.

# OBJECTIVE, SCOPE, AND METHODOLOGY

The objectives of our review were to (1) identify any current management
controls required by the Department over scoring of the state
assessments under the *No Child Left Behind* Act of 2001 and (2) provide
the Department information on the types of management controls over
scoring used for NAEP and the benefits of having standards for a minimum
level of controls. To accomplish our review objectives, we

1\. interviewed officials from OESE, NCS, and ETS to gain an
understanding of current management control standards over scoring of
the state assessments required under the *No Child Left Behind* Act of
2001,

2\. reviewed background materials, such as:

a.  Prior OIG, GAO, and Texas State Auditor's Office audit reports;

b.  *No Child Left Behind* Act of 2001;

c.  U.S. Department of Education Strategic Plan for 2002 -- 2007;

> d\. U.S. Department of Education, Information Quality Guidelines;
>
> e\. Peer Reviewer Guidance For Evaluating Evidence of Final
> Assessments Under Title I of the Elementary and Secondary Act, dated
> November 1999;

f\. Minnesota State Performance Report for the 2000-2001 school year;

> g\. State of West Virginia Consolidated State Application
> Accountability Workbook, approved April 11, 2003;

h\. Consolidated State Performance Report for State Formula Grant
Programs under the Elementary and Secondary Education Act and Goals
2000: Educate America Act, For reporting on School Years 1999-2000 and
2000-2001, dated October 26, 1999;

> i\. Planning and Evaluation Service, State ESEA Title I Participation
> Information for 1999-2000, Final Summary Report, dated 2002;
>
> j\. Planning and Evaluation Service, State Education Indicators with a
> Focus on Title I 1999-2000; and

3\. relied on prior OIG audit work, *Review of Management Controls Over
Scoring of the National Assessment of Educational Progress 2000*, Audit
Control Number: ED-OIG/A05-C0010.

We performed our fieldwork at OESE's offices in Washington, DC, from
April 29, 2003, through May 1, 2003. We held an exit conference with
OESE officials on June 16, 2003. Our review was performed in accordance
with established OIG standards found in Inspector General Bulletin No.
91-4 (standards for performing services other than audits) that are
appropriate to the scope of the review described above.

No response from your office is necessary regarding the information
contained herein. If you would like to discuss the information presented
in this memorandum or obtain additional information, please contact me
at (312) 886-6503.

# ADDITIONAL MANAGEMENT CONTROL INFORMATION

## NAEP 2000 Audit Work

We performed extensive audit work to determine whether management
controls over scoring of the NAEP 2000 assessment were in place and
adequate to provide reasonable assurance that the assessment results
could be relied on during the period October 1, 1999, through September
30, 2000. Based on the work performed, we determined that the management
controls over scoring of the NAEP 2000 assessment were adequate and
generally working as intended. For additional details, see the OIG final
audit report entitled *Review of Management Controls Over Scoring of the
National Assessment of Educational Progress 2000*, issued June 2003,
under Audit Control Number: ED-OIG/A05-C0010.

## Review of Prior Audit Reports

The prior audit reports that we reviewed included (1) OIG report,
*Improving Title I Data Integrity for Schools Identified for
Improvement*, dated March 2002; (2) GAO report, *Education Needs to
Monitor States\' Scoring of Assessments*, dated April 2002; (3) Texas
State Auditor's Office report, *The Quality of the State\'s Public
Education Accountability Information*, dated May 2002; (4) Joint Audit
Report on *The Status of State Student Assessment Systems and the
Quality of Title I School Accountability Data*, dated August 2002; and
(5) GAO report, *Title I - Characteristics of Tests Will Influence
Expenses; Information Sharing May Help States Realize Efficiencies*,
dated May 2003. The entire reports are available on the OIG, GAO, and
Texas State Auditor's Office web sites respectively,
<http://www.ed.gov/about/offices/list/oig/areports.html>,
[www.gao.gov](http://www.gao.gov/), and
[www.sao.state.tx.us](http://www.sao.state.tx.us/).

OIG Final Audit report, *Improving Title I Data Integrity for Schools
Identified for Improvement*, Audit Control Number ED-OIG/A03-B0025,
dated March 2002, recommended that the Department take steps to improve
its management controls over school improvement data to ensure the data
are reliable, valid, and timely. The Department generally concurred with
the findings and recommendations.

GAO report, *Education Needs to Monitor States\' Scoring of
Assessments*, GAO-02-393, dated April 2002, recommended that when the
Department monitors state compliance with federal programs, it include
checks for contractor monitoring related to Title I, Part A.
Specifically, the Department should include in its new compliance
reviews a check on the controls states have in place to ensure proper
test scoring and the effective implementation of these controls by
states. The Department agreed with GAO's recommendation.

Texas State Auditor's Office audit report, *The Quality of the State\'s
Public Education Accountability Information*, Report No. 02-044, dated
May 2002, identified specific improvements that can be made to further
enhance data collection, processing, and reporting for student
assessment records and results and for the calculation and reporting of
passing rates for accountability purposes. The audit report identified
the need for Texas to monitor the assessment test vendor's data security
and quality controls. Management agreed with the recommendations.

The Joint Audit Report, on *The Status of State Student Assessment
Systems and the Quality of Title I School Accountability Data*, SAO
Report No. 02-064, dated August 2002, indicated that monitoring methods
states use to ensure the accuracy of test contractors' scoring and
reporting do not always provide adequate assurance of complete and
accurate scoring results. Of the 44 states that hire contractors for
test scoring, 16 (about one-third) have no monitoring mechanism to
ensure the accuracy of the contractor's test scoring and reporting.
Management concurred with most of the findings.

GAO report, *Title I - Characteristics of Tests Will Influence Expenses;
Information Sharing May Help States Realize Efficiencies*, GAO-03-389,
dated May 2003, is a GAO study of the costs associated with implementing
the test required under the *No Child Left Behind* Act of 2001. GAO
recommended that the Department facilitate the sharing of information on
states\' experiences in attempting to reduce expenses. The Department
agreed with GAO's recommendation.