> ![](media/image1.wmf)**UNITED STATES DEPARTMENT OF EDUCATION**

OFFICE OF INSPECTOR GENERAL

Audit Services

April 4, 2008

> **Control Number**
>
> **ED-OIG/A06H0001**

Kerri L. Briggs

Assistant Secretary

Office of Elementary and Secondary Education

U.S. Department of Education

400 Maryland Avenue, SW

Washington, DC 20202

Dear Dr. Briggs:

This **Final** **Audit Report**, entitled *Audit of Selected Portions of
the U.S. Department of Education's Oversight of the Consolidated State
Performance Reports,* presents the results of our audit. The purpose of
the audit was to determine whether the Department of Education's Office
of Elementary and Secondary Education (OESE) provided sufficient
oversight of graduation and dropout rates submitted by states in their
Consolidated State Performance Reports to ensure the rates were
supported by reliable data. Our review covered the reporting period of
July 1, 2003, through June 30, 2004.

BACKGROUND

Sections 9302 and 9303 of the Elementary and Secondary Education Act of
1965 (ESEA), as amended by the No Child Left Behind Act of 2001 (NCLB),
Public Law 107-110, enacted January 8, 2002, provide to states the
option of applying for and reporting on multiple ESEA programs through a
single consolidated application and report.

The NCLB Consolidated State Performance Reports (CSPRs) are required to
be submitted annually to the Department and consist of two information
collections. Part I of the CSPR provides information from the prior
school year related to the five ESEA Goals.[^1] Part II of the CSPR
consists of information related to state activities and the outcomes of
specific ESEA programs.

States report graduation rates for all students and by student subgroups
in Part I of the CSPR. The CSPR instructions require the state to report
graduation rates computed in accordance with the definition approved as
part of the state's accountability plan (i.e., approved Consolidated
State Application Accountability Workbook).

> Graduation Rate is identified as a specific accountability indicator
> for secondary education and is defined in Section 1111 (b)(2)(C)(vi)
> of NCLB ". . . as the percentage of students who graduate from
> secondary school with a regular diploma in the **standard** number of
> years . . . ." (emphasis added)

The National Center for Education Statistics (NCES)[^2] preferred
indicator calculates graduation rates by dividing the number of cohort
graduates in the reporting year by the cohort for that year. A cohort is
defined as "Students who started high school (i.e., ninth grade) plus
student transfers in, less student transfers out in year Y; plus student
transfers in, less student transfers out in year Y+1; plus student
transfers in, less student transfers out in year Y+2; plus student
transfers in, less student transfers out in year Y+3."

The CSPR instructions also require states to report the annual dropout
rates for high school students using the definition of a high school
dropout from the NCES. The annual dropout rate is calculated by dividing
the number of grade 9-12 dropouts during the school year by the total
number of students served in those grades during the school year. The
classification (leaver code) of a student's status (e.g., graduate,
dropout, transfer, or continuing) determines their usage in the
graduation and dropout rate calculations.

AUDIT RESULTS

OESE could have provided better oversight to ensure that graduation and
dropout rates submitted by states in their Consolidated State
Performance Reports were supported by reliable data. Specifically, we
determined that the Department did not put enough emphasis on data
reliability and comparability across states.

In its comments to the draft report, OESE generally agreed with the
finding and most of the recommendations. The comments are summarized at
the end of the finding. The full text of OESE's comments on the draft
report is included as an Attachment to the report. Based on OESE's
comments, we modified four of our recommendations.

FINDING -- More Emphasis Needed on Data Reliability and Comparability
Across States

We conducted four audits (Oklahoma, South Dakota, Texas, and Washington)
of the reliability of the graduation and dropout rates submitted in the
states' 2003-2004 CSPRs. The data used by the states to compute
graduation and dropout rates were not always sufficiently reliable.

### **Graduation Rates Not Supported by Reliable Data**

To determine if states were reporting graduation rates that were
supported by reliable data, we selected two samples of students at each
state audited---one of graduates and one of cohort or graduating year
leavers. We compared our samples of graduates reported by each state to
the supporting documentation and found that the reporting of the number
of graduates for all four states was accurate and supportable. However,
in comparing our second sample of dropouts and transfers during the
cohort period to the supporting documentation, we found significant
errors in the numbers of dropouts and transfers reported by all four
states. Additionally, only one state used a true cohort to compute the
graduation rate. The other three states used data from only one year
instead of four years of data tracked longitudinally that would be
necessary to calculate a cohort.

Although approved by the Department, the different graduation rate
formulas used by the three states did not provide a graduation rate that
was consistent with NCLB. The formulas used by the three states were---

-   A one year cohort alternative graduation rate, which only captured
    dropouts in grade 12 and excluded dropouts in grades 9-11;

-   A synthetic graduation rate, which used dropout data for grades 9-12
    for one school year; and

-   A four-year cohort reported in the state's Accountability Workbook
    that was in fact computed with data from a single year.

These formulas were given approval with no qualifications by the
Department and thus would require no further review and approval unless
changed by a state in its submission of a new Consolidated State
Application Accountability Workbook. Therefore the three states could
continue to use graduation rate formulas that were not consistent with
NCLB. We acknowledge that at the time NCLB was enacted only a few states
had the capacity to calculate true cohort graduation rates because most
states' data systems were not capable of tracking individual students
across years. We believe provisional approval of graduation rate
formulas for these states would provide a means to track states while
they are developing longitudinal tracking systems.[^3]

Additionally, three of the four states erroneously included students who
took longer than the standard number of years to graduate in the
numerator of the graduation rate calculations. Students who take longer
than the standard number of years to graduate do not meet the definition
of an "on-time graduate" and should not be included in the calculation.
Further, the states did not include students who did not graduate but
continued their high school education into the following year in their
graduation rate calculations. Continuing students should be included in
the denominator of the graduation rate calculation because they are part
of the total population served. Including students in the numerator when
they should not be included and excluding students from the denominator
when they should be included result in inflated graduation rates.
Lastly, three of the four states did not have a data collection system
in place to compute graduation rates in accordance with NCLB.

As a result of these problems, we determined that the graduation rates
for all four states were not supported by reliable data.

#### Dropout Rates Not Supported by Reliable Data

To determine if the four states' dropout rates were supported by
reliable data, we selected samples of students in grades 9-12 who were
reported as dropouts for the reporting year, transfers between
districts, and transfers out of state. We compared our samples to
supporting documentation. The dropout rate for only one state was
supported by sufficiently reliable data. The three remaining states'
dropout rates were not supported by reliable data because the states did
not provide adequate guidance, did not adequately monitor the districts,
and did not adequately train personnel.

#### Department Oversight and Guidance Needs To Be Strengthened

We reviewed the applicable portions of the Accountability Workbooks for
the 50 states, the District of Columbia, and Puerto Rico. Only 11 of 52
(21 percent) Accountability Workbooks (1) used an actual cohort where
the students were tracked during their progression through high school
and (2) were in compliance with the law. The majority of the
Accountability Workbooks (32) used an estimated/synthetic cohort
formula. The remaining 9 Accountability Workbooks used 7 different
formulas which did not track a cohort and, among other things, did not
account for continuing students, excluded General Education Development
(GED) certificate recipients from the denominator, used only one year of
data, or did not use the NCES definition of a dropout.

OESE reviewed graduation rates as part of its Title I monitoring. We
reviewed the 52 reports covering school years 2003-2004 through
2005-2006. Only one report identified an error in the calculation of the
state's graduation rate, GED students were erroneously included as
graduates. This is in sharp contrast to the results of our audits, where
none of the four states we reviewed had reliable data to support their
graduation rates, and one state was not using the graduation rate
formula approved by the Department.

We also reviewed the current *Student Achievement and School
Accountability Programs Monitoring Plan for Formula Grant Programs* and
noted steps to check for the presence of procedures addressing data
quality of elements within Adequate Yearly Progress (AYP) and state
report cards. However, we did not find any steps designed to check the
accuracy of the graduation rate formulas or the accuracy of the
underlying supporting data.

Department officials were very careful not to require individual student
tracking because they do not believe NCLB requires it. Although NCLB
does not specifically require individual student tracking, the only way
to know if a student has graduated in the standard number of years is to
track an individual student from entry into high school through
graduation. Since the implementation of NCLB in 2002, the Secretary and
Deputy Secretary issued 32 Policy Letters that deal with some aspect of
NCLB. None of the policy letters provide any specific guidance to the
states on graduation or dropout rates or the importance of a
longitudinal tracking system. If the Department had been more assertive
in requiring states to implement a longitudinal student tracking system
shortly after the enactment of NCLB, all states now could have four
years of student data. Instead, less than a quarter of the states are
using a system that complies with the requirements of the law.

The Department issued Non-Regulatory Guidance, titled *Improving Data
Quality for Title I Standards, Assessments, and Accountability
Reporting,* in April 2006. The Department states in this document---

> A fundamental piece of any data quality infrastructure is a
> standardized set of precise data definitions that all providers use. A
> "data dictionary," which identifies all data elements and describes
> their content, coding options, and format, is essential to
> establishing consistent collection and reporting. Adhering to a
> standard data dictionary improves data quality by fostering
> interoperability of different reporting systems and promoting the use
> of comparable data across the entire State.

The Department has not developed a standard set of data definitions that
all states would be required to follow. Because there is no standard set
of data definitions for the states to follow, states are handling some
identical student statuses in differing manners. An example observed
during our audits related to students returning to their home country.
One state requires only a withdrawal form signed by a parent/guardian
and a school official and then that student is counted as a transfer.
Another state requires a transcript request from the student's new
school before that student can be counted as a transfer; otherwise, the
student must be counted as a dropout.

NCES published a technical report in August 2006, titled "*Users Guide
to Computing High School Graduation Rates"* (NCES 2006-604). It states
that NCLB

> . . . included an on-time graduation rate as an accountability
> reporting requirement. The accurate reporting of such a rate requires
> student record data on student progression from grade to grade, data
> on graduation status, and data on students who transfer in and out of
> a school, district, or state during the high school years, or in other
> words cohort data (National Institute of Statistical Sciences (NISS)
> 2004 Task Force, NCES 2005-105).

The NCLB places emphasis on and strengthens accountability for results.
It also increases the importance of the Department having reliable and
valid data. The data reliability problems we identified occurred because
the Department did not provide adequate guidance on such things as
tracking students from entry into high school through graduation and the
standardization of the data elements used in the graduation rate
formulas. Additionally, the Department did not assess the reliability of
the graduation rate data submitted by the states. Lastly, the Department
approved graduation rate formulas that did not track true cohorts.

Allowing states to use varying formulas not only results in graduation
rates that are inaccurate, but it also results in graduation rates that
are not comparable between states. It is important that the data be
reliable because states, and the public, use the graduation and dropout
rates in evaluating schools' performances. The information can also be
used to assess school, district, and state accountability. Finally, as
one of the indicators in AYP, inaccurate graduation rates could result
in schools being identified as not making AYP when they are or vice
versa.

**Recommendations**

We recommend that the Assistant Secretary for Elementary and Secondary
Education---

1.1 Stress to states the importance of using a longitudinal student
tracking system.

1.2 Disapprove any future graduation rate formulas that do not use a
true cohort except for states that are in the process of developing data
systems that would enable them to calculate a cohort graduation rate.
For those states, we recommend granting provisional approval pending
full implementation of the necessary data systems.

1.3 Continue to impress upon states the importance of data quality and
the need to provide regular guidance, training, and monitoring of their
local educational agencies.

1.4 Publish a list of data definitions that states will be required to
use in NCLB reports. In addition, the published list should include
procedures on how school districts should handle different
classifications of students, (e.g., transfers and returning to home
country) and the documentation requirements to support the
classification.

1.5 Continue to develop and implement written policies and procedures to
monitor states that are in the process of developing data collection
systems to ensure the systems will collect student data in accordance
with the requirements in NCLB.

6.  Periodically review its procedures to monitor the accuracy of the
    data used by the states and districts to compute graduation rates
    and make any necessary adjustments.

**OESE's General Comments**

In its response, OESE generally agreed with our finding, acknowledging
that the subject of the IG Report, graduation rates, is an important
NCLB accountability measure. However, OESE stated that the report, which
looks at data from the 2003-04 school year, does not accurately reflect
the data capacity of states at the time NCLB was signed into law in
January 2002 and does not mention the progress that has been made since.

OESE agreed that reliable graduation rates are a key factor in holding
high schools accountable for graduating their students within four
years. It further believes that a high-quality, uniform graduation rate
would improve the reliability of graduation rate data among the states.
However, OESE stated it was concerned that the draft report does not
provide sufficient perspective on the matter with regard to the early
years of implementing NCLB. The OIG audit focuses on state graduation
data from the 2003-04 school year, the first year that the new
accountability measures under NCLB, including measuring graduation
rates, were in effect.

OESE further stated that it is important to acknowledge that, at the
time NCLB was enacted, only a few states had the capacity to calculate
true cohort graduation rates because most states' data systems were not
capable of tracking individual students across years. OESE also pointed
out that states have made substantial progress in developing their data
systems for collecting and reporting information on graduation rates.

**OIG's Response to General Comments**

We are aware that most states did not have the systems in place to
capture longitudinal data and that progress is being made in this area.
However, our focus was the reliability of the data. Our report addresses
the lack of supporting documentation for and the non-comparability of
the data being reported. This is why our recommendations address the
development of data collection systems which will provide reliable and
comparable information.

**OESE's Comments on the Finding**

OESE agreed that states need to continue their efforts to improve the
quality of data used to compute graduation and dropout rates. However,
OESE stated the report should make clear that NCLB did not establish a
single method of computing graduation rates. There is no one definition
of graduation rate that complies with NCLB, and NCLB does not mandate a
definition that is comparable across states.

OESE further stated that with the states' progress in data systems and
the recent coalescence around the National Governors Association's
(NGA\'s) suggested graduation rate definition, it acknowledged the
benefits of establishing a more uniform and accurate definition of
graduation rate. Also, the Department\'s reauthorization proposal would
require use of the NGA definition. The NCES has calculated and reported
the \"averaged freshman graduation rate\" (AFGR) for each state to
provide more accurate, comparable graduation rate information across all
states. In addition, OESE takes seriously issues that compromise data
quality and has augmented its review of data quality.

**OIG's Response**

It is apparent that the Department desires comparability when it
developed the AFGR. The Deputy Secretary stated in a press release in
November 2005, when the AFGR was posted, that it was an "important first
step towards a national perspective on the success of our high schools .
. . and is comparable across states." While NCLB does not specifically
require states to use definitions of graduation rates that can be
compared across states, the intent of Congress was for the public to use
the information to compare one school to others, even across state
lines. The House Committee stated in House Report 107-404 that:

> *the Committee intends for the Director to consider what would
> comprise an appropriate uniform standard, building upon prior efforts
> by the Department and the National Research Council, by which States
> could report dropout and graduation data, as well as examine
> longitudinal measurements that follow students from 7th grade through
> graduation from secondary school, and support more accurate and
> consistent measures that avoid problems such as distinguishing between
> transfers and dropouts that will allow for meaningful comparisons
> across schools, districts, and States.*

**OESE's Comments on Recommendations and OIG's Response**

**Recommendation 1.1** Stress to states the importance of using a
longitudinal student tracking system.

+---------+------------------------------------------------------------+
| **OESE  | > OESE did not comment on Recommendation 1.1.              |
| Com     |                                                            |
| ments** |                                                            |
+---------+------------------------------------------------------------+
| **OIG   | > No change to the recommendation.                         |
| Res     |                                                            |
| ponse** |                                                            |
+---------+------------------------------------------------------------+

**Recommendation 1.2** Disapprove any future graduation rate formulas
that do not use a true cohort.

+---------+------------------------------------------------------------+
| **OESE  | > OESE stated it recognizes the value and preference for a |
| Com     | > cohort but reasonable flexibility is still necessary for |
| ments** | > states who are unable to calculate a cohort graduation   |
|         | > rate until necessary data systems are in place.          |
+---------+------------------------------------------------------------+
| **OIG   | > We are aware that the Department has the flexibility to  |
| Res     | > waive requirements under the law. However, in our        |
| ponse** | > opinion granting a provisional approval would be more    |
|         | > appropriate than a full approval. Therefore, we modified |
|         | > the recommendation to "Disapprove any future graduation  |
|         | > rate formulas that do not use a true cohort except for   |
|         | > states that are in the process of developing data        |
|         | > systems that would enable them to calculate a cohort     |
|         | > graduation rate. For those states, we recommend granting |
|         | > provisional approval pending full implementation of the  |
|         | > necessary data systems."                                 |
+---------+------------------------------------------------------------+

**Recommendation 1.3** Impress upon states the importance of data
quality and the need to provide regular guidance, training, and
monitoring of their local educational agencies.

+---------+------------------------------------------------------------+
| **OESE  | > OESE agreed with Recommendation 1.3, noting that OESE    |
| Com     | > already has augmented efforts to further impress upon    |
| ments** | > states the importance of data quality.                   |
+---------+------------------------------------------------------------+
| **OIG   | > Based on OESE's comments, we have modified               |
| Res     | > Recommendation 1.3 for OESE to "Continue to impress upon |
| ponse** | > states the importance of data quality and the need to    |
|         | > provide regular guidance, training, and monitoring of    |
|         | > their local educational agencies."                       |
+---------+------------------------------------------------------------+

**Recommendation 1.4** Publish a list of data definitions that states
will be required to use in NCLB reports. In addition, the published list
should include procedures on how school districts should handle
different classifications of students, (e.g., transfers and returning to
home country) and the documentation requirements to support the
classification.

+---------+------------------------------------------------------------+
| **OESE  | > OESE agreed having states use uniform data definitions   |
| Com     | > for NCLB reporting is a worthwhile goal but needs to be  |
| ments** | > balanced with the flexibility the statute grants to      |
|         | > states. However, it stated the non-regulatory guidance,  |
|         | > noted in the body of this report, more appropriately     |
|         | > addressed these concerns.                                |
+---------+------------------------------------------------------------+
| **OIG   | > Flexibility for states to define data elements negates   |
| Res     | > the likelihood of having data that is comparable.        |
| ponse** | > Further, the Non-Regulatory Guidance indicates that a    |
|         | > standardized set of data definitions is a fundamental    |
|         | > piece of any data quality infrastructure but does not    |
|         | > identify or define the data elements. In addition to     |
|         | > noting the importance of standard definitions, OESE      |
|         | > should define those data elements that all states are    |
|         | > required to report to the Department. We did not change  |
|         | > the recommendation.                                      |
+---------+------------------------------------------------------------+

**Recommendation 1.5** Develop and implement written policies and
procedures to monitor states that are in the process of developing data
collection systems to ensure the systems will collect student data in
accordance with the requirements in NCLB.

+---------+------------------------------------------------------------+
| **OESE  | > OESE stated it is not clear that the Department has the  |
| Com     | > authority to monitor the development of state data       |
| ments** | > systems. It further stated that during its monitoring of |
|         | > states it examines whether states have the data          |
|         | > necessary to meet NCLB requirements and require          |
|         | > corrective actions when they do not.                     |
+---------+------------------------------------------------------------+
| **OIG   | > While OESE is not clear it has the authority, its        |
| Res     | > response indicates that it requires states to take       |
| ponse** | > corrective action if the states do not have data         |
|         | > necessary to meet NCLB requirements. It appears from its |
|         | > response that it is requiring states to have the data    |
|         | > necessary to meet NCLB requirements. We have therefore   |
|         | > modified the recommendation for OESE to "Continue to     |
|         | > develop and implement written policies and procedures to |
|         | > monitor states that are in the process of developing     |
|         | > data collection systems to ensure the systems will       |
|         | > collect student data in accordance with the requirements |
|         | > in NCLB."                                                |
+---------+------------------------------------------------------------+

**Recommendation 1.6** Modify the monitoring procedures to check the
accuracy of the data used by the states and districts to compute
graduation rates.

+---------+------------------------------------------------------------+
| **OESE  | > OESE agreed with Recommendation 1.6, stating it has      |
| Com     | > already revised its monitoring indicators for reviewing  |
| ments** | > the procedures states and school districts have in place |
|         | > to ensure data quality.                                  |
+---------+------------------------------------------------------------+
| **OIG   | > Based on OESE's comments, we have modified               |
| Res     | > Recommendation 1.6 to "Periodically review its           |
| ponse** | > procedures to monitor the accuracy of the data used by   |
|         | > the states and districts to compute graduation rates and |
|         | > make any necessary adjustments."                         |
+---------+------------------------------------------------------------+

**OESE Summary of Issues Identified in Individual State Audit Reports**

As part of its response, OESE reiterated several positions taken in the
state audits.

**OIG Response to Summary of Issues**

Comments regarding prior audits were addressed during the process of
issuing the respective reports. Those reports can be obtained from the
OIG Audit Service Reading Room on the internet at
<http://www.ed.gov/about/offices/list/oig/areports.html> or will be
provided upon request. The states reported on, their audit control
numbers, and issue dates are: Texas (Audit Report A06F0020, dated March
21, 2006); South Dakota (Audit Report A06F0021, dated May 7, 2006);
Oklahoma (Audit Report A06G0008, dated October 23, 2006); and Washington
(Audit Report A09G0009, dated November 14, 2006.)

**OESE Miscellaneous Comments**

OESE reiterated its desire to emphasize that no one specific definition
of graduation rate is required by the current statute or regulations and
that the report also appears to imply that it should require states to
implement longitudinal student tracking systems though NCLB does not
require such systems. OESE stated that it was not clear whether the
problem the report is attempting to present is what was approved in
states\' Accountability Workbooks or their implementation of what was
approved.

**OIG Response to Miscellaneous Comments**

While NCLB does not specifically require states to use definitions of
graduation rates that can be compared across states, the intent of
Congress was for the public to use the information to compare one school
to others, even across state lines. While OESE emphasizes that no one
definition is required, in its General Comments at the beginning of its
response, it agreed with the OIG position stating ". . . that a
high-quality, uniform graduation rate would improve the reliability of
graduation rate data among States." Also, it has proposed requiring
every state to use the cohort graduation rate endorsed by the NGA during
reauthorization of NCLB. In addition, in our Perspective Paper, *An OIG
Perspective on Improving Accountability and Integrity in ESEA Programs,*
issued in October, 2007, we address amending the General Education
Provisions Act to establish standard definitions for data quality terms
in ESEA and other laws authorizing Federal education programs.

One of our concerns dealt with the approval of Accountability Workbooks
with graduation rate formulas that did not track a true cohort. We are
aware of the waiver authority. However, granting a provisional approval
would be more appropriate than a full approval.

OBJECTIVE, SCOPE, AND METHODOLOGY

The objective of our audit was to determine whether OESE provided
sufficient oversight of graduation and dropout rates submitted by states
in their CSPRs to ensure the rates were supported by reliable data.

To accomplish our objective, we---

-   Reviewed applicable laws, regulations, and other guidance;

-   Reviewed pertinent pages from the Accountability Workbooks for all
    50 states, the District of Columbia, and Puerto Rico;

-   Reviewed Policy Letters issued by the Secretary and Deputy
    Secretary;

-   Reviewed the Department's monitoring procedures;

-   Analyzed the results of our audits of four states;

-   Reviewed 52 Title I monitoring reports prepared by OESE; and

-   Interviewed OESE officials.

We conducted our fieldwork at the Department of Education in Washington,
D.C., from November 6, 2006, through December 6, 2006. An exit
conference was held with OESE officials on April 16, 2007. We provided
the draft audit report to OESE on June 18, 2007. OESE provided a
response to the draft audit report on August 7, 2007. OESE subsequently
requested to revise its response and provided its revised response on
February 13, 2008.

Our audit was performed in accordance with generally accepted government
auditing standards appropriate to the scope of audit described above.

ADMINISTRATIVE MATTERS

Corrective actions proposed (resolution phase) and implemented (closure
phase) by your office will be monitored and tracked through the
Department's Audit Accountability and Resolution Tracking System
(AARTS). ED policy requires that you develop a final corrective action
plan (CAP) for our review in the automated system within 30 days of the
issuance of this report. The CAP should set forth the specific action
items, and targeted completion dates, necessary to implement final
corrective actions on the findings and recommendations contained in this
final audit report. An electronic copy of this report has been provided
to your Audit Liaison Officer.

In accordance with the Inspector General Act of 1978, as amended, the
Office of Inspector General is required to report to Congress twice a
year on the audits that remain unresolved after six months from the date
of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552),
reports issued by the Office of Inspector General are available to
members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

If you have any questions, please call Gary D. Whitman, Regional
Inspector General for Audit, at 312-730-1620.

Sincerely,

/s/

> Keith West
>
> Assistant Inspector General
>
> for Audit

Attachment

**Attachment** **--** **OESE Response**

+---------+------------------------------------------------------------+
| !       | UNITED STATES DEPARTMENT OF EDUCATION                      |
| [](medi |                                                            |
| a/image | > OFFICE OF ELEMENTARY AND SECONDARY EDUCATION             |
| 2.png){ |                                                            |
| width=" |                                                            |
| 1.01319 |                                                            |
| 4444444 |                                                            |
| 4445in" |                                                            |
| he      |                                                            |
| ight="1 |                                                            |
| .016666 |                                                            |
| 6666666 |                                                            |
| 666in"} |                                                            |
+---------+------------------------------------------------------------+

> FEB 13 2008
>
> TO: George A. Rippey
>
> Acting Assistant Inspector General for Audit

FROM: Kerri L. Briggs, Ph.D /s/

SUBJECT: Draft Audit Report, entitled *Audit of Selected Portions of the
U.S. Department of*

> *Education\'s Oversight of the Consolidated State Performance
> Reports,* Control Number ED-OIG/A0 6H0001

We appreciate the opportunity to review and provide comments on the
draft audit report entitled *Audit of Selected Portions of the U.S.
Department of Education\'s Oversight of the Consolidated State
Performance Reports.* Attached are the Office of Elementary and
Secondary Education\'s updated comments on the draft report, which are
intended to replace earlier comments that we provided. Please let me
know if you have questions.

> Attachment

400 MARYLAND AVE., S.W. WASHINGTON, D.C. 20202\
[[www.ed.gov]{.underline}](http://www.ed.gov/)

> *Our mission is to* ensure *equal access* to education *and to promote
> educational excellence throughout the Nation.*

Draft Audit Report, *Audit of Selected Portions of the U.S. Department
of Education\'s Oversight of the Consolidated State Performance Reports*
(Control Number ED-OIG/A06H0001)

[General Comments]{.underline}

In this draft audit report, the Office of Inspector General (OIG)
examines a very important topic: graduation rates, which are a critical
accountability measure under the No Child Left Behind Act (NCLB).
Through its audit work, OIG raises a number of issues for consideration
by policymakers, both within the Department and in Congress during the
upcoming reauthorization of NCLB. Specifically, OIG recommends the use
of a uniform definition of graduation rate in order to compare
graduation rates among States. OIG also recommends that such uniform
definition follow a cohort of students from entry in ninth grade through
twelfth grade who graduate with a regular diploma in the standard number
of years, and track students who transfer into the cohort and those who
leave the cohort to ensure that they enroll in another school or
educational program that confers a regular diploma; otherwise, they must
be counted as dropouts. OIG also highlights the need for common
definitions of the components used to calculate graduation rates to
facilitate accurate computation and emphasizes the need for high-quality
reliable data in reporting graduation rates, particularly regarding
dropouts and transfers.

We agree with OIG that reliable graduation rates are a key factor in
holding high schools accountable for graduating their students within
four years. Moreover, we believe that a high-quality, uniform graduation
rate would improve the reliability of graduation rate data among the
States. That is why the Administration proposed in its Blueprint for
reauthorizing NCLB that the reauthorization include requiring every
State to use the cohort graduation rate endorsed by the National
Governors Association (NGA). As the Secretary recently noted in her
speech at the National Press Club on the sixth anniversary of NCLB, if
Congress does not act quickly to reauthorize the law, she intends to
take available administrative actions to address areas of concern to
her, including a uniform graduation rate. Although we are hopeful that
reauthorization will occur this year, we are also exploring what
existing authority we may have with regard to securing more uniform and
reliable graduation rate data. We appreciate that the OIG helped to
focus attention on this important issue.

We are concerned, however, that the draft report does not provide
sufficient perspective on the matter with regard to the early years of
implementing NCLB. The OIG audit focused on State graduation data from
the 2003-04 school year, the first year that the new accountability
measures under NCLB, including measuring graduation rates, were in
effect. The Department\'s regulations published in December 2002 made
clear that a State must measure graduation rate from the beginning of
high school. At the same time, the regulations did not require a uniform
definition in order to afford flexibility and because there was not
clear consensus at that time as to what that definition should be.
Moreover, the Department did not mandate that States adopt a definition
that would require individual tracking of students across grades, taking
into account the data systems that States had at the time.

It is important to acknowledge that, at the time NCLB was enacted, only
a few States had the capacity to calculate a true cohort graduation rate
because their data systems were not capable of tracking individual
students across years. Even under the Department\'s more modest
regulations, in 2003-04, few States had the data collection and
reporting capability to calculate a graduation rate that measured
students from entry in ninth grade through high school. Moreover, they
did not have data for the 12th grade class in 2003-04 on dropouts, and
transfers, etc. from the prior three years. As a result, the Department
used its \"transition authority\" to grant most States permission under
section 4(c) of NCLB to use \"proxy\" or \"transitional\" definitions
until States could accumulate four years of data on a given class. The
audit report would be improved significantly if it acknowledged this
perspective more fully, so that readers fully understand the context in
which the OIG\'s findings emerged.

1

In addition, the report should note that substantial progress has been
made by States in developing their data systems for collecting and
reporting information on graduation rates since 2003-04. We believe it
is important for readers to know that improvements have been made to
mitigate the level of concern from the earliest years of the
implementation of NCLB that is cited in the draft report.

[Specific Comments on Findings and Recommendations are as
follows:]{.underline}

1.  [Finding (page 3): More Emphasis Needed on Data Reliability,
    Comparability. and Oversight]{.underline}

The Department agrees that States need to continue their efforts to
improve the quality of data used to compute graduation and dropout
rates. As with any new endeavor, it takes time and iterative efforts to
effectively implement a new system or process.

The report should make clear, however, that NCLB did not establish a
single method of computing graduation rates. There is no one definition
of graduation rate that complies with NCLB and NCLI3 does not mandate a
definition that is comparable across states, a point made in OESE\'s
determinations regarding the audit report on South Dakota (see
additional comments below). It would be very helpful if the report were
clearer on this important fact; otherwise, it may be misleading.

Taking into account the progress States have made in developing more
sophisticated data systems and the recent coalescence around the NGA\'s
suggested definition, we acknowledge the benefits of establishing a more
uniform and accurate definition of graduation rate. As noted above, the
Department\'s reauthorization proposal would require use of the NGA
definition. Moreover, the Department\' s National Center for Educational
Statistics (NCES) has calculated and reported the \"averaged freshman
graduation rate\" for each State to provide more accurate, comparable
graduation rate information across all States. In addition, we note that
issues that compromise data quality are also concerns OESE takes
seriously and that OESE has augmented its review of data quality, both
through monitoring efforts and other efforts including the provision of
guidance on data quality.

2.  [Recommendations (page 6).]{.underline}

[Recommendation 1.2: Disapprove any future graduation rate formulas that
do not use a true cohort.]{.underline} The process OESE has established
for reviewing and approving State graduation rate formulas centers on
State Accountability Workbooks. All States have approved graduation rate
definitions in their current Accountability Workbooks. Any State wishing
to change its definition of graduation rate must request an amendment to
its Accountability Workbook, and we review the amendment requests
against the current statutory and regulatory requirements. We also
recognize the value and preference for a cohort graduation rate.
However, although every State has made progress in enhancing its data
capacity (including support and funding from the Department through the
Institute of Educational Sciences (IES) Statewide Longitudinal Data
Systems Grants), reasonable flexibility is still necessary for States
who are unable to calculate a cohort graduation rate until necessary
data systems are in place. For example, even if we were to require that
States adopt the NGA definition of graduation rate today, we believe
States would need until the 2012-13 school year to fully implement it,
given the need to document students who transfer to another educational
program that culminates in the award of a regular high school diploma in
order to distinguish them from dropouts.

[Recommendation 1.3: Impress upon States the importance of data quality
and the need to provide regular guidance, training, and monitoring of
their local educational agencies.]{.underline} We agree with the
importance of data quality and note that OESE already has augmented
efforts to further impress upon States the importance of data quality.
These augmented efforts include revisions to the monitoring indicators
(i.e., that focus on data quality and holding States accountable for
monitoring activities in their LEAs), the release of *Improving Data
Quality for Title I Standards, Assessments, and Accountability
Reporting: Guidelines for States, LEAs, and Schools, Non-Regulatory
Guidance, April 2006,* and the use of more systematic procedures to
check data submitted through the Consolidated State Performance Reports.

Additionally, we have supported the development and enhancement of State
data systems through the Statewide Longitudinal Data System (SLDS) Grant
Program, administered by the Department\'s IES. As of June 2007, 27
States have received grains totaling nearly S115 million to increase
their ability to efficiently and accurately manage, analyze, and use
education data, including individual student records. The Department has
also worked extensively with States to streamline and improve submission
of State and district data, including achievement data through the
*EDFacts* project. One of the specific purposes of this initiative is to
\"improve State data capabilities by providing resources and technical
assistance.\"

[Recommendation 1.4: Publish a list of data definitions that States will
be required to use in NCLB reports. In addition, the published list
should include procedures on how school districts should handle
different classifications of students. (e.g. transfers and returning to
home country) and the documentation requirements to support the
classification.]{.underline} We agree that having States use uniform
data definitions for NCLB reporting is a worthwhile goal. However, on
this issue, we need to balance the authority the statute grants to the
Department and the flexibility it provides to States. We believe the
guidance OESE issued last year in *Improving Data Quality for Title I
Standards, Assessments, and Accountability Reporting: Guidelines for
States, LEAs, and Schools, Non-Regulatory Guidance, April 2006* strikes
the right balance of providing direction to States yet permitting them
to work through the very complicated issues of defining and collecting
data that yield reliable and valid measures of student and school
performance.

[Recommendation 1.5:Develop and implement written policies and
procedures to monitor States that are in the process of developing data
collection systems to ensure the systems will collect student data in
accordance with the requirements in NCLB.]{.underline} It is not clear
that the Department has the authority to monitor the \"development\" of
State data systems. OESE does examine whether States have the data
necessary to meet NCLB requirements, and OESE requires corrective
actions from States when this is not the case. The Department also has
provided funding through the Statewide Longitudinal Data System Grants
to States who wish to enhance their data systems, which includes the
implementation of individual student records.

[Recommendation 1.6: Modify the monitoring procedures to check the
accuracy of the data used by]{.underline} the [States and districts to
compute graduation rates.]{.underline} OESE has revised its monitoring
indicators for reviewing the procedures States and districts use to
ensure data quality. These revisions provide for a more in-depth
examination of the procedures for ensuring data quality in the areas of
assessments (e.g., administration and scoring), data disseminated, and
AYP data.

[Summary of Issues Identified in Individual State Audit
Reports]{.underline}

The report would be strengthened by providing context for State
reporting of graduation rate data in 2003­04. For example, South Dakota,
one of the four case study states for this audit, is one such State that
is now able to use an improved data system and has amended the
graduation rate section of its Accountability Workbook since data were
collected for this report (see page 28 of the current South Dakota
Accountability Workbook at
[[www.ecl.gov/admins/leal/account/stateplans03/sdcsa.doc).]{.underline}](http://www.ecl.gov/admins/leal/account/stateplans03/sdcsa.doc).)

Finding 1 Approved Graduation Rate Does Not Meet NCLB Requirements

South Dakota, like most States, did not have four years of data
available in 2003-04, the initial year of NCLB implementation and the
year audited, to calculate a graduation rate in accordance with the
regulatory definition in §200.19(a)(1)(i)(A). (Although SDDE had
collected graduation data in prior years, it did so for reporting
purposes only. It did not use these data for NCLB accountability
purposes because it believed the data were not sufficiently reliable.)
As a result, the Secretary used his transition authority in section 4(c)
of NCLB to permit the orderly transition from requirements under the
Improving America\'s Schools Act to NCLB.

In a letter to SDDE dated July 1, 2003, former Under Secretary Eugene
Hickok authorized SDDE to calculate graduation rate on the basis of only
12^th^ grade graduates and dropouts and to continue to do so, adding a
grade each year, until it had four years of data to calculate a cohort
graduation rate consistent with the regulatory definition. The letter
also provided approval of South Dakota\'s plan, consistent with §200.19
of the Title I regulations, to use a definition of graduation rate that
would follow a cohort of students from entry in ninth grade through
graduation in four years. It emphasized that South Dakota\'s graduation
rate had to include all recipients of any type of certificate or diploma
(as well as students who had dropped out of or transferred into a high
school) in the denominator and could include only those students
receiving a regular diploma in the standard number of years in the
numerator.

Given that SDDE, like most other States, received permission from the
Secretary to deviate from the statutory and regulatory definition
because it did not have sufficient data in the early years of NCLB to
calculate a proper rate, we disagree with the auditors\' finding that
South Dakota\'s 2003-04 graduation rate was non-compliant and that South
Dakota\'s approved graduation rate definition violates NCLB\'s
requirements.

As noted above, there is no one definition of graduation rate that
complies with NCLB. The auditors favor a definition like the one
developed by the NGA, as do we now. We have encouraged States to
consider the use of the NGA definition, and have proposed it as the
definition for NCLB reauthorization. The NGA definition is an acceptable
way to compute graduation rate and may, in fact, have some advantages
over the definition used by SDDE. By actually requiring schools to focus
specifically on \"transfers in\" and \"transfers out,\" it would likely
lead to better data quality as students who dropout of school are
sometimes mistakenly coded as transfers. However, South Dakota\'s
definition is consistent with the current statute and regulations. South
Dakota\'s definition:

-   Tracks a cohort of students from entry in 9^th^ grade through
    graduation with a regular diploma after 12^th^ grade.

-   Excludes dropouts as high school completers, as required by
    §200.19(a)(1)(ii). The dropouts who are excluded each year are from
    the class that began in a given school in ~9~th grade. That is why
    SDDE needed to phase in its definition over four years; in
    2002-2003, SDDE had only data for 12^th^ grade completers plus
    dropouts in that cohort of students.

-   Takes into consideration the students that the auditors contend are
    not included.

    -   South Dakota\'s definition does not count students who graduate
        with a GED as graduates; rather, they are included in the
        definition of \"dropout\" because a student cannot qualify to
        take the GED unless the student has already dropped out.

    -   South Dakota\'s definition of \"dropout\" does not include
        \"transfers to another public school district, private school,
        or state- or district-approved educational program (including
        correctional or health facility programs); temporary absence due
        to suspension or school-excused illness; or death.\"

    -   South Dakota\'s definition of \"dropout\" includes students who
        have not \"completed a state approved educational program.\"
        This language includes students who exit high school with only a
        certificate, rather than a regular diploma, and those who do not
        graduate in the standard number of years.

[Miscellaneous Comments]{.underline}

We wish to emphasize that no one specific definition of graduation rate
is required by the current statute or regulations; rather the
regulations allow States some flexibility in defining graduation rate.
Therefore, we suggest that the report acknowledge the flexibilities in
the law related to the findings and recommendations. The report also
appears to imply that we should require States to implement longitudinal
student tracking systems (see page 5), though NCLB does not require such
systems. Findings and recommendations that do not take into account the
proper legal context, including the flexibilities allowed by NCLB, are
not very helpful.

It would be helpful if the methodology section indicated how the four
case-study States were selected.

Paragraph spanning pages 3 to 4: It is not clear whether the problem the
report is attempting to present is what was approved in States\'
accountability workbooks or their implementation of what was approved.

Page 4, Reference to the lack of State data system capacity: This needs
to be put in context of the 2003-04 school year. Additionally, the
report should acknowledge the improvements that have been put in place
by numerous States to enhance their data systems since the passage of
NCLB.

Page 4, second and third full paragraphs, references to data being
unreliable: It appears that the concern is that there was not enough
information to document reliability; this does not confirm
unreliability, however.

Page 10, Washington, graduation rates, and last sentence: It\'s not
clear what \"these deficiencies\" are.

We wish to incorporate by reference all of the comments we have made on
the draft audits that are part of this management audit (i.e., South
Dakota, Texas, Oklahoma and Washington) as well as comments made by
Department staff during interviews and conference calls with OIG staff.
These additional comments, particularly those on the audit reports for
the four states reviewed, should be applied to this report.

We appreciate your efforts in this important area.

[^1]: **Performance Goal 1:** By 2013-2014, all students will reach high
    standards, at a minimum attaining proficiency or better in
    reading/language arts and mathematics. **Performance Goal 2**: All
    limited English proficient students will become proficient in
    English and reach high academic standards, at a minimum attaining
    proficiency or better in reading/language arts and mathematics.
    **Performance Goal 3**: By 2005-2006, all students will be taught by
    highly qualified teachers. **Performance Goal 4**: All students will
    be educated in learning environments that are safe, drug free, and
    conducive to learning. **Performance Goal 5**: All students will
    graduate from high school.

[^2]: NCES is a part of the Institute of Education Sciences within the
    U.S. Department of Education. NCES is the primary federal entity
    responsible for collecting and analyzing data related to education.

[^3]: A data system which tracks a cohort of students over five to seven
    years, from the time they enter Grade 9 or Grade 7 until the fall
    following their anticipated graduation date. (Texas Education
    Agency, *Secondary School Completion and Dropouts in Texas Public
    Schools 2002-03*)