No Child Left Behind Act: Education Could Do More to Help States Better Define Graduation Rates and Improve Knowledge about Intervention Strategies
Released: 2005-09-20T00:00:00

Letter

Background

Despite the increasing importance of a high school education, only an estimated two thirds of students graduate from high schools nationwide. Students in certain subgroups, such as the economically disadvantaged and certain racial and ethnic groups, have historically graduated from high school at substantially lower rates than their peers. Students who do not graduate from high school are at a serious disadvantage compared to their peers who do. They are much less likely to obtain good jobs or attend college. The NCLBA includes several requirements for states to improve school and student performance, including measuring high school graduation rates.

NCLBA Requirements

NCLBA expanded the requirements of the Improving America’s Schools Act of 1994 (IASA) for states, school districts, and schools to demonstrate that their students are making adequate progress toward their state’s academic goals. IASA required testing in each of three grade spans to determine whether a school made adequate yearly progress (AYP). NCLBA requires, by the 2005-06 school year, that annual tests in math and reading be administered to students in grades 3 through 8 and once in high school; by 2007-08, students must also be tested in science. In order to make AYP, schools are to show that increasing numbers of students reach the proficient level on state tests and that every student is proficient by 2014. NCLBA also designated specific groups of students for particular focus. These four groups are students who (1) are economically disadvantaged, (2) represent major racial and ethnic groups, (3) have disabilities, and (4) are limited in English proficiency. For a school to make AYP, its student body as a whole and each of the student groups must, at a minimum, meet the state targets for testing proficiency.

Under NCLBA, schools must also use at least one other academic indicator, in addition to annual tests, to measure AYP. High schools must use graduation rate as one of their other academic indicators. The law defines graduation rate as the percentage of students who graduate from secondary school with a regular diploma in the standard number of years. Education officials told us that standard number of years is determined by a state and is generally based on the structure of the school. For example, a high school with grades 9 through 12 would have 4 as its standard number of years while a school with grades 10 through 12 would have 3 as its standard number of years.

NCLBA regulations specifically require a high school, in order to make AYP, to meet or exceed its other academic indicators, including what the state has set as the graduation rate for public high schools. NCLBA does not specify a minimum graduation rate that states must set. States have used a variety of methods to measure AYP on their graduation rate indicator. For example, states have set graduation rate targets or goals or have allowed schools to show progress toward a target or goal as a way for schools to meet the graduation rate indicator requirement. The law does not require states to increase their graduation rate over time.

The law requires states to demonstrate that their definitions produce graduation rates that are valid and reliable. A valid rate would be one that measures what it intends to measure. A reliable rate is one which, with repeated data collections and calculations, produces the same result each time such collections and calculations are performed. A key aspect of the reliability of graduation rates is the quality of the data used to calculate them. The National Center for Education Statistics (NCES), Education’s chief statistical agency, has funded a document that describes the following dimensions for ensuring that data are of high quality: Accuracy. The information must be correct and complete. Data entry procedures must be reliable to ensure that a report will have the same information regardless of who fills it out.

Security. The confidentiality of student and staff records must be ensured and data must be safe.

Utility. The data must provide the right information to answer the question asked.

Timeliness. Deadlines are discussed, and data are entered in a timely manner.

This document suggests that school staff members are responsible for entering data accurately and completely and maintaining data security. It provides ideas for assisting staff to accomplish these tasks, such as sharing best practices with a peer and implementing school-district policies on data security, such as changing passwords frequently.

If schools receiving funding under Title I, Part A of the act do not make AYP—including meeting the state’s requirements for graduation rates—for 2 consecutive years or more, they are “identified for improvement.” They must take certain actions such as offering parents an opportunity to transfer students to a school that had made AYP (school choice). If these schools continue not to make AYP, they must take additional actions, such as providing supplemental services to students—including transportation, tutoring, and training. States and school districts are required to provide funding for such actions up to a maximum specified in law. However, according to Education officials, most high schools do not receive Title I funding, and therefore, if these schools do not make AYP, they are not required to take improvement actions, such as offering school choice or supplemental services. However, NCLBA requires each school district receiving Title I funds to prepare a report card that must contain graduation rates for high school students and is available to the public.

Education’s Responsibilities

Education has responsibility for general oversight of Title I of NCLBA. As part of its oversight effort, Education has implemented the Student Achievement and School Accountability Program for monitoring each states’ administration of Title I programs. This monitoring effort was designed to provide regular and systematic reviews and evaluations of how states provide assistance in terms of funding, resources, and guidance to school districts to ensure that they administer and implement programs in accordance with the law. Monitoring is conducted on a 3-year cycle and addresses high school graduation rates among other requirements. Teams of federal officials visit state offices, interview state officials, and review documentation on how states comply with federal law and regulations. NCLBA also requires the Secretary of Education to report to the Congress annually regarding state progress in implementing various requirements, including the number of schools identified for improvement. Education has required states to report their graduation rates for the state as a whole and for designated student groups.

All states submitted plans to Education as required under NCLBA, which were to include their definitions of graduation rates. By June 2003, Education reviewed and approved all state plans, including their definitions of graduation rates and their statements regarding how such rates were valid and reliable. Education provided many states with approval to use a definition of their choosing until they are able to develop ones that better meet the law’s requirements for defining and measuring graduation rates. Education has also reviewed and approved many amendments to plans submitted by states, including those that make changes to the state’s definition of its graduation rate.

Additionally, NCES commissioned a task force to review issues about definitions, data, and implementation. In its report, the Task Force discussed the data challenges faced by states in calculating their graduation rates. Regarding data used to measure student performance generally, GAO and Education’s Inspector General have commented on the importance of data accuracy.

Dropout Prevention

To attempt to improve graduation rates in high schools or keep students from dropping out of school, Education, state governments, school districts, schools, and foundations have funded or implemented various interventions to address the educational needs of students. Such interventions are based on the idea that many factors influence a student’s decision to drop out of school, such as low grades, socio-economic challenges, and disciplinary problems. These factors may be evident as early as elementary school, and therefore some interventions are designed for these students.

During the late 1980s and through the mid-1990s, Education supported dropout prevention programs across the country. In an attempt to determine which programs effectively reduced the drop out rate, Education conducted several evaluations of these programs. The largest of these was the evaluation of the second phase (1991 to 1996) of the School Dropout Demonstration Assistance Program. This evaluation looked at more than 20 dropout prevention programs including school within a school, alternative middle and high schools, restructuring initiatives, tutoring programs, and GED programs. While two of these programs showed promise in reducing dropout rates—alternative high schools and middle schools—the major finding was that most programs did not reduce dropping out.

In our 2002 report, we identified three intervention approaches to prevent students from dropping out of school: Restructuring schools. This approach modifies a school or all schools in a district through such initiatives as curriculum reform or dividing schools into smaller, more individualized learning communities.

Providing supplemental services. This approach provides additional services such as tutoring or mentoring in language and math; interventions attempt to raise student academic achievement and self esteem.

Creating alternative learning environments. These interventions target at-risk students and attempt to create personalized learning environments, such as career academies that focus the entire school around a specific career theme.

However, our 2002 report found that additional research was needed to document which interventions were particularly successful for certain groups of students. Education agreed that additional rigorous evidence is needed and that it would consider commissioning a systematic review of the literature.

Many States Moving toward Using A Definition That Follows Students over Time; Education’s Guidance Regarding NCLBA Requirements Is Limited

A majority of states used or planned to use a graduation rate definition based on the group of students entering high school who graduate on time, referred to as the cohort definition. Education has assisted states, approved their graduation rate definitions, and given some states more time to develop planned definitions intended to produce more precise results. However, states faced challenges in resolving common data issues and in providing information on how to modify definitions to better account for certain students, such as for those with disabilities.

A Majority of States Used or Planned to Use a Definition That Follows Students over Time

According to state plans, 12 states used a definition that followed a group of students over time from when they entered high school until they left— referred to as the cohort definition. An additional 18 states using other definitions planned to adopt the cohort definition no later than the 2007-08 school year. The cohort definition compares the number of 12th grade graduates with a standard diploma, with the number of students enrolled as 9th graders 4 years earlier, while also taking into account those who left the cohort, such as those who transferred in and out. A study commissioned by NCES found that a cohort definition designed to track individual students over time—from when they enter high school until they leave—could result in a more precise high school graduation rate than one calculated with other definitions. The data in figure 1 show a hypothetical high school class from the time students enrolled in 9th grade until they graduated with a standard diploma, including those who dropped out, transferred, received alternative degrees, continued in school, or took 5 years to graduate.

If the school was in a state that used the cohort definition and considered 4 years to be on-time graduation, its graduation rate would be 60 percent. The 60 percent figure comes from using the number of students who started (100), the net number of transfers over the 4 years, and the number who graduate in 4 years (60). Figure 2 shows the formula of the cohort definition. The year students in the cohort graduate is denoted by “y,” while “T” signifies the net number of students who transfer in and out in any given year. The cohort definitions actually used by states may vary somewhat from the basic definition. For example, Kansas used dropout and transfer data in its definition. Additionally, some states track individual students, while others track groups of students based on the entering 9th grade cohort.

According to state plans, 32 states used a definition of high school graduation rate, referred to as the departure classification definition, based primarily on the number of dropouts over a 4-year period and graduates. Essentially, this definition looks back from a 12th grade class at those who (1) graduated (regardless of when they started high school), (2) dropped out in 9th, 10th, 11th, and 12th grades (including those who enrolled in GED programs) and (3) did not graduate, but received some form of alternative completion certificate. So, using this definition, the data from the high school shown in figure 1 would result in a graduation rate of 65 percent. The 65 percent figure comes from using the number of students who graduated (65), the number who received an alternative certificate (5), and the number who dropped out (30), as shown in Figure 3. Unlike the cohort definition, this definition does not take into consideration the number of students entering high school 4 years earlier. As noted earlier, some of these states (13) planned to adopt the cohort definition by school year 2007-08.

The departure classification definition includes students who drop out. Each of the “D” designations refers to the number of dropouts during one year. For example “D y-2g10” stands for the number of students who dropped out in the 10th grade.

Prior to NCLBA, many states had been using a similar version of this formula, which NCES developed in collaboration with several states. However, earlier definitions used by states may have also included as graduates those who receive GED certificates. Under NCLBA, Education required states to modify the formula so that GED recipients were not counted as graduates.

Different data systems accommodated the use of different definitions. The departure classification definition allowed many states to continue using existing data systems, according to Education officials. Such systems generally collect aggregate data, rather than data at the student level. The cohort definition generally requires states to implement a state-level student tracking system, often with a mechanism that can uniquely identify each student. Such a system identifies students in the 9th grade and tracks them throughout high school, indicating whether they graduate, transfer or drop out. This system also allows for students who transfer into a school to be placed in the proper cohort.

The more specific information required by the cohort definition may result in the calculation of more precise graduation rates than those produced by the departure classification definition. Since the cohort definition follows students entering high school, either by individual students or groups of students, it can better be used to include only on-time graduates. However, how it is implemented may affect the level of precision of the rate calculated. Tracking individual students may result in a more precise rate than tracking groups of students.

In our analysis of one state’s school year 2002-03 data, we found that the variations in data collection and calculations between the two types of definitions, produced different graduation rates. Our analysis showed that the departure classification definition produced a graduation rate that was 12 percent greater than when we used the cohort definition. Because the departure classification definition does not track the entering cohort, it does not account for students who were held back, and therefore differences may result. Our findings are consistent with observations made by other researchers that show differences in graduation rates based on the definition used. In addition, NCES plans to complete a study this year that examines high school graduation rate definitions and how rates differ depending on the definition used.

According to state plans, the remaining eight states that did not use either a departure classification or cohort definition used a variety of other definitions. Five of these states plan to adopt cohort definitions no later than 2007-08.

Figure 4 shows the definitions each state used as April 2005 and planned to use by school year 2007-08.

Wash.

Mont.

N.Dak.

Oreg.

Minn.

Vt.

N.H.

S.Dak.

Wisc.

N.Y.

Mass.

Wyo.

Mich.

R.I.

Nev.

Nebr.

Pa.

Conn.

N.J.

Ill.

Ind.

Calif.

Colo.

Del.

Kans.

Mo.

W.Va.

Va.

Ky.

Md.

D.C.

Tenn.

N.C.

Ariz.

Okla.

N.Mex.

Ark.

S.C.

Miss.

Ala.

Ga.

Tex.

La.

Fla.

Cohort definition (12)

Departure classification definition (32)

Other definition (8)

Wash.

Mont.

N.Dak.

Oreg.

Minn.

Vt.

N.H.

S.Dak.

Wisc.

N.Y.

Mass.

Wyo.

Mich.

R.I.

Nev.

Nebr.

Pa.

Conn.

N.J.

Ill.

Ind.

Calif.

Colo.

Del.

Kans.

Mo.

W.Va.

Va.

Ky.

Md.

D.C.

Ariz.

Okla.

Tenn.

N.C.

N.Mex.

Ark.

S.C.

Miss.

Ala.

Ga.

Tex.

La.

Fla.

Cohort definition (30)

Departure classification definition (19)

Other definition (3)

Most States Allowed Schools to Show Progress toward State Graduation Rate Targets in Order to Meet Graduation Rate Requirements

Most states set graduation rate targets, and many allowed schools to show progress toward these targets as a way for schools to make AYP. NCLBA requires that states set a graduation rate indicator. Most states have set such rates to help determine which schools make AYP. Additionally, many states allow schools to make AYP even if their graduation rates are not as high as the state’s required rate, so long as the school shows progress toward the required rate. States’ graduation rate targets ranged from 50 percent in Nevada to 100 percent in South Carolina, with about half at 80 percent or greater, as shown in figure 5.

Valid comparisons of graduation rate targets across states cannot be made, in part, because of differences in rates used. For example, Alabama and North Carolina both had targets of 90 percent graduation rates. However, Alabama arrived at its target by using a departure classification definition that accounted for dropouts, while North Carolina used a definition that did not account for dropouts.

According to state plans, 36 states considered their schools as meeting their graduation rate requirements if the schools increased their graduation rates from the previous year, known as “showing progress.” In addition, two states allowed their schools to meet such requirements if they maintained the previous year’s rates. A majority of states that allowed progress as a way for schools to demonstrate they met state graduation rate requirements had set no minimum rate of progress. We found instances in which very little progress, less than 1 percent, enabled a school to meet such requirements. Table 1 shows the number of states that allow schools to show progress toward the state goals as a means of meeting state graduation rate requirements, for all states as of the time we completed our review.

By showing progress toward state graduation rate targets, schools can still make AYP even though they do not meet target rates. For example, our analysis of one state’s data from the 2002-03 school year showed that 46 out of 444 high schools made AYP by increasing their graduation rates toward the state graduation rate target of 66 percent rather than by meeting or exceeding this target. Specifically, these schools met or exceeded the state’s requirement for 1 percentage point progress in increasing the graduation rate, even though the schools were below the 66 percent target. Another 232 schools made AYP for the year by meeting or exceeding the target of 66 percent.

In addition, allowing schools to use progress as the NCLBA graduation rate indicator could result in schools making AYP annually, while not meeting state graduation rate targets for decades, if at all. For example, a hypothetical school with a graduation rate of 56 percent can meet the state high school graduation indicator by increasing its graduation rate by 0.1 percent each year. At this rate, the school would not make the state graduation rate target of 66 percent for 100 years.

Education’s Guidance Did Not Specify Modifications Available to Account for Certain Students

Education provided states with assistance with their graduation rate definitions; however, Education’s guidance did not specify modifications available to account for certain types of students. To help states with their definitions, Education developed some guidance and provided support such as on-site peer reviews, conferences, and information posted on its Web site. Education also commissioned a task force that published a report identifying the advantages and disadvantages of different definitions. In addition, Education officials told us they granted states time to develop definitions that met the law’s requirements better for defining and measuring graduation rates.

Education has provided information on how to account for students in special programs and students with disabilities to states that have requested it. Education’s approach has been to provide such information on a case-by-case basis rather than to all states. Education officials stated that they preferred to work with each state’s specific circumstances. However, we found that issues raised, such as students enrolled in 5-year programs, were common to many states. States varied in how they included students enrolled in these programs in their graduation rate definitions. For example, one state counted students in 5 year programs who graduated as dropouts until it received approval to count them as graduates. Another state planned to count such students as graduates without requesting approval to do so. Officials in that state said that since it was unclear what the actual requirements for counting graduates were, they were doing what they believed was allowable under the law. Without guidance on how to account for students in special programs and students with disabilities, there is less consistency among states in how students in these programs are included in graduation rates.

Education also has not provided information to all states on how their definitions can be modified to better accommodate students with disabilities. State plans in 16 of the 52 states indicated that Education approved these states to allow students with disabilities more than the standard number of years to graduate based on the number of years in their Individualized Education Plans. In the 20 states we contacted, we found that they varied in whether they sought approval from Education on how to include students with disabilities in their graduation rate definitions. For example, six of the states we contacted had sought approval from Education to include students with disabilities who need more than the standard number of years to graduate in their graduation rate definitions. In contrast, officials in seven other states contacted told us they did not seek approval for the same issue. Officials in the remaining seven states provided no information on this topic or said it did not apply to them.

Several Factors Affected the Accuracy of Graduation Rates, and Data Quality Remains a Key Challenge

State, school district, and school officials and experts we interviewed reported several factors that affect the accuracy of data used to calculate graduation rates, especially student mobility. While Education has taken steps to assist states and districts in improving the quality of their data, the Department has not reviewed the accuracy of all states’ data, because, at the time of our review, many states were in the process of implementing new definitions, data collection strategies, or both.

Several Factors, Especially Student Mobility, Compromise the Accuracy of Data Used to Calculate High School Graduation Rates

Officials in six schools, three school districts, and three states we visited and several experts we interviewed cited challenges in tracking student mobility, the key factor in calculating accurate high school graduation rates. Some inaccuracies may lead to the reporting of lower graduation rates, such as recording all students with “unknown” status as dropouts or counting students who drop out, return to school, and then drop out again as a dropout each time, as may happen in schools in states that use the departure classification definition. Other inaccuracies may lead to the reporting of higher graduation rates, such as schools’ recording students who drop out as transfers. This may occur when school staff record such students as transfers before they receive documentation that the student actually enrolled in a different school. Since the number of dropouts counts against a school in calculating its graduation rate in many states, schools that record such students as transfers—because they were unaware that the students had actually dropped out—may be reporting inflated graduation rates.

A second factor that affects data accuracy is how staff members understand and follow policies and procedures for recording students as transfers to other schools. For example, staff members in schools in two states reported that they electronically record a student as having transferred to another school on the day that student withdraws from their schools. However, the policy in these states is that a student is to be recorded as having transferred only upon receiving a request for records from the school to which the student transfers. In one of these schools, staff assigned to record student data reported contradictory practices and beliefs about state policy regarding when to record a student as a transfer. One staff member stated that the policy and her practice was to record the student as a transfer upon receiving the records request while another staff member said that no such policy existed and that she recorded the student as a transfer on the day of withdrawal. Therefore, how a student transferring out the school was counted depended on which staff member recorded the student’s data.

The accuracy of data may be further compromised when schools have large numbers of students who transfer in a given year because the more students come and go, the more difficult it is for schools to accurately account for them. Some schools are in areas where families tend to move more frequently. For example, officials in one school we visited near an Army base reported that their school had an enrollment of about 1,200 students and that 187 students had left the school by December of the academic year. The status of 19 of those 187 students was recorded as “unknown” because of difficulty in maintaining contact with these families. The policy in that state was for students whose status is “unknown” (because they could not be contacted) to be counted as dropouts, even if, in fact, the student had transferred to another school. Staff in another school reported the presence of several children from another country. Their experience has been that these particular students report plans to return to their country of origin, but they often do not know the status of these students once they leave the school. The school’s procedure is to record such students as having an “unknown” status, and these are eventually counted as dropouts, unless another school requests their records. Research has shown higher mobility rates among certain subgroups of students compared to all other students, including those who are African-American, Hispanic, Native American, and those classified as having limited English proficiency and as children from migrant families. Consequently, schools with higher concentrations of these subgroups would likely report less accurate graduation rates.

Another factor affecting the accuracy of graduation rate data is the absence of state audits or verification checks. For example, in our survey of state officials, over half (27) reported that their states did not audit the data received from local officials that the state used to calculate high school graduation rates. The lack of such auditing or verification implies that states were likely to be unaware of the extent of certain errors in data—such as students’ indicating they were transferring to another school but not actually doing so—and consequently were unable to ensure that data they received from schools and districts were accurate. Officials in only one of the six schools we visited reported that their data on student transfers had been audited or verified by an outside party, leaving the accuracy of transfer data in the other schools uncertain.

A fourth factor that contributes to challenges in assuring accurate data is the lack of a unique identifier for each student. In our survey, officials in 22 states reported that their state did not have a unique identifier for each of their students. Concerns about using student identifiers include the cost of implementing data systems that support such identifiers and privacy issues. The lack of a unique identifier for students made it difficult to obtain accurate data. Officials in one state that did not use unique identifiers stated that they had to compute graduation rates based on aggregating student data and as a result, they could not track on-time graduates. Officials in another state estimated that they were only 90 percent accurate in identifying students, because, without a unique identifier for each student, they had to use other information. Using this information, such as the student’s name or birth date, can lead to identifying more than one student with the same characteristics, resulting in inaccurate data used in calculating graduation rates.

A fifth factor we found that may affect data accuracy is variation in security and accountability practices. For example, we found that while some schools restricted the ability to change student enrollment information (such as transfers) to one or two people in the building (e.g., a registrar), others allowed many staff members to do so. Further, while some schools’ data systems kept a record of each person who accessed a student’s record and the changes made, other systems did not maintain such information. Without sufficient security and record monitoring, there is a greater risk of inaccurate data being entered and used to calculate graduation rates.

Data Inaccuracies May Affect Schools’ Meeting State Graduation Rate Goals

We analyzed data from one state to estimate the effect of errors of various sizes in reporting dropouts on school graduation rates and found that such errors could raise or lower a school’s graduation rate substantially. This state used a high school graduation definition that incorporated the number of graduates and dropouts in calculating its graduation rate. For example, its median high school in school year 2002-03, with 924 students, reported 41 dropouts and had a graduation rate of 75 percent. We re- estimated its graduation rate after assuming that the school had more dropouts, up to twice as many more than reported. In this case, if the school had 82 dropouts, its graduation rate fell to 64 percent. We also re- estimated its graduation rate after assuming that it had fewer dropouts, as few as half as many dropouts as reported. Thus, if it had 21 dropouts, its graduation rate rose to 88 percent. Figure 6 shows how the estimates of graduation rates were affected by assumed errors in counting dropouts for this school.

Our analysis was performed for all high schools in the state. As expected, when we assumed the number of dropouts was higher than what schools reported, their estimated graduation rates decreased.

Our analysis also found the extent to which schools miscount their dropouts affects their likelihood of reaching the state’s graduation rate target. We estimated that an additional 70 of 444 high schools in the state in school year 2002-03 would not have reached the state target if they were in fact reporting only half of their dropouts. On the other hand, an additional 77 high schools would have reached the state target if in fact their dropout counts were overreported at twice the actual level. According to the NCLBA, high schools that do not meet the state’s requirements for its graduation rate are designated as not making AYP. Such designations if made for 2 or more consecutive years would result in the district’s providing technical assistance to schools that receive Title I funding. Thus, schools that undercount their dropouts may be precluded from receiving the attention and assistance from the state they need to improve students’ school retention and graduation while those with over counts may receive such services unnecessarily.

Education Has Taken Some Steps to Help States with Data Issues, but Data Accuracy Remains a Key Challenge

Education has taken steps to help states address data collection issues. First, Education helped states prepare information to address how their graduation rate definitions were valid and reliable. Education gave instructions in its regulations and in a template given to each state to help states prepare the accountability plans they were to submit to Education for approval in 2003. Education also worked with states on an as-needed basis when state officials had questions about what information the Department needed to review. Education officials indicated that they reviewed information in each state’s plan when they conducted site visits to states as part of the state plan approval process. According to Education, most states were in some stage of transition in calculating their graduation rates: some were implementing plans to transition from their current definition to a cohort indicator; others were improving their data systems; and some were collecting information on designated student groups for the first time. For these states, Education reported that it was unable to meaningfully examine the reliability of data used to calculate the graduation rate because such definitions of such rates had not been in place for a sufficient number of years necessary to determine whether the rate would produce consistent results.

Second, Education, as part of its state monitoring, introduced a data review component to examine data states used for graduation rates, among other aspects of their participation in the Title I program. As of August 2005, Education had monitored and reported on 29 states, and expected to monitor the remaining states by the end of fiscal year 2006 as part of its 3-year monitoring plan. This monitoring consisted of broad questions intended to collect information about how states corrected or addressed errors in student data received from districts and schools, including data used to calculate graduation rates. The monitoring was also designed to identify written procedures states used to confirm the accuracy of their data, the extent to which these procedures were communicated to districts, and how data validity issues related to schools and districts have been addressed. According to Education officials, their reviews of the nine states identified no significant problems with data systems these states used to calculate high school graduation rates.

Third, in response to recommendations from GAO and Education’s Inspector General, Education contracted with a firm to develop a guide to help states improve data collection processes. According to Education officials, this guide is to consist of three parts. One part is designed for state officials and is to focus on the design and implementation of data systems. A second part, which focuses on data management issues such as methods for verifying the accuracy of data, is designed for district and school officials. A third part summarizes the first two parts and is to be suitable for oral presentation to state, district, and school officials. According to department officials, this guide will be issued by the end of 2005.

Although Education monitors states to determine if they have written procedures for ensuring data quality and have methods to address data quality issues, it does not evaluate other methods of ensuring data accuracy. For example, it does not assess whether states ensure that districts and schools have effective controls to accurately record student status, including transfers. Further, Education’s monitoring approach does not capture whether states ensure that schools have computer controls that allow only authorized staff to make changes to student data. Department officials said that the guide it is developing is planned to address these issues.

However, departmental efforts have not resolved immediate data accuracy problems. In July 2005, Education announced that it planned to calculate and report interim graduation rate estimates for each state to provide a nationwide, comprehensive perspective. Education stated that the interim rate that it developed, based on data NCES collects from states, will provide more accurate on-time graduation rates. Some states’ graduation rates rely on the same data reported to NCES, while other states rely on different data. However, these states also provide data that are requested by NCES. The quality of the data states provide to NCES varies across states depending, in part, on the extensiveness and rigor of their internal controls and other data verification checks. Because Education plans to rely on state-reported data to calculate interim graduation rates, the accuracy of such data is critical.

Few Interventions Have Been Rigorously Evaluated, and Education Has Done Little to Evaluate and Disseminate Existing Research

While states and school districts have implemented numerous interventions designed to increase high school graduation rates, few of these programs have been rigorously evaluated, and Education has done little to evaluate and disseminate existing research. Several of the interventions that have been rigorously evaluated have shown potential to increase graduation rates. In addition to these interventions, schools are trying other approaches to enhance students’ chances of success, though the effectiveness of these approaches has not been demonstrated.

About one third of students who enter high school do not graduate and are likely to earn less money, are more frequently unemployed, and are more likely to receive public assistance compared with those who graduate from high school. In response, some schools and districts have implemented programs to address the factors that influence a student’s decision not to complete high school.

Research has shown that a student’s decision to leave school may be affected by experiences that begin as early as elementary school. For example, studies have shown that students who are not at least moderately skilled at reading by the end of 3rd grade are less likely to graduate from high school. Besides basic literacy skills, there are a variety of other academic and family-related factors that contribute to whether a student graduates. For example, poor grades and attendance, school disciplinary problems, and failure to advance to the next grade can all gradually lead to disengagement from school and result in a student not finishing high school. In addition to these academic factors, students from low-income backgrounds, students with low levels of self esteem, or students with a learning or behavioral disability drop out at a much higher rate than other students.

Schools and districts have implemented a range of interventions to address these factors and they vary in scope from redesigning the structure of an entire school to an individual school’s mentoring program. While there is variability among interventions, most generally fall into one of the three following categories that we identified in our 2002 report: (1) school wide restructuring efforts; (2) alternative forms of education for students who do not do well in a regular classroom; and (3) supplemental services, such as mentoring or tutoring services, for at-risk students. While most of the schools we visited used interventions from only one of the three categories identified above, some schools combined aspects of these categories. (See table 2 for a complete list).

Few Interventions Have Been Rigorously Evaluated, Though Some Showed Potential to Increase Graduation Rates

Several of the programs at schools we visited have conducted evaluations of how they affect high school completion, while others are reporting positive results on other outcomes such as attendance or academic performance. We identified and reviewed five intervention evaluations that used a rigorous research design and have shown potential to increase graduation rates. We visited schools that had implemented three of these programs. In addition, we visited other schools that were trying other interventions that experts and Education officials noted were promising for improving high school graduation rates. While the effectiveness of these approaches to increase graduation rates had not been demonstrated, research does point towards the possibility that these interventions may help increase high school completion.

The three schools we visited whose programs displayed positive results all used a rigorous research design. However, evaluations of the effectiveness of these interventions are not as strong as they need to be for results to be conclusive. For example, design limitations or data collection concerns were evident during our review of these evaluations. It is worth keeping in mind that research of this nature is limited in the education field due to a variety of factors, and these studies represent some of the most promising research on graduation rate interventions available.

Promising Approaches: Check and Connect, Project GRAD, Help One Student to Succeed (HOSTS), Talent Development, and First Things First

In our visits to 16 school programs in 6 states, we observed 3 interventions where research has indicated potential for improving high school graduation rates. These interventions addressed a variety of student risk factors and provided services to students in elementary through high school.

One school we visited in Minneapolis, Minnesota, had implemented the Check and Connect program which provides mentoring services in an alternative-learning environment. The program began in 1990 with a model developed for urban middle school students with learning and behavioral challenges. It has since been expanded to serve additional at-risk populations as well. This intervention is designed around a mentor who acts as both an advocate and service coordinator for students who have been referred into the program due to excessive absences combined with poor academic performance and behavioral problems. Program officials noted that the mentors offer around-the-clock services including monitoring school performance, regularly checking student data (attendance, grades, and suspensions), and identifying and addressing out of school issues. The mentor also regularly communicates with the student’s parents or relatives to ensure that the whole family is engaged in the student’s education.

The mentoring is built into a program model that relies on several inter- related features including relationship building, individualized and timely intervention, and long-term commitment. A complete listing of program features can be seen in table 3.

The school we visited in Minneapolis had 220 students in the program during the 2004-05 school year. Program officials noted that students in the program were divided among four mentors and had two separate classrooms they could use to meet with their mentor or to study between classes. The program had no set schedule for the student—it was the responsibility of the mentor to make sure they followed up with the students, parents, teachers, courts or counselors on a regular basis. A student in the program noted that Check and Connect helps because it “provides someone who cares how you do and keeps after you about coming to school and doing well academically.”

A school official remarked that both attendance and retention rates had improved significantly since the program was implemented. An evaluation of program impacts on students with emotional and behavioral disabilities showed that students participating in Check and Connect were more likely than students not participating to have either completed high school, including GED certification, or be enrolled in an educational program. While graduation rates are not available yet for the first Check and Connect cohort at the school we visited, a teacher at the school commented that the staff knows that the program is working “because the students are coming to class everyday.” School officials noted that the program is funded through a renewable grant from a private foundation.

Another program we visited, Project GRAD (Graduation Really Achieves Dreams), seeks to ensure a quality public education for students in economically disadvantaged communities through school restructuring, curriculum reform, and social services. The goal of the program is to increase high school graduation rates in Project GRAD schools to at least 80 percent, with 50 percent of those students entering and completing college.

Originally established in 1989 as a scholarship program, it has since developed into a replicable and comprehensive k-12 school reform model. The reform design relies on two components—a structural model and an instructional model. Structural components include an independent local organization to provide implementation oversight, and community involvement such as mentoring, tutoring, and financial support. Figure 7 shows Project GRAD’s structural components.

Local Project GRAD sites—such as one located in Atlanta—also used the instructional component of the model, which emphasizes specific reading and math programs for students in kindergarten through 8th grade. Program officials commented that this component also incorporates campus based social services (which focus on dropout prevention as well as family case management), classroom management techniques, and college scholarships to all high school students who qualify.

In 2004, the local Atlanta site served 29 schools and approximately 17,000 students in the inner city. Officials at one of Atlanta’s schools noted that the program provided additional outreach staff to advocate on behalf of students and address other issues that may interfere with a student’s ability to attend school and learn. Students at the school, commenting on the program’s effect on their lives, noted that the program should be expanded to all of the schools in the district because of the opportunities it offers students. Project GRAD-Atlanta officials noted that the effectiveness of the program has been demonstrated through higher test scores and increased college attendance since implementing Project GRAD in these schools. Additionally, the results of an independent evaluation of Project GRAD also suggest an increase in students’ test scores and graduation rates. However, aspects of the study’s design may limit the strength of study findings.

The Project GRAD—Atlanta model relies on a mix of public funding and private local fundraising. As of school year 2003-04, Project GRAD had also been replicated in feeder systems in Akron, Ohio; Brownsville, Tex.; Cincinnati, Ohio; Columbus, Ohio; Houston, Tex.; Kenai Peninsula, Alaska; Knoxville, Tenn.; Lorain, Ohio; Los Angeles, Calif.; Newark, N.J. and Roosevelt, N.Y.

We also visited a school that had implemented the language arts component of the HOSTS program, an intervention focused on literacy, an area that research has linked to students’ graduating. This program is a structured tutoring program in reading and language arts that targets low performing elementary students whose reading skills are below grade level. School officials at the elementary school we visited noted that they had been using the program for 7 years to increase at-risk student’s reading scores as well as raise their self esteem. The 90 students in the program worked individually with a tutor 4 days a week for 30 minutes each day. School officials considered the program a success because of the number of students who successfully transitioned into grade level reading in the regular classroom. The program, which has been replicated in schools or districts in 12 states, was cited in the report language of the NCLBA as a scientifically based intervention that has assisted schools in improving student achievement. A recent study of the program in nine Michigan elementary schools suggests reading improvement for students at schools participating in HOSTS programs. While this study displayed some promising results for elementary literacy, students were not tracked over time to determine its effect on high school graduation rates.

Two recently completed rigorous program evaluations also displayed promising results for increasing graduations rates. These two programs, the Talent Development Model and First Things First, are both comprehensive school reform initiatives with numerous components.

The Talent Development program in Philadelphia, Pennsylvania, is designed to improve large urban high schools that face serious problems with attendance, discipline, achievement scores, and graduation rates. The program has been implemented in twenty districts nationwide and consists of several different components including a separate career academy for all 9th graders, career academies for students in 10th through 12th grades, block scheduling (4 courses a semester, each 80-90 minutes long) and an after hours program for students with attendance or behavioral problems. An evaluation of the first five schools in Philadelphia to implement the Talent Development program suggest that it may have contributed to increasing the graduation rate for two high schools compared with other high schools in the district that did not implement the program.

The First Things First program was first launched in Kansas City, Kansas, and has since been tested in 12 middle schools and high schools in four additional districts. The program has three central components: small learning communities of up to 350 students, a family advocate system that pairs students with a staff member who monitors their progress, and instructional improvement that aims to make lessons more rigorous and better aligned with state and local standards. A recent evaluation in Kansas City schools suggests that students in the four high schools with First Things First had increased reading and math scores, improved attendance, lowered dropout rates, and increased graduation rates compared with schools that did not participate in the program. For middle schools in Kansas City, the study found increased reading and math scores and somewhat improved attendance compared with other scores. However, the research did not show significance differences in the First Things First schools when compared with other schools in two other school districts.

Approaches Selected Schools Are Trying to Enhance Students Chances for Success

In addition to the 3 school programs we visited whose rigorous evaluations displayed potential for increasing graduation rates, we also visited 13 other school programs which experts, Education officials, and evaluations noted were promising. While the effectiveness of these approaches has not been demonstrated, research points toward the possibility that these interventions may help increase high school completion. These other school programs generally focused on one specific approach which generally fell into one of three categories—school restructuring, alternative learning environment, and supplemental services. Selected programs that illustrate these approaches are discussed below.

School-Restructuring Efforts: Making Schools Smaller

Schools and districts used schoolwide restructuring to change a school or all schools in the district to provide a more personalized education and increase graduation rates. Schoolwide restructuring efforts are generally implemented in schools or districts that have a history of high dropout rates.

One restructuring approach is to create many small schools from larger low performing schools. For example, the New Century High Schools Consortium for New York City is a New York Public School’s small schools initiative that is funded by the Bill and Melinda Gates Foundation, the Carnegie Corporation of New York, and the Open Society Institute. School officials commented that the project began in the Bronx with the conversion of six low performing high schools that served between 1,500 and 3,000 students each. This intervention began in 2001 and, as of September 2004, New York City had created 77 small schools.

One of those schools, Morris High School, has been a part of this program since the small schools program begun in 2001. School officials noted that the school has been divided into several small schools including the Bronx International High School and the Bronx Leadership Academy, which serve 300 and 252 students respectively. While housed in the same building, each school has a different curriculum and student population. For example, the Bronx International High School provides an intensive English language program for recent immigrants while the Bronx Leadership Academy offers a science-based curriculum for college bound students. The core concepts for both these programs are the small school size, team approach to teaching, and school-based learning that also has relevance within their community. A student at the school noted that the small groups they work in allow students to help and support each other, something that did not happen in junior high school. School officials commented that teacher investment in the school is expected and is often displayed by working overtime, serving as counselors to students, and participating in school governance. Additionally, the project-based curriculum is developed by teacher teams who work collaboratively to plan activities for incoming students.

School officials did not indicate a plan for a formal outcome-based evaluation of the schools; however, they did consider the intervention a success based on positive improvement in a number of areas including higher percentages of students meeting state standards, higher attendance rates, and higher passing grades. The New York City Department of Education reported similar results for small schools throughout the district including more students advancing from 9th to 10th grade and higher attendance rates. While these results provide a snapshot of some possible benefits of New York’s school reform initiative, it is still too early to look at student outcomes. The Gates Foundation has commissioned an 8-year evaluation of the small schools program.

Alternative-Learning Environment: Providing Individualized Education

States and school districts are also using alternative learning environments for students at risk of school failure. These interventions are designed to foster a supportive school environment through small enrollments, one-on- one interaction, flexible schedules, and structures, and a curriculum that appeals to students’ interests. Often, enrollment is limited and the programs are tailored to individual students’ needs to ensure that they graduate.

One type of alternative learning environment, the career academy, is focused on keeping students in school by providing an interesting curriculum focused on a specific career theme. For example, Aviation High School in Washington State is an aviation-themed public high school housed at a local community college. School officials noted that the school addresses a range of student risk factors, including those related to academics (learning and literacy), social issues (attendance and behavior), and family (counseling and strategies for living with drug addicted family members). With a 2004 enrollment of only 103 students, Aviation High School offers small class sizes, aviation themed curriculum, and mentoring opportunities. (See figure 8 for an example of a school event focused on aviation).

Additionally, school officials report that each teacher at the high school serves as a student advisor who assists students with academic, social, and emotional development. Students noted that while transportation to the school was challenging due to its distance from their home, they still selected the program because of the aviation curriculum, the personalized attention they received, and the highly motivated students at the school.

Aviation High School officials indicated that it is too soon to tell the impact of the program, but they noted that the school will be included in a national evaluation to be conducted by the Gates foundation. Research on career academies has demonstrated positive gains for employment and earnings for graduates, but also found that high school completion rates of career and non career academy students were not significantly different.

Alternative learning environments may also allow students to tailor their learning experience to individual needs that are not being met in traditional schools. For example, we visited an alternative high school in Atlanta, Georgia, that uses a computer-based instructional program designed for students to learn the state-certified curriculum at their own pace. Students rotate through classrooms, each of which contains a different computer module for the particular subject being taught. Students received assistance from teachers as needed. According to officials, the school is made up of a team of 6 teachers and 75 at-risk 11th and 12th grade students (for the 2004-05 school year). The school’s enrollment is composed of students who were referred to the school either through other schools, court tribunals, or parents. School officials noted that the program also includes a motivational component. For example, each school morning begins with an assembly where students discuss the obstacles they have had to overcome and the people who have helped make a difference in the world. After the assembly, students get up and shake hands with each other and then move to their first hour class. School personnel stated that this allows students to begin each day with confidence and prepares them to learn. School officials noted that the school’s graduation rate, which they stated was consistently over 90 percent, indicated that the program was effective.

Research on alternative programs in general has shown some promising outcomes. For example, an evaluation of 8 middle school dropout prevention programs showed some positive impacts on dropout rates, grade promotion, grades, and test scores for students in alternative programs. The same study also looked at five alternative high school programs and found limited evidence that these programs reduced dropout rates, but did note that alternative programs oriented toward GED certificates experience were more effective than those oriented toward high school diplomas.

Supplemental Services: Targeting Literacy and Self- Esteem

Several schools we visited used targeted supplemental services to provide at-risk students with extra help. These services aim to improve students’ academic performance, acclimate them to a new culture, or increase their self-esteem. Supplemental service programs are offered at all grade levels, with research showing the importance of building academic and social skills at an early age.

Supplemental services can focus on the needs of a specific group of students, such as immigrant students or students with limited English proficiency. One such intervention we visited in Georgia was designed to provide educational and cultural services to immigrant students with low level English skills and limited formal schooling. These interventions, often referred to as “newcomer” models, provide intensive language development courses and may also offer a cultural orientation component. Newcomer programs can take place within a school or at a separate site and vary in the amount of time a student is enrolled. The benefits of the newcomer program is supported by research on English language learners that notes one major factor that decreases risk of dropping out of school is level of understanding and mastery of the English language.

At the program we visited, international students who were new to the district were registered, tested, and placed depending on their skill level. Students with no English language skills were placed in an intensive 3 - to 6-week English program that helped ease the transition into school. Students who were 14 years or older and had fewer than 7 years of formal schooling in their native country were placed in the English for Speakers of Other Languages (ESOL) lab program. School officials noted that the lab served 132 students in school year 2004-05 and is designed to help students achieve grade level proficiency within 3 years. The ESOL lab focused on listening, speaking, reading, and writing English in addition to other core high school courses such as math, science, and social studies. Additionally, several district schools have added Saturday school tutorials for parents and students. Students can study language arts while their parents attend citizenship classes, orientation, and career awareness sessions. School officials noted that they believe the number of ESOL students graduating has increased, based on state-reported rates as well as the numbers of students who pass the ESOL tests and exit the program.

Other supplemental services incorporate cultural elements as a means of addressing student self-esteem. For example, a k-8 school located on the Arapahoe Indian reservation in Wyoming offers all students services that include after-school academic programs, drug awareness events, and a 2- week summer cultural camp focusing on Native American traditions. School personnel emphasized that the path to high school graduation begins with helping students address their self-esteem issues. School officials mentioned that students already have a mindset that they are not going to graduate from high school and do not have a future on or off the reservation. The cultural element of the school’s programs is a significant component of building up the student’s self-esteem and instilling a pride about their Native American identity. Students commented that they participated in the program because of the Native American cultural activities offered, including clogging, dancing, and drumming. Program officials noted that since implementing interventions designed specifically to address the issues of Native Americans, they have noticed general improvement in student attitudes and performance. While studies suggest that self-esteem affects dropout rates, a study over time of the intervention programs used by the Arapahoe school would be needed to determine its effectiveness.

Education Has Done Little to Evaluate and Disseminate Knowledge about Interventions

Graduation rates have become increasingly important since the passage of NCLBA, but Education has done little to evaluate and disseminate knowledge about interventions that could help increase such rates. The increased interest in high school reform by the National Governors Association, combined with concerns about low graduation rates, have set the stage for designing strategies that encourage more students to graduate. While many types of interventions are available for school districts, most have not been rigorously evaluated, and there is little information on which are successful and for what student subgroups. Most officials from the 20 states we included in our study told us that such information would be useful. For example, one school official noted that little information exists on what interventions increase graduation rates among Native American students and that such information would be helpful in designing interventions.

Education has made some efforts to address the problem of high school completion by sponsoring research and disseminating information through conferences and on its Web site. For example, Education officials noted that Education’s Office of Special Education Programs has supported research papers on dropout interventions for youth with disabilities. These studies are currently being completed and will be available in late 2005. In terms of dissemination, Education’s 2nd Annual High School Leadership Summit held in December 2004 included sessions on dropout prevention and recovery as well as strategies for creating higher-performing schools. Additionally, Education’s Office of Vocation and Adult Education has dedicated a part of its Web site to the High School Initiative. The pages on the Web site contain information on high school reform models, adolescent literacy initiatives as well as information on research based practices that may help high schools.

While Education has made some efforts to help states and districts address the dropout problem, the agency has not acted on its commitment to implement the recommendation, contained in our 2002 report on interventions, that Education evaluate results from research. Agency officials have commented several times that they plan to evaluate the research on dropout prevention efforts and then disseminate the results through the agency’s What Works clearinghouse. However, the Web space for this effort still contains placeholder information. Agency officials indicated that reviews of other topics, such as elementary reading and math, have come before the reviews necessary for the dropout section of the Web site.

Conclusions

The nation’s public school systems are responsible for educating 48 million students, the majority of our future workforce. Providing them with the skills needed to succeed is vital to the nation’s economic strength and ability to compete in a global economy. NCLBA was passed to ensure that all students have access to a high-quality education and to increase the likelihood that these students will graduate. In particular, the act seeks to make significant changes in public education by asking federal, state, and local education officials to reconsider how they assess the academic achievement of the nation’s students. NCLBA specifies that states must set high school graduation rate indicators as an additional benchmark, along with test results, for measuring schools’ progress. However, increasing and accurately calculating graduation rates have been formidable challenges for many states and districts. Many states have used flexibility to define their indicators as both numerical goals as well as progress toward those goals, where progress has generally ranged from no increase to a 1 percent increase from the previous year. Therefore, some states have set expectations that their schools may not graduate many more students than previously.

Education has addressed these challenges by developing some guidance and providing support such as on-site peer reviews, conferences, and information on its Web site. However, because Education’s approach has been to provide guidance on how to deal with specific student circumstances on a case-by-case basis, not all states have received such guidance. Without guidance, state officials may not appropriately include students in these specific circumstances in their graduation rate definitions, resulting in graduation rates that may be inaccurate. Such inconsistent calculations raise questions about the quality of graduation rates reported by states.

A key challenge for states is to ensure that student data used for calculating state graduation rates, as well as data provided to NCES, are accurate and that state systems have the internal controls and data verification checks to promote data reliability. As some states transition to new graduation rate definitions, it is important that they ensure that such controls are part of new student data systems. Student data accuracy is particularly important because Education plans to use those state data reported to NCES to develop interim graduation rate estimates, which are intended to promote consistency across states and provide a nationwide perspective.

Finally, little is known about the success of interventions that are designed to increase high school graduation rates. While some programs have shown potential to increase such rates, few have been rigorously evaluated. Some interventions have conducted limited evaluations of a variety of different outcomes (attendance, test scores, job attainment), but more comprehensive evaluations are necessary to understand programs’ effects on graduation rates. As a result, schools and districts may not be using the most effective approaches to help their students stay in school and graduate. Education could play an important role in evaluating existing research, which was a recommendation we made in our 2002 dropout report. Although Education agreed with this recommendation, the agency has not established a clear plan or timetable for carrying it out.

Additionally, Education should disseminate the results of research, since such information will be critical as high school reform moves forward.

Recommendations for Executive Action

To assist states in improving their definitions of high school graduation rates and enhancing the consistency of these rates, we recommend that the Secretary of Education make information available to all states on modifications available to account for students in special programs and students with disabilities in their graduation rate calculations. This information could include fuller explanations or examples of available flexibilities.

We recommend that the Secretary of Education, before developing interim graduation rate estimates, assess the reliability of data submitted by states used for this purpose. This assessment could include specific criteria that demonstrate that states’ data systems can produce accurate data.

We recommend that the Secretary establish a timetable for carrying out the recommendation in our 2002 report that Education evaluate research on dropout interventions, including those interventions that focus on increasing graduation rates. In addition, we recommend that the Secretary disseminate research on programs shown to be effective in increasing graduation rates.

Agency Comments and Our Evaluation

We provided a draft of this report to Education for review and comment. In its letter, Education concurred with two of our three recommendations: (1) about making information available to all states on modifications available to account for students in special programs and students with disabilities in their graduation rate calculations and (2) about evaluating research on dropout interventions and disseminating such research on those programs shown to be effective in increasing graduation rates. Regarding our recommendation that that the department assess the reliability of data submitted by states that it plans to use to develop interim graduation rate estimates, Education noted that it has taken a number of steps to conduct such reliability assessments. However, it is not clear whether these efforts include those data that Education will be using to develop interim graduation rate estimates. Although data submitted to Education are publicly available and have been reported by states for years, their reliability has not been determined. We believe that Education should take additional steps to ensure the reliability of these data before they are used in calculating such estimates. Education officials also provided technical comments that we incorporated into the report where appropriate. Education's written comments are reproduced in appendix II.

We are sending copies of this report to the Secretary of Education, relevant congressional committees, and other interested parties. We will also make copies available to others upon request. In addition, the report will be made available at no charge on GAO’s Web site at http://www.gao.gov.

Please contact me at (202) 512-7215 if you or your staff have any questions about this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and major contributors are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To address the objectives of this study, we used a variety of methodological approaches. We analyzed the plans states were required to submit to Education to identify the graduation rate definitions states used and graduation rate indicators set by states, reviewed updates to plans submitted through July 2005 and reviewed letters from Education to states regarding its decisions about state plans and updates.

As part of another GAO review, we surveyed officials in 50 states, the District of Columbia, and Puerto Rico to obtain information about two issues—the extent to which (1) states verify school and district data used to calculate high school graduation rates and (2) have unique student identifiers. The surveys were conducted using self-administered electronic questionnaires posted on the World Wide Web. We sent e-mail notifications to all 52 state Performance Based Data Management Initiative coordinators (50 U.S. states, the District of Columbia, and Puerto Rico) beginning on November 15, 2004. We closed the survey on January 13, 2005, after the 50th respondent had replied. Washington state and the District of Columbia did not complete the survey in time to be included in our analysis.

We selected 20 states for further analysis. States were selected to capture variation in high school graduation rate definitions, geographic location, and types of interventions with the potential to increase graduation rates. We conducted a case study in 1 state (Washington state) to calculate graduation rates; site visits in 3 states (Georgia, North Carolina, and Washington) to review site visits in 6 states (Georgia, Illinois, Minnesota, New York, Washington, and Wyoming) to observe interventions and interview program staff; and semi structured telephone interviews in all 20 states to obtain information on definitions used, implementation status, and guidance provided by Education.

See table 4 for a list of states selected for site visits and phone interviews based on the research objective we studied.

In our case study we used student data from Washington state for the 2002-03 school year, the most recent school year for which data were available at the time of our review. Using these data, we conducted an analysis comparing the results of calculating the high school graduation rate using two different graduation rate definitions—the cohort definition and the departure classification definition. Washington state used a modified cohort formula that was based on tracking student dropouts rather than on tracking student transfers. It also required all students with “unknown” status to be reported as dropouts. We also used these data to analyze the effects of allowing schools to make progress toward the graduation rate target as a means of making AYP and using an estimated miscount of the number of dropouts on the graduation rate. We interviewed experts to determine reasonable rates at which dropouts may be in error. We analyzed data using a set of 444 out of 547 of the state’s high schools. The 103 high schools that were not included in our analysis were those with graduation rates of 10 percent or less. These were generally alternative high schools, such as those designed to serve students who had committed serious crimes. We also interviewed a state official who confirmed our understanding of the omitted schools and agreed with the reasonableness of the criterion.

Although our analyses were based on a 4-year period, we used the 1 year of student data and estimated information for the 3 prior years. We did not obtain student data from prior years because state officials told us that data accuracy had improved significantly in the 2002-03 school year. We assessed the reliability of the Washington state data by (1) performing electronic testing of required data elements for missing data and for obvious errors, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing Washington state officials knowledgeable about the data. However, we did not check the data to source information. We determined that the data were sufficiently reliable for the purposes of this report.

To identify interventions with the potential to increase graduation rates, we used a “snowballing” approach. Using this approach, we reviewed the literature on interventions and interviewed Education officials and dropout prevention experts and reviewed Web sites, such as the National Dropout Prevention Centers Web site (http://www.dropoutprevention.org/), to identify those that have the potential to increase high school graduation rates. Based on the research we reviewed and on recommendations from experts, we selected several interventions at various locations around the country. For those interventions we selected to visit we reviewed available evaluations, including the findings related to outcomes, such as increased graduation rates and improved literacy. We also assessed the methodological approaches of these evaluations. Based on our review, we identified 3 interventions that had been rigorously evaluated and have shown potential to increase graduation rate and visited 3 schools that had implemented these programs. (Rigorous evaluations of 2 other interventions which showed promising results were released subsequent to our field work. We reviewed the results of these evaluations and reported their findings.) We also visited schools that had implemented 13 other interventions that experts and research showed promise in affecting factors that may improve grad rates. However, rigorous evaluations on these programs had not been done at the time of our review.

To determine how Education assists states, we reviewed Education regulations, guidance, and other documents and interviewed Education and state agency officials. We also interviewed these officials to determine the degree to which Education’s actions have enhanced and disseminated knowledge about interventions. Finally, we interviewed officials from the National Governors Association, national education organizations, and other experts in the area of high school graduation rates and reviewed related research to obtain an understanding of the issues surrounding these rates and high school reform efforts to address them. We conducted our work between September 2004 and July 2005 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Education

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Harriet Ganson (Assistant Director), Julianne Hartman Cutts (Analyst-in- Charge), and Jason Palmer (Senior Analyst) managed all aspects of the assignment. Dan Klabunde made significant contributions to this report, in all aspects of the work. In addition, Sheranda Smith-Campbell, Nagla’a El- Hodiri, and Greg Kato provided analytic assistance. Jean McSween, Karen O’Conor, and Beverly Ross provided technical support. Jim Rebbe and Sheila McCoy provided legal support, and Corinna Nicolaou assisted in the message and report development.

Related GAO Products

No Child Left Behind Act: Improvements Needed in Education’s Process for Tracking States’ Implementation of Key Provisions. GAO-04-734. Washington, D.C.: September 30, 2004.

No Child Left Behind Act: Additional Assistance and Research on Effective Strategies Would Help Small Rural Districts. GAO-04-909. Washington, D.C.: September 23, 2004.

Special Education: Additional Assistance and Better Coordination Needed among Education Offices to Help States Meet the NCLBA Teacher Requirements. GAO-04-659. Washington, D.C.: July 15, 2004.

Student Mentoring Programs: Education’s Monitoring and Information Sharing Could Be Improved. GAO-04-581. Washington, D. C.: June 25, 2004.

Title I: Characteristics of Tests Will Influence Expenses; Information Sharing May Help States Realize Efficiencies. GAO-03-389. Washington, D.C.: May 8, 2003.

Title I: Education Needs to Monitor States’ Scoring of Assessments. GAO-02-393. Washington, D.C.: April 1, 2002.

School Dropouts: Education Could Play a Stronger Role in Identifying and Disseminating Promising Prevention Strategies. GAO-02-240. Washington, D.C.: February 1, 2002.

Elementary School Children: Many Change Schools Frequently, Harming Their Education. GAO/HEHS-94-45. Washington, D.C.: February 4, 1994.

Bibliography

Burns, Matthew K., Barbara V. Senesac, and Todd Symington. “The Effectiveness of the HOSTS Program in Improving the Reading Achievement of Children At-Risk for Reading Failure.” Reading Research and Instruction, vol. 43, no. 2 (2004): 87-103 Dynarski, Mark, Philip Gleason, Anu Rangarajan, and Robert Wood. Impacts of Dropout Prevention Programs, Final Report. Princeton, New Jersey: Mathematica Policy Research, Inc., 1998.

Dynarski, Mark, Philip Gleason, Anu Rangarajan, and Robert Wood. Impacts of School Restructuring Initiatives, Final Report. Princeton, New Jersey: Mathematica Policy Research, Inc., 1998.

Dynarski, Mark and Philip Gleason. How Can We Help? What We Have Learned From Evaluations of Federal Dropout Prevention Programs? A Research Report from the School Dropout Demonstration Assistance Program Evaluation. Princeton, New Jersey: Mathematica Policy Research, Inc., 1998.

Gingras, Rosano, and Rudy Careaga. Limited English Proficient Students at Risk: Issues and Prevention Strategies. Silver Spring, Maryland: National Clearinghouse for Bilingual Education, 1989.

Greene, J. P. and Marcus A. Winters. Public School Graduation Rates in the United States (New York: Manhattan Institute for Policy Research, 2002), http://www.manhattan-institute.org/pdf/cr_31.pdf (accessed June 21, 2005).

Kemple, James J. Career Academies: Impacts on Labor Market Outcomes and Educational Attainment. New York: Manpower Demonstration Research Corporation, December 2001.

Kemple, James J., Corinne M. Herlihy, and Thomas J. Smith. Making Progress towards Graduation: Evidence from the Talent Development High School Model. New York: Manpower Demonstration Research Corporation, May 2005.

Kerbow, David. “Patterns of Urban Student Mobility and Local School Reform.” Journal of Education for Students Placed at Risk. vol. 1, no. 2 (1996): 149-171.

Lehr, Camilla A. and Cheryl M. Lange. “Alternative Schools Serving Students with and without Disabilities: What Are the Current Issues and Challenges.” Preventing School Failure, vol. 47, no. 2 (2003): 59-65.

Opuni, K. A. Project GRAD Newark: 2003-2004 Program Evaluation Report, Houston, Texas: Center for Research on School Reform, February 2005.

Quint, Janet, Howard S. Bloom, Alison Rebeck Black, LaFleur Stephens LaFleur, and Theresa M. Akey. The Challenge of Scaling Up Educational Reform: Findings and Lessons from First Things First, New York: Manpower Demonstration Research Corporation, July 2005.

Rumberger, Russell, and Scott Thomas. “The Distribution of Dropout and Turnover Rates among Urban and Suburban High Schools.” Sociology of Education, vol. 73, no. 1 (2000): 39-69.

Sinclair, M. F., S. L. Christenson, and M. L. Thurlow. “Promoting School Completion of Urban Secondary Youth with Emotional or Behavioral Disabilities.” Exceptional Children, (in press).

Snow, Catherine E., M. Susan Burns, and Peg Griffin, Eds. Preventing Reading Difficulties in Young Children. Washington, D.C.: National Academy Press, 1998.

Swanson, Christopher B. Keeping Count and Losing Count: Calculating Graduation Rates for All Students under NCLB Accountability. Washington, D.C.: Urban Institute, 2003, http://www.urban.org/url.cfm?ID=410843 (downloaded June 21, 2005).

Shannon, Sue G., and Pete Bylsma. Helping Students Finish School: Why Students Drop Out, and How to Help Them Graduate. Olympia, Washington: Office of Superintendent of Public Instruction, 2003.

U.S. Department of Education, National Center for Education Statistics, National Forum on Education Statistics. Forum Guide to Building a Culture of Quality Data: A School and District Resource. NFES 2005-801. Washington, D.C.: 2004.

U.S. Department of Education, National Center for Education Statistics. National Institute of Statistical Sciences / Education Statistics Services Institute Task Force on Graduation, Completion, and Dropout Indicators. NCES 2005-105. Washington, D.C.: 2004.

Wagner, Mary. Dropouts with Disabilities: What Do We Know? What Can We Do? A Report from the National Longitudinal Transition Study of Special Education Students. Menlo Park, California: SRI International, 1991.