No Child Left Behind Act: Education Actions Needed to Improve Local Implementation and State Evaluation of Supplemental Educational Services
Released: 2006-08-04T00:00:00

Letter

Background

Under NCLBA, SES primarily includes tutoring provided outside of the regular school day that is designed to increase the academic achievement of economically disadvantaged students in low-performing Title I schools. These services must consist of high-quality, research-based instruction that aligns with state educational standards and district curriculum.

Title I

Title I of ESEA, as amended and reauthorized by NCLBA, authorizes federal funds to help elementary and secondary schools establish and maintain programs that will improve the educational opportunities of economically disadvantaged children. Title I is the largest federal program supporting education in kindergarten through 12th grade, supplying $12.7 billion in federal funds in fiscal year 2006. According to Education, during the 2005-06 school year, nearly all U.S. school districts and approximately half of public schools received some Title I funding. In addition, the latest national data available from Education counted 16.5 million students as Title I participants in the 2002-2003 school year.

Title I funds are distributed by formula to state education agencies, which retain a share for administration and school improvement activities before passing most of the funds on to school districts. Districts are required to distribute Title I funds first to schools with poverty rates over 75 percent, with any remaining funds distributed at their discretion to schools in rank order of poverty either districtwide or within grade spans. A school’s Title I status can change from year to year because school enrollment numbers and demographics may vary over time.

Enactment of NCLBA strengthened accountability by requiring states and schools to improve the academic performance of their students so that all students are proficient in reading and math by 2014. Under NCLBA, each state creates its own content standards, academic achievement tests, and proficiency levels. In 2005-2006, states were required to test all children for reading and mathematics achievement annually in grades 3-8 and once in high school to determine whether schools are making adequate yearly progress (AYP).

In addition to meeting the state’s performance goals by grade, subject, and overall student population, schools are responsible for meeting those goals for designated groups. These groups are students who (1) are economically disadvantaged, (2) are part of a racial or ethnic group that represents a significant proportion of a school’s student population, (3) have disabilities, or (4) have limited English proficiency. To make AYP, each school must also show that each of these groups met the state proficiency goals for both reading and math. In addition, schools must show that at least 95 percent of students in grades required to take the test have done so. Schools must also demonstrate that they have met state targets for at least one other academic indicator, including graduation rate in high schools and a state-selected measure in elementary or middle schools.

For Title I schools that do not meet state AYP goals, NCLBA requires the implementation of specific interventions, and these interventions must continue until the school has met AYP for 2 consecutive years. Table 1 outlines the interventions applied after each year a Title I school misses state performance goals. At their discretion, states may also implement interventions for public schools that do not receive Title I funds and do not make AYP. Although districts are not required to offer SES until a Title I school has missed performance goals for 3 years, because some schools had not met state goals set under ESEA before the enactment of NCLBA, some Title I schools were first required to offer SES in 2002-2003, the first year of NCLBA implementation.

States are also required to establish and implement AYP standards for school districts based on the performance of all of the schools in the district. If districts fail to meet these standards for 2 consecutive years, states may classify districts as needing improvement. A district identified for improvement must develop and implement an improvement plan and remain in this status until it meets AYP standards for 2 consecutive years. If a district remains in improvement status for 2 or more years, it may be identified for corrective action as deemed necessary by the state.

Students are eligible for SES if they attend Title I schools that have missed AYP for 3 consecutive years and are from low-income families. School districts must determine family income on the same basis they use to make allocations to schools under Title I, for which many have historically used National School Lunch Program (NSLP) data. The NSLP is a federally funded program that annually collects family income data from students’ parents to determine student eligibility for free and reduced-priced lunch. A student’s state assessment scores, grades, and other academic achievement information are generally not considered when determining SES eligibility. However, if sufficient funds are not available to provide SES to all eligible children, school districts must give priority to the lowest-achieving eligible students.

SES providers may include nonprofit entities, for-profit entities, school districts, public schools, public charter schools, private schools, public or private institutions of higher education, educational service agencies, and faith-based organizations. Under the Title I regulations that govern SES, a district identified as in need of improvement or corrective action may not be an SES provider, though its schools that are not identified as needing improvement may. In addition, individual teachers who work in a school or district identified as in need of improvement may be hired by any state- approved provider to serve as a tutor in its program.

A district must set aside an amount equal to 20 percent of its Title I allocation to fund both SES and transportation for students who elect to attend other schools under school choice. This set-aside cannot be spent on administrative costs for these activities, and the district may reallocate any unused set-aside funds to other Title I activities after ensuring all eligible students have had adequate time to opt to transfer to another school or apply for SES. Funding available for SES is, therefore, somewhat dependent on costs for choice-related transportation, though as we found in our 2004 report on NCLBA’s school choice provisions, few students are participating in the school choice option. If a district does not incur any choice-related transportation costs, it must use the full 20 percent set- aside amount to pay for SES if sufficient demand for services exists. In addition, if the Title I set-aside is not sufficient to fund SES for interested students, both states and districts may direct other funds for these services at their discretion. For each student receiving SES, a district must spend an amount equal to its Title I per-pupil allocation or the actual cost of provider services, whichever is less.

Education oversees SES implementation by monitoring states and providing technical assistance and support. OII leads SES policy development and coordinates the publication of SES guidance, and OESE oversees and monitors Title I, including SES. NCLBA and the Title I regulations and SES guidance outline the roles and responsibilities states, school districts, parents, and service providers have in ensuring that eligible students receive additional academic assistance through SES (see table 2).

During the 2005-2006 school year, Education announced the implementation of two pilot programs intended to increase the number of eligible students receiving SES and generate additional information about the effectiveness of SES on students’ academic achievement. In the first, Education permitted four districts in Virginia to offer SES instead of school choice in schools that are in their first year of needs improvement.

In the second, Education entered into flexibility agreements with the Boston and Chicago school districts, enabling them to act as SES providers while in improvement status. OII and OESE coordinated implementation of the pilots for the department. Both pilots were subject to review at the end of the 2005-2006 school year, at which time Education planned to evaluate their effect on student academic achievement.

SES Participation Has Increased as Districts Have Taken Steps to Ease Access, but Challenges Remain

SES participation increased between 2003-2004 and 2004-2005, and most students receiving services were among the lower achieving students in school. Districts have taken multiple actions to encourage participation, such as offering services on or near the school campus or at various times. Despite these efforts, challenges to increasing participation remain, including notifying parents in a timely and effective manner, ensuring there are providers to serve certain areas and students, and encouraging student attendance.

Participation among Eligible Students Increased from 12 to 19 Percent between 2003-2004 and 2004-2005, and Most Participants Were Low Achieving

Nationally, the participation rate increased substantially from 12 percent of eligible students receiving SES in 2003-2004 to 19 percent in 2004-2005. In addition, the number of students receiving services almost quadrupled between 2002-2003 and 2004-2005 from approximately 117,000 to 430,000 students nationwide, based on the best available national data (see fig. 1). This increase may be due in part to the increase in the number of schools required to offer SES over that time period. Specifically, between 2004- 2005 and 2005-2006 the number of schools required to offer SES increased from an estimated 4,509 to 6,584.

Although nationally SES participation is increasing, some districts required to offer SES have no students receiving services. Specifically, we estimate that no students received services in about 20 percent of the approximately 1,000 districts required to offer SES in 2004-2005. A majority of these districts were rural or had a total enrollment of fewer than 2,500 students. Our survey did not provide sufficient information to explain why these districts had no students receiving services in 2004- 2005; therefore, it is unclear whether their lack of participation was related to district SES implementation or other issues.

Nationwide, we estimate that districts required to offer SES spent the equivalent of 5 percent of their total Title I funds for SES in 2004-2005 excluding administrative expenditures. Districts set aside an amount equal to 20 percent of their Title I funds for SES and choice-related transportation at the beginning of the school year, and the proportion of the set-aside spent on SES varied by district. Specifically, in 2004-2005, about 40 percent of districts spent 20 percent or less of the set-aside to provide SES and almost one-fifth of districts spent over 80 percent. Nationwide, of the total amount districts set-aside for SES, we estimate they spent 42 percent on SES, excluding administrative expenditures.

Further, an estimated 16 percent of districts reported that the required Title I set-aside was not sufficient to fund SES for all eligible students whose parents requested services. For example, during our site visit to Newark, N.J., district officials reported budgeting the entire 20 percent Title I set-aside to fund SES in 2004-2005, but with this amount of funding, the district was only able to fund SES for 17 percent of the students eligible for services. In addition, according to Chicago, Ill., district officials, the district budgeted the entire 20 percent Title I set-aside to fund SES in 2005-2006, and because parents’ demand for services significantly exceeded the amount of funding available, the district also allocated $5 million in local funds to provide SES.

While approximately 1,000 of the over 14,000 districts nationwide were required to offer SES in 2004-2005, SES recipients are concentrated in a small group of large districts, as 56 percent of recipients attended school in the 21 districts required to offer SES with more than 100,000 total enrolled students (see fig. 2). Further, states ranged from having 0 districts to 257 districts required to offer SES in 2004-2005, with most states having fewer than 10 districts required to offer SES. State differences in the number of districts required to offer SES may have resulted from differences in performance or differences in state proficiency standards and methods used to measure adequate yearly progress.

Students receiving SES in 2004-2005 shared certain characteristics, as districts reported that most students receiving services were among the lower achieving students in school. Specifically, an estimated 91 percent of the districts that reviewed the academic records of students receiving SES classified most or all of the students receiving SES as academically low achieving. For example, Hamilton County, Tenn., school officials said that students receiving SES are frequently behind grade level in their skills and require special attention to increase their academic achievement.

Further, we estimate that over half of SES recipients were elementary school students in the majority of districts and about 60 percent of schools required to offer SES in 2004-2005 were elementary schools. Districts varied in the percentage of students with limited English proficiency receiving services. In about one-third of districts, less than 5 percent of SES recipients were students with limited English proficiency; however, in about one-fifth of districts, over half of SES recipients were students with limited English proficiency. Students with disabilities made up less than 20 percent of students receiving services in about two-thirds of districts. Finally, in some districts, the majority of SES recipients were African- American or Hispanic. In about 40 percent of districts, over half of SES recipients were African-American, and in about 30 percent of districts, over half of SES recipients were Hispanic. Because we were unable to obtain comparable data on the characteristics of Title I students enrolled in these districts in 2004-2005, we were unable to determine whether certain groups of students were underserved.

We estimate that about 2,800 providers delivered services to students nationwide in 2004-2005, and more providers were available to deliver services in the districts with the largest student enrollments. Specifically, about 80 percent of districts had between 1 and 5 providers delivering services in 2004-2005. However, the number of providers delivering services in the 21 districts with more than 100,000 total enrolled students ranged from 4 to 45, and averaged 15 providers per district in 2004-2005.

Districts Used Several Methods to Contact Parents and Offered Services on School Campuses and at Various Times to Increase Participation

Districts have taken multiple actions to encourage participation, as shown in table 3. In line with the federal statutory requirement that districts notify parents in an understandable format of the availability of SES, over 90 percent of districts provided written information in English, held individual meetings with parents, and encouraged school staff to talk with parents about SES. Some districts collaborated with providers to notify parents. For example, during our site visit, Illinois state officials described a provider and district sharing administrative resources to increase participation, which involved the provider printing promotional materials and the district addressing and mailing the materials to parents. In addition, we estimate that over 70 percent of districts lengthened the period of time for parents to turn in SES applications, held informational events for parents to learn about providers, and provided written information to parents in languages other than English. During our site visit to Woodburn, Ore., district officials reported extending the time parents had to sign up their children for SES and hosting an event where providers presented their programs to parents in English and Spanish. Further, Newark, N.J., district officials told us during our site visit that the district provided transportation for parents to attend informational events to increase participation. Also to encourage participation, an estimated 90 percent of districts offered services at locations easily accessible to students, such as on or near the school campus, and almost 80 percent of districts offered services at a variety of times, such as before and after school or on weekends. For example, Hamilton County, Tenn., worked with providers to offer an early morning tutoring program located at the school site in addition to providing services after school. Providers also reported delivering SES on school campuses and at various times. Specifically, over three-fourths of the 22 providers we interviewed reported delivering services at the school site, although providers also offered services off-site, such as in the home, online, or at the provider’s facility. In addition, providers generally delivered SES after school and some also offered SES at alternative times, such as before school, on weekends, or during the summer. Finally, about one-third of districts provided or arranged for transportation for participating students or worked with a local community partner to raise awareness of the services. For example, in Newark, N.J., the district worked with a local community organization to inform parents and students living in public housing and homeless shelters about SES.

States also reported taking actions to increase participation in 2005-2006, as shown in table 4. Regarding parent notification, all states encouraged district staff to communicate with parents about SES. In addition, almost 90 percent of states provided guidance to districts on the use of school campuses for service delivery to encourage participation.

Remaining Challenges Include Notifying Parents in a Timely and Effective Manner and Attracting More Providers for Certain Areas and Students

Despite some districts’ promising approaches to encourage participation, notifying parents in a timely manner remains a challenge for some districts. An estimated 58 percent of districts did not notify parents that their children may be eligible to receive SES before the beginning of the 2005-2006 school year, which may be due in part to delays in states reporting which schools were identified for improvement. Specifically, about half of districts that did not notify parents before the beginning of the 2005-2006 school year did not receive notification from the state of the schools identified for improvement by that time. Moreover, district officials in three of the states we visited experienced delays in receiving school improvement information, and state officials agreed that providing timely information about whether schools have met state performance goals has been a challenge. Almost all of the districts that did not notify parents before the beginning of the 2005-2006 school year did so within the first 2 months of the year.

Effectively notifying parents is also a challenge for some districts. For example, officials in all four districts we visited reported difficulties contacting parents to inform them about SES in part because some families frequently move and do not always update their mailing address with districts. In addition, some providers we interviewed indicated that confusing parental notification letters do not effectively encourage SES participation. For example, some of the providers we interviewed said some districts use confusing and poorly written letters to inform parents of SES or send letters to parents of eligible children but conduct no further outreach to encourage participation in SES. Four of the providers we interviewed also indicated that complicated district enrollment processes can discourage participation. For example, one provider said certain districts send parents multiple documents to complete in order for their child to receive SES, such as an enrollment form to select an SES provider and a separate contract and learning plan.

Another challenge to increasing SES participation is attracting more SES providers for certain areas. Some rural districts surveyed indicated that no students received services last year because of a lack of providers in the area. We estimate that the availability of transportation for students attending supplemental services was a moderate, great or very great challenge for about half of rural districts. For example, one rural district commented in our survey that there are no approved providers within 200 miles of its schools. A few other rural districts commented in our survey that it was difficult to attract providers to their area because there were few students to serve or providers had trouble finding staff to serve as tutors.

In addition, ensuring there are providers to serve students with limited English proficiency or disabilities has been a challenge for some districts. There were not enough providers to meet the needs of students with limited English proficiency in an estimated one-third of districts, and not enough providers to meet the needs of students with disabilities in an estimated one-quarter of districts. Many states also indicated that the number of providers available to serve these groups of students was inadequate. While over half of the providers we interviewed reported serving students with limited English proficiency or disabilities, some providers served these students on a limited basis and reported difficulties meeting their needs. For example, one provider reported serving few students with limited English proficiency and disabilities because the amount of funding the provider receives for SES was not sufficient to pay for specialized tutors to serve these students. Another provider said it was difficult to find tutors to meet the needs of students with limited English proficiency and its program was not designed for students with disabilities. Another provider said that it was difficult to serve students with disabilities because it required significantly modifying the tutoring lessons to meet their needs.

Encouraging student attendance has also been a challenge, in part because students may participate in other afterschool activities, such as sports or work. Low parent and student demand for SES has been a challenge in about two-thirds of districts. For example, about one-quarter of districts reported that both competition from other afterschool programs and the availability of services that are engaging to students were challenges to implementing SES. In addition, providers, district and school officials in all four districts we visited told us that SES is competing for students with extracurricular and other activities. For example, a Chicago, Ill., high school official indicated that student attendance at SES sessions declined significantly as the school year progressed. To address this problem, providers sometimes offer students incentives for participation. For example, while 2 of the 22 providers we interviewed offered incentives for students to sign up for services, 19 providers used incentives to encourage student attendance, such as school supplies and gift certificates.

Providers Have Taken Steps to Deliver Quality Services, but Contracting and Coordination Remain Challenges to Working with Districts and Schools

To promote improved student academic achievement, providers aligned their curriculum with district instruction primarily by hiring district teachers and communicating with the teachers of participating students. Providers reported communicating with teachers and parents in person as well as mailing information and progress reports to them; however, districts indicated the extent of provider efforts varied, as some did not contact teachers and parents in 2004-2005. In addition, both providers and districts experienced contracting and coordination difficulties. In part because SES is often delivered in school facilities, providers and officials in the districts and schools we visited reported that involvement of school administrators and teachers can improve SES delivery and coordination.

Providers Worked to Align Curriculum and Communicate with Parents, though the Extent of Their Efforts Varied

In order to increase student academic achievement, providers took steps to align their curriculum with school instruction by hiring and communicating with teachers, though the extent of their efforts varied. A majority of the 22 providers we interviewed hired certified teachers in the district as tutors. Some providers said hiring district teachers promoted curriculum alignment, in part because district teachers were apt to draw on district curriculum during tutoring sessions. School officials in three of our site visits also said providers’ use of district teachers as tutors helped to align the tutoring program with what the student learned in the classroom. In addition, some providers reported aligning curriculum by communicating with the teachers of participating students to identify student needs and discuss progress. The frequency of contact between tutors and teachers ranged from mailing teachers information once prior to the beginning of the program to contacting teachers at least weekly, according to the providers we interviewed. A few providers also used other methods to align their curriculum with district instruction, such as using the same tests to evaluate student progress and allowing principals to choose components of the tutoring curriculum for students receiving SES in their schools. However, not all providers worked with schools to align curriculum, as we estimate that some, most, or all providers did not contact teachers to align curriculum with school instruction in almost 40 percent of districts in 2004-2005. For example, Woodburn, Ore., district and school officials indicated during our site visit that instead of aligning their services with the district curriculum, certain providers openly questioned the district’s curriculum and teaching methods, which caused confusion among some parents and students.

Providers reported mailing information as well as meeting with parents over the phone and in-person to communicate about student needs and progress; however, the frequency of communication with parents varied by provider. A majority of the providers we interviewed communicated with parents about student progress repeatedly, sometimes by sending home progress reports on a monthly basis or holding parent-tutor conferences. The frequency of contact between tutors and parents reported by the 22 providers we interviewed ranged from meeting with parents twice during the tutoring program to giving parents a weekly progress report. A few providers also reported communicating with parents by holding workshops for parents to learn about the SES program and in some cases having the parents sign their students’ learning plans. For example, one provider conducted workshops where parents received reading materials to share with their children and a parent guide in English and Spanish that explained the program and how to use the materials to enhance student learning. Some providers also reported hiring staff dedicated in part to maintaining communication with parents. However, some providers faced difficulties when communicating with parents about SES, such as language barriers or incorrect contact information. Districts confirmed that the degree to which providers communicated with parents varied, as we estimate that some, most, or all providers did not contact parents to discuss student needs and progress in about 30 percent of districts in 2004-2005.

Despite these challenges, most districts had positive relationships with providers. Specifically, an estimated 90 percent of districts indicated that their working relationships with providers during 2004-2005 were good, very good, or excellent. In addition, many of the providers we interviewed during our site visits also reported having positive working relationships with district officials. Although other studies have found that districts reported certain difficulties working with providers, relatively few districts reported that their providers signed up ineligible students or billed for services not performed in 2004-2005, as shown in figure 3.

Generally, states did not hear about these provider issues very often. Almost half of states said the issue of providers not showing up for SES sessions was rarely brought to their attention. Similarly, half of states said the issue of providers billing the district for services not performed was rarely brought to their attention. In addition, about 40 percent of states said the issue of providers using excessive incentives was rarely brought to their attention. Further, about 40 percent of states said the issue of providers signing up ineligible students rarely arose. Almost one-third of states heard about each of these issues sometimes, while few states had these issues brought to their attention very often. For example, during our site visits, state officials provided examples of issues that had been brought to their attention regarding provider practices, but these issues were often isolated incidents particular to one or a few providers in certain districts.

Providers and Districts Experienced Contracting and Coordination Difficulties

While providers have taken steps to deliver quality services, both providers and districts reported experiencing difficulties during the contracting process. For example, some of the providers we interviewed said certain districts imposed burdensome contract requirements, such as requiring substantial documentation to be submitted with invoices, limiting the marketing they could do to parents and students, or restricting the use of school facilities to deliver services. Specifically, 7 of the 22 providers we interviewed experienced difficulties with districts restricting provider access to school facilities, by for example, not allowing providers to deliver services in school buildings or by charging providers substantial fees to do so. A few providers also said contracting with districts was a resource-intensive process, in part because contract requirements vary by district and state. Some of the multi-state providers we interviewed reported dedicating a team of staff to help them finalize and manage contracts with districts. These providers commented that, while they have the administrative capacity to manage this process, smaller providers may not have such capacity. In addition, one provider that delivered services exclusively online commented that contracting with districts across the country was a challenge, particularly because some states and districts require provider representatives to attend meetings in-person and be fingerprinted in their states.

Contracting with providers was also a challenge for some districts. Specifically, negotiating contracts with providers was a moderate, great, or very great challenge in about 40 percent of districts nationwide. For example, Woodburn, Ore., district officials described having contractual discussions with providers around whether the district would charge fees for the use of school facilities, the types of incentives providers could use to encourage students to sign up, and whether the district would pay for services when students did not attend SES sessions. While states may review and define program design parameters as part of the provider approval process, district officials in three of our site visits expressed concern about their lack of authority to set parameters in provider contracts around costs and program design, such as tutor-to-student ratios and total hours of instruction. For example, during our site visit, a Hamilton County, Tenn., district official expressed frustration with providers that charged the maximum per-pupil amount but varied in the level of services provided, such as the number of instructional hours and tutor-to-student ratio. Chicago, Ill., district officials also expressed concern about the variation among providers in the hours of instruction and cost of services because the district does not have sufficient funds to serve all eligible students and officials would like to maximize the number of students they can serve. In part to help address district concerns, in 2005-2006, Illinois required approved providers to submit information on the cost of providing services in each of the districts they served and made this information available to districts and the public in order to improve transparency and accountability. While Tennessee state officials told us they were hesitant to set restrictions on providers and would like more federal direction on this issue, other states have set restrictions on the cost and design of SES programs. For example, Georgia set a maximum tutor-to-student ratio of 1:8 for non-computer based instruction and 1:10 for computer based instruction, and New Mexico adopted a sliding fee scale based on the educational level of tutors.

Coordination of service delivery has also been a challenge for providers, districts, and schools. About 70 percent of states reported that the level of coordination between providers, districts, and schools implementing SES was a moderate to very great challenge. Sometimes these coordination difficulties have resulted in service delays. For example, services were delayed or withdrawn in three of the districts we visited because not enough students signed up to meet the providers’ enrollment targets and districts were not aware of these targets. In one district we visited, services were delayed because school teachers hired to be tutors did not provide evidence of their background checks and teaching certificates to providers in a timely manner. Coordination difficulties also occurred during the enrollment process. Though districts are responsible for arranging SES for eligible students, in two districts we visited, both the district and providers sent parents enrollment forms, which caused confusion among parents as well as additional work for the district staff processing the forms. In addition, a few providers told us they do not know how many students they will serve until enrollment forms are returned to district officials, which hinders planning and may delay services since they do not know how many tutors they will need to hire and train to deliver SES in each district.

Providers, District and School Officials Reported That a Greater Role for Schools Would Improve Local SES Implementation

In part because SES can be delivered in school facilities, providers and officials in the districts and schools we visited reported that involvement of school administrators and other staff improves SES implementation. Although schools do not have federally-defined responsibilities for administering SES, many officials said SES implementation is hindered when school officials are not involved. Some providers we interviewed said that a lack of involvement of school principals can make it difficult for them to coordinate with schools to encourage student participation. In addition, a few providers said certain districts contributed to this problem by restricting communication with school officials or not defining a role for schools in SES implementation. Officials in one of the districts we visited also told us that encouraging participation and administering the program was more difficult when they did not designate school staff to assist the district with SES implementation. School officials from all four of our site visits also said the lack of a clear role for school officials, including principals, in administering SES has been a challenge. For example, Illinois and Oregon school principals told us they found it difficult to manage afterschool activities because they didn’t have sufficient authority to oversee SES tutors operating in their buildings at that time. Further, problems can arise when school officials are not fully informed about the SES program. For example, Woodburn, Ore., school officials told us that although the school was not provided SES tutoring schedules for students, parents and students have come to school officials for help when they were unclear about the schedule or when tutors failed to show up.

A majority of the providers we interviewed told us that involvement of school principals can improve participation and make it easier to deliver services, in part because principals are familiar with the students and manage school staff. For example, certain providers reported providing principals with information about the tutoring program so that school staff can assist with the enrollment process, involving principals in selecting the curriculum used in their schools, and sending principals student progress reports. In addition, all four districts we visited had school site coordinators to assist with SES, such as helping with the enrollment process and assisting with the day-to-day administration of the SES program in the schools. For example, Woodburn, Ore., district officials told us implementation improved when the district designated school site coordinators to assist with parental notification and events where providers present their programs, and meet with parents and providers to help them complete individual student learning plans. A few providers we interviewed also assigned a staff person at the school site to communicate with teachers and parents. While helping to administer the SES program adds additional administrative burden on schools, school officials in all four of the districts we visited said they welcomed a stronger or more clearly defined role.

States’ SES Monitoring Has Been Limited Though Their Efforts Are Increasing, and Many States Struggle to Develop Meaningful Evaluations

While state monitoring of SES had been limited, more states reported taking steps to monitor both district and provider efforts to implement SES in 2005-2006. In addition, districts have taken a direct role in monitoring providers, and their monitoring activities similarly increased during this time. Although states are required to withdraw approval from providers that fail to increase student academic achievement for 2 years, many states reported challenges evaluating SES providers. In addition, the few states that have completed an evaluation have not yet produced reports that provided a conclusive assessment of SES providers’ effect on student academic achievement.

More States Reported Conducting On-Site Reviews of Districts in 2005-2006, and Many also Collected Information from Other Sources to Monitor District SES Implementation

State monitoring of district SES implementation, which is sometimes performed as part of state Title I monitoring, had been limited prior to 2005-2006, though more states reported conducting on-site reviews of districts in that year. While less than one-third of states conducted on-site reviews of districts to monitor their implementation of SES in 2004-2005, almost three-fourths reported conducting such reviews in 2005-2006. This increase reflects both those states that had already begun monitoring district SES implementation for 2005-2006 at the time of our survey and those states planning to conduct monitoring activities before the end of that school year. Because our data were collected during the middle of the 2005-2006 school year, we do not know whether all of the states that planned to complete these activities before the end of the year did so. In both years, a majority of the states that conducted on-site reviews visited few or some of their districts. Therefore, while more states reported conducting such reviews in 2005-2006 than in 2004-2005, the number of districts per state receiving reviews does not appear to have increased.

In addition to on-site reviews, many states also reported reviewing information collected from several other sources to assess district SES implementation in 2005-2006. The most common source used by states was district officials, as almost all states reported reviewing or planning to review information collected from district officials to monitor their implementation of SES in 2005-2006. Further, many states were also collecting information from school principals, parents, and providers to monitor districts, with between 67 and 81 percent of states reviewing or planning to review information collected from these sources in 2005-2006.

States also reported reviewing or planning to review information related to several aspects of district SES implementation in 2005-2006. For example, almost all states reported reviewing district notification of parents and SES expenditures, as shown in figure 4.

States we visited reported that some districts have had difficulties implementing SES, in part because of district staff capacity limitations and the complexities of administering SES at the local level. When states find that a district is having difficulty implementing SES, most hold a meeting with the district and provide or arrange for assistance, including consultations or training. Half of the states also develop an action plan and time line with the district to help improve its efforts. During our site visits, state officials reported that notifying parents, maintaining a fair and competitive environment for providers, ensuring providers understand their SES roles and responsibilities, and determining an appropriate role for schools continue to challenge some districts as they implement SES.

Though States and Districts Are Challenged to Monitor Providers, They Are Moving from Limited Monitoring of Providers to a More Active Approach

Although states and districts reported increasing their efforts to monitor SES providers between 2004-2005 and 2005-2006, over two-thirds of states reported that on-site monitoring of providers has been a challenge. In addition, several districts commented in our survey that more provider monitoring is needed. During all four of our site visits, state and district officials expressed concerns about their capacity to fully administer and oversee all required aspects of SES implementation, including provider monitoring. Officials explained that state and district capacity to implement SES is limited, because there is typically one staff person at each level coordinating all of SES, and sometimes that person may also oversee implementation of additional federal education programs. Further, several states commented in our survey that additional training, technical assistance, and national monitoring protocols from the federal government would assist their efforts to monitor providers.

During our site visits, state officials noted that while they did not initially have structured plans or procedures in place to monitor SES providers, they took steps to develop and formalize procedures starting with the 2004-2005 and 2005-2006 school years. Nationally, in 2004-2005, states monitored providers primarily by collecting data from district officials, though many states reported using a more active monitoring approach in the next year. For example, while less than one-third of states conducted on-site reviews of providers in 2004-2005, over three-fourths had conducted or planned to conduct such reviews in 2005-2006, as shown in figure 5. In addition, while one-third or fewer states reviewed information collected from school staff, parents, and students in 2004-2005, the percentage that reported reviewing or planning to review information collected from these sources more than doubled the next year. Similar to 2004-2005, many states continued to rely on information collected from district officials to monitor providers in 2005-2006, with almost all states reviewing or planning to review information collected from districts in that year.

Federal guidance suggests states may request district assistance in collecting data from providers to assist state monitoring activities. While the state is ultimately responsible for monitoring providers, most states reported that districts have taken a direct role in monitoring providers. Similar to states, although district monitoring of providers was limited in 2004-2005, districts used a more extensive and active approach in the next year, as shown in figure 6. For example, while we estimate that less than half of districts collected information from on-site reviews, school staff, parents, and students to monitor providers in 2004-2005, 70 percent or more were collecting or planning to collect information from these sources in 2005-2006.

Although states and districts collected information from similar sources to monitor providers, districts collected information from more providers than states. Specifically, while a majority of the states that conducted on- site reviews observed only some or few providers, we estimate that a majority of districts that conducted on-site reviews observed most or all of their providers in 2004-2005. While states and districts may both have capacity limitations that affect their ability to conduct on-site reviews to monitor providers, conducting such reviews is likely easier for districts because services are often delivered on or near school campuses.

States and districts collected information on several aspects of SES programs to monitor providers, as shown in table 5. While federal regulations provide states flexibility to design their own SES monitoring systems, over two-thirds or more of states and districts monitored or planned to monitor all program elements listed, including those related to service delivery and use of funds. For example, 94 percent of states monitored or planned to monitor parent or student satisfaction with providers, and 93 percent of districts monitored or planned to monitor billing and payment for services and student attendance records.

Many States Struggle to Develop Meaningful Evaluations, and the Few State Evaluations Completed to Date Were Inconclusive

Many states struggle to develop evaluations to determine whether SES providers are improving student achievement, though states are required to evaluate and withdraw approval from providers that fail to do so after 2 years. Specifically, federal law requires states to develop standards and techniques to evaluate the services delivered by approved providers, but it does not require states to use specific evaluation methods or criteria for determining provider effectiveness. Through our survey, states reported several challenges to evaluating SES providers. Specifically, over three-fourths of states reported that determining sufficient academic progress of students, having the time and knowledge to analyze SES data, and developing data systems to track SES information have been challenges. Further, during our site visits to Illinois and New Jersey, state officials noted they were currently in the process of improving their data collection systems to more effectively capture and analyze data for SES evaluations. In addition, several state officials reported that while they have collected some information to assess provider effectiveness, they have done little with that data. Others noted that they have not received sufficient federal guidance on effective models for SES provider evaluations, and because developing and implementing such evaluations can be both time-consuming and costly, additional assistance from Education would improve their efforts.

At the time of our survey in early 2006, only a few states had drafted or completed an evaluation report addressing individual SES provider’s effects on student academic achievement, and we found that no state had produced a report that provided a conclusive assessment of this effect. New Mexico and Tennessee were the only two states that had completed final or draft SES evaluation reports that attempted to assess the impact of all SES providers serving students in their states in previous years. To measure student academic achievement, New Mexico’s report analyzed students’ grades as well as their scores on state assessments and provider assessments, which often differ by provider and are administered both before SES sessions begin and at the end of SES sessions each year.

However, the report noted that these assessments produced different results related to student academic achievement gains. While Tennessee also planned to review students’ state assessment scores, the draft available at the time of our analysis did not include this information. In addition, both reports drew on information obtained through other sources, such as teacher surveys, to assess provider effectiveness. Due to their limitations, neither evaluation provided a conclusive assessment of SES providers’ effect on student academic achievement.

In addition, at the time of our survey, over half of the states reported that they were in the process of conducting an evaluation of SES providers in order to meet the federal requirement of assessing each provider’s effect on student academic achievement. Similar to the state evaluations already undertaken, officials reported using different methods and criteria to evaluate SES providers. For example, some states were collecting each provider’s pre- and post-SES assessments of students while others were collecting student achievement data from districts or students’ state assessment scores. Further, while one state defined adequate student progress as 80 percent of a provider’s students making one-grade level of improvement after a year of SES, another state defined adequate student progress as 50 percent or more of a provider’s students having any positive academic achievement gain after a year of SES. While these states have yet to produce final results from their SES provider evaluations, it is unclear whether any of these efforts will produce a conclusive assessment of SES providers’ effect on student academic achievement.

Likely because states are struggling to complete evaluations to assess SES providers’ effect on student academic achievement, states did not report that they have withdrawn approval from providers because their programs were determined to be ineffective at achieving this goal. Rather, though over 40 percent of states reported that they had withdrawn approval from some providers, they most frequently reported withdrawing provider approval because the provider was a school or district that had entered needs improvement status, the provider asked to be removed from the state-approved provider list, or because of provider financial impropriety.

Several Education Offices Monitor and Support SES Implementation, but States and Districts Cite the Need for Additional Assistance and Flexibility

Several offices within Education monitor various aspects of SES activity across the country and provide support, but states and districts reported needing additional assistance and flexibility with program implementation. Education conducts SES monitoring in part through policy oversight and compliance reviews of states and districts, and provides SES support through guidance, grants, research, and technical assistance. However, many states and districts reported needing additional assistance and guidance regarding evaluation and administration of SES. Further, some states and districts voiced interest in expansion of Education’s pilot programs that increase SES flexibility, including the pilot that allows certain low-achieving districts to serve as SES providers.

Education Monitors SES and Provides States and Districts with Guidance and Technical Assistance

OII and OESE are primarily responsible for monitoring and supporting state and district SES implementation, and other Education offices also contribute to these efforts (see fig. 7).

OII, which leads SES policy development and provides strategic direction, monitors SES policy issues primarily through what it calls “desk monitoring.” This monitoring is performed at its federal office and includes the review of SES-related research and media reports. OII also conducts more intensive monitoring of specific SES implementation challenges when states, districts, and providers bring them to Education’s attention and keeps a log documenting these issues. For example, during 2004-2005, Illinois and New Jersey officials contacted OII to clarify guidance regarding providers affiliated with districts in need of improvement, and OII staff provided assistance and clarification. In addition, several providers we interviewed also mentioned that they have contacted OII directly to discuss implementation challenges associated with enrollment, district contracts, and provider access to school facilities.

OESE, which oversees and supports NCLBA implementation, is also involved in monitoring SES implementation through its overall monitoring of state compliance with Title I and NCLBA. To monitor Title I, OESE staff visit state departments of education and selected districts within each state to interview officials and review relevant documents. Once the review is complete, OESE issues a report to the state outlining any instances of Title I non-compliance, including those related to SES, and actions needed to comply with regulations. As of June 2006, OESE had visited and issued reports to over three-fourths of the states.

In addition to its Title I monitoring, OESE also oversees the collection of state NCLBA data, including data on SES, through the annual Consolidated State Performance Report (CSPR). For the CSPR, each state is required to report the number of schools with students receiving SES, the number of students eligible for services, and the number that received services. However, although almost all states reported that they are collecting information on district SES expenditures as part of their oversight, Education does not require states to submit information on the percent or amount of Title I funds districts spent for SES through the CSPR or other means. Therefore, while Education tracks the extent to which states are providing SES to eligible students, the department does not collect data on the relative costs of serving them. Further, under NCLBA, Education is required to present an annual summary of the CSPR data to Congress. As of June 2006, the most recent report to Congress was for the 2002-2003 school year, though Education officials indicated they expect to submit reports for 2003-2004 and 2004-2005 in the near future.

While OII and OESE monitoring of SES has been either limited to desk monitoring or combined with general Title I monitoring, OIG has conducted audits specifically focused on SES. During 2003-2004 and 2004- 2005, the office performed a series of SES implementation audits in six states, which involved site visits to state offices and selected districts within each state. Also during 2004-2005, OIG performed audits of five California districts and one SES provider within each district. These audits included an examination of district SES contracts with providers, provider services, and provider compliance with state SES regulations.

Several Education officials reported coordinating internally to share information, including SES monitoring results. To facilitate coordination, OII leads an internal group comprised of staff members from other Education offices, who meet bi-weekly to exchange information. OESE shares its state Title I monitoring results and CSPR data with other Education offices. In addition, OIG makes recommendations to both OII and OESE in its state and district SES auditing reports and disseminates the reports throughout Education and on the department’s Web site. Since 2002, OII has coordinated the publication of four versions of non-regulatory SES guidance, each updated to address ongoing challenges with SES implementation. The latest and most comprehensive version of non-regulatory SES guidance was published in June 2005. In May 2006, Education issued a separate supplement to the guidance containing additional information on private school participation in providing SES and a policy letter clarifying the definition of a district-affiliated provider.

In addition to its monitoring efforts, OII also provides SES implementation assistance, in part through presentations at conferences, and through grants to external organizations that assist states and districts. For example, OII staff have presented information on SES policy and promising practices at national meetings attended by SES coordinators and others involved in SES implementation. In addition, the office has issued grants to the Black Alliance for Educational Options, the Hispanic Council for Reform and Educational Options, and through the Star Schools Program to promote SES to minority students and those in rural areas. Further, OII funded the Supplemental Educational Services Quality Center (SESQC), which offered SES technical assistance through tool-kits, issue briefs, and a Web site containing SES information for state and district officials, schools, parents, and providers. SESQC also periodically convened representatives of organizations working on education issues to discuss SES national coordination, challenges, and promising practices. However, those meetings and all SESQC activities were discontinued in December 2005 when SESQC’s grant period ended.

Other Education offices also provide SES support through various means. For example, OESE funded the Comprehensive Centers Program through grants that established technical assistance centers across the country to help low-performing schools and districts close achievement gaps and meet the goals of NCLBA. Of these, the Center on Innovation and Improvement provides support to regional centers that assist states with Education’s programs, including SES. In addition, Education’s Policy and Program Studies Service, within the Office of Planning, Evaluation and Policy Development, oversees several research studies that examine SES, either in whole or in part. These reports, such as the National Assessment of Title I: Interim Report and Case Studies of Supplemental Services under the No Child Left Behind Act, provide states and districts with information on SES implementation, challenges, and promising practices. Further, Education’s Center for Faith-Based and Community Initiatives offers technical assistance to faith- and community-based organizations interested in becoming state-approved SES providers.

Given the technical assistance and support Education has already provided to states and districts for implementation of SES and school choice, and the department’s view that implementation of these provisions has been uneven throughout the country, in May 2006, Education issued a policy letter announcing the department’s plans to take significant enforcement action. Specifically, Education plans to use the data collected through its monitoring and evaluation efforts to take enforcement actions such as placing conditions on state Title I grants, withholding federal funds, or entering into compliance agreements. In the letter, the department noted that its various monitoring activities have identified several areas of noncompliance with SES requirements. For example, the OIG’s audits found that each of the six states reviewed failed to adequately monitor their districts for compliance. Consequently, nearly all of the parental notification letters reviewed failed to include the required key components, and several districts failed to budget sufficient funding for services. Through our own analysis of Education’s monitoring reports, we also found that some of the states reviewed were found to have inadequate or incomplete processes for monitoring and evaluating SES providers.

Many States and Districts Reported Needing SES Evaluation Support and Additional Technical Assistance, and Some Suggested Increasing Flexibility through Education’s Pilot Programs

Despite Education’s efforts, many states and districts reported needing more information and assistance to better comply with certain aspects of SES implementation, including SES evaluation (see table 6). Specifically, 85 percent of states and an estimated 70 percent of districts needed additional assistance with methods for evaluating SES, and over 60 percent also needed assistance with developing data systems. Many districts also needed more information on provider quality and effectiveness. Although OESE and OIG monitoring results have also continually indicated that states and districts struggle with SES evaluation, Education has yet to provide comprehensive assistance in this area, and during our site visits, officials mentioned that they have been relying on other states, organizations, or individuals for evaluation assistance. States and districts also indicated a need for more support and technical assistance to help them administer SES. Specifically, approximately three-fourths of states and two-thirds of districts reported needing funding to increase their capacity to implement SES. Many states also reported needing tool kits and model/sample documents, as well as training from Education, and a majority of districts needed effective parent outreach strategies. Further, most states reported a need for conferences or meetings where they could share lessons learned and promising practices with other states. A few Tennessee officials mentioned that conferences hosted by national organizations have been an effective means of allowing SES officials to gather and share knowledge.

While three-fourths of states reported that the most recent version of Education’s SES guidance, published in June 2005, has been very or extremely useful, several states commented through our survey that they needed additional or clearer guidance on certain SES provisions. This included guidance on managing SES costs and fees, implementing SES in rural areas, and handling provider complaints. During three of our site visits, officials also expressed some concern about the lack of clarity in the SES guidance with regards to student eligibility requirements and how to craft a parental SES notification letter that is both complete and easy for parents to understand.

Regarding parental notification letters, though both OESE and OIG found many states and districts to be non-compliant with the federal requirement that district SES parental notification letters include several specific elements, Education’s SES guidance, which is coordinated by OII, provides a sample that does not clearly specify all of the key elements required by SES law and regulations. For example, the sample letter does not include information on provider services, qualifications, and effectiveness. Furthermore, a few state and district officials commented that, when followed, the Title I regulations governing SES yield a letter that is unreasonably long and complex, which may be difficult for parents to understand.

Many states and districts expressed interest in the flexibility offered through two pilot programs that Education implemented during 2005-2006. The department designed these pilots to increase the number of eligible students receiving SES and to generate additional information about the effect of SES on student academic achievement. For example, several state and district SES coordinators expressed interest in Education’s pilot program that allowed two districts in needs improvement status to act as SES providers in exchange for their expansion of student access to SES providers and collection of achievement data to determine SES program effectiveness. During three of our four site visits, state and district officials expressed concern that districts identified for needs improvement are excluded from delivering SES, and one state official noted that removing districts from the state approved provider list may result in lower SES participation. Further, in our surveys, three state SES coordinators and 17 district SES coordinators wrote in comments that permitting districts in needs improvement status to provide services would assist their efforts. Through both our surveys and site visits, officials suggested that allowing districts to act as providers may ease student access to SES for rural districts that do not have providers located nearby, allow more students to participate in SES because district costs to provide services are sometimes lower than other providers’ costs, and enable districts to continue their existing tutoring programs that they feel are effective and meet the same goals as SES. Overall, we estimate that 15 percent of districts were state- approved providers in 2004-2005. However, another national survey recently found the percentage of urban and suburban districts that are state-approved SES providers is declining.

The other SES pilot allowed four districts in Virginia to offer SES instead of school choice in schools that have missed state performance goals for 2 years and are in their first year of needs improvement. During our site visits and through our surveys, many states and districts expressed interest in adjusting the order of the SES and school choice interventions. Specifically, half of states and over 60 percent of districts suggested that SES should be made available before school choice (see table 7). Further, approximately three-quarters of both states and districts indicated that SES should be offered either before or simultaneously with choice. As we found through our previous work on school choice, few students are opting to transfer schools in the first year of needs improvement, and therefore this change would provide students with another option to receive additional academic support in that year. Further, in a recent national study, district and school officials noted that parents and students are often not interested in changing schools, in part because of potential long commutes and satisfaction with their current schools, which suggests that parents and students would likely prefer to receive SES in their own schools and neighborhoods rather than school choice. In line with interest in increased flexibility with these interventions, in May 2006, Education announced that due to the positive results in Virginia districts under the pilot, the department plans to extend and expand this pilot in 2006-2007.

Conclusions

Over the last few years, almost all states and approximately 1,000 districts have been required to implement SES across the country and, if current trends continue, more schools will be required to offer services in the future. Although some states and districts are beginning to gain experience in implementing SES and use promising approaches to increase SES participation, many students are still not receiving services, in part because providers are sometimes not available to serve certain areas and groups. In addition, some districts are unsure how to involve school officials in facilitating local coordination of SES implementation and providing effective parental notification. While Education has provided support to states and districts through guidance and technical assistance, many report needing additional assistance to address these challenges. Further, the lack of clarity between policy guidance issued by OII and criteria used by OESE in their compliance reviews of states’ implementation efforts creates additional challenges in meeting the federal requirements for parental notification letters. Providing states with clear guidance that has been coordinated across Education offices is especially important now that the department has announced plans to take significant enforcement actions to ensure states comply with federal SES requirements.

While some districts do not have any students receiving services and, therefore, are not spending any Title I funds for SES, other districts are spending more than their entire set-aside and still have students on waiting lists to receive services. Two districts have been able to participate in Education’s pilot program waiving federal regulations that preclude districts in need of improvement from providing SES, which may help them provide services to more students at a lower cost. However, extending this flexibility to more districts depends on the evaluation of the quality of these services to determine if SES is having a positive impact on student academic achievement. In addition, the absence of strategies that states can use to set parameters around program design and costs further hinders their ability to stretch available funding to serve more students. Federal and state oversight of district efficiency in using federal funds to provide SES services is hindered by incomplete reporting requirements that require states to report on the number of eligible children receiving SES, but not the data they collect on the amount of Title I funding used to serve them. This information gap limits Education’s ability to track state and district compliance in spending funds for SES. Further, Education’s ability to ensure that federal dollars are effectively spent to improve student academic achievement is limited until states increase their capacity to monitor and evaluate provider performance. In the absence of additional federal technical assistance and access to information about state and district promising practices, some states may continue to struggle with implementation and evaluation of SES.

Recommendations

To help states and districts implement SES more effectively, we recommend that the Secretary of Education use the department’s Web site and the Center on Innovation and Improvement, as well as other means of communication, to: Provide federal guidance on SES parental notification letters that is clear and has been coordinated internally between OII and OESE to provide additional assistance to states and districts to help them comply with federal requirements and ensure that letters are easy for parents to understand. Education might consider providing several samples of actual district notification letters that meet these criteria. Collect and disseminate information on promising practices used by states to attract more providers for certain areas and groups and promising practices used by districts to improve parental notification of SES services and providers’ ability to serve specific groups of students and to encourage student attendance.

Provide examples of how districts can involve schools and school officials to facilitate local coordination with providers.

To improve states’ and districts’ ability to make the most of funding for SES and provide services to the maximum number of students, we recommend that the Secretary of Education: Consider expanding the 2005-2006 pilot that allowed two districts in need of improvement to enter into flexibility agreements to serve as SES providers if evaluation results show that these districts can provide quality SES services.

Clarify what states can do through the provider approval process to set parameters around program design and costs. For example, Education could issue guidance to states that clarifies their authority to set parameters on issues such as minimum hours of SES per student, minimum tutor qualifications, and cost ranges. In addition, Education could suggest to states that they coordinate these discussions with districts to address their concerns about program design and costs.

To improve federal and state monitoring of SES, we recommend that the Secretary of Education: Require states to report information necessary to determine the amount of funds spent by districts to provide SES and the percentage of their Title I allocations that this amount represents. Because almost all states reported that they are planning to monitor district SES expenditures, Education could require the states to submit these data through the annual NCLBA Consolidated State Performance Reports.

Provide states with technical assistance and guidance on how to evaluate the effect of SES on student academic achievement. For example, Education might require the Center on Innovation and Improvement to focus its SES assistance on providing states with suggested evaluation methods, measures to assess the impact of SES on achievement, and criteria for using this information to monitor and withdraw state approval from providers. Further, lessons learned and promising practices on evaluation could also be shared with states on the Center on Innovation and Improvement’s Web site or during national or regional meetings, trainings, or conferences.

Agency Comments

We provided a draft of this report to Education for review and comment. Educations’ written comments are reprinted in appendix II, and the department's technical comments were incorporated into the report as appropriate. In its written comments, Education expressed appreciation for the report’s recommendations and cited actions the department has already initiated or plans to take in addressing them. Specifically, Education noted several projects under development that might assist in carrying out our recommendations to provide more assistance to states on notifying parents, attracting providers for certain areas and groups, and involving schools. The department also said that is currently considering expanding the pilot allowing districts in need of improvement to apply to become SES providers, per our recommendation. Regarding our recommendation that Education clarify what states can do to set parameters around program design and costs, Education said it would consider addressing this issue further in the next set of revisions to the SES non-regulatory guidance. In addition, Education said it would address our recommendations to improve federal and state monitoring of SES by proposing that districts report on their SES spending and by providing more SES evaluation assistance to states through an updated issue brief as well as technical assistance provided by the Comprehensive Center on Innovation and Improvement and at a conference this fall.

We are sending copies of this report to appropriate congressional committees, the Secretary of Education, and other interested parties. Copies will also be made available upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions about the report, please contact me at (202) 512-7215. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To obtain nationally representative information on supplemental educational services (SES) participation, state and local implementation, and federal oversight, we conducted a Web-based survey of state SES coordinators and a mail survey of district SES coordinators from a nationally representative sample of districts with schools required to offer SES. We also conducted site visits during which we interviewed state, district, and school officials representing four school districts, and we conducted interviews with 22 SES providers both during the site visits and separately. In addition, we spoke with staff at Education involved in SES oversight and implementation and reviewed Education’s data on SES. We conducted our work from August 2005 through July 2006 in accordance with generally accepted government auditing standards.

Survey of States

To better understand state SES implementation, particularly how states are monitoring and evaluating SES, we designed and administered a Web- based survey of state SES coordinators in all 50 states, the District of Columbia, and Puerto Rico. The survey was conducted between January and March 2006 with 100 percent of state SES coordinators responding. The survey included questions about student participation in SES, actions taken to increase participation, SES funding and expenditures, methods used to monitor and evaluate implementation, implementation challenges, and assistance received from Education.

Because this was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce nonsampling errors, such as variations in how respondents interpret questions and their willingness to offer accurate responses. We took steps to minimize nonsampling errors, including pre-testing draft instruments and using a Web-based administration system. Specifically, during survey development, we pre-tested draft instruments with officials in Oregon, Maryland, and Washington between October and November 2005. In the pre-tests, we were generally interested in the clarity of the questions and the flow and layout of the survey. For example, we wanted to ensure definitions used in the surveys were clear and known to the respondents, categories provided in closed-ended questions were complete and exclusive, and the ordering of survey sections and the questions within each section was appropriate. On the basis of the pre-tests, the Web instrument underwent some slight revisions. A second step we took to minimize nonsampling errors was using a Web-based survey. By allowing respondents to enter their responses directly into an electronic instrument, this method automatically created a record for each respondent in a data file and eliminated the need for and the errors (and costs) associated with a manual data entry process. To further minimize errors, programs used to analyze the survey data and make estimations were independently verified to ensure the accuracy of this work.

While we did not fully validate specific information that states reported through our survey, we took several steps to ensure that the information was sufficiently reliable for the purposes of this report. For example, after the survey was closed, we made comparisons between select items from our survey data and other national-level data sets. We found our survey data were reasonably consistent with the other data sets. On the basis of our checks, we believe our survey data are sufficient for the purposes of our work.

Survey of School Districts

To obtain national-level information on district implementation of SES, we administered a mail survey to a nationally representative sample of districts that had schools required to offer SES in school year 2004-2005. The survey was conducted between January and March 2006. To obtain the maximum number of responses to our survey, we sent a reminder postcard to nonrespondents approximately 1 week after the initial mailing of the survey instrument, a follow-up mailing with the full survey instrument to nonrespondents approximately 3 weeks after the initial mailing, and a second follow-up mailing with the full survey instrument approximately 4 weeks later. The survey included questions about student participation in SES, characteristics of students receiving SES, actions taken to increase participation, SES funding and expenditures, methods used to monitor and evaluate implementation, implementation challenges, and assistance received and still needed.

Population

The target population of 1,095 districts consisted of public school districts with at least one school in each of their jurisdictions required to provide SES in the 2004-2005 school year. To define our population, we collected school improvement information from state education agency Web sites and the NCLBA Consolidated State Performance Reports: Part I for Reporting on School Year 2003-2004 that states submitted to Education. When available, we checked both sources for school improvement information and used the source that provided the most recently updated data, as this data is often updated by states over the course of the school year. After constructing our population of districts, we used Education’s Common Core of Data Local Education Agency (School District) preliminary file and the Public Elementary/Secondary School preliminary file for the 2003-2004 school year to further define the characteristics of our population. On the basis of our review of these data, we determined these sources to be adequate for the purposes of our work.

Sample Design and Errors

The sample design for the mail survey was a stratified random sample of districts with one certainty stratum containing all of the districts with 100,000 students or more and one stratum containing all other districts in the universe. We included the 21 districts with 100,000 or more students with certainty in the sample to ensure we gathered information from the largest districts required to offer SES.

We selected a simple random sample of districts in the non-certainty stratum and calculated the sample size to achieve a precision of plus and minus 7 percent at the 95 percent confidence level for an expected proportion of 50 percent. To ensure the sample sizes were adequate, we increased the sample size assuming we would obtain a 70 percent response rate. The total sample size for this stratum was 237 districts. In the sample, each district in the population had a known, nonzero probability of being selected. Each selected district was subsequently weighted in the analysis to account statistically for all the schools in the population, including those that were not selected. Table 8 provides a description of the universe and sample of districts.

Because we surveyed a sample of districts, our results are estimates of a population of districts and thus are subject to sampling errors that are associated with samples of this size and type. Our confidence in the precisions of the results from this sample is expressed in 95 percent confidence intervals, which are expected to include the actual results in 95 percent of the samples of this type. We calculated confidence intervals for this sample based on methods that are appropriate for a stratified random sample.

We determined that 10 of the sampled districts were out of scope because they did not have any schools required to provide SES in the 2004-2005 school year. All estimates produced from the sample and presented in this report are for the estimated target population of 1,034 districts with at least one school required to provide SES in the 2004-2005 school year. All percentage estimates included in this report have margins of error of plus or minus 8 percentage points or less, except for those shown in table 9. All other numerical estimates, such as the total number of schools required to offer SES in 2004-2005, included in this report have margins of error of plus or minus 18 percent or less.

We took steps to minimize nonsampling errors that are not accounted for through statistical tests, like sampling errors. In developing the mail survey, we conducted several pretests of draft instruments. We pretested the survey instrument with district officials in Woodburn, Ore.; Tacoma, Wash.; Baltimore, Md.; and Alexandria, Va., between October and November 2005. These pre-tests were similar to the state Web survey pre- tests in design and content. On the basis of the pre-tests, the draft survey instrument underwent some slight revisions.

While we did not fully validate specific information that districts reported through our survey, we took several steps to ensure that the information was sufficiently reliable for the purposes of this report. For example, data from the surveys were double-keyed to ensure data entry accuracy, and the information was analyzed using statistical software. After the survey was closed, we also made comparisons between select items from our survey data and other national-level data sets. We found our survey data were reasonably consistent with the external sources. On the basis of our checks, we believe our survey data are sufficient for the purposes of our work.

Response Rate

We received survey responses from 73 percent of the 258 district Title I/SES coordinators in our sample. The response rate, adjusted for the known and estimated districts that were out of scope, was 77 percent.

After the survey was closed, we analyzed the survey respondents to determine if there were any differences between the responding districts, the nonresponding districts, and the population. We performed this analysis for three characteristics—total number of students enrolled, total number of special education students, and total number of English language learner students. We determined whether sample-based estimates of these characteristics compared favorably with the known population values. The population value for all of the characteristics we examined fell within the 95 percent confidence intervals for the estimates from the survey respondents. On the basis of the 77 percent response rate and this analysis, we chose to include the survey results in our report and produce sample-based estimates to the population of districts required to provide SES in the 2004-2005 school year.

Site Visits

To understand SES implementation at the local level, we conducted site visits to four districts between October 17, 2005, and February 16, 2006. The districts visited included Woodburn School District (Woodburn, Ore.), Hamilton County Schools (Chattanooga, Tenn.), Newark Public Schools (Newark, N.J.), and Chicago Public Schools (Chicago, Ill.). The four districts visited were selected because they had experience implementing SES in their schools and were recommended by stakeholders as having promising parent outreach and/or monitoring practices. When viewed as a group, the districts also provided variation across characteristics such as geographic location, district size, student ethnicity, and the percentage of students with limited English proficiency or disabilities.

During the site visits, we interviewed state officials, including the state SES coordinator, and district officials, including the superintendent and SES coordinator. We also interviewed officials representing 12 schools, including principals and other school staff involved with SES. In total, we visited several schools of each level, from elementary to high, and though district officials selected the schools we visited, all of the schools had experience implementing SES. Through our interviews with state, district, and school officials, we collected information on district efforts to notify parents and fulfill implementation responsibilities, student participation, providers, local implementation challenges, and implementation assistance received and needed. During the visits, we also interviewed providers and observed tutoring sessions in order to better understand implementation. During our visit to Woodburn, Ore., we also observed a provider fair.

In addition to our site visits to four districts, we also visited the Rhode Island Department of Elementary and Secondary Education in March 2006 to gather additional information on state efforts to monitor and evaluate SES. Rhode Island invited us to attend two meetings the state held with districts implementing SES and providers serving students in Rhode Island, during which SES challenges, ways to improve implementation, and state efforts to evaluate providers were discussed.

Provider Interviews

In total, we conducted interviews with 22 providers, including 15 providers during the site visits and 7 providers operating in multiple states. The Education Industry Association assisted our efforts to contact multi-state providers, and most of the multi-state providers we interviewed were association members. Multi-state provider interviews were conducted between November and December 2005.

Through all of our provider interviews, we collected information on provider efforts to increase participation in SES, align services with state standards and district curriculum, and communicate with parents and schools to ensure students are receiving needed services. We also collected information on students served, tutor and program characteristics, and provider challenges to SES implementation. While the providers we interviewed reflect some variety in provider characteristics, our selections were not intended to be representative. Thus, the findings from our interviews cannot be used to make inferences about all providers.

Education Data Analyses

We analyzed state data submitted to Education through the NCLBA Consolidated State Performance Reports (CSPR) for school years 2002- 2003, 2003-2004, and 2004-2005. State reports from all 3 years included the number of students receiving SES and the number of schools those students attended, and state reports from 2003-2004 and 2004-2005 also included the number of students eligible for SES. Data from the 2003-2004 CSPRs were used to assist our analysis of SES participation. To assess the reliability of the 2003-2004 data, we performed a series of tests, which included checking to ensure that data were consistent, that subtotals added to overall totals and that data provided for 1 year bore a reasonable relationship to the next year’s data and to data reported elsewhere, including state education reports. We also spoke with Education officials about their follow-up efforts to verify the data. At the time of our review, Education was in the process of completing efforts to verify the 2003-2004 data.

While we compared the 2004-2005 CSPR data to data obtained through our state and district surveys to further verify our data, we generally did not use the 2004-2005 CSPR data for our analysis. During this comparison analysis, where we found discrepancies or sought clarification, we followed up with state officials. In several states, officials revised the numbers that they had initially reported to us or to Education. On the basis of our review of these data, we determined these sources to be adequate for the purposes of our work.

We also considered SES-related findings from Education studies, including the Evaluation of Title I Accountability Systems and School Improvement Efforts: Findings From 2002-03 (2005) and the National Assessment of Title I: Interim Report (2006). To ensure the findings from these studies were generally reliable, we reviewed each study’s methodology, including data sources and analyses, limitations, and conclusions. In addition, in designing our state and district surveys, we reviewed SES-related survey questions used by Education in these studies.

Appendix II: Comments from the Department of Education

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

Cindy Ayers, Assistant Director, and Rachel Frisk, Analyst-in-Charge, managed this assignment and made significant contributions to all aspects of this report. Cathy Roark, Ted Burik, and David Perkins also made significant contributions. Kevin Jackson, Jean McSween, Jim Ashley, and Jerry Sandau provided methodological expertise and assistance; Rachael Valliere assisted with message and report development; and Rasheeda Curry made contributions during study design. In addition, Jessica Botsford assisted in the legal analysis.

Related GAO Products

No Child Left Behind Act: Assistance from Education Could Help States Better Measure Progress of Students with Limited English Proficiency. GAO-06-815. Washington, D.C.: July 26, 2006.

No Child Left Behind Act: States Face Challenges Measuring Academic Growth that Education’s Initiatives May Help Address. GAO-06-661. Washington, D.C.: July 17, 2006.

No Child Left Behind Act: Most Students with Disabilities Participated in Statewide Assessments, but Inclusion Options Could Be Improved. GAO-05-618. Washington, D.C.: July 20, 2005.

No Child Left Behind Act: Education Needs to Provide Additional Technical Assistance and Conduct Implementation Studies for School Choice Provision. GAO-05-7. Washington, D.C.: December 10, 2004.

No Child Left Behind Act: Improvements Needed in Education’s Process for Tracking States’ Implementation of Key Provisions. GAO-04-734. Washington, D.C.: September 30, 2004.

No Child Left Behind Act: Additional Assistance and Research on Effective Strategies Would Help Small Rural Districts. GAO-04-909. Washington, D.C.: September 23, 2004.

Disadvantaged Students: Fiscal Oversight of Title I Could Be Improved. GAO-03-377. Washington, D.C.: February 28, 2003.

Title I Funding: Poor Children Benefit Though Funding Per Poor Child Differs. GAO-02-242. Washington, D.C.: January 31, 2002.