School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for Schools and Contractors
Released: 2012-04-11T00:00:00

Letter

Background

Education’s changes to SIG requirements in 2010 have led to new responsibilities for the agency, states, and school districts. These entities all play key roles in the SIG award and implementation process, with Education supporting and overseeing state SIG efforts. Before awarding formula grants to states, Education reviews each state’s application and approves the state’s proposed process for competitively awarding SIG grants and monitoring implementation. As part of the state application process, states identify and prioritize eligible schools into three tiers:

Tier I schools. Receive priority for SIG funding and are the state’s lowest-achieving 5 percent of Title I schools (or 5 lowest-achieving schools, whichever number is greater) in improvement status.

Tier II schools. Secondary schools eligible for, but not receiving, Title I funds with equivalently poor performance as Tier I schools.

Tier III schools. Title I schools in improvement status that are not Tier I or Tier II schools.

After states receive SIG funding, school districts submit applications to states describing their SIG reform plans for eligible schools. Education has required that districts base their plans on an analysis of each school’s needs, called a needs assessment. After reviewing district applications, states distribute their SIG dollars using their approved competitive grant award process, giving priority to districts seeking funding for Tier I and Tier II schools. Education’s regulations and guidance require districts receiving SIG awards for Tier I or Tier II schools to implement one of four intervention models in each school. Select aspects of each model are as follows:

Transformation. Transformation schools must replace the principal, implement a transparent and equitable teacher and principal evaluation system that incorporates student academic growth, identify and reward staff who are increasing student outcomes, and provide increased student learning time, among other requirements.

Turnaround. In addition to implementing many requirements of the transformation model, turnaround schools must use locally adopted competencies to screen existing staff and rehire no more than 50 percent of the existing staff.

Restart. The district must reopen the school under the management of a contractor, such as a charter school operator, charter management organization, or education management organization.

Closure. The district must close the school and enroll its students in a higher achieving school within a reasonable proximity.

Districts may choose to use contractors to implement aspects of their reform plans. Schools enacting a restart model are required to contract with an organization that will assume many of the decision-making and leadership functions in that school. Districts employing other models may also contract with external organizations for services that could include data analysis, teacher professional development, and efforts to create safe school environments. Our work notes the importance of screening potential contractors before awarding contracts, as well as regular evaluation in order to ensure contractors are providing timely and quality services with government funds.

In addition to reviewing district applications, states are also responsible for monitoring grant implementation. States make decisions about whether to renew funding for each SIG school for an additional year, based on factors such as whether schools meet annual student achievement goals that districts set for the schools. Pursuant to Education’s guidance, if a school meets its annual goals, the state must renew the school’s SIG grant. If a school does not meet one or more annual goals, Education’s guidance gives states the flexibility to consider other factors such as the “fidelity with which the school is implementing” its chosen intervention model.

Education provides states with technical assistance and oversight regarding SIG implementation. For example, Education funds 21 Comprehensive Centers that help build states’ capacity to assist school districts and schools. Sixteen of these organizations serve states in designated regions, and 5 provide technical assistance on specific issues, such as teacher quality. In addition, Education funds Regional Educational Laboratories, a network of 10 laboratories that serve designated regions by providing access to applied education research and development projects, studies, and other related technical assistance activities. Education also monitors states’ implementation of SIG. This monitoring process consists of visits to selected states and several SIG districts and schools within the monitored states, followed by reports documenting any findings. States have an opportunity to respond to any findings before the release of Education’s monitoring reports.

States have awarded funding to two cohorts of schools since the program was modified and expanded in 2010. In the first cohort, 867 schools received SIG funding to implement one of the four intervention models in SY 2010-2011, and in the second cohort 488 schools received funding to implement one of the intervention models in SY 2011-2012. Seven have received waivers from Education to delay awarding funding statesto their second cohort of schools until SY 2012-2013 because of various issues, such as turnover of key staff in state educational agencies. The proportion of schools choosing each model was similar in both cohorts and, as shown in figure 1, most schools chose to implement the transformation model.

Successful SIG Implementation Has Posed a Number of Challenges

Staff Capacity and Commitment to Reform Posed Implementation Challenges

Although most states have increased the amount of staff time devoted to SIG since the program was expanded, some states have struggled to develop the necessary staff capacity to successfully support and oversee SIG implementation because of budget constraints. In our survey, 29 states told us that they have increased the staff time devoted to SIG since they first applied for SIG funds in the expanded SIG program. However, officials from four of the eight states we visited—California, Nebraska, Rhode Island, and Texas—told us that because of budgetary constraints, the time staff could devote to administering the SIG program and monitoring district implementation was significantly limited. For example, officials in California said that as a result of the state budget crisis, the state legislature reduced the amount of SIG funds available for state administration from the allowable 5 percent to 0.5 percent, limiting the number of staff available to administer the program and monitor districts. Several state officials we spoke with also reported that their existing workloads made it difficult to focus on SIG. For example, in several states the program was administered by officials who also had responsibilities for other major education programs, such as Race to the Top.

In addition, state officials sometimes did not have expertise in supporting school turnaround efforts. Officials from Education and several states and research groups told us that SIG required states to support local reform efforts to a much greater extent than they had in the past, and staff in some states had not yet developed the knowledge base to fulfill these responsibilities. Even when states were able to develop expertise in school reform and hire necessary staff, officials from Education told us that personnel turnover in many states made it difficult to retain such knowledge. For example, Rhode Island officials said they encountered difficulties filling vacated positions because many nearby states were also recruiting from the same small pool of qualified applicants. Several states increased their capacity through actions such as contracting with nationally recognized experts to help them run their grant competitions, establishing school turnaround offices, or hiring turnaround specialists that regularly handled an individualized caseload of SIG schools. For example, 18 states created new turnaround offices to help districts implement SIG, according to our survey.

In addition to these state level issues, many districts also struggled to develop the necessary staff capacity to implement successful school reforms. It was particularly difficult for schools to recruit and retain qualified staff members, according to many stakeholders, including officials from several states and districts we visited. They told us that SIG schools were sometimes in rural areas or needed staff to have expertise that was in short supply, such as experience with reform or specialized academic subjects. Among the 12 states that Education monitored during SY 2010-2011, they found that 4 states did not ensure that turnaround schools met requirements to remove at least half of the schools’ staff and hire new staff for those positions based on staff effectiveness. For example, Education found that one monitored district in Minnesota did not base hiring decisions on prospective teachers’ instructional effectiveness. In addition, Education found that three of the monitored states did not ensure appropriate replacement of the principal in turnaround or transformation schools.

Moreover, some districts did not have staff with expertise in using performance and evaluation data—such as data on student performance—to inform plans for reforming schools and ongoing instructional improvements. Education officials said that, in many cases, school district staff were able to collect data, but did not have experience linking data to needed interventions. In addition, our review of the needs assessments districts were required to develop when planning SIG interventions showed that some were more extensive than others. Also, in one district we visited, the new teacher evaluation process did not include state assessment data on student achievement as one of the evaluation criteria, as required by Education. Several states, districts, and researchers identified promising practices for recruiting and retaining staff or improving data usage, such as developing “grow-your-own” leadership programs, conducting priority hiring for SIG schools, or hiring data coaches to help teachers collect and analyze student data.

Districts also varied in their commitment to use SIG funds to enact major reforms. According to our survey, 35 of 51 states awarded grants to all or most Tier I applicants who applied for grants starting in SY 2010-2011, but several officials from states we visited and research organizations reported that some districts receiving SIG grants were not prepared to make significant reforms. For example, officials in one large school district we visited told us they followed turnaround model requirements to rehire no more than 50 percent of teachers at a SIG school. However, the district officials said they relocated the released teachers to other SIG schools in their district because those schools had almost all of the vacancies. Similarly, in two states we visited, district officials moved a school’s previous principal into another leadership position on site so that person could continue to work in the school even after a new principal was assigned. State and district officials also cited instances where districts chose their SIG model for reasons other than its likelihood of improving student success. For example, the superintendent in one district told us they chose the restart model because they considered it less restrictive than other models. Although many states responding to our survey told us that all or most of their transformation model schools were operating very differently after the first year of SIG, 33 states said that at least some of these schools choosing the transformation model were not. Figure 2 shows responses from these 33 states about whether inadequate action by SIG schools or districts was a reason the schools were not operating very differently. increased learning time requirements in about half of the states that it monitored during SY 2010-2011 and in both states for which it had completed SY 2011-2012 monitoring reports by February 2012..According to district officials, at least half the districts we spoke with will not have fully implemented new teacher evaluation systems by the end of their second year of SIG. In addition, during its SY 2011-2012 monitoring visit to Iowa, Education found that student growth was not always incorporated in new teacher evaluation systems, as required. Our analysis showed that increased learning time and teacher evaluation requirements were challenging because the planning needed to implement them was complex and time-consuming, and stakeholders, such as unions and parents, were sometimes reluctant to embrace the changes. Some districts struggled to develop increased learning time initiatives that would be sustainable after their 3-year SIG grant ended. More specifically: Interventions required extensive planning. Effectively implementing increased learning time and teacher evaluations required extensive planning. Several stakeholders stressed the importance of carefully designing increased learning time schedules because, for the intervention to be successful, it must provide quality instruction rather than simply increasing the amount of poor instruction. Officials from several districts said they were unable to fully implement their plans for increased learning time at the beginning of the first year of increased SIG funding because, for example, they first needed to fully analyze their existing schedules and curricula and adapt them to meet SIG requirements. Officials from states and districts we visited often stressed that developing a teacher evaluation system is time- consuming because it requires districts to accurately and comprehensively identify, collect, and analyze information about teachers’ performance and students’ academic growth. In response to the challenges involved in planning and implementing teacher evaluation systems, Education allowed states to apply for a waiver to extend the planning period for this requirement, and 27 states applied for and received the waiver as of February 2012. In its final SIG requirements, Education required schools implementing the transformation model to implement new teacher evaluation systems within the first year of the grant. Districts in states receiving these waivers must develop their evaluation systems during SY 2011-2012; pilot or fully implement them by SY 2012-2013; and use them to make decisions about retention, promotion, and compensation by SY 2013- 2014. The timeline is the same regardless of whether the SIG schools in the district are from the first or second SIG cohort.

Stakeholders sometimes reluctant to embrace required changes.

Implementation was also delayed or otherwise challenged by concerns from various stakeholder groups. Teachers and teachers’ unions were sometimes concerned about increasing student learning time or implementing new teacher evaluations in SIG schools, according to Education, state, and district officials. For example, these officials said unions were concerned about whether teacher evaluation systems that incorporated student academic growth could do so in a manner that would not penalize teachers working with the most challenging students. Such concerns sometimes led to delays in finalizing evaluation systems. In a few cases, officials told us that other stakeholder groups such as parents and school board members were also resistant to SIG requirements. For example, an official in Virginia said that some schools trying to increase learning time had met resistance from parents because students often had jobs or responsibilities at home once the traditional school day was over.

Difficulty designing sustainable approaches for increasing learning time. State and district officials also questioned whether increased learning time initiatives would be sustainable after SIG funds were exhausted. For example, survey respondents from 26 states said the costs of increased learning time were unlikely or very unlikely to be sustainable after the SIG grant ends, compared with 10 states that reported it was likely or very likely to be sustainable. Rhode Island officials noted that increased learning time benefits students enrolled in SIG schools during the grant cycle, but state and local financial constraints will make it difficult to sustain the increased learning time for future students. Due in part to these concerns, one of the two districts in the state with SIG schools limited the amount of learning time it added in order to avoid significant cuts in this time after grant funding ends.

While many officials stressed the complexity of effectively implementing these requirements, some states and districts that we visited found ways to address the challenges they posed. This was particularly true in districts that had started to plan for and implement similar school reforms prior to applying for SIG funds. Many officials from Education, state, and districts stressed the importance of stakeholder involvement while designing and implementing SIG reforms in order to enhance buy-in and strengthen reform initiatives. In order to increase students’ learning time without increasing teacher workloads and salaries, Education officials and researchers told us that a few districts were working with community partners to fund or staff additional learning time or were staggering teachers’ schedules so that students would be in class longer but teachers would not. In addition, a few states developed sample teacher evaluation system that met SIG requirements so that districts could use it as a framework for developing their own systems.

Some States Had Limited Evidence about SIG Grant Performance When Making Grant Renewal Decisions

States often had limited evidence for making decisions about whether to renew schools’ SIG funding. For example, in our survey, officials from 10 states told us that they did not use schools’ achievement of annual goals to make grant renewal decisions after SY 2010-2011. According to state officials, at least half the states we interviewed did not have the annual student achievement data available at the time they had to make renewal decisions because assessment results only became available at the end of the summer. Officials from two of these states told us that timely access to annual achievement data will continue to be a problem in future years. In addition, even when these data were available, states frequently chose not to base their decisions on schools’ achievement of annual goals. Twenty-three of 44 states responding to our survey question said that, among schools that had their funding renewed, all or most did not meet their annual goals. Several officials from our site visits questioned the usefulness of annual goal data in determining whether progress was made, particularly because districts set their own performance targets. For example, California officials said they did not find annual goals data useful because districts often included generic annual goals in their applications for SIG funding instead of proposing goals based on schools’ unique circumstances.

Regardless of whether annual goals information was available, states almost always considered “fidelity of implementation”—the extent to which the school is implementing the requirements of its intervention model—when making grant renewal decisions. However, states did not always base decisions about this criterion on extensive information. In our survey, 48 of 51 states identified fidelity of implementation as an important factor in their decision-making process, more than any other factor. Several states we spoke with said that qualitative information about implementation was important for assessing grant progress because the first steps of school reform, such as efforts to change school culture, do not always result in measurable student achievement gains.

However, making this assessment can involve a high degree of subjectivity and states’ determinations were not always developed based on extensive interaction with schools or systematic monitoring of their implementation efforts. For example, officials in California told us they used fidelity of implementation as their key criterion for making grant renewal decisions, and that the primary method for evaluating this criterion was one telephone conversation with each district at the end of the year. Prior to those conversations, the state had limited interaction with most districts for the purpose of assessing their implementation and was unable to conduct SIG monitoring visits for budgetary reasons. In addition, a Virginia official told us the state used fidelity of implementation for making renewal decisions but would benefit from guidance on how to define and measure it.

States were in some cases reluctant to discontinue SIG funding even when information they collected showed that schools were not implementing key requirements with fidelity. Several officials from states we visited said they renewed all schools’ SIG funding even if the schools were struggling to fulfill key SIG requirements because tight implementation timeframes made the officials reluctant to eliminate funding after the first year of the grant. In our survey, 21 states reported that half or fewer of their Tier I and Tier II schools were able to implement major aspects of their plan by the beginning of SY 2010-2011, such as extending the school day or having new staff in place. In the 19 cases where these states had made renewal decisions, the state renewed all or most grants. Furthermore, officials in several states we visited identified instances where they chose to renew schools’ funding despite significant problems at the district or school level, such as having administrators who were not committed to enacting major reforms or were not ensuring that planned reforms were fully implemented. For example, officials from Nevada said they renewed such grants after the first year because they did not want to negatively impact students and teachers when significant district-level problems were outside their control.

Although Education reviewed states’ proposed grant renewal procedures through the state SIG application process, the agency did not provide written guidance after grant renewal challenges arose. Education required states to submit their renewal processes for review as part of their SIG applications. Nonetheless, in several state applications we reviewed, descriptions of renewal processes and criteria did not align with the practices the state actually implemented. For example, states that told us they were unable to use annual goals data to make renewal decisions had originally identified these goals as a key renewal criterion in their applications to Education. In its work with states, Education officials told us they found some had difficulty using annual goals data or fidelity of implementation and that the agency provided technical assistance to several states that asked for help. However, agency officials were not aware of how states ultimately addressed these issues, and said the agency has not provided any additional technical assistance on grant renewal.

States renewed almost all SIG grants at the end of SY 2010-2011, and in some cases imposed conditions on schools for renewal. According to Education, 39 states chose to renew funding to every SIG school in their state. Eleven states and the District of Columbia chose not to renew funding to one or more schools, for a total of 16 nonrenewed schools overall.problems with fidelity of implementation. Several states we spoke with chose to renew grants with conditions or required changes. For example, officials in Ohio told us that struggling schools were required to take corrective actions in the second year of the 3-year grant and that their level of success in taking such actions will be a key criterion in future renewal decisions. In addition, New York officials renewed all grants after SY 2010-2011 under the condition that transformation and restart schools would implement state and federal SIG teacher evaluation requirements by December 30, 2011. Once that deadline passed, state officials determined that no districts had met the requirements and suspended all SIG funding until they were met. In February 2012, the state commissioner reinstated funds to half of the SIG districts after determining that the districts had made the necessary changes. In our survey, 23 states reported that at least a few of their SIG schools were required to make major changes to their SIG plans as a condition of having funding renewed.

States and Districts Screen Contractors Before Selecting Them to Receive SIG Funding, but Inconsistent Approach to Reviewing Performance Reduces Contractor Accountability

Contractors Provide a Wide Range of Services, and Many Receive Significant Amounts of SIG Funding

Contractors provide a wide range of services with SIG funds, and school districts have often given contractors major roles in schools using the restart, turnaround, and transformation models. Education’s guidance identifies a clear role for contractors in schools using the restart model. Specifically, districts must hire a contractor to take over school operations. For example, in the Los Angeles Unified School District, the Partnership for Los Angeles Schools has been given full management authority over five restart schools. In contrast, Education allows districts with schools using the turnaround and transformation models—which include more than 90 percent of schools receiving SIG funds—to use contractors, but does not identify a specific role for them. Most turnaround and transformation schools we visited were working with contractors. Although in some cases turnaround and transformation schools used these contractors for minor tasks, in other cases the contractors played a major role in school operations. For example, in Virginia, the state required schools implementing the turnaround and transformation intervention models to use a contractor for a range of services that could include improving teacher performance, principal and management leadership, or changing school culture.

Among the school districts we visited, several planned to spend significant amounts of their SIG grants on hiring contractors. These included districts using the restart, turnaround, and transformation models. For example, a district with one SIG school using the transformation model planned to spend about $450,000 for contractors in one school year. In addition, a district that we visited with three SIG schools planned to spend approximately $1.5 million on contractors over the 3-year period for services that included data analysis and curriculum planning.

Our prior work and reports regarding services acquisition have shown the importance of building safeguards into acquisition processes to ensure accountability. These leading practices include screening potential contractors prior to award using a thorough selection process that evaluates their ability to achieve results and the contractors’ past performance. Once a contractor has been selected, officials should routinely review contractors’ work to help ensure they are providing timely and quality services and to help mitigate any contractor performance problems.

Education and States Required Contractors to Be Screened, but Screening Procedures Varied

Education required and states reported requiring that potential contractors be selected after a thorough screening process. Education required that either states or districts screen contractors prior to contract award to ensure their quality. Although Education’s guidance does not provide specific criteria for approval, Education requires each state to describe in its state application how it will ensure that school districts screen contractors. Each of the eight states we reviewed required districts to describe their plans to screen contractors in their applications for SIG funding.

In addition, states varied in how they approached contractor screening at the state level, either taking an active role in the process or delegating screening responsibilities to districts. According to our survey, 17 of the 51 developed approved lists of contractors from which districts could choose. For example, Virginia officials told us they enacted statewide contracts with four organizations, and strongly encouraged districts to choose one of those four organizations. Ohio officials said they developed a list of approximately 100 state-screened organizations from which districts could choose, but districts were free to use other contractors, provided that they screened those organizations. States that we visited that did not develop a list of approved contractors reported requiring districts to screen contractors. For example, Texas officials told us they required all districts to use a formal competitive process in selecting contractors, which included a process to evaluate contractor proposals, in order to be approved by the state.

Education Does Not Require that All Contractors’ Performance Be Reviewed, and State Review Approaches Vary

Education’s monitoring protocols for the SIG program require the review of contractors in schools using the restart model, but they do not require review of contractors during contract performance for the other school improvement models. Education’s protocol for monitoring states’ SIG implementation asks whether districts have included accountability measures in the contract for restart schools and also asks for the district’s current assessment of the contractor. The protocol does not include a similar question for turnaround and transformation schools.

States varied in their approaches to the review of contractors, and in some cases reported that they did not require that districts review contractors during contract performance. Among the eight states we spoke with, none assessed districts’ plans to review contractors in their SIG applications. In addition, several states reported not having any state-level review requirements. For example, Nebraska state officials said their districts conduct informal reviews of the contractors, but the state does not require reviews or provide districts with a formal process or metrics to assess performance. Similarly, in follow up calls for our state survey, state officials in several states said they do not require districts to review contractor performance and were unaware of whether districts conducted any reviews. In contrast, Nevada officials told us they require districts to add accountability steps for contractors in each phase of work.

Inconsistent review of contractors during contract performance reduces states’ and districts’ ability to ensure that they are receiving the services they have paid for. In our work, one stakeholder told us that in the absence of stronger guidance or oversight, the extent to which contracts include accountability measures is largely dependent on the knowledge and experience of the individual contract manager. Although some district officials in our site visits described efforts to include accountability measures or regular review in the contracts, others indicated that contractors are reviewed informally, if at all.

Education’s Assistance and Oversight Are Supporting SIG Implementation

Education’s Technical Assistance Was Well Received by States

Education’s guidance and technical assistance on SIG implementation was well received by nearly all states. In our survey, nearly all states responded favorably about Education’s guidance and various technical assistance offerings for SY 2011-2012. Most states reported that Education’s guidance and technical assistance were helpful and many reported they were very helpful (see fig. 3). In our survey, we also inquired about the amount and timeliness of guidance provided by Education. Forty-one states reported that Education provided about the right amount of guidance for the second year of SIG. In addition, 33 states responded that in SY 2011-2012, Education’s guidance was timely, allowing the state to meet its needs, while 14 states commented that the guidance was not timely.

Although most states told us that Education’s guidance was helpful, some identified additional technical assistance that would assist with SIG implementation. In an open-ended question on our survey that asked about the types of additional guidance that Education could provide, 15 states indicated they wanted additional information about other states’ SIG implementation efforts that are working well. Several states that we met with also mentioned wanting more information on successful and sustainable implementation strategies, proven contractors, increased learning time strategies, and teacher/principal evaluation systems.

To provide additional support and enhance information sharing among the states, Education has recently begun three new assistance efforts. First, Education selected nine states to participate in the SIG “implementation support initiative” as an optional technical assistance resource. Under this initiative, each participating state receives a visit from an Education representative as well as officials from the eight other participating states. These site visits have two purposes—first, to provide technical assistance to the states, and second, to enable states to engage in peer-to-peer information sharing. Education reported that they have used information from these site visits to produce targeted technical assistance reports. Second, in December 2011, Education began conducting monthly check-in calls with state officials to better manage SIG implementation. Each state was assigned an Education program officer responsible for providing oversight and technical assistance support, including outreach and monthly check-in calls. Lastly, Education launched the School Turnaround Learning Community—an on-line forum to provide states and districts with access to resources and to facilitate networking. According to Education, this initiative offers research-based practices and practical examples from states, districts and schools for developing and implementing SIG.

Education is Now Using a Risk-Based Approach to Monitor All States within 3 Years

Education’s oversight strategy is to monitor all states during the 3-year period—starting with SY 2010-2011—in which the first cohort of schools will receive SIG funding. In selecting states for on-site monitoring for SY 2010-2011, Education did not use a SIG-specific risk-based approach and instead used the existing Title I monitoring schedule.to resource constraints, Education suspended its Title I monitoring and However, due instead focused exclusively on SIG monitoring. Education also delayed SY 2010-2011 monitoring to allow states and districts time to implement SIG before beginning monitoring in February 2011. For SY 2010-2011, Education conducted on-site monitoring in 12 states uncovering 28 deficiencies. At least one deficiency was identified in 11 of the 12 monitored states, with California and Pennsylvania having the most deficiencies with seven and five, respectively. Half of the monitored states had deficiencies in ensuring appropriate district implementation of the increased learning time requirement. In addition, two states did not ensure that all SIG funds were used consistent with the SIG requirements.

In SY 2011-2012, Education selected states with a risk based approach tailored for SIG based on factors such as the size of a state’s SIG grant. For SY 2011-2012, Education officials initially selected 12 states to conduct on-site monitoring. As of February 2012, Education had issued SY 2011-2012 monitoring reports for Iowa and Florida, containing seven and two deficiencies respectively. For example, in Iowa, Education found that funds were not used consistently with SIG grant requirements nor was the state monitoring SIG as written in its approved SIG application. Education also set aside a portion of its oversight resources so that additional states could be selected for monitoring as more information became available. As of February 2012, Colorado and South Carolina were also selected to receive an on-site review. To maximize its oversight resources, Education plans to conduct some limited “desk monitoring” in five additional states in SY 2011-2012.monitoring protocol is similar to the on-site visit protocol, but—unlike the on-site monitoring—does not include interviews with school officials. Finally, Education officials told us that they plan to monitor the remaining states in SY 2012-2013, and that these states represent a small percentage of SIG funds.

Conclusion

Dramatic funding increases in a short period of time—such as those made to SIG—can subject federal programs to considerable financial risk. While states and school districts carry a large share of the responsibility for planning and implementing successful SIG reforms, Education also plays a critical role in supporting these efforts and mitigating risk through strong oversight and accountability. For example, it is important that Education have rigorous processes for reviewing state SIG applications, conducting oversight, and providing technical assistance when needed. The ability to successfully carry out these functions is vital to ensuring the long-term success of the SIG program and protecting taxpayer funds from waste and abuse. Although SIG has been challenging to implement, in part due to the short implementation timeframes we highlighted in our July 2011 report, Education has reviewed state SIG applications, distributed funds to states, begun its monitoring activities, and provided technical assistance. However, the agency’s guidance in some cases has not been sufficient to ensure that schools and contractors are fully accountable. For example, given the implementation issues we and Education’s monitoring have found, it is critical that states have rigorous SIG grant renewal procedures in place to identify schools that are not making progress. Education has provided limited guidance to states about how to make renewal decisions. Some states are using highly subjective review processes to renew nearly all grants, often without key information on SIG schools’ performance. Until Education provides additional support about how states should make evidence-based renewal decisions when, for example, state assessment results are received too late to be factored into these decisions, schools that are not making progress may continue receiving SIG funds. In addition, although contractors are receiving large amounts of many schools’ SIG funds, Education has not ensured that states or districts review contractor performance during the terms of their contracts. Unless Education takes action to ensure that states or districts review contractor performance, districts may not receive an appropriate level of contractor services for their SIG funds and funds may not be well spent.

Recommendations for Executive Action

To ensure that SIG grant renewal decisions serve to hold districts and schools accountable, we recommend that the Secretary of Education provide additional support to states about how to make evidence-based grant renewal decisions, particularly when states do not have annual student achievement goal information available at the time renewal decisions are made.

To ensure that contractors hired with SIG funding are accountable for their performance, we recommend that the Secretary of Education take steps to ensure that the performance of SIG funded contractors, including those in turnaround and transformation schools, is reviewed during contract performance. In developing such requirements and to ensure that those reviews are targeted to contractors receiving large amounts of SIG funding, Education could consider setting a dollar threshold amount for contracts, above which contractor performance should be reviewed.

Agency Comments and Our Evaluation

We provided a draft copy of this report to the Department of Education for review and comment. Education’s comments are reproduced in appendix II. Education generally supported our recommendation about SIG grant renewal and outlined how the agency is planning to address this recommendation.

Education did not agree with our draft recommendation that it should require states to ensure that the performance of all SIG-funded contractors be reviewed, including contractors in turnaround and transformation schools. In its comments, Education said that it believed that existing provisions and requirements address this issue appropriately. For example, Education cited a federal regulation that requires districts to follow their existing procurement procedures, and noted that districts and states have their own requirements for evaluating contractors to ensure accountability. Education also said that the type of evaluation process needed for a contractor should depend on the contractor’s role, and that contractors used by schools implementing the turnaround or transformation models may be working on small, discrete projects and may require less provider-specific reviews than contractors in schools implementing the restart model.

We agree with Education that the need for performance reviews should be dependent on the specific role of the contractor, and we modified our recommendation to address some of Education’s concerns. Specifically, Education may wish to create a dollar threshold above which performance reviews are required. We continue to believe, however, that the current monitoring framework is inadequate. As noted in our report, schools implementing the turnaround and transformation models account for the overwhelming majority of SIG schools, and contractors operating in these schools are performing a range of functions, including some that are large or complex. In our view, there is a need for additional steps to ensure adequate review of contractor performance. Furthermore, our work shows that as a practical matter, states varied in their approaches to contractor review, with some imposing no requirements on districts. Education says that it will clarify in existing guidance the requirement for SIG recipients to follow state and local procurement procedures. Education could use this opportunity to implement our recommendation through additional guidance on contractor performance reviews.

In addition, Education implied that our report was based only on the first year of SIG implementation. This is inaccurate. We also conducted interviews with all eight states, reviewed SIG documents, received finalized survey responses, and interviewed Education officials several times during the second year of implementation, thereby enabling us to reflect activities beyond the first year. Based on the number and significance of deficiencies identified in Education’s SIG monitoring reports—including some completed during the SY 2011-2012—as well as our own findings, we continue to believe that Education should take additional steps to increase program accountability.

Education also provided technical comments that we have incorporated into the report as appropriate.

We are sending copies of this report to relevant congressional committees, the Secretary of Education, and other interested parties. In addition, this report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This study’s objectives were to answer the following questions: (1) What, if any, aspects of the School Improvement Grant (SIG) program pose challenges to successful implementation? (2) How do U.S. Department of Education (Education) and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices? (3) To what extent are Education’s oversight and technical assistance activities effectively supporting SIG implementation? To meet these objectives, we used a variety of methods, including document reviews of Education and state documents; a web-based survey of the 50 states and the District of Columbia; interviews with Education officials and stakeholders; site visits to and teleconferences with 8 states; and a review of the relevant federal laws, regulations, and guidance. The survey we used was reviewed by Education and several external reviewers, and we incorporated their comments as appropriate.

We conducted this performance audit from January 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Providing Information on Challenges to Successful SIG Implementation

To identify aspects of the SIG program that pose challenges for successful SIG implementation, we analyzed responses to our survey of state educational agency officials with responsibility for SIG in the 50 states and the District of Columbia. The wed-based survey was in the field from August to October 2011. In the survey, we asked states to provide information on challenges they faced in implementing the SIG program and on other aspects of the program, such as SIG grant renewal. We received responses from all 50 states and the District of Columbia, for a 100 percent response rate. We reviewed state responses and followed up by telephone and e-mail with select states for additional clarification and context.

Nonsampling error could affect data quality. Nonsampling error includes variations in how respondents interpret questions, respondents’ willingness to offer accurate responses, and data collection and processing errors. We included steps in developing the survey, and collecting and analyzing survey data to minimize such nonsampling error. In developing the web survey, we pretested draft versions of the instrument with state officials in various states to check the clarity of the questions and the flow and layout of the survey. Education officials also reviewed the draft survey and provided comments. On the basis of the pretests and reviews, we made minor revisions of the survey. Using a web-based survey also helped remove error in our data collection effort. By allowing state SIG directors to enter their responses directly into an electronic instrument, this method automatically created a record for each SIG director in a data file and eliminated the errors (and costs) associated with a manual data entry process. In addition, the program used to analyze the survey data was independently verified to ensure the accuracy of this work. Detailed survey results are available at GAO-12-370SP.

We also conducted site visits to and teleconferences with eight states— California, Delaware, Nebraska, Nevada, Ohio, Rhode Island, Texas, and Virginia—that reflect a range of population size, number of SIG schools, and use of the four SIG intervention models. In each state, we interviewed state officials, as well as district or school officials from one to three districts that had Tier I or Tier II SIG schools. Districts were selected in consultation with state officials to cover heavily and sparsely populated areas, and a variety of SIG intervention models. We also reviewed documents, such as state and district applications for SIG funds, and the relevant federal laws, regulations and guidance. We interviewed Education officials and stakeholders, such as teachers’ union officials from the national and local levels.

Identifying How Education and State Guidance on Contractors Compare with Leading Practices

To gather information about state policies and procedures for selecting and overseeing contractors, we analyzed state survey results. Our survey questions included whether states had developed a list of approved contractors, the SIG turnaround models for which they required that districts work with contractors, and whether states reviewed contractor performance. We reviewed Education documents, including SIG guidance, the state application template, and monitoring protocols, and interviewed Education officials responsible for reviewing state applications and providing oversight of states. Further, we reviewed state and district SIG applications from the eight states to identify their selection and review processes for contractors, and proposed contract expenditures. We also spoke with state and local officials about their procedures for selecting and overseeing contractors, as well as with several contractors working with districts we visited. We compared Education and state requirements for selecting and overseeing SIG contractors to leading contracting practices that were identified through collaboration with our contracting experts and review of GAO-09-374GAO-05-274 .

Describing the Extent to Which Education Has Supported SIG Implementation

To address the extent of Education’s support and oversight of SIG implementation, we reviewed Education guidance, summaries of Education assistance, monitoring time frames, monitoring protocols, and monitoring reports from SY 2010-2011. In addition, we analyzed survey results. We asked states to provide information on the federal role in SIG, including their perspectives on technical assistance offered by Education and Education’s monitoring process. We also talked with officials from Comprehensive Centers and Regional Educational Laboratories serving several of the eight states we worked with. The technical assistance providers were selected to include those working with large, medium, and small rural states. In addition, we interviewed Education officials in charge of the Comprehensive Centers Program and in charge of SIG monitoring efforts.

Appendix II: Comments from the Department of Education

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff members made important contributions to this report: Elizabeth Sirois, Assistant Director; Scott Spicer, Analyst-in-Charge; Jacques Arsenault; Melissa King; Salvatore Sorbello; and Barbara Steel-Lowney. In addition, Jean McSween, James Rebbe, Tom James, William Woods, and Kathleen Van Gelder provided guidance on the study.