School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for Schools and Contractors
Released: 2012-04-11T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: The School Improvement Grant (SIG) program funds reforms in low performing schools. Congress provided $3.5 billion for SIG in fiscal year 2009, and a total of about $1.6 billion was appropriated in fiscal years 2010-2012. SIG requirements changed significantly in 2010. Many schools receiving SIG funds must now use the funding for specific interventions, such as turning over certain school operations to an outside organization (contractor). GAO examined (1) what, if any, aspects of SIG pose challenges for successful implementation; (2) how Education and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices; and (3) to what extent Education’s technical assistance and oversight activities are effectively supporting SIG implementation. GAO surveyed SIG directors in all 50 states and the District of Columbia; analyzed Education and state documents; and interviewed officials from 8 states and school districts in those states, SIG contractors, and education experts.

section_title: What GAO Found
paragraphs: Successful SIG implementation posed a number of challenges. Specifically, state and district officials were challenged to build staff capacity and commitment for reform, facing difficulties such as recruiting and retaining strong staff members. In addition, the SIG requirements to develop teacher evaluations and increase student learning time were difficult to implement quickly and effectively because they required extensive planning and coordination. Furthermore, states sometimes had limited evidence about the performance of SIG schools when making grant renewal decisions. For example, although Education’s guidance identifies meeting annual student achievement goals as a key criterion for making renewal decisions, some states did not receive student achievement data by the time decisions had to be made. States also made decisions through qualitative assessments of schools’ implementation efforts, but such determinations were not always based on extensive interaction with schools or systematic monitoring. Education did not provide written guidance to states about making evidence-based grant renewal decisions after they encountered these challenges.

Districts used a significant portion of their SIG funds to hire contractors for a range of services, such as managing school operations and conducting teacher professional development. Leading practices show that screening potential contractors and then reviewing their performance are important for ensuring accountability and quality of results. Education required screening of contractors before contract awards were made. However, Education did not require review of contractors during contract performance, and states varied in whether they ensured that contractors were reviewed during the course of contract performance.

Education’s assistance and oversight activities are generally supporting SIG implementation. In our survey, nearly all states reported they were satisfied with Education’s technical assistance, particularly the agency’s SIG guidance and conferences. In addition, many states reported that Education’s guidance was timely. With respect to oversight, Education monitored 12 states in school year (SY) 2010-2011 and found deficiencies in 11 of the 12 states. Education is working with states to correct these deficiencies. For SY 2011-2012, the agency plans to use a risk-based approach to conduct on-site monitoring in 14 additional states. To maximize its oversight resources, Education also plans to conduct some limited monitoring in five additional states in SY 2011-2012. Education officials told us that they plan to monitor the remaining states in SY 2012-2013 and that these states represent a small percentage of SIG funds.

section_title: What GAO Recommends
paragraphs: GAO recommends that Education (1) provide additional support to states about making evidence-based grant renewal decisions and (2) ensure that contractor performance is reviewed. Education generally supported our first recommendation but disagreed with the second. We modified our recommendation to address some of Education’s concerns.

section_title: Letter
paragraphs: The School Improvement Grant (SIG) program is designed to fund  significant reforms in low performing schools. Schools that receive SIG  funding (SIG schools) receive up to $2 million annually for 3 years to  improve student outcomes, such as standardized test scores and  graduation rates. While the program—which provides funds to states on a  formula basis—was first authorized in fiscal year 2002, it changed  significantly in fiscal year 2009. Specifically, Congress greatly increased  funding for the program from $125 million in fiscal year 2007 to $3.5  billion in fiscal year 2009. From this amount, $3 billion was provided  under the American Recovery and Reinvestment Act of 2009, and schools began spending these funds during school year (SY) 2010-2011.Additionally, a total of approximately $1.6 billion was appropriated in fiscal  years 2010-2012.

75 Fed. Reg. 66,363 (Oct. 28, 2010). organization (contractor). Also, states are now required to award SIG  funds to school districts competitively, rather than by formula. States  evaluate grant applications using several criteria, including the proposed  intervention model for each school and the district’s budget and reform  implementation plan, as well as their capacity and commitment to  effectively implement the reforms. After grant awards are made, states  are also responsible for deciding whether to renew schools’ SIG funding  for additional years. State renewal decisions are to be based on criteria  that can include whether school assessment results meet annual student  achievement goals, and whether the school has implemented its chosen  intervention model. Education oversees the program by reviewing state  grant applications, monitoring implementation and providing technical  assistance.

The recent changes to the SIG program have raised questions about how  grants are being implemented. The Senate Appropriations Committee  and its Subcommittee on Labor, Health and Human Services, Education,  and Related Agencies requested that we conduct a broad review of the  SIG program, and in July 2011 we reported on early implementation  issues in six states during the first year of the expanded SIG program.  That report found that the program’s implementation time frames in some  cases did not allow schools sufficient time to plan and fully enact reforms,  and we recommended that the Secretary of Education consider options to  award SIG grants to school districts earlier in the school year. In this  review, we addressed the following questions: (1) What, if any, aspects of  SIG pose challenges for successful implementation? (2) How do  Education and state guidance and procedures for screening potential  contractors and reviewing contractor performance compare with leading  practices? (3) To what extent are Education’s technical assistance and  oversight activities effectively supporting SIG implementation?

To identify aspects of SIG that pose challenges to successful school  turnaround, we administered a survey to the 50 states and the District of  Columbia, and received a 100 percent response rate.Education documents and interviewed Education officials. We gathered  in-depth information from eight states that were selected to represent a  range of size, geographic diversity, and the intervention models being  used. These states were California, Delaware, Nebraska, Nevada, Ohio,  Rhode Island, Texas, and Virginia. In each of the states, we also  interviewed school district and school officials from one to three districts  that had SIG schools. To collect information about policies and  procedures for screening potential contractors and reviewing contractor  performance, we examined Education guidance and state monitoring  policies. We also interviewed Education officials and state and local  officials from our site visit states. We identified leading practices for  screening and reviewing contractors based on our past work. To evaluate  Education’s technical assistance and oversight efforts, we reviewed  relevant documents, such as SIG monitoring protocols, and interviewed  relevant Education officials. We also gathered information from states  through our survey and state site visits. We interviewed several  Education-funded technical assistance providers that served our site visit  states and various stakeholders, such as teachers’ union officials. For  more information about our scope and methodology, see appendix I.

We conducted this performance audit from January 2011 to April 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Education’s changes to SIG requirements in 2010 have led to new  responsibilities for the agency, states, and school districts. These entities  all play key roles in the SIG award and implementation process, with  Education supporting and overseeing state SIG efforts. Before awarding  formula grants to states, Education reviews each state’s application and  approves the state’s proposed process for competitively awarding SIG  grants and monitoring implementation. As part of the state application  process, states identify and prioritize eligible schools into three tiers:

Tier I schools. Receive priority for SIG funding and are the state’s  lowest-achieving 5 percent of Title I schools (or 5 lowest-achieving  schools, whichever number is greater) in improvement status.

Tier II schools. Secondary schools eligible for, but not receiving, Title I  funds with equivalently poor performance as Tier I schools.

Tier III schools. Title I schools in improvement status that are not Tier  I or Tier II schools.

After states receive SIG funding, school districts submit applications to  states describing their SIG reform plans for eligible schools. Education  has required that districts base their plans on an analysis of each school’s  needs, called a needs assessment. After reviewing district applications,  states distribute their SIG dollars using their approved competitive grant  award process, giving priority to districts seeking funding for Tier I and  Tier II schools. Education’s regulations and guidance require districts  receiving SIG awards for Tier I or Tier II schools to implement one of four  intervention models in each school. Select aspects of each model are as  follows:

Transformation. Transformation schools must replace the principal,  implement a transparent and equitable teacher and principal  evaluation system that incorporates student academic growth, identify  and reward staff who are increasing student outcomes, and provide  increased student learning time, among other requirements.

Turnaround. In addition to implementing many requirements of the  transformation model, turnaround schools must use locally adopted   competencies to screen existing staff and rehire no more than 50  percent of the existing staff.

Restart. The district must reopen the school under the management of  a contractor, such as a charter school operator, charter management  organization, or education management organization.

Closure. The district must close the school and enroll its students in a  higher achieving school within a reasonable proximity.

Districts may choose to use contractors to implement aspects of their  reform plans. Schools enacting a restart model are required to contract  with an organization that will assume many of the decision-making and  leadership functions in that school. Districts employing other models may  also contract with external organizations for services that could include  data analysis, teacher professional development, and efforts to create  safe school environments. Our work notes the importance of screening  potential contractors before awarding contracts, as well as regular  evaluation in order to ensure contractors are providing timely and quality  services with government funds.

In addition to reviewing district applications, states are also responsible  for monitoring grant implementation. States make decisions about  whether to renew funding for each SIG school for an additional year,  based on factors such as whether schools meet annual student  achievement goals that districts set for the schools. Pursuant to  Education’s guidance, if a school meets its annual goals, the state must  renew the school’s SIG grant. If a school does not meet one or more  annual goals, Education’s guidance gives states the flexibility to consider  other factors such as the “fidelity with which the school is implementing”  its chosen intervention model.

Education provides states with technical assistance and oversight  regarding SIG implementation. For example, Education funds 21  Comprehensive Centers that help build states’ capacity to assist school  districts and schools. Sixteen of these organizations serve states in  designated regions, and 5 provide technical assistance on specific issues,  such as teacher quality. In addition, Education funds Regional  Educational Laboratories, a network of 10 laboratories that serve  designated regions by providing access to applied education research  and development projects, studies, and other related technical assistance  activities. Education also monitors states’ implementation of SIG. This  monitoring process consists of visits to selected states and several SIG  districts and schools within the monitored states, followed by reports  documenting any findings. States have an opportunity to respond to any  findings before the release of Education’s monitoring reports.

States have awarded funding to two cohorts of schools since the program  was modified and expanded in 2010. In the first cohort, 867 schools  received SIG funding to implement one of the four intervention models in  SY 2010-2011, and in the second cohort 488 schools received funding to  implement one of the intervention models in SY 2011-2012. Seven   have received waivers from Education to delay awarding funding  statesto their second cohort of schools until SY 2012-2013 because of various  issues, such as turnover of key staff in state educational agencies. The  proportion of schools choosing each model was similar in both cohorts  and, as shown in figure 1, most schools chose to implement the  transformation model.

section_title: Successful SIG Implementation Has Posed a Number of Challenges
subsections: section_title: Staff Capacity and Commitment to Reform Posed Implementation Challenges
paragraphs: Although most states have increased the amount of staff time devoted to  SIG since the program was expanded, some states have struggled to  develop the necessary staff capacity to successfully support and oversee  SIG implementation because of budget constraints. In our survey, 29  states told us that they have increased the staff time devoted to SIG since  they first applied for SIG funds in the expanded SIG program. However,  officials from four of the eight states we visited—California, Nebraska,  Rhode Island, and Texas—told us that because of budgetary constraints,  the time staff could devote to administering the SIG program and  monitoring district implementation was significantly limited. For example,  officials in California said that as a result of the state budget crisis, the  state legislature reduced the amount of SIG funds available for state  administration from the allowable 5 percent to 0.5 percent, limiting the  number of staff available to administer the program and monitor districts.  Several state officials we spoke with also reported that their existing  workloads made it difficult to focus on SIG. For example, in several states  the program was administered by officials who also had responsibilities  for other major education programs, such as Race to the Top.

In addition, state officials sometimes did not have expertise in supporting  school turnaround efforts. Officials from Education and several states and  research groups told us that SIG required states to support local reform  efforts to a much greater extent than they had in the past, and staff in  some states had not yet developed the knowledge base to fulfill these  responsibilities. Even when states were able to develop expertise in  school reform and hire necessary staff, officials from Education told us  that personnel turnover in many states made it difficult to retain such  knowledge. For example, Rhode Island officials said they encountered  difficulties filling vacated positions because many nearby states were also  recruiting from the same small pool of qualified applicants. Several states  increased their capacity through actions such as contracting with  nationally recognized experts to help them run their grant competitions,  establishing school turnaround offices, or hiring turnaround specialists  that regularly handled an individualized caseload of SIG schools. For  example, 18 states created new turnaround offices to help districts  implement SIG, according to our survey.

In addition to these state level issues, many districts also struggled to  develop the necessary staff capacity to implement successful school  reforms. It was particularly difficult for schools to recruit and retain  qualified staff members, according to many stakeholders, including  officials from several states and districts we visited. They told us that SIG  schools were sometimes in rural areas or needed staff to have expertise  that was in short supply, such as experience with reform or specialized  academic subjects. Among the 12 states that Education monitored during  SY 2010-2011, they found that 4 states did not ensure that turnaround  schools met requirements to remove at least half of the schools’ staff and  hire new staff for those positions based on staff effectiveness. For  example, Education found that one monitored district in Minnesota did not  base hiring decisions on prospective teachers’ instructional effectiveness.  In addition, Education found that three of the monitored states did not  ensure appropriate replacement of the principal in turnaround or  transformation schools.

Moreover, some districts did not have staff with expertise in using  performance and evaluation data—such as data on student  performance—to inform plans for reforming schools and ongoing  instructional improvements. Education officials said that, in many cases,  school district staff were able to collect data, but did not have experience  linking data to needed interventions. In addition, our review of the needs  assessments districts were required to develop when planning SIG  interventions showed that some were more extensive than others. Also, in  one district we visited, the new teacher evaluation process did not include  state assessment data on student achievement as one of the evaluation  criteria, as required by Education. Several states, districts, and  researchers identified promising practices for recruiting and retaining staff  or improving data usage, such as developing “grow-your-own” leadership  programs, conducting priority hiring for SIG schools, or hiring data  coaches to help teachers collect and analyze student data.

Districts also varied in their commitment to use SIG funds to enact major  reforms. According to our survey, 35 of 51 states awarded grants to all or  most Tier I applicants who applied for grants starting in SY 2010-2011,  but several officials from states we visited and research organizations  reported that some districts receiving SIG grants were not prepared to  make significant reforms. For example, officials in one large school district  we visited told us they followed turnaround model requirements to rehire  no more than 50 percent of teachers at a SIG school. However, the  district officials said they relocated the released teachers to other SIG  schools in their district because those schools had almost all of the  vacancies. Similarly, in two states we visited, district officials moved a  school’s previous principal into another leadership position on site so that  person could continue to work in the school even after a new principal  was assigned. State and district officials also cited instances where  districts chose their SIG model for reasons other than its likelihood of  improving student success. For example, the superintendent in one  district told us they chose the restart model because they considered it  less restrictive than other models. Although many states responding to  our survey told us that all or most of their transformation model schools  were operating very differently after the first year of SIG, 33 states said  that at least some of these schools choosing the transformation model  were not. Figure 2 shows responses from these 33 states about  whether inadequate action by SIG schools or districts was a reason the  schools were not operating very differently. increased learning time requirements in about half of the states that it  monitored during SY 2010-2011 and in both states for which it had  completed SY 2011-2012 monitoring reports by February 2012..According  to district officials, at least half the districts we spoke with will not have  fully implemented new teacher evaluation systems by the end of their  second year of SIG. In addition, during its SY 2011-2012 monitoring visit  to Iowa, Education found that student growth was not always incorporated  in new teacher evaluation systems, as required. Our analysis showed that  increased learning time and teacher evaluation requirements were  challenging because the planning needed to implement them was  complex and time-consuming, and stakeholders, such as unions and  parents, were sometimes reluctant to embrace the changes. Some  districts struggled to develop increased learning time initiatives that would  be sustainable after their 3-year SIG grant ended. More specifically:  Interventions required extensive planning. Effectively implementing  increased learning time and teacher evaluations required extensive  planning. Several stakeholders stressed the importance of carefully  designing increased learning time schedules because, for the  intervention to be successful, it must provide quality instruction rather  than simply increasing the amount of poor instruction. Officials from  several districts said they were unable to fully implement their plans  for increased learning time at the beginning of the first year of  increased SIG funding because, for example, they first needed to fully  analyze their existing schedules and curricula and adapt them to meet  SIG requirements. Officials from states and districts we visited often  stressed that developing a teacher evaluation system is time- consuming because it requires districts to accurately and  comprehensively identify, collect, and analyze information about  teachers’ performance and students’ academic growth. In response to  the challenges involved in planning and implementing teacher  evaluation systems, Education allowed states to apply for a waiver to  extend the planning period for this requirement, and 27 states applied  for and received the waiver as of February 2012. In its final SIG  requirements, Education required schools implementing the  transformation model to implement new teacher evaluation systems  within the first year of the grant. Districts in states receiving these  waivers must develop their evaluation systems during SY 2011-2012;  pilot or fully implement them by SY 2012-2013; and use them to make  decisions about retention, promotion, and compensation by SY 2013- 2014. The timeline is the same regardless of whether the SIG schools  in the district are from the first or second SIG cohort.

Stakeholders sometimes reluctant to embrace required changes.

Implementation was also delayed or otherwise challenged by  concerns from various stakeholder groups. Teachers and teachers’  unions were sometimes concerned about increasing student learning  time or implementing new teacher evaluations in SIG schools,  according to Education, state, and district officials. For example, these  officials said unions were concerned about whether teacher  evaluation systems that incorporated student academic growth could  do so in a manner that would not penalize teachers working with the  most challenging students. Such concerns sometimes led to delays in  finalizing evaluation systems. In a few cases, officials told us that  other stakeholder groups such as parents and school board members  were also resistant to SIG requirements. For example, an official in  Virginia said that some schools trying to increase learning time had  met resistance from parents because students often had jobs or  responsibilities at home once the traditional school day was over.

Difficulty designing sustainable approaches for increasing learning  time. State and district officials also questioned whether increased  learning time initiatives would be sustainable after SIG funds were  exhausted. For example, survey respondents from 26 states said the  costs of increased learning time were unlikely or very unlikely to be  sustainable after the SIG grant ends, compared with 10 states that  reported it was likely or very likely to be sustainable. Rhode Island  officials noted that increased learning time benefits students enrolled  in SIG schools during the grant cycle, but state and local financial  constraints will make it difficult to sustain the increased learning time  for future students. Due in part to these concerns, one of the two  districts in the state with SIG schools limited the amount of learning  time it added in order to avoid significant cuts in this time after grant  funding ends.

While many officials stressed the complexity of effectively implementing  these requirements, some states and districts that we visited found ways  to address the challenges they posed. This was particularly true in  districts that had started to plan for and implement similar school reforms  prior to applying for SIG funds. Many officials from Education, state, and  districts stressed the importance of stakeholder involvement while  designing and implementing SIG reforms in order to enhance buy-in and  strengthen reform initiatives. In order to increase students’ learning time  without increasing teacher workloads and salaries, Education officials and  researchers told us that a few districts were working with community  partners to fund or staff additional learning time or were staggering  teachers’ schedules so that students would be in class longer but  teachers would not. In addition, a few states developed sample teacher  evaluation system that met SIG requirements so that districts could use it  as a framework for developing their own systems.

section_title: Some States Had Limited Evidence about SIG Grant Performance When Making Grant Renewal Decisions
paragraphs: States often had limited evidence for making decisions about whether to  renew schools’ SIG funding. For example, in our survey, officials from 10  states told us that they did not use schools’ achievement of annual goals  to make grant renewal decisions after SY 2010-2011. According to state  officials, at least half the states we interviewed did not have the annual  student achievement data available at the time they had to make renewal  decisions because assessment results only became available at the end  of the summer. Officials from two of these states told us that timely  access to annual achievement data will continue to be a problem in future  years. In addition, even when these data were available, states frequently  chose not to base their decisions on schools’ achievement of annual  goals. Twenty-three of 44 states responding to our survey question said  that, among schools that had their funding renewed, all or most did not  meet their annual goals. Several officials from our site visits questioned  the usefulness of annual goal data in determining whether progress was  made, particularly because districts set their own performance targets.  For example, California officials said they did not find annual goals data  useful because districts often included generic annual goals in their  applications for SIG funding instead of proposing goals based on schools’  unique circumstances.

Regardless of whether annual goals information was available, states  almost always considered “fidelity of implementation”—the extent to  which the school is implementing the requirements of its intervention  model—when making grant renewal decisions. However, states did not  always base decisions about this criterion on extensive information. In our  survey, 48 of 51 states identified fidelity of implementation as an  important factor in their decision-making process, more than any other  factor. Several states we spoke with said that qualitative information  about implementation was important for assessing grant progress  because the first steps of school reform, such as efforts to change school  culture, do not always result in measurable student achievement gains.

However, making this assessment can involve a high degree of  subjectivity and states’ determinations were not always developed based  on extensive interaction with schools or systematic monitoring of their  implementation efforts. For example, officials in California told us they  used fidelity of implementation as their key criterion for making grant  renewal decisions, and that the primary method for evaluating this  criterion was one telephone conversation with each district at the end of  the year. Prior to those conversations, the state had limited interaction  with most districts for the purpose of assessing their implementation and  was unable to conduct SIG monitoring visits for budgetary reasons. In  addition, a Virginia official told us the state used fidelity of implementation  for making renewal decisions but would benefit from guidance on how to  define and measure it.

States were in some cases reluctant to discontinue SIG funding even  when information they collected showed that schools were not  implementing key requirements with fidelity. Several officials from states  we visited said they renewed all schools’ SIG funding even if the schools  were struggling to fulfill key SIG requirements because tight  implementation timeframes made the officials reluctant to eliminate  funding after the first year of the grant. In our survey, 21 states reported  that half or fewer of their Tier I and Tier II schools were able to implement  major aspects of their plan by the beginning of SY 2010-2011, such as  extending the school day or having new staff in place. In the 19 cases  where these states had made renewal decisions, the state renewed all or  most grants. Furthermore, officials in several states we visited identified  instances where they chose to renew schools’ funding despite significant  problems at the district or school level, such as having administrators who  were not committed to enacting major reforms or were not ensuring that  planned reforms were fully implemented. For example, officials from  Nevada said they renewed such grants after the first year because they  did not want to negatively impact students and teachers when significant  district-level problems were outside their control.

Although Education reviewed states’ proposed grant renewal procedures  through the state SIG application process, the agency did not provide  written guidance after grant renewal challenges arose. Education required  states to submit their renewal processes for review as part of their SIG  applications. Nonetheless, in several state applications we reviewed,  descriptions of renewal processes and criteria did not align with the  practices the state actually implemented. For example, states that told us  they were unable to use annual goals data to make renewal decisions  had originally identified these goals as a key renewal criterion in their  applications to Education. In its work with states, Education officials told  us they found some had difficulty using annual goals data or fidelity of  implementation and that the agency provided technical assistance to  several states that asked for help. However, agency officials were not  aware of how states ultimately addressed these issues, and said the  agency has not provided any additional technical assistance on grant  renewal.

States renewed almost all SIG grants at the end of SY 2010-2011, and in  some cases imposed conditions on schools for renewal. According to  Education, 39 states chose to renew funding to every SIG school in their  state. Eleven states and the District of Columbia chose not to renew  funding to one or more schools, for a total of 16 nonrenewed schools  overall.problems with fidelity of implementation. Several states we spoke with  chose to renew grants with conditions or required changes. For example,  officials in Ohio told us that struggling schools were required to take  corrective actions in the second year of the 3-year grant and that their  level of success in taking such actions will be a key criterion in future  renewal decisions. In addition, New York officials renewed all grants after  SY 2010-2011 under the condition that transformation and restart schools  would implement state and federal SIG teacher evaluation requirements  by December 30, 2011. Once that deadline passed, state officials  determined that no districts had met the requirements and suspended all  SIG funding until they were met. In February 2012, the state  commissioner reinstated funds to half of the SIG districts after  determining that the districts had made the necessary changes. In our  survey, 23 states reported that at least a few of their SIG schools were  required to make major changes to their SIG plans as a condition of  having funding renewed.

section_title: States and Districts Screen Contractors Before Selecting Them to Receive SIG Funding, but Inconsistent Approach to Reviewing Performance Reduces Contractor Accountability
subsections: section_title: Contractors Provide a Wide Range of Services, and Many Receive Significant Amounts of SIG Funding
paragraphs: Contractors provide a wide range of services with SIG funds, and school  districts have often given contractors major roles in schools using the  restart, turnaround, and transformation models. Education’s guidance  identifies a clear role for contractors in schools using the restart model.  Specifically, districts must hire a contractor to take over school  operations. For example, in the Los Angeles Unified School District, the  Partnership for Los Angeles Schools has been given full management  authority over five restart schools. In contrast, Education allows districts  with schools using the turnaround and transformation models—which  include more than 90 percent of schools receiving SIG funds—to use  contractors, but does not identify a specific role for them. Most turnaround  and transformation schools we visited were working with contractors.  Although in some cases turnaround and transformation schools used  these contractors for minor tasks, in other cases the contractors played a  major role in school operations. For example, in Virginia, the state  required schools implementing the turnaround and transformation  intervention models to use a contractor for a range of services that could  include improving teacher performance, principal and management  leadership, or changing school culture.

Among the school districts we visited, several planned to spend  significant amounts of their SIG grants on hiring contractors. These  included districts using the restart, turnaround, and transformation  models. For example, a district with one SIG school using the  transformation model planned to spend about $450,000 for contractors in  one school year. In addition, a district that we visited with three SIG  schools planned to spend approximately $1.5 million on contractors over  the 3-year period for services that included data analysis and curriculum  planning.

Our prior work and reports regarding services acquisition have shown the  importance of building safeguards into acquisition processes to ensure  accountability. These leading practices include screening potential  contractors prior to award using a thorough selection process that  evaluates their ability to achieve results and the contractors’ past  performance. Once a contractor has been selected, officials should  routinely review contractors’ work to help ensure they are providing timely  and quality services and to help mitigate any contractor performance  problems.

section_title: Education and States Required Contractors to Be Screened, but Screening Procedures Varied
paragraphs: Education required and states reported requiring that potential contractors  be selected after a thorough screening process. Education required that  either states or districts screen contractors prior to contract award to  ensure their quality. Although Education’s guidance does not provide  specific criteria for approval, Education requires each state to describe in  its state application how it will ensure that school districts screen  contractors. Each of the eight states we reviewed required districts to  describe their plans to screen contractors in their applications for SIG  funding.

In addition, states varied in how they approached contractor screening at  the state level, either taking an active role in the process or delegating  screening responsibilities to districts. According to our survey, 17 of the  51 developed approved lists of contractors from which districts could  choose. For example, Virginia officials told us they enacted statewide  contracts with four organizations, and strongly encouraged districts to  choose one of those four organizations. Ohio officials said they developed  a list of approximately 100 state-screened organizations from which  districts could choose, but districts were free to use other contractors,  provided that they screened those organizations. States that we visited  that did not develop a list of approved contractors reported requiring  districts to screen contractors. For example, Texas officials told us they  required all districts to use a formal competitive process in selecting  contractors, which included a process to evaluate contractor proposals, in  order to be approved by the state.

section_title: Education Does Not Require that All Contractors’ Performance Be Reviewed, and State Review Approaches Vary
paragraphs: Education’s monitoring protocols for the SIG program require the review  of contractors in schools using the restart model, but they do not require  review of contractors during contract performance for the other school  improvement models. Education’s protocol for monitoring states’ SIG  implementation asks whether districts have included accountability  measures in the contract for restart schools and also asks for the district’s  current assessment of the contractor. The protocol does not include a  similar question for turnaround and transformation schools.

States varied in their approaches to the review of contractors, and in  some cases reported that they did not require that districts review  contractors during contract performance. Among the eight states we  spoke with, none assessed districts’ plans to review contractors in their  SIG applications. In addition, several states reported not having any  state-level review requirements. For example, Nebraska state officials  said their districts conduct informal reviews of the contractors, but the  state does not require reviews or provide districts with a formal process or  metrics to assess performance. Similarly, in follow up calls for our state  survey, state officials in several states said they do not require districts to  review contractor performance and were unaware of whether districts  conducted any reviews. In contrast, Nevada officials told us they require  districts to add accountability steps for contractors in each phase of work.

Inconsistent review of contractors during contract performance reduces  states’ and districts’ ability to ensure that they are receiving the services  they have paid for. In our work, one stakeholder told us that in the  absence of stronger guidance or oversight, the extent to which contracts  include accountability measures is largely dependent on the knowledge  and experience of the individual contract manager. Although some district  officials in our site visits described efforts to include accountability  measures or regular review in the contracts, others indicated that  contractors are reviewed informally, if at all.

section_title: Education’s Assistance and Oversight Are Supporting SIG Implementation
subsections: section_title: Education’s Technical Assistance Was Well Received by States
paragraphs: Education’s guidance and technical assistance on SIG implementation  was well received by nearly all states. In our survey, nearly all states  responded favorably about Education’s guidance and various technical  assistance offerings for SY 2011-2012. Most states reported that  Education’s guidance and technical assistance were helpful and many  reported they were very helpful (see fig. 3). In our survey, we also  inquired about the amount and timeliness of guidance provided by  Education. Forty-one states reported that Education provided about the  right amount of guidance for the second year of SIG. In addition, 33  states responded that in SY 2011-2012, Education’s guidance was timely,  allowing the state to meet its needs, while 14 states commented that the  guidance was not timely.

Although most states told us that Education’s guidance was helpful, some  identified additional technical assistance that would assist with SIG  implementation. In an open-ended question on our survey that asked  about the types of additional guidance that Education could provide, 15  states indicated they wanted additional information about other states’  SIG implementation efforts that are working well. Several states that we  met with also mentioned wanting more information on successful and  sustainable implementation strategies, proven contractors, increased  learning time strategies, and teacher/principal evaluation systems.

To provide additional support and enhance information sharing among  the states, Education has recently begun three new assistance efforts.  First, Education selected nine states to participate in the SIG  “implementation support initiative” as an optional technical assistance  resource. Under this initiative, each participating state receives a visit  from an Education representative as well as officials from the eight other  participating states. These site visits have two purposes—first, to provide  technical assistance to the states, and second, to enable states to  engage in peer-to-peer information sharing. Education reported that they  have used information from these site visits to produce targeted technical  assistance reports. Second, in December 2011, Education began  conducting monthly check-in calls with state officials to better manage  SIG implementation. Each state was assigned an Education program  officer responsible for providing oversight and technical assistance  support, including outreach and monthly check-in calls. Lastly, Education  launched the School Turnaround Learning Community—an on-line forum  to provide states and districts with access to resources and to facilitate  networking. According to Education, this initiative offers research-based  practices and practical examples from states, districts and schools for  developing and implementing SIG.

section_title: Education is Now Using a Risk-Based Approach to Monitor All States within 3 Years
paragraphs: Education’s oversight strategy is to monitor all states during the 3-year  period—starting with SY 2010-2011—in which the first cohort of schools  will receive SIG funding. In selecting states for on-site monitoring for SY  2010-2011, Education did not use a SIG-specific risk-based approach  and instead used the existing Title I monitoring schedule.to resource constraints, Education suspended its Title I monitoring and   However, due  instead focused exclusively on SIG monitoring. Education also delayed  SY 2010-2011 monitoring to allow states and districts time to implement  SIG before beginning monitoring in February 2011. For SY 2010-2011,  Education conducted on-site monitoring in 12 states uncovering 28  deficiencies. At least one deficiency was identified in 11 of the 12  monitored states, with California and Pennsylvania having the most  deficiencies with seven and five, respectively. Half of the monitored states  had deficiencies in ensuring appropriate district implementation of the  increased learning time requirement. In addition, two states did not  ensure that all SIG funds were used consistent with the SIG  requirements.

In SY 2011-2012, Education selected states with a risk based approach  tailored for SIG based on factors such as the size of a state’s SIG grant.  For SY 2011-2012, Education officials initially selected 12 states to  conduct on-site monitoring. As of February 2012, Education had issued  SY 2011-2012 monitoring reports for Iowa and Florida, containing seven  and two deficiencies respectively. For example, in Iowa, Education found  that funds were not used consistently with SIG grant requirements nor  was the state monitoring SIG as written in its approved SIG application.  Education also set aside a portion of its oversight resources so that  additional states could be selected for monitoring as more information  became available. As of February 2012, Colorado and South Carolina  were also selected to receive an on-site review. To maximize its oversight  resources, Education plans to conduct some limited “desk monitoring” in  five additional states in SY 2011-2012.monitoring protocol is similar to the on-site visit protocol, but—unlike the  on-site monitoring—does not include interviews with school officials.  Finally, Education officials told us that they plan to monitor the remaining  states in SY 2012-2013, and that these states represent a small  percentage of SIG funds.

section_title: Conclusion
paragraphs: Dramatic funding increases in a short period of time—such as those  made to SIG—can subject federal programs to considerable financial risk.  While states and school districts carry a large share of the responsibility  for planning and implementing successful SIG reforms, Education also  plays a critical role in supporting these efforts and mitigating risk through  strong oversight and accountability. For example, it is important that  Education have rigorous processes for reviewing state SIG applications,  conducting oversight, and providing technical assistance when needed.  The ability to successfully carry out these functions is vital to ensuring the  long-term success of the SIG program and protecting taxpayer funds from  waste and abuse. Although SIG has been challenging to implement, in  part due to the short implementation timeframes we highlighted in our  July 2011 report, Education has reviewed state SIG applications,  distributed funds to states, begun its monitoring activities, and provided  technical assistance. However, the agency’s guidance in some cases has  not been sufficient to ensure that schools and contractors are fully  accountable. For example, given the implementation issues we and  Education’s monitoring have found, it is critical that states have rigorous  SIG grant renewal procedures in place to identify schools that are not  making progress. Education has provided limited guidance to states  about how to make renewal decisions. Some states are using highly  subjective review processes to renew nearly all grants, often without key  information on SIG schools’ performance. Until Education provides  additional support about how states should make evidence-based  renewal decisions when, for example, state assessment results are  received too late to be factored into these decisions, schools that are not  making progress may continue receiving SIG funds. In addition, although  contractors are receiving large amounts of many schools’ SIG funds,  Education has not ensured that states or districts review contractor  performance during the terms of their contracts. Unless Education takes  action to ensure that states or districts review contractor performance,  districts may not receive an appropriate level of contractor services for  their SIG funds and funds may not be well spent.

section_title: Recommendations for Executive Action
paragraphs: To ensure that SIG grant renewal decisions serve to hold districts and  schools accountable, we recommend that the Secretary of Education  provide additional support to states about how to make evidence-based  grant renewal decisions, particularly when states do not have annual  student achievement goal information available at the time renewal  decisions are made.

To ensure that contractors hired with SIG funding are accountable for  their performance, we recommend that the Secretary of Education take  steps to ensure that the performance of SIG funded contractors, including  those in turnaround and transformation schools, is reviewed during  contract performance. In developing such requirements and to ensure  that those reviews are targeted to contractors receiving large amounts of  SIG funding, Education could consider setting a dollar threshold amount  for contracts, above which contractor performance should be reviewed.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft copy of this report to the Department of Education for  review and comment. Education’s comments are reproduced in appendix  II. Education generally supported our recommendation about SIG grant  renewal and outlined how the agency is planning to address this  recommendation.

Education did not agree with our draft recommendation that it should  require states to ensure that the performance of all SIG-funded  contractors be reviewed, including contractors in turnaround and  transformation schools. In its comments, Education said that it believed  that existing provisions and requirements address this issue  appropriately. For example, Education cited a federal regulation that  requires districts to follow their existing procurement procedures, and  noted that districts and states have their own requirements for evaluating  contractors to ensure accountability. Education also said that the type of  evaluation process needed for a contractor should depend on the  contractor’s role, and that contractors used by schools implementing the  turnaround or transformation models may be working on small, discrete  projects and may require less provider-specific reviews than contractors  in schools implementing the restart model.

We agree with Education that the need for performance reviews should  be dependent on the specific role of the contractor, and we modified our  recommendation to address some of Education’s concerns. Specifically,  Education may wish to create a dollar threshold above which  performance reviews are required. We continue to believe, however, that  the current monitoring framework is inadequate. As noted in our report,  schools implementing the turnaround and transformation models account  for the overwhelming majority of SIG schools, and contractors operating  in these schools are performing a range of functions, including some that  are large or complex. In our view, there is a need for additional steps to  ensure adequate review of contractor performance. Furthermore, our  work shows that as a practical matter, states varied in their approaches to  contractor review, with some imposing no requirements on districts.  Education says that it will clarify in existing guidance the requirement for  SIG recipients to follow state and local procurement procedures.  Education could use this opportunity to implement our recommendation  through additional guidance on contractor performance reviews.

In addition, Education implied that our report was based only on the first  year of SIG implementation. This is inaccurate. We also conducted  interviews with all eight states, reviewed SIG documents, received  finalized survey responses, and interviewed Education officials several  times during the second year of implementation, thereby enabling us to  reflect activities beyond the first year. Based on the number and  significance of deficiencies identified in Education’s SIG monitoring  reports—including some completed during the SY 2011-2012—as well as  our own findings, we continue to believe that Education should take  additional steps to increase program accountability.

Education also provided technical comments that we have incorporated  into the report as appropriate.

We are sending copies of this report to relevant congressional  committees, the Secretary of Education, and other interested parties. In  addition, this report will be available at no charge on GAO’s website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This study’s objectives were to answer the following questions: (1) What,  if any, aspects of the School Improvement Grant (SIG) program pose  challenges to successful implementation? (2) How do U.S. Department of  Education (Education) and state guidance and procedures for screening  potential contractors and reviewing contractor performance compare with  leading practices? (3) To what extent are Education’s oversight and  technical assistance activities effectively supporting SIG implementation?  To meet these objectives, we used a variety of methods, including  document reviews of Education and state documents; a web-based  survey of the 50 states and the District of Columbia; interviews with  Education officials and stakeholders; site visits to and teleconferences  with 8 states; and a review of the relevant federal laws, regulations, and  guidance. The survey we used was reviewed by Education and several  external reviewers, and we incorporated their comments as appropriate.

We conducted this performance audit from January 2011 through April  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Providing Information on Challenges to Successful SIG Implementation
paragraphs: To identify aspects of the SIG program that pose challenges for  successful SIG implementation, we analyzed responses to our survey of  state educational agency officials with responsibility for SIG in the 50  states and the District of Columbia. The wed-based survey was in the  field from August to October 2011. In the survey, we asked states to  provide information on challenges they faced in implementing the SIG  program and on other aspects of the program, such as SIG grant  renewal. We received responses from all 50 states and the District of  Columbia, for a 100 percent response rate. We reviewed state responses  and followed up by telephone and e-mail with select states for additional  clarification and context.

Nonsampling error could affect data quality. Nonsampling error includes  variations in how respondents interpret questions, respondents’  willingness to offer accurate responses, and data collection and  processing errors. We included steps in developing the survey, and  collecting and analyzing survey data to minimize such nonsampling error.  In developing the web survey, we pretested draft versions of the  instrument with state officials in various states to check the clarity of the  questions and the flow and layout of the survey. Education officials also  reviewed the draft survey and provided comments. On the basis of the  pretests and reviews, we made minor revisions of the survey. Using a  web-based survey also helped remove error in our data collection effort.  By allowing state SIG directors to enter their responses directly into an  electronic instrument, this method automatically created a record for each  SIG director in a data file and eliminated the errors (and costs) associated  with a manual data entry process. In addition, the program used to  analyze the survey data was independently verified to ensure the  accuracy of this work. Detailed survey results are available at   GAO-12-370SP.

We also conducted site visits to and teleconferences with eight states— California, Delaware, Nebraska, Nevada, Ohio, Rhode Island, Texas, and  Virginia—that reflect a range of population size, number of SIG schools,  and use of the four SIG intervention models. In each state, we  interviewed state officials, as well as district or school officials from one to  three districts that had Tier I or Tier II SIG schools. Districts were selected  in consultation with state officials to cover heavily and sparsely populated  areas, and a variety of SIG intervention models. We also reviewed  documents, such as state and district applications for SIG funds, and the  relevant federal laws, regulations and guidance. We interviewed  Education officials and stakeholders, such as teachers’ union officials  from the national and local levels.

section_title: Identifying How Education and State Guidance on Contractors Compare with Leading Practices
paragraphs: To gather information about state policies and procedures for selecting  and overseeing contractors, we analyzed state survey results. Our survey  questions included whether states had developed a list of approved  contractors, the SIG turnaround models for which they required that  districts work with contractors, and whether states reviewed contractor  performance. We reviewed Education documents, including SIG  guidance, the state application template, and monitoring protocols, and  interviewed Education officials responsible for reviewing state  applications and providing oversight of states. Further, we reviewed state  and district SIG applications from the eight states to identify their  selection and review processes for contractors, and proposed contract  expenditures. We also spoke with state and local officials about their  procedures for selecting and overseeing contractors, as well as with  several contractors working with districts we visited. We compared  Education and state requirements for selecting and overseeing SIG  contractors to leading contracting practices that were identified through  collaboration with our contracting experts and review of GAO-09-374GAO-05-274 .

section_title: Describing the Extent to Which Education Has Supported SIG Implementation
paragraphs: To address the extent of Education’s support and oversight of SIG  implementation, we reviewed Education guidance, summaries of  Education assistance, monitoring time frames, monitoring protocols, and  monitoring reports from SY 2010-2011. In addition, we analyzed survey  results. We asked states to provide information on the federal role in SIG,  including their perspectives on technical assistance offered by Education  and Education’s monitoring process. We also talked with officials from  Comprehensive Centers and Regional Educational Laboratories serving  several of the eight states we worked with. The technical assistance  providers were selected to include those working with large, medium, and  small rural states. In addition, we interviewed Education officials in charge  of the Comprehensive Centers Program and in charge of SIG monitoring  efforts.

section_title: Appendix II: Comments from the Department of Education

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff members  made important contributions to this report: Elizabeth Sirois, Assistant  Director; Scott Spicer, Analyst-in-Charge; Jacques Arsenault; Melissa  King; Salvatore Sorbello; and Barbara Steel-Lowney. In addition, Jean  McSween, James Rebbe, Tom James, William Woods, and Kathleen Van  Gelder provided guidance on the study.


Question-summary annotations: questions: question: What are the challenges of School Improvement Grants (SIG) for staff capacity?
answer: Successful SIG implementation posed a number of challenges. Specifically, state and district officials were challenged to build staff capacity and commitment for reform, facing difficulties such as recruiting and retaining strong staff members.
child_questions: question: What are the challenges of SIGs for planning?
answer: In addition, the SIG requirements to develop teacher evaluations and increase student learning time were difficult to implement quickly and effectively because they required extensive planning and coordination.

question: What what the state's opinion on SIGs?
answer: Furthermore, states sometimes had limited evidence about the performance of SIG schools when making grant renewal decisions.

question: How may the lack of data affect decision-making?
answer: For example, although Education’s guidance identifies meeting annual student achievement goals as a key criterion for making renewal decisions, some states did not receive student achievement data by the time decisions had to be made. States also made decisions through qualitative assessments of schools’ implementation efforts, but such determinations were not always based on extensive interaction with schools or systematic monitoring.

question: What guidance has Education provided for addressing these challenges?
answer: Education did not provide written guidance to states about making evidence-based grant renewal decisions after they encountered these challenges.

questions: question: What is the main use of SIG funds?
answer: Districts used a significant portion of their SIG funds to hire contractors for a range of services, such as managing school operations and conducting teacher professional development.
child_questions: question: What did the leading practices show of screening potential contractors?
answer: Leading practices show that screening potential contractors and then reviewing their performance are important for ensuring accountability and quality of results.

question: What was the viewpoint of education screening on contractors?
answer: Education required screening of contractors before contract awards were made.

question: In what way did Education not require review of contractors?
answer: However, Education did not require review of contractors during contract performance, and states varied in whether they ensured that contractors were reviewed during the course of contract performance.

questions: question: What is the statistical analysis of Education's assistance on SIG implementation?
answer: Education’s assistance and oversight activities are generally supporting SIG implementation. In our survey, nearly all states reported they were satisfied with Education’s technical assistance, particularly the agency’s SIG guidance and conferences.

question: How was the guidance of Education's guidance observed in respect to oversight?
answer: In addition, many states reported that Education’s guidance was timely. With respect to oversight, Education monitored 12 states in school year (SY) 2010-2011 and found deficiencies in 11 of the 12 states.
child_questions: question: How is the state working to address these oversights?
answer: Education is working with states to correct these deficiencies. For SY 2011-2012, the agency plans to use a risk-based approach to conduct on-site monitoring in 14 additional states.

question: How is Education attempting to monitor oversight and for how long?
answer: To maximize its oversight resources, Education also plans to conduct some limited monitoring in five additional states in SY 2011-2012. Education officials told us that they plan to monitor the remaining states in SY 2012-2013 and that these states represent a small percentage of SIG funds.

questions: question: What is the SIG program?
answer: The School Improvement Grant (SIG) program funds reforms in low performing schools.
child_questions: question: How is congress funding the SIG from 2009 to 2012?
answer: Congress provided $3.5 billion for SIG in fiscal year 2009, and a total of about $1.6 billion was appropriated in fiscal years 2010-2012.

question: When did the SIG requirements change the most?
answer: SIG requirements changed significantly in 2010.
child_questions: question: What are these changes?
answer: Many schools receiving SIG funds must now use the funding for specific interventions, such as turning over certain school operations to an outside organization (contractor). GAO examined (1) what, if any, aspects of SIG pose challenges for successful implementation; (2) how Education and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices; and (3) to what extent Education’s technical assistance and oversight activities are effectively supporting SIG implementation. GAO surveyed SIG directors in all 50 states and the District of Columbia; analyzed Education and state documents; and interviewed officials from 8 states and school districts in those states, SIG contractors, and education experts.