Race to the Top: Education Could Better Support Grantees and Help Them Address Capacity Challenges
Released: 2015-04-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Education created RTT under the American Recovery and Reinvestment Act of 2009. From 2010 through 2011, Education awarded $4 billion in competitive grant funds to 19 states to reform core areas of K-12 education. RTT states also committed to building capacity to implement and sustain reforms. GAO and others previously reported that capacity challenges had adversely affected RTT implementation and could hinder efforts to sustain the reforms. GAO was asked to further examine these challenges.

This report examines: (1) the effect of RTT on reform and capacity challenges states and districts faced, (2) how helpful Education's assistance was to states in building and sustaining capacity, and (3) lessons learned that could inform future reform efforts.

GAO surveyed all 19 RTT states and a generalizable sample of RTT districts; held an expert panel; reviewed RTT applications, progress reports, relevant federal laws and regulations, and literature; and interviewed officials from seven selected states and districts, chosen based on survey responses. GAO selected expert panelists based on research or experience with RTT, capacity issues, and federal grants.

section_title: What GAO Found
paragraphs: The Department of Education's (Education) Race to the Top (RTT) program encouraged states to reform their K-12 educational systems, but states and districts faced various capacity challenges in implementing the reforms. RTT accelerated education reforms underway and spurred new reforms in all 19 RTT states and in an estimated 81 percent of districts, according to GAO's surveys of RTT grantees and districts that received RTT funds. At the same time, states and districts noted various challenges to their capacity to successfully support, oversee, and implement these reform efforts. For example, about one-quarter to one-third of RTT states reported that their greatest challenges involved obtaining support from stakeholders such as teacher organizations. In contrast, districts primarily reported that their greatest challenges involved financial and human capital capacity, especially with competitive compensation and standards and assessments. Additionally, rural districts reported facing greater challenges than urban and suburban districts. Education is to assist grantees in achieving successful project outcomes according to its grants handbook, while holding them accountable for their RTT reform plans. Yet, GAO found no specific activities tailored to rural needs in areas grantees identified as most challenging. A better understanding of the capacity challenges rural districts face could help Education better target its technical assistance to districts that need it the most.

In response to GAO's survey, many RTT states reported that technical assistance from Education officials and its contractor was more helpful than other RTT resources, such as web-based materials. Ten states also reported they would benefit from additional support in areas such as training and professional development. Education created a new office to oversee and provide coordinated support to RTT and other programs, and intends to develop office-wide coordinated technical assistance policies. Federal internal control standards note that adequate policies help ensure that actions are taken to address risks to achieving an agency's objectives. However, Education has not determined the type or amount of technical assistance to be provided and its policies are still being developed. RTT's $43 million technical assistance contract ends in June 2015, which may create a gap in assistance to states. Unless Education focuses on technical assistance activities that states found most useful, it risks providing ineffective assistance to programs supporting these education reforms.

GAO's panel of RTT and grant experts identified key lessons learned, such as leveraging existing funding flexibilities under federal formula grants, to help address capacity needs and sustain reforms when RTT ends in September 2015. Districts and schools may not, however, be using these flexibilities to their fullest extent, in part because of uncertainty about what is allowed under federal requirements. Federal internal control standards state that information should be communicated in a form that enables an agency to achieve its objectives. Education lacks time frames for finalizing and disseminating new guidance for states to clarify federal formula grant flexibilities; and recognizes the need for, but has not developed guidance to help auditors better understand these flexibilities. Such guidance, when finalized, may help states and districts sustain education reforms, thereby raising student achievement – a primary objective of reform.

section_title: What GAO Recommends
paragraphs: GAO recommends that Education incorporate into its coordinated policies technical assistance grantees found most useful, target assistance to rural districts, and issue guidance to help states and auditors with funding flexibilities. Education did not explicitly agree or disagree with GAO's recommendations, but outlined steps to address many aspects of them.

To view the e-supplements online, click:

section_title: Letter
paragraphs: From 2010 through 2011, the Department of Education (Education)  awarded over $4 billion in Race to the Top (RTT) grant funds to 19 states.  These RTT grantees serve about 22 million students in more than 40,000  schools, representing 45 percent of all K-12 students and 42 percent of all  low-income students nationwide. Through a competitive grant process— the largest ever administered by Education—these states had to  demonstrate a commitment to reforming four core areas of K-12  education: (1) standards and assessments; (2) data systems; (3) effective  teachers and leaders; and (4) school turnaround. States also committed  to build strong statewide capacity to implement, scale up, and sustain  their plans. Many RTT states are also implementing similar reform efforts  as a condition of obtaining Elementary and Secondary Education Act of  1965 (ESEA) waivers or funds through other Education programs, such  as School Improvement Grants.governments continue to face funding gaps that threaten to adversely  affect educational services.

At the same time, states and local  GAO and others have reported that state and district capacity challenges  have adversely affected the implementation of RTT. For example, in  2011, we reported that most RTT states faced a variety of challenges that  led to implementation delays, including difficulty hiring qualified  personnel. In September 2013, we found that most RTT states did not  meet their target date for implementing teacher and principal evaluation  reforms, citing capacity issues, such as staff expertise, among their  challenges.we interviewed for our 2013 report also cited concerns about their ability  to sustain evaluation systems after RTT grant funds are no longer  available. You asked us to examine broader capacity challenges states  and districts face as they implement all RTT reform efforts.

Education officials from most of the 12 states and 12 districts  In this report, we examine: (1) the effect RTT had on education reform,  and the capacity challenges states and districts faced in implementing  and sustaining RTT reform efforts; (2) states’ perspectives on how helpful  Education’s assistance was in helping them build and sustain capacity to  implement RTT reforms; and (3) lessons learned from RTT that could  inform future education reform efforts.

To examine capacity challenges at the state and district level we  conducted two surveys. We sent one survey to state educational agency  officials in all 19 RTT grantee states, including the District of Columbia.  We sent a second survey to a stratified random sample of 643 school  districts that received RTT funds—the results of which were generalizable  to all districts receiving RTT funds. We administered our web-based  surveys from May through September 2014. All state officials and 76.7  percent of district officials in our sample responded to the surveys.

The  surveys and a more complete tabulation of the results can be viewed at  GAO-15-316SP and GAO-15-317SP. We also conducted follow-up  interviews with officials from four state educational agencies and three  districts that received RTT funds to obtain specific examples of their  challenges, support received, and efforts to build and sustain their  capacity for reform. In addition, we reviewed documentation, including  states’ RTT applications and annual performance reports to understand  their capacity challenges and efforts to build capacity before receiving  RTT funds and throughout the grant period. To understand the assistance  Education provided to states, we met with Education’s RTT program  officials to discuss their efforts, particularly related to capacity, monitoring,  oversight, and technical assistance. We also gathered information on  Education’s assistance from our survey and in our follow-up interviews  with selected state officials.

To ascertain how lessons learned from RTT could inform future education  reform efforts, we held an expert panel that addressed the implications of  state and district capacity issues on sustaining RTT reforms and future  competitive grants. We developed discussion topics and questions for the  panelists based on information gathered from our surveys, interviews,  and academic literature. We selected experts with research or  professional experience related to RTT and other competitive grants,  state and district capacity, and federal grant making. See appendix V for  a list of panelists and appendix I for more details on our scope and  methodology, including specifics on our survey and expert panel.

We conducted this performance audit from November 2013 to April 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Overview of RTT
paragraphs: The American Recovery and Reinvestment Act of 2009 (Recovery Act)  required the Secretary of Education to provide grants to states that show  promise in meeting the objectives of four broad education reform areas  outlined in law. Education subsequently established the RTT grant fund  to encourage states to reform their K-12 education systems and to reward  states for improving certain student outcomes, such as making  substantial gains in student achievement and improving high school  graduation rates. The reforms contained in RTT were expected to help  prepare students to graduate ready for college and career, and enable  them to successfully compete with workers in other countries. Providing a  high-quality education for every student is also vital to a strong U.S.  economy.

States competed for RTT grant funds based on reforms across the  following four core reform areas:  1.  Standards and assessments: adopting standards and assessments  that prepare students to succeed in college and the workplace and to  compete in the global market;  2.  Data systems: building data systems that measure student academic  growth and success and inform teachers and principals about how  they can improve instruction;  3.  Effective teachers and leaders: recruiting, developing, rewarding,  and retaining effective teachers and principals, especially where they  are needed most; and  4.  School turnaround: turning around the lowest-achieving schools.

Education awarded RTT grants to states in three phases, with award  amounts ranging from approximately $17 million to $700 million (see  appendix II for list of grantees and award amounts). States are generally  required to sub-grant at least 50 percent of their RTT funds to school  districts within their state that signed a Memorandum of Understanding  stating their agreement to implement all or significant portions of the  state’s RTT plan (participating districts). According to Education  officials, providing a competitive grant with substantial funding to  implement ambitious plans in the four core education reform areas was  meant to encourage states to create the conditions for reform and  achieve significant improvement in student outcomes (see fig. 1). The 4- year grant period began on the date funds were awarded to the state.  Education officials stated that, of the Recovery Act funding used in 2010  for the first two phases of RTT, under federal law, any funds not obligated  and liquidated by September 30, 2015, will no longer be available.  Education made grants for the third phase of RTT from fiscal year 2011  funding, and officials told us that those funds must be liquidated by  September 30, 2017.

In awarding the RTT grants, Education used a peer review process to  evaluate applications. Capacity to implement, scale up, and sustain RTT  reforms was one of 19 primary criteria Education used to guide the  selection of RTT grantees (see appendix III for a list of these criteria).  Education did not provide a definition of capacity, but it provided guidance  to peer reviewers on how to assess the specific criterion related to  capacity: building strong statewide capacity to implement, scale up, and  sustain proposed plans. Peer reviewers evaluated states on the extent to  which they demonstrated that they would: (1) provide strong leadership  and dedicated teams to implement the reforms; (2) support participating  districts in implementing the reforms through a variety of activities, such  as identifying and disseminating promising practices; (3) provide efficient  and effective operations and processes for grant administration and  performance measurement, among other functions; (4) use RTT funds to  accomplish the state’s plans; and (5) use fiscal, political, and human  capital resources to continue successful grant-funded reforms after RTT  funds are no longer available.

section_title: Capacity to Implement RTT
paragraphs: The capacity of grantees is a key issue in grants management that can  affect program success. Capacity involves both maintaining appropriate  resources and the ability to effectively manage those resources. For the  purposes of this report, we defined capacity as the ability to successfully  support, oversee, and implement reform efforts. It includes the following  types of capacity:

Organizational Capacity: degree of preparedness for grants  management and implementation including having the appropriate  leadership, management, and structure to efficiently and effectively  implement the program and adapt as needed.

Human Capital Capacity: the extent to which an organization has  sufficient staff, knowledge, and technical skills to effectively meet its  program goals.

Financial Capacity: the extent to which an organization has sufficient  financial resources to administer or implement the grant.

Stakeholder Capacity: the extent to which an organization has  sufficient support from its stakeholders, including their authority and  commitment to execute reform efforts.

We and other researchers have noted that capacity concerns may have  important implications for competitive grants generally. For example, in  2011 and 2012, we reported on the School Improvement Grant program,  another competitive grant awarded by Education, and found that human  capital and stakeholder capacity issues influenced the implementation of   In addition, a 2011 Journal of  School Improvement Grant interventions. Federalism study demonstrated that applicant capacity is an important  factor likely to influence how competitive grants are administered and that  an applicant’s chances of winning competitive grants are strongly related  to their capacity.capacity given relatively modest levels of investment in school  improvement activities, as well as human resources, organization, and  political challenges. In a January 2014 report, Education’s Inspector  General identified common capacity-related causes for delays, such as  changes in state leadership; staffing and organizational challenges at   Other researchers also raised concerns about states’  state educational agencies; acquisitions issues; and stakeholder issues,  particularly regarding the new evaluation systems.

section_title: Role of the Department of Education
paragraphs: In 2011, Education established the Implementation and Support Unit,  within the Office of the Deputy Secretary, to administer the RTT program.  The purpose of the Implementation and Support Unit was to support the  implementation of comprehensive reforms at the state level, pilot new  approaches to strengthen and support state reforms, and act as a single  point of contact for the Education programs that were housed in that  office.of all aspects of RTT, including monitoring and technical assistance.

The office was responsible for fiscal and programmatic oversight  The Implementation and Support Unit established a program review  process to monitor RTT states’ progress toward meeting their RTT goals  and to tailor support based on individual state needs. The program review  process emphasized outcomes and the quality of RTT implementation by  states rather than focusing solely on a compliance-driven approach.  Program officials and other staff in the Implementation and Support Unit  were to work directly with states to understand their RTT plans and  objectives, observe benchmarks, and monitor the quality of  implementation. Education considered each state’s progress toward its  goals and timelines, risk factors and strategies for addressing them, and  the state’s own assessment of its quality of implementation, among other  factors. In October 2014, Education established a new Office of State  Support, which replaced the Implementation and Support Unit in the  administration and oversight of RTT.

Education provides technical assistance to RTT states via the Reform  Support Network (RSN), which it established in 2010 through a 4-year,  $43 million technical assistance contract with ICF International. The  RSN is intended to work with RTT states to build capacity to implement  and sustain reform efforts and achieve improvements in educational  outcomes, identify and share promising and effective practices, as well as  facilitate collaboration across states and among the many education  stakeholders who implement and support state reform efforts. RSN is to  provide RTT grantees one-on-one technical assistance that is tailored to  the grantee’s RTT reform plans. RSN is to ensure that the state  requesting individualized technical assistance receives the best available  and relevant expertise by identifying specific experts that a state can  contact for help.

RSN also provides collective technical assistance to RTT states through  communities of practice. Communities of practice use a variety of  mechanisms to support states in meeting their RTT goals, including the  use of working groups, publications, and various forms of direct technical  assistance, such as webinars and individualized technical assistance.  RSN established a capacity-building community of practice designed to  strengthen the organizational capacity of RTT states and a working group  to help states assess the sustainability of their reform initiatives and take  action if needed.

section_title: RTT Spurred Reform amid Different Types of State and District Capacity Challenges
subsections: section_title: RTT Both Accelerated Reforms Already Under Way and Spurred New Reform Efforts
paragraphs: RTT accelerated reforms under way or spurred new reforms in all 19  states and in an estimated 81 percent of districts that were awarded RTT  grants, according to states and districts we surveyed (see fig. 2 for district  survey responses). For example, several state officials reported in their  survey comments that their states began implementing reform activities— such as developing standards, longitudinal data systems, and new  teacher evaluation systems—before they received RTT funds. In addition,  16 states reported that RTT provided the opportunity to accelerate or  enhance existing reform plans or existing priorities. For example, one  state official reported that RTT allowed their state to increase courses in  science, technology, engineering, and math for students and teachers  and provide professional development opportunities for pre-kindergarten  teachers.

In addition, RTT may have helped promote reforms not only within the 19  states that received RTT grants, but also in the states that applied but did  not receive RTT funding. A 2014 Education study found that although  RTT states implemented more reform activities in the four core reform  areas than non-RTT states, many non-RTT states also adopted similar  reforms. Specifically, many of the 47 states that applied for the grant  had aligned their educational policies and actions to RTT’s four core  education reform areas to develop a competitive application. For  example, 43 states had adopted Common Core State Standards  (Common Core) in both math and reading/English language arts in the  2010-11 school year. Adopting college- and career-ready standards was  one of the 19 criteria peer reviewers used to select RTT grantees.  Similarly, our prior work on RTT found that four states that applied for but  were not awarded a RTT grant reported enacting new state legislation or  making formal executive branch policy changes to be more competitive  for RTT. Further, our 2011 report found that sharing information with all  states carrying out initiatives similar to RTT initiatives can accelerate the  pace and scope of reform efforts. Education developed RTT resources  and subsequently made them available to all states on its website.

section_title: States and Districts Faced Moderate but Different Capacity Challenges Implementing RTT
paragraphs: In our survey of states and districts that received RTT funds, we asked  officials to identify capacity challenges they faced in implementing and  sustaining RTT and the level of difficulty associated with each challenge  identified. In general, capacity issues posed a somewhat moderate level  of challenge to states and currently participating districts implementing  RTT, according to our survey of states and districts that received RTT  funds. However, some states and districts described particular aspects  of the four types of capacity—organizational, human capital, financial and  stakeholder—as very or extremely challenging.

For example, RTT states rated stakeholder capacity as the greatest  challenge faced while implementing RTT reform initiatives. Overall, they  rated this challenge as moderate; however, about one-quarter to one-third  of RTT states reported that obtaining support from state legislatures,  organizations that represent teachers and/or administrators, and district  leaders was very or extremely challenging. Further, in implementing  changes in two of the four core reform areas—standards and  assessments and effective teachers and leaders—more than one-third of  RTT states found stakeholder capacity to be very or extremely  challenging. Although states were encouraged to show in their grant  applications that they had garnered support for reforms from  stakeholders, some states said that they had difficulty maintaining that  support throughout the grant period. One state official told us that the  state’s teachers’ union was seeking to reverse elements of their  evaluation system linking teacher performance to student achievement,  and the legislature was seeking to reverse the adoption of the Common  Core—key elements of the state’s RTT application.

RTT states rated organizational capacity as the second greatest  challenge faced while implementing RTT. Although they rated this  challenge as moderate overall, officials from 4 of the 19 states reported  that consistency in leadership at the state educational agency was a  specific aspect of organizational capacity that was very or extremely  challenging. One state official we spoke with explained that frequent turnover at the superintendent level made implementing its teacher  evaluation system difficult because they had to constantly educate new  superintendents on how to use the evaluations to improve instruction.

School districts reported facing different types of capacity challenges than  did states. For example, school districts currently participating in RTT  reforms reported that financial capacity was the most challenging. In each  of the four core reform areas, about one-third of currently participating  districts reported that financial capacity was very or extremely challenging  to implementing RTT initiatives (see appendix IV). District officials we  surveyed stated in their written comments that decreased state funding,  the effects of the 2008 recession, and increasing enrollments affected  their financial capacity to fund reform at the local level. While RTT grant  funding to currently participating districts represented an estimated 1 to 2  percent of their budgets during each school year of the grant period,  district officials told us that RTT funds were crucial to their ability to  implement reforms.

Districts also reported particular difficulties with human capital capacity— the second greatest challenge they faced implementing RTT. currently participating in RTT reported the most challenging aspect of  human capital capacity was recruiting staff through competitive  compensation, with an estimated 45 percent of districts reporting that  doing so was very or extremely challenging. An estimated one-third of  currently participating districts also cited retaining staff and having the  appropriate number of staff among the most challenging aspects of  human capital capacity, as well as issues related to Common Core  implementation, such as having staff prepared to develop and/or  implement curricula meeting the new standards.

Human capital capacity is the extent to which an organization has sufficient staff,  knowledge, and technical skills to effectively meet its program goals.

section_title: Most States and Districts Increased Capacity During the Grant Period but Anticipate Challenges Sustaining Their Efforts
paragraphs: States and districts reported taking various actions to build and increase  their capacity overall throughout the grant period (see fig. 3). However,  both indicated that human capital and financial capacity would be the  most challenging to sustain after the RTT grant period ends. State and  district officials we spoke with explained that these issues were inter- related; that is, staff shortages and skill gaps required continued funds for  professional development.

Throughout the grant period, more than half of the 19 states reported  putting great or very great effort into building stakeholder capacity—the  area that state officials cited as the most challenging—most frequently by  consulting with organizations that represent teachers and/or  administrators (17 states), consulting with district leadership (16 states),  and building political relationships (15 states).

Similarly, most states reported building organizational capacity—another  area that presented great challenges as they implemented reforms—by,  for example, establishing an RTT point of contact or office (18 states) and  establishing communication mechanisms for RTT staff, such as group  email lists (17 states). To a lesser extent, states reported that  reorganizing an existing office (12 states) and appointing new RTT  leadership (13 states) were also helpful in building organizational  capacity. According to one state official we spoke with, the state  reorganized its entire state educational agency into departments aligned  with its RTT reforms. The official noted that the RTT grant helped the  state fund the reorganization which, in turn, helped them mitigate capacity  challenges throughout implementation. Another state official explained  that the state focused on reorganizing how staff conduct their work by  fostering collaboration among program officers.

School districts—whose second greatest capacity challenge related to  human capital—reported making great or very great effort to build human  capital capacity for RTT reform by training existing staff (80 percent),  expanding the responsibilities of current staff (74 percent), and shifting  responsibilities among staff (64 percent). Similarly, all three district  officials we spoke with in our follow-up interviews noted that efforts to  build human capital capacity focused on training and shifting the roles of  their current staff. One district official explained that they avoided funding  new staff positions that they might not be able to retain after RTT funds  ended. To build financial capacity, an estimated 23 percent of currently  participating districts reported receiving supplemental funding from their  state general fund. Additionally, an estimated 7 percent of districts  reported receiving funds from foundations to build capacity.

Despite their efforts, state and district officials reported that capacity  struggles would likely remain once the RTT grant period ends. For both  states and districts, financial capacity and human capital capacity  represented the greatest challenges to sustaining reforms (see fig. 4).  However, states and districts also reported planning to take various  actions to help sustain their capacity for reform. All 19 states, as well as  an estimated 84 percent of currently participating districts, indicated that  retaining staff with requisite knowledge and skills is part of their plan to  sustain RTT reform efforts. For example, one district official explained  that they used a large portion of their RTT funds on training for teachers  and administrators. Using the RTT funds for this purpose—as opposed to  hiring many new staff—helped them build capacity and institutional  knowledge that would be easier to sustain once the RTT funding ends.  Additionally, 17 states indicated that modifying existing staff roles and  responsibilities was the second most planned action to sustain RTT  reforms. An estimated 72 percent of districts indicated that building  institutional knowledge was their second most planned action to sustain  RTT reforms.

section_title: Rural Districts Faced Greater Capacity Challenges Than Urban and Suburban Districts in Many of the Reform Areas
paragraphs: Rural school districts reported facing significantly greater challenges than  urban districts in the standards and assessments and data systems core  reform areas when implementing RTT, according to our survey results  (see fig. 5).

These survey results are consistent with our past work on the capacity  challenges rural districts face. For example, in a 2013 report, we found  that a rural district in New York faced unique difficulties implementing its  teacher evaluation system because its small student population required  some teachers to teach more than one subject, which made the  evaluation process more complex and time-consuming. Similarly, our  prior work on implementation of School Improvement Grants showed that  rural districts faced difficulties because attracting and retaining high- quality teachers and implementing increased learning time requirements  were difficult, in part due to higher transportation costs in rural areas.

In addition, in responding to our survey, rural districts reported  anticipating more difficulty than urban districts in sustaining all four types  of capacity after the RTT grant period ends; and anticipated more  difficulty than suburban districts in sustaining three of the four capacity  types. For example, according to our survey, an estimated 40 percent of  rural districts anticipated that human capital capacity would be very or  extremely challenging in sustaining RTT reform efforts compared to 26  percent for urban and suburban districts (see fig. 6). One expert  participating on our panel agreed, noting that rural districts would also  face challenges sustaining reforms because constrained budgets and a  lack of human capital capacity are often particularly challenging for rural  districts. In addition, a rural district official told us that they have a small  number of employees, and attracting and retaining skilled employees who  can perform multiple work functions can be more difficult for them. The  official also noted that recruiting staff is a challenge because rural districts  are often also among the poorer districts and do not have the resources  to implement large-scale hiring efforts.

Although states and districts across the country likely face capacity  challenges and resource limitations to some degree, research suggests  that some rural districts—and states that have many rural districts—may  be less likely to have the skills, knowledge, or expertise to overcome  these challenges. For example, one 2013 report recommended that  states may have to play a much more direct role in guiding school  improvement in smaller, rural districts, where capacity is lacking.addition, a 2014 Education Office of Inspector General report indicated   In  this approach may be effective in reducing project delays and provided an  example of a state that planned to help districts build capacity in order to  better support low-performing schools in rural areas.

Our prior work and other research demonstrate that states with many  rural districts need additional supports in this area. Given that rural  districts reported that they faced challenges implementing and sustaining  reforms that were statistically significantly greater than urban and  suburban districts, a greater understanding of these challenges could  help Education provide more targeted support to rural districts. According  to Education’s Handbook for the Discretionary Grant Process, Education  is to provide technical assistance to grantees to help them achieve  successful project outcomes. also required to hold grantees accountable for meeting the commitments  made in their approved RTT applications. Education has recognized and  reported on challenges facing rural districts. In addition, Education  officials stated that they have supported RTT grantees and their rural  districts through a series of convenings, work groups, publications,  webinars, and individual technical assistance, and provided examples of  these activities. However, we reviewed RSN’s technical assistance  documents and found that most of the activities were not provided in the  manner that RTT states reported finding most helpful—as discussed later  in this report—nor were they tailored to helping states address the unique  capacity challenges that rural districts reported facing in the reform areas  identified in our survey. Unless Education provides assistance specifically  designed to help states support their rural districts in addressing their  capacity challenges in implementing and sustaining high-quality reform,  states may not be able to help the districts that need it the most.

U.S. Department of Education, Handbook for the Discretionary Grant Process,  Handbook OS-01 (Washington, D.C.: January 2009).

section_title: Many RTT States Found Education’s Technical Assistance the Most Helpful Resource
subsections: section_title: States Found Technical Assistance Most Helpful in Building and Sustaining Capacity
paragraphs: According to our state survey, individualized technical assistance  provided by Education program officers was the most helpful resource  when building capacity to implement and sustain reform plans (see fig. 7).  This was consistent with the views of officials we interviewed in four RTT  states, who described very positive interactions with their Education  program officer. For example, state officials explained that the program  officers practiced collaborative problem-solving and provided a significant  amount of support to the state as it implemented reform activities.

The next most helpful resources, according to our state survey, were  technical assistance provided by other staff in the Implementation and  Support Unit and RSN. One state official we spoke with noted that  Implementation and Support Unit staff provided useful information on how  other states were implementing their reform activities. An official from  another state explained that the state is working closely with RSN to  better understand how to work with its participating RTT districts to better  leverage federal funding to improve student outcomes.

As shown in figure 7, RSN’s communities of practice ranked fourth in  terms of helpfulness to build capacity to implement and sustain RTT  reform. According to state officials and one expert participating on our  panel, these communities of practice encouraged collaboration across  states, which has helped them leverage knowledge, talent, and  resources, as well as facilitate the sharing of promising practices.  Education officials observed similar value in RSN’s communities of  practice, noting that through them, states had a forum in which to learn  from each other and discuss RTT implementation issues. It is worth  noting that state officials we interviewed commented that communities of  practice may have been more helpful to states that were in the early  stages of implementing RTT reforms. For example, one official noted that  their state was farther along in implementing its teacher and principal  evaluation system and school turnaround efforts and therefore did not  gain as much from those communities of practice.

State officials ranked RSN’s capacity-building community of practice and  web-based resources from Education and RSN among the least helpful to  states. Education officials similarly noted that while webinars were an  easy way to disseminate information, they are likely not as valuable as  other RTT resources because they are not as tailored to a particular  state’s needs. Two experts participating on our panel noted that although  an abundance of school reform-related information exists on websites,  little is known about the effectiveness of the information.

In December 2013, RSN published the results of an evaluation of its  technical assistance activities that generally aligned with the results of our  state survey. For example, according to RSN’s evaluation report,  participants indicated they were satisfied with the quality of the support,  the format and content of the technical assistance activities provided by  RSN. Individualized technical assistance had the highest ratings because,  according to the evaluation report, it was designed to address a state’s  specific implementation challenges. In addition, participants in the RSN  evaluation indicated that on average, technical assistance activities had a  moderate effect on states’ ability to build capacity overall. The results of  the RSN evaluation also showed that while webinars were useful for  disseminating information to larger audiences and convening states on a  regular basis, they received lower ratings than other forms of assistance.  Our body of work on performance measures and evaluations has shown  that successful organizations conduct periodic or ad hoc program  evaluations to examine how well a program is working. These types of  evaluations allow agencies to more closely examine aspects of program  operations, factors in the program environment that may impede or  contribute to its success, and the extent to which the program is operating  as intended. Information from periodic reviews of RSN’s technical  assistance efforts are an important factor in determining if adjustments  are needed to help grantees meet their goals for education reform.

State officials we surveyed also identified additional activities that  Education could undertake that would better assist states with  implementing RTT. Specifically, 10 of 19 states reported wanting ongoing  professional development throughout the grant period, as opposed to  during the early stages of the grant. Ten of 19 states reported wanting  training to be provided in their respective states to make it more easily  accessible, rather than having to travel to Washington, D.C. Further, 11 of  19 states reported wanting assistance identifying skilled contractors who  could assist with reform efforts. Education officials stated that any  assistance it provides to identify contractors cannot compromise the  fairness and objectivity of the states’ procurement processes. Education  officials also pointed out other legal challenges to identifying contractors,  such as prohibitions against endorsements of private entities. However,  Education officials stated they can assist grantees by, for example,  helping them to develop objective criteria, analysis, or research regarding  the qualifications of skilled contractors. They said they can also provide  resource lists using objective criteria, as well as technical assistance in  this area.

section_title: Education Established a New Office and Aims to Develop Policies that Reflect a Coordinated Approach to Providing Technical Assistance
paragraphs: In October 2014, Education created the Office of State Support to expand  and sustain the collaborative approach to providing oversight and  technical assistance that began under the Implementation and Support  Unit. More specifically, the purpose of the Office of State Support is to  design a coordinated approach across multiple Education programs to  reduce redundancy and improve the efficiency and effectiveness of  Education’s oversight efforts. The Office of State Support will provide  states with one point of contact for multiple education programs that will  provide support and technical assistance. The Office of State Support  plans to establish advisory committees, involve staff from other education  programs in decision making, and maintain close communication with  staff from other education programs that have similar goals and activities  as programs covered under the new office.

Officials from the Office of State Support stated that the lessons learned  from the RTT monitoring and technical assistance processes will inform  their work in the new office for programs they oversee—many of which  are helping states to facilitate comprehensive education reforms similar to  those started under RTT. However, officials stated that they will need to  eventually transition to a longer-range plan for monitoring and reconsider  how they provide technical assistance because Education’s contract with  RSN ends on June 30, 2015. Education officials noted that it was unlikely  that the department would receive such a large amount of funding ($43  million) for technical assistance again. They explained that the type and  extent of technical assistance efforts to states after the end of the RSN  contract will, in turn, be dependent upon the funding available for that  purpose. Lastly, they said that they will look to leverage existing technical  assistance funds, such as those provided for the Comprehensive Centers  program, to help increase state capacity to assist districts and schools.

Education’s Handbook for the Discretionary Grant Process requires  program offices to develop a monitoring and technical assistance plan for  each grant program. In addition, according to Federal Standards for  Internal Control, policies and procedures help ensure that necessary  actions are taken to address risks to achieving the entity’s objectives.  Education has a monitoring and technical assistance plan for RTT, which  it has been using for the past four years and has continued to use during  the transition from the Implementation and Support Unit to the Office of  State Support. However, officials from the Office of State Support stated  that they planned to establish coordinated technical assistance processes  and procedures for all of the programs administered by the new office,  while meeting the needs of the states and their particular initiatives. For  example, they said they need to consider how to bring the various kinds  of monitoring and technical assistance conducted by different program  offices together to provide support for and make connections across  programs, and be less burdensome for states. Officials stated that they  formed a working group of staff from various Education program offices,  including former Implementation and Support Unit staff, to help inform the  new office’s coordinated technical assistance policies. However, officials  noted that the working group was in the early stages of this process, and  had not yet developed any draft policies or established a definitive  deadline for accomplishing this task.

Given the valuable technical assistance that RSN provided to states, and  that Education has not determined the type or amount of technical  assistance to be provided, there could be a gap in the type of support that  Education can provide to states when the contract expires. Until the  Office of State Support develops and finalizes policies and procedures  that include support activities states identified as most helpful, Education  runs the risk of not providing the most effective assistance to its grantees  to help them successfully implement and sustain reform efforts.

section_title: Experts Identified Lessons Learned to Help Sustain Reform and Made Observations to Consider in Future Efforts
subsections: section_title: Key Lessons Learned from RTT May Be Critical to Sustaining Reform
paragraphs: Our analysis of our expert panel transcript revealed key lessons that  could help states and districts address their greatest capacity challenges  and help sustain reforms after the RTT grant period ends.

To address challenges with financial capacity, five of the 10 experts  participating on our panel noted that federal formula grants are better  suited than competitive grants for building and sustaining capacity  because they provide a more stable funding source. Three experts  stated that there are several ways that states and districts can leverage  the funds they receive annually in formula grants to help sustain  reforms. The Title I formula grant—designed to improve schools with  high concentrations of students from low-income families—gives districts  and schools flexibility to use federal funds to support instructional  strategies and methods that best meet local needs. For example, schools  where at least 40 percent of students are from low-income families may  operate “school-wide” Title I programs, which allow schools to combine  Title I funds with other federal, state, and local funds to improve the  overall instructional program for all children in a school. In the 2012–2013  school year, approximately 40,632 schools, or 74 percent of all Title I  schools, operated school-wide programs.

Despite the large number of schools running a school-wide program,  districts and schools may not be using the flexibilities to combine Title I  funds with other federal funds to their fullest extent due, in part, to a lack  of organizational capacity at the state and district levels. According to  Education officials and two experts on our panel, states and districts are  often uncertain about whether they are allowed to combine federal  formula grants in new ways to support comprehensive reforms. For  example, Education officials told us that historically, states and districts  have used Title II funds—formula grants designed in part to increase  student academic achievement through strategies such as improving  teacher and principal quality—to reduce class size. However, according  to Education’s guidance, states and districts could also choose to  combine Title I and Title II funds to sustain reforms initiated under RTT,  such as providing academic support coaches and financial incentives and  rewards to attract and retain qualified and effective teachers to help low- performing schools. According to five experts on our panel,  uncertainties about what is allowed may stem from lack of communication  and coordination among the multiple federal education program and  financial management offices, and because these offices are not always  focused on helping states and districts better leverage their funds.

GAO/AIMD-00-21.3.1. programs to support the four core reform areas. Further, in 2013, the  Council of Chief State School Officers developed a toolkit for states to  help clarify how districts and schools may spend K-12 federal formula  grants. This toolkit encourages states to improve collaboration among  offices supported by federal grants to help ensure they effectively  leverage federal funds. Currently, Education is working with RSN to  develop another toolkit for states and districts on ways to leverage federal  formula grants to sustain educational reforms. Education officials could  not provide definitive time frames for the release and dissemination of the  toolkit, but noted that they are hoping to release it sometime in 2015. This  toolkit, when finalized, may help states and districts better understand  how to leverage their formula grants to sustain reform activities and help  raise student achievement—a primary objective of education reform.

Education officials and one expert participating on our panel also said that  states and districts do not use funding flexibilities to their fullest extent  because they have concerns about compliance with state audit  requirements. Education officials explained that states and auditors may  believe that federal law prohibits certain activities, even when the law and  its implementation rules do not. Education officials told us they tried to  address these uncertainties by issuing guidance to clarify how states and  districts can leverage federal funds to support reforms. According to this  guidance, states may use Title I funds to provide technical assistance to  low-achieving schools, and districts may consolidate Title I, Title II, and  IDEA funds in schools under the school-wide program to support  comprehensive reforms by, for example, extending the school day or  school year. However, Education officials said that there is still confusion about this issue, particularly among the audit community, and  that it needs to provide new guidance to help auditors better understand  allowable spending within federal formula grants, especially with Title I  funds. However, it does not have a definitive plan for developing and  implementing this guidance. Such guidance—when developed and fully  implemented—may help auditors better understand funding flexibilities in  existing formula grants and help states and districts fully leverage these  flexibilities.

Further, the pending reauthorization of ESEA also provides an  opportunity to address these capacity issues. Education told us that it is  exploring new options to help states and districts build capacity to  implement comprehensive reforms, including increasing the portion of  Title I grant funds that can be set aside for administrative purposes.  Currently, two of the set-asides in the Title I program limit the maximum  percentage of funds that can be set aside to support state administrative  functions and districts’ school improvement activities. Specifically, ESEA  requires that a state generally spend no more than 1 percent (or  $400,000, whichever is greater) of its Title I funds on state  administration and 4 percent on district school improvement activities.

Education told us that the current portion of funds under the ESEA Title I  grant that may be used for administrative functions may be inadequate  given the range and complexity of state-level work in supporting effective  implementation of local Title I projects. In its fiscal year 2016 budget  proposal, the Administration proposed increasing the funds a state can  spend on administration from 1 percent to 3 percent. According to  Education officials, the trade-off, particularly in a tight fiscal environment,  is that larger set-asides may reduce the portion of available funds that  would transfer to districts and schools to implement programs. In the  current Congress, the Student Success Act, which was reported out of  the House Committee on Education and the Workforce, would make  changes to both of these set-asides.

To help address human capital and stakeholder capacity challenges, five  experts on our panel noted the importance of fostering partnerships  between a state and its districts, among districts within a state, and with  non-governmental entities by, for example, convening groups of experts  across the state to share expertise, solve problems, and share lessons  learned to help leverage knowledge and talent. They further noted the  potential for such a strategy to solve common challenges, such as how to  develop effective strategies for evaluating teachers who teach subjects  that are not assessed using standardized tests (e.g. foreign language or  art). Universities with research and professional development institutes  are another potential resource to help states and districts build and  sustain human capital capacity. For example, one expert noted that  strong relationships with higher education institutions and teacher unions  are needed to revamp teacher, principal and superintendent training  programs and teacher licensure requirements. Lastly, three panelists said  that to maintain key stakeholder support for reforms, states need to show  progress in meeting their established time frames for RTT reform, or  increase student achievement.

section_title: Observations from Experts May Help Education Address Capacity Concerns in Future Grant Competitions
paragraphs: Three experts on our panel noted that competitive grants may be better  suited than formula grants for spurring reforms and innovative  approaches, but varying levels of capacity among states and districts  raises concerns about their ability to win competitive grants and  successfully implement large-scale education reforms. Research  suggests that states’ capacity was an important variable in helping to  predict who applied for RTT funds and which states scored well during  the competition. In particular, a 2011 study found that states with quality  standards and accountability procedures, and that had achieved overall  student gains, were more likely to receive higher scores during the RTT  grant competition. When making competitive grant awards in the future,  Education officials told us they expect to look at demonstrated capacity as  evidenced by a state’s performance under previous grants and may offer  a competitive priority for previous success. To help states and districts  that may be struggling in these areas, experts participating on our panel  made four observations that they believe could be incorporated into the  design of future competitive grants to help level the playing field between  high- and low-capacity states and districts. Education has incorporated  some of the observations into its competitive grant programs to varying  degrees and pointed out some advantages and disadvantages of each.

Observation 1: Allow joint applications so that states and  districts with greater capacity can partner with those with less  capacity. Education noted that it used this approach in recent grant  competitions. Education encouraged states that opted to adopt a  common set of college- and career-ready standards to form  collaborative groups to apply for RTT assessment grants to develop  assessments aligned with the new standards. A 2011 study  proposed that such arrangements could help states with less capacity  more easily benefit from the initiatives of ones with more capacity by  helping them identify partners and providing them access to funds that  may help valuable reforms gain traction. Education officials told us,  however, that when they have allowed joint applications or consortia  for some competitive grants, the complexity of implementing the  grants increased because states have different procurement rules  which take longer to navigate. Education officials also noted that  these joint initiatives sometimes take longer to implement because  states have to establish a framework for how they are going to  coordinate.

Observation 2: Staggering or “phasing” competitive grant  funding to allow for varying capacity needs of grantees.  Education officials told us that they have had mixed success using  planning grants to allow grantees additional time to build capacity to  implement plans. For example, Education used a two-phase strategy  for awarding competitive grants under its Promise Neighborhoods  grant program, including 1-year planning grants to organizations to  enhance the grantees’ capacity and a separate competition for a 5- year implementation grant to organizations that demonstrated they  were ready to implement their plans. However, we recently reported  that Education did not communicate clearly to grantees about its  expectations for the planning grants and the likelihood of receiving  implementation grants. Education officials told us that they do not  always have the authority to offer this feature, but they consider it  where it is possible. Education officials told us that they are  considering adding a planning year to the School Improvement Grant,  which is federal money awarded to states that states, in turn, award to  districts using a competitive process. Education officials told us that  they believe that low-capacity districts could benefit from this  approach, but noted that it will be important to emphasize their  expectation that grantees use the planning year to build capacity to  implement their reform plans.

Observation 3: Allowing intermediary entities that often help  coordinate or provide technical assistance to districts to apply  for competitive grants. Education officials told us that they see a  benefit to using partners such as nonprofit organizations to drive  reform, noting, for example, that the Investing in Innovation program  allows nonprofits to partner with school districts as part of the  application process and throughout the grant period. Research  supports such an approach as well. A 2011 RAND study examining  the federal and state role in improving schools in 15 states found that  although some states assumed primary responsibility for assisting  low-performing schools, others relied on regional organizations, area  education agencies, or intermediate school districts to fill this role.However, Education officials noted that applicant eligibility is generally  defined in statute.

Observation 4: Streamlining Education’s grant application  processes to make it easier for states and districts with less  capacity to apply. Education officials told us that one example of  streamlining the grant process was allowing states that did not win an  award in the first phase of a competition to revise the same  application and resubmit for subsequent phases. Education adopted  this strategy in the RTT grant competition. Another way to streamline  the grant application process is by encouraging shorter applications.  Education officials said it used this approach in a grant competition for  the Investing in Innovation program. Education officials noted that, in  general, one disadvantage to shorter applications is that there may  not be sufficient detail in the applications to hold grantees accountable  for implementing their plans.

section_title: Conclusions
paragraphs: As Education’s technical assistance contract for RSN comes to a close,  and it develops new processes for technical assistance under the new  Office of State Support, it has an opportunity to apply the technical  assistance that RTT states reported as most helpful, such as  individualized technical assistance and professional development, to  other grant programs that the office oversees. Such technical assistance  could help states implement and sustain the comprehensive education  reforms which will continue to be supported by other grant programs  managed by the Office of State Support.

In addition, because rural districts face unique challenges implementing  and sustaining RTT reforms, focusing efforts to enhance Education’s  understanding of the types of additional supports they may need could  help these districts successfully implement and sustain their reform  efforts, and ultimately improve student achievement. Further, as the RTT  grant period comes to an end, RTT states may need to better leverage  their federal formula grants to continue to support comprehensive reform  in the absence of RTT funds. Education officials and other experts have  emphasized the importance of leveraging existing funding flexibilities in  education formula grants to help states implement and sustain large-scale  reform efforts. However, concerns about a lack of communication  between states’ program and financial management offices, as well as  concerns about non-compliance with state and federal requirements may  be limiting states’ willingness to use the funding flexibilities present in  current law to develop and implement strategies tailored to their unique  local needs. By taking actions to address these issues, Education can  help states and districts better use their federal funding in the most  effective way to improve student achievement and to support  comprehensive school reform.

section_title: Recommendations
paragraphs: To help ensure that states are better able to sustain RTT reforms and that  Education can effectively support other grant programs managed by the  Office of State Support, we recommend that the Secretary of Education  direct the Office of State Support to fully implement and incorporate into  its coordinated technical assistance policies and procedures the types of  support that would be useful in sustaining RTT reforms and providing  effective support to grantees in other programs supporting education  reform that the Office of State Support oversees. These could include: providing individualized technical assistance to states, such as that  currently provided by Education program officers;  facilitating communities of practice to promote opportunities for  collaboration across states; providing professional development (or training) throughout the grant  period, as opposed to only during the early stages of the grant; making training more easily accessible by conducting training locally  in their respective states, when possible; and  to the extent permissible in the context of federal and state  requirements and restrictions, exploring the possibility of assisting  states in identifying skilled contractors to help implement reform  efforts.

To help states address capacity challenges as they sustain  comprehensive education reforms similar to RTT, we recommend that the  Secretary of Education direct the Office of State Support to take steps,  such as: providing ongoing individualized technical assistance to states to help  them target assistance to rural districts, particularly in the reform  areas that were most challenging for rural districts;  finalizing and disseminating guidance to be included in Education’s  toolkit to help states leverage federal formula grants to sustain  education reforms; and  clarifying and improving understanding of how funding flexibilities in  existing formula grants could be used to support education reform  efforts to help states and the audit community address impediments to  using formula grants in different ways.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Education for  comment. Education provided technical comments, which we  incorporated into the report as appropriate. Education’s written comments  are reproduced in appendix VI and summarized below. Education did not  explicitly agree or disagree with our recommendations, but outlined steps  to address many elements contained in them. It also provided additional  information related to our findings and recommendations.

In response to our first recommendation, Education stated that it shares  our interest in supporting states as they sustain RTT reforms and  supporting other grant programs under the Office of State Support  through performance management and technical assistance. To this end,  Education described plans to build on its generally successful RTT  monitoring strategy to develop a consolidated technical assistance  strategy for all programs under the auspices of the Office of State  Support. We have added clarifying language in the body of the report to  better reflect existing elements of the RTT monitoring and technical  assistance plan.

Education’s plan to provide coordinated policy development, performance  management, technical assistance, and data analysis services through a  structure intended to more effectively support the implementation of key  reforms and provide individualized support is a positive step. These  coordinated policies and procedures could continue to support RTT  grantees as well as other grantees under other Office of State Support  programs that have a role in helping states implement comprehensive  education reforms. However, we continue to believe that until these  policies are fully implemented, Education risks providing less effective  support than it otherwise might. Further, as Education’s technical  assistance contract for RSN comes to an end, we continue to believe that  Education should take explicit steps to incorporate into its new  consolidated assistance strategy for all programs under the Office of  State Support the technical assistance activities that RTT grantees  identified as being most helpful to them in sustaining their reforms. In  addition, Education should incorporate those additional supports that  states reported as desirable. We have clarified the intent of our  recommendation accordingly.

In response to our second recommendation, Education agreed that it is  important to identify ways to help states target assistance to rural districts.  Education stated, however, that the draft report does not adequately  recognize the actions it has taken to support RTT grantees in rural states  and districts, and provided a list of 17 activities it has undertaken through  RSN to support rural areas. We acknowledge Education’s efforts to  provide support to rural areas and have incorporated additional  information in the draft report, as appropriate, to reflect this. However, in  further reviewing these 17 activities, we found significant limitations and  believe our overall finding and corresponding recommendation is still  warranted. Specifically:

Nearly all of the activities (16 of 17) were in the form of working  groups, convenings, webinars, toolkits, and publications developed by  the RSN, many of which were located on the RSN website. According  to our survey of all 19 RTT states, web-based resources were among  the least helpful to RTT states in building and sustaining the  necessary capacity to implement reforms. Only one of the 17 activities  provided individualized technical assistance which, according to our  survey, was the most helpful form of assistance to RTT states. We  realize that Education formed RSN to provide support in a variety of  formats and agree that RSN has generally well supported RTT  grantees. However, given the unique capacity challenges that rural  districts face, we believe there is value in offering technical assistance  tailored to the individual needs of rural areas.

According to our generalizable survey of districts that received RTT  funds, rural districts faced statistically significantly greater challenges  than urban districts in implementing reforms in two areas: standards  and assessments and data systems. However, 14 of the 17 RSN  activities focused on the other two reform areas (school turnaround  and effective teachers and leaders). RSN’s efforts to focus resources  on assisting states in implementing RTT reforms are important ones,  and we believe that many states and districts may have benefitted  from these efforts. However, in order to best support states that are  working to implement and sustain reforms in their rural districts,  Education should target future support in the reform areas in which  rural districts most struggled: standards and assessments and data  systems. Accordingly, we modified our recommendation to clarify that  Education should take steps to provide targeted assistance to states  in those reform areas that we have identified as statistically  significantly more challenging for rural districts.

Many of the activities undertaken to support rural districts were  conducted in 2012 and 2013 (6 of the 11 that included specific dates)  when states and districts were fully engaged in implementing RTT  reforms. However, our survey of districts that received RTT funds was  deployed from June through September 2014, and the results  indicated that rural districts continued to face challenges long after  they would have availed themselves of these resources.

Some of the activities (6 of 17) provided support that was not  specifically tailored for rural districts; rather, it could be applied in  rural, suburban, and urban school settings alike. We continue to  believe that opportunities exist to help states better target support to  rural districts. Without a better understanding of the unique capacity  challenges that rural districts face, and a more focused approach to  providing support, Education may not be able to help the states and  districts that need it the most.

Finally, Education recognized the importance of clarifying its guidance on  the use of funding flexibilities and provided several examples of “Dear  Colleague” letters it has provided to states. We referenced one of these  letters in the draft of the report. We did not include the other two “Dear  Colleague” letters (guidance related to leveraging federal funds to support  school counselors and digital education) because they do not address the  use of funding flexibilities in support of education reform initiatives, which  was at the heart of our finding and corresponding recommendation. To  address this apparent confusion we have clarified our recommendation  accordingly. We noted in our report, and Education emphasized, that it is  working with RSN to release new guidance in 2015 on ways to leverage  federal grants to sustain educational reforms. However, as stated in our  report, Education officials could not provide definitive time frames for the  release and dissemination of the toolkit. We continue to believe that until  this guidance is fully implemented, states and districts will continue to lack  clarity on how to leverage their formula grants to sustain reform activities.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Education, and other interested parties. In  addition, the report will be available at no charge on GAO’s web site at  http://www.gao.gov.

If you or your staff should have any questions about this report, please  contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix VII.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We framed our study of capacity challenges faced by states and districts  implementing Race to the Top (RTT) reforms around three objectives: (1)  What effect did RTT have on education reform, and what capacity  challenges did states and districts face in implementing and sustaining  RTT initiatives?; (2) How helpful was the assistance the U.S. Department  of Education provided to states to build capacity to implement and sustain  RTT reforms?; and (3) What lessons have been learned from RTT that  could inform future education reform efforts?

In addressing these objectives, we incorporated elements of “grounded  foresight,” a methodological approach developed by GAO to examine  future implications by identifying key trends, emerging challenges, and  opportunities to inform government’s future role and responsibilities.  According to GAO’s internal grounded foresight methodology paper, the  heart of the proposed approach consists of three elements of grounding,  designed to support GAO’s core values of integrity and reliability: (1) a  strong factual-conceptual base, (2) one or more methods for discussing  or anticipating the future, and (3) transparent communication of the  outcomes. We developed a strong factual-conceptual base to assure that  relevant trends and occurrences related to capacity issues and  competitive grants are documented, recognized, and understood as part  of the study. We reviewed and analyzed existing literature on capacity  issues and competitive grants in K-12 education using GAO’s prospective   We examined the features of RTT, and  evaluation synthesis approach.reviewed findings from published reports to identify capacity challenges.  We also deployed two web-based surveys of state educational agency  and district officials; reviewed relevant federal laws, regulations, and  guidance; and conducted interviews with a variety of federal, state, and  local officials. We then convened a panel of experts who were  knowledgeable about capacity issues and federal grants to obtain their  views on the implications of capacity challenges on the sustainability of  RTT reform efforts and potential future competitive grants. We made the  results of the two web-based surveys publicly available to help ensure  transparent communication of the capacity challenges states and districts  reported facing.
subsections: section_title: Surveys of RTT Grantees and Districts
paragraphs: To obtain information on capacity challenges states faced in  implementing and sustaining RTT reforms we conducted a web-based  survey of RTT points of contact at each state educational agency in all 19   We conducted the survey from May through July 2014. In  grantee states.the survey, we asked RTT states about their capacity to implement RTT  efforts, the support received to do so, and efforts to build and sustain  capacity for RTT reform, among other things. We received responses  from all 19 RTT states for a 100 percent response rate. We reviewed  state responses and followed up by telephone and e-mail with selected  states for additional clarification and context. We also published survey  responses in an e-publication supplemental to this report, RACE TO THE  TOP: Survey of State Educational Agencies’ Capacity to Implement  Reform (GAO-15-316SP, April 2015).

To obtain information on capacity challenges districts faced in  implementing and sustaining RTT reform efforts we conducted a web- based survey of a sample of district officials whose districts received RTT  funds. We selected a stratified random sample of 643 from 3,251 school  districts that received RTT funds from a population of 18,541 school  districts in the 19 RTT states (see table 1). Although the focus was on  districts that currently receive RTT funds, we also included districts that  initially were participating in RTT but later decided to formally withdraw.

We obtained data from Education’s National Center for Education  Statistics, which maintains the Common Core of Data for public school  districts, for the 2011-12 school year. Our sample allowed us to make  estimates to all RTT districts and to subpopulations by urban status of the  district.

We conducted the school district survey from June through September  2014 and had a 76.7 percent final weighted response rate. Because we  followed a probability procedure based on random selections, our sample  is only one of a large number of samples that we might have drawn.  Since each sample could have provided different estimates, we  expressed our confidence in the precision of our particular sample’s  results as a 95 percent confidence interval (e.g., plus or minus 6  percentage points). This is the interval that would contain the actual  population value for 95 percent of the samples we could have drawn.  Unless otherwise noted, all percentage estimates in this report have  confidence intervals within plus or minus 6 percentage points. For other  estimates, the confidence intervals are presented along with the  estimates themselves. In the survey, we asked questions about school  districts’ capacity to implement RTT efforts, the support received to do so,  and efforts to build and sustain capacity for RTT reform, among other  things. We reviewed survey responses and followed up by telephone and  e-mail with selected districts, as needed for additional clarification and to  determine that their responses were complete, reasonable, and  sufficiently reliable for the purposes of this report. We also published  survey responses in an e-publication supplement to this report, RACE TO  THE TOP: Survey of School Districts’ Capacity to Implement Reform  (GAO-15-317SP, April 2015).

The quality of the state and district survey data can be affected by  nonsampling error, which includes variations in how respondents interpret  questions, respondents’ willingness to offer accurate responses, and data  collection and processing errors. To minimize such error, we included the  following steps in developing the survey and in collecting and analyzing  survey data. We pretested draft versions of the instrument with state  educational agency officials in three states and officials in four districts to  check the clarity of the questions and the flow and layout of the survey.  On the basis of the pretests, we made revisions to both surveys. We  contacted respondents to clarify any questions or responses where  appropriate. Further, using a web-based survey and allowing state and  district officials to enter their responses into an electronic instrument  created an automatic record for each state and district and eliminated the  errors associated with a manual data entry process. In addition, the  programs used to analyze the survey data were independently verified to  ensure the accuracy of this work.

section_title: Expert Panel
paragraphs: To obtain information on lessons learned from RTT that could inform  future education reform efforts, we convened a group of knowledgeable  individuals for an expert panel. In identifying the experts, we compiled a  preliminary list of 15 individuals with research or professional experience  related to RTT reforms, state and district capacity, federal grant making,  and state or federal education policy. These experts represented the  following entities: state educational agencies, school districts, education  associations, academia, and education think tanks. They also included a  former Education official and a representative from Education’s Office of  Inspector General. We identified a state educational agency official based  on participation in RTT and the state’s proximity to Washington, D.C.  where the panel was convened.

To obtain a different local perspective, we selected a school district  official from a different state. In addition, we selected the school district  based on proximity to Washington, D.C. and the extent to which the  district had completed questions in our district survey. An external expert  who conducted extensive research on K-12 education and federal policy  vetted our initial list of panelists. We used feedback from this expert,  along with biographical information about the experts, to determine which  experts would be invited to participate.

The resulting 10 experts participated in a 1-day panel focused on capacity  challenges and their implications for RTT reforms and future competitive  grants (see appendix V for list of participants). Each panelist completed a  questionnaire to document any conflicts of interest. This information was  not used to determine the qualification of the expert for the panel, but to  ensure that we were aware of circumstances that could be viewed by  others as affecting the expert’s point of view on these topics. We  developed discussion topics and questions for the panelists based on  information gathered from the surveys, interviews, and academic  literature. A contractor recorded the panel and transcribed the discussion.  We performed a content analysis of the transcript of the panel discussion  to develop common themes among the experts on lessons learned from  RTT that could help sustain reform efforts, inform the design or  implementation of future education competitive grants, and inform future  education reform efforts. We tallied responses for each panelist who  commented on those themes. This analysis was independently verified to  ensure the accuracy of this work.

section_title: Review of Laws, Regulations, and Guidance and Interviews with Officials
paragraphs: For all three objectives, we reviewed relevant federal laws, regulations,  and guidance—including federal internal control standards and  Education’s Handbook for the Discretionary Grant Process—and  interviewed federal, state, and district officials and other experts regarding  capacity to implement and sustain RTT reforms. We reviewed RTT  applications to identify commitments states made to build capacity to  implement RTT initiatives. To identify actions taken to build capacity, we  compared the states’ commitments to information provided in their  progress reports for school year 2012-2013. We also reviewed  information on Education’s efforts to assist states with building capacity,  such as guidance, technical assistance, webinars, and other information  on the RTT website. We interviewed federal officials from the  Implementation and Support Unit in Education’s Office of the Deputy  Secretary and staff from the newly established Office of State Support. In  addition, we conducted interviews with a variety of interested parties,  such as educational organizations, researchers, and university  professors. For example, we met with representatives from the American  Association of School Administrators, the Council of Chief State School  Officers, and the Center on Reinventing Public Education, among others.  We also conducted follow-up interviews with officials in four state  educational agencies and three districts to obtain more detailed  information and illustrative examples. We selected these state and district  officials based on their responses to our surveys and representation  across award phase.

We conducted this performance audit from November 2013 to April 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Race to the Top (RTT) Grant Awards by Phase

section_title: Appendix III: Race to the Top State Selection Criteria

section_title: Appendix IV: Capacity Challenges by Race to the Top Reform Area and Type of Capacity, as Reported by States and Estimated by Districts
paragraphs: Appendix IV: Capacity Challenges by Race to  the Top Reform Area and Type of Capacity, as  Reported by States and Estimated by Districts STATES (Percent and Number)

DISTRICTS (Estimated Percentage)

11% (2)  17% (3)  11% (2)  39% (7)  24% (4)  18% (3)

Effective Teachers and Leaders   Organizational  24% (4)  24% (4)  22% (4)  11% (2)  22%(4)
subsections: section_title: School Turnaround Organizational
paragraphs: 44% (8)

22%  33%  16%  40% (6)  27% (4)  33% (5)  33% (5)

section_title: Appendix V: List of Participants on GAO’s Panel on Implications of Race to the Top Capacity Challenges

section_title: Appendix VI: Comments from the U.S. Department of Education

section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Morrison (Assistant  Director), Jamila Jones Kennedy (Analyst-in-Charge), Sheranda  Campbell, Kathryn O’Dea Lamas, Amanda Parker, and Stacy Spence  made significant contributions to this report. Assistance, expertise, and  guidance were provided by David Chrisinger, Nancy Donovan, Alexander  Galuten, Catherine Hurley, Jill Lacey, Jean McSween, Mark Ramage,  Walter Vance, and Mimi Nguyen.


Question-summary annotations: questions: question: How successful was the Race to the Top program?
answer: The Department of Education's (Education) Race to the Top (RTT) program encouraged states to reform their K-12 educational systems, but states and districts faced various capacity challenges in implementing the reforms.
child_questions: question: What did GAO's assessment of RTT determine?
answer: RTT accelerated education reforms underway and spurred new reforms in all 19 RTT states and in an estimated 81 percent of districts, according to GAO's surveys of RTT grantees and districts that received RTT funds. At the same time, states and districts noted various challenges to their capacity to successfully support, oversee, and implement these reform efforts.
child_questions: question: What challenges did states report?
answer: For example, about one-quarter to one-third of RTT states reported that their greatest challenges involved obtaining support from stakeholders such as teacher organizations.

question: What did districts report as a main challenge?
answer: In contrast, districts primarily reported that their greatest challenges involved financial and human capital capacity, especially with competitive compensation and standards and assessments.
child_questions: question: How did these challenges compare between rural and urban districts?
answer: Additionally, rural districts reported facing greater challenges than urban and suburban districts.

question: How is Education expected to be involved with its grantees?
answer: Education is to assist grantees in achieving successful project outcomes according to its grants handbook, while holding them accountable for their RTT reform plans.
child_questions: question: To what extent did Education meet this expectation?
answer: Yet, GAO found no specific activities tailored to rural needs in areas grantees identified as most challenging.
child_questions: question: How would improving in this area bolster Education's RTT effort?
answer: A better understanding of the capacity challenges rural districts face could help Education better target its technical assistance to districts that need it the most.

questions: question: What type of assistance did most RTT states report was the most helpful from Education officials?
answer: In response to GAO's survey, many RTT states reported that technical assistance from Education officials and its contractor was more helpful than other RTT resources, such as web-based materials.
child_questions: question: In what other areas did states say they would benefit from more support?
answer: Ten states also reported they would benefit from additional support in areas such as training and professional development.

question: How has Education worked to increase their level of support?
answer: Education created a new office to oversee and provide coordinated support to RTT and other programs, and intends to develop office-wide coordinated technical assistance policies.
child_questions: question: How do these efforts compare to federal internal control standards?
answer: :Federal internal control standards note that adequate policies help ensure that actions are taken to address risks to achieving an agency's objectives. However, Education has not determined the type or amount of technical assistance to be provided and its policies are still being developed.

question: Why should Education prioritize the assistance that states found most useful?
answer: Unless Education focuses on technical assistance activities that states found most useful, it risks providing ineffective assistance to programs supporting these education reforms.

question: What is the future of RTT's technical assistance contract?
answer: RTT's $43 million technical assistance contract ends in June 2015, which may create a gap in assistance to states.

questions: question: What did GAO's panel of RTT and grant experts identify?
answer: GAO's panel of RTT and grant experts identified key lessons learned, such as leveraging existing funding flexibilities under federal formula grants, to help address capacity needs and sustain reforms when RTT ends in September 2015.
child_questions: question: Why may districts and schools not be taking full advantage of these funding flexibilities?
answer: Districts and schools may not, however, be using these flexibilities to their fullest extent, in part because of uncertainty about what is allowed under federal requirements.

question: To what extent does Education's program match federal internal control standards?
answer: Federal internal control standards state that information should be communicated in a form that enables an agency to achieve its objectives. Education lacks time frames for finalizing and disseminating new guidance for states to clarify federal formula grant flexibilities; and recognizes the need for, but has not developed guidance to help auditors better understand these flexibilities.
child_questions: question: How would better guidance help states and districts?
answer: Such guidance, when finalized, may help states and districts sustain education reforms, thereby raising student achievement – a primary objective of reform.

questions: question: Under what act was RTT created?
answer: Education created RTT under the American Recovery and Reinvestment Act of 2009.
child_questions: question: What was the purpose of the funding Education awarded?
answer: From 2010 through 2011, Education awarded $4 billion in competitive grant funds to 19 states to reform core areas of K-12 education.

question: To what did the RTT states commit?
answer: RTT states also committed to building capacity to implement and sustain reforms.

question: What had GAO previously reported as a challenge to sustaining reforms?
answer: GAO and others previously reported that capacity challenges had adversely affected RTT implementation and could hinder efforts to sustain the reforms.
child_questions: question: What was GAO asked to do regarding these challenges?
answer: GAO was asked to further examine these challenges.