Race to the Top: Education Could Better Support Grantees and Help Them Address Capacity Challenges
Released: 2015-04-13T00:00:00

Letter

Background

Overview of RTT

The American Recovery and Reinvestment Act of 2009 (Recovery Act) required the Secretary of Education to provide grants to states that show promise in meeting the objectives of four broad education reform areas outlined in law. Education subsequently established the RTT grant fund to encourage states to reform their K-12 education systems and to reward states for improving certain student outcomes, such as making substantial gains in student achievement and improving high school graduation rates. The reforms contained in RTT were expected to help prepare students to graduate ready for college and career, and enable them to successfully compete with workers in other countries. Providing a high-quality education for every student is also vital to a strong U.S. economy.

States competed for RTT grant funds based on reforms across the following four core reform areas: 1. Standards and assessments: adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global market; 2. Data systems: building data systems that measure student academic growth and success and inform teachers and principals about how they can improve instruction; 3. Effective teachers and leaders: recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most; and 4. School turnaround: turning around the lowest-achieving schools.

Education awarded RTT grants to states in three phases, with award amounts ranging from approximately $17 million to $700 million (see appendix II for list of grantees and award amounts). States are generally required to sub-grant at least 50 percent of their RTT funds to school districts within their state that signed a Memorandum of Understanding stating their agreement to implement all or significant portions of the state’s RTT plan (participating districts). According to Education officials, providing a competitive grant with substantial funding to implement ambitious plans in the four core education reform areas was meant to encourage states to create the conditions for reform and achieve significant improvement in student outcomes (see fig. 1). The 4- year grant period began on the date funds were awarded to the state. Education officials stated that, of the Recovery Act funding used in 2010 for the first two phases of RTT, under federal law, any funds not obligated and liquidated by September 30, 2015, will no longer be available. Education made grants for the third phase of RTT from fiscal year 2011 funding, and officials told us that those funds must be liquidated by September 30, 2017.

In awarding the RTT grants, Education used a peer review process to evaluate applications. Capacity to implement, scale up, and sustain RTT reforms was one of 19 primary criteria Education used to guide the selection of RTT grantees (see appendix III for a list of these criteria). Education did not provide a definition of capacity, but it provided guidance to peer reviewers on how to assess the specific criterion related to capacity: building strong statewide capacity to implement, scale up, and sustain proposed plans. Peer reviewers evaluated states on the extent to which they demonstrated that they would: (1) provide strong leadership and dedicated teams to implement the reforms; (2) support participating districts in implementing the reforms through a variety of activities, such as identifying and disseminating promising practices; (3) provide efficient and effective operations and processes for grant administration and performance measurement, among other functions; (4) use RTT funds to accomplish the state’s plans; and (5) use fiscal, political, and human capital resources to continue successful grant-funded reforms after RTT funds are no longer available.

Capacity to Implement RTT

The capacity of grantees is a key issue in grants management that can affect program success. Capacity involves both maintaining appropriate resources and the ability to effectively manage those resources. For the purposes of this report, we defined capacity as the ability to successfully support, oversee, and implement reform efforts. It includes the following types of capacity:

Organizational Capacity: degree of preparedness for grants management and implementation including having the appropriate leadership, management, and structure to efficiently and effectively implement the program and adapt as needed.

Human Capital Capacity: the extent to which an organization has sufficient staff, knowledge, and technical skills to effectively meet its program goals.

Financial Capacity: the extent to which an organization has sufficient financial resources to administer or implement the grant.

Stakeholder Capacity: the extent to which an organization has sufficient support from its stakeholders, including their authority and commitment to execute reform efforts.

We and other researchers have noted that capacity concerns may have important implications for competitive grants generally. For example, in 2011 and 2012, we reported on the School Improvement Grant program, another competitive grant awarded by Education, and found that human capital and stakeholder capacity issues influenced the implementation of In addition, a 2011 Journal of School Improvement Grant interventions. Federalism study demonstrated that applicant capacity is an important factor likely to influence how competitive grants are administered and that an applicant’s chances of winning competitive grants are strongly related to their capacity.capacity given relatively modest levels of investment in school improvement activities, as well as human resources, organization, and political challenges. In a January 2014 report, Education’s Inspector General identified common capacity-related causes for delays, such as changes in state leadership; staffing and organizational challenges at Other researchers also raised concerns about states’ state educational agencies; acquisitions issues; and stakeholder issues, particularly regarding the new evaluation systems.

Role of the Department of Education

In 2011, Education established the Implementation and Support Unit, within the Office of the Deputy Secretary, to administer the RTT program. The purpose of the Implementation and Support Unit was to support the implementation of comprehensive reforms at the state level, pilot new approaches to strengthen and support state reforms, and act as a single point of contact for the Education programs that were housed in that office.of all aspects of RTT, including monitoring and technical assistance.

The office was responsible for fiscal and programmatic oversight The Implementation and Support Unit established a program review process to monitor RTT states’ progress toward meeting their RTT goals and to tailor support based on individual state needs. The program review process emphasized outcomes and the quality of RTT implementation by states rather than focusing solely on a compliance-driven approach. Program officials and other staff in the Implementation and Support Unit were to work directly with states to understand their RTT plans and objectives, observe benchmarks, and monitor the quality of implementation. Education considered each state’s progress toward its goals and timelines, risk factors and strategies for addressing them, and the state’s own assessment of its quality of implementation, among other factors. In October 2014, Education established a new Office of State Support, which replaced the Implementation and Support Unit in the administration and oversight of RTT.

Education provides technical assistance to RTT states via the Reform Support Network (RSN), which it established in 2010 through a 4-year, $43 million technical assistance contract with ICF International. The RSN is intended to work with RTT states to build capacity to implement and sustain reform efforts and achieve improvements in educational outcomes, identify and share promising and effective practices, as well as facilitate collaboration across states and among the many education stakeholders who implement and support state reform efforts. RSN is to provide RTT grantees one-on-one technical assistance that is tailored to the grantee’s RTT reform plans. RSN is to ensure that the state requesting individualized technical assistance receives the best available and relevant expertise by identifying specific experts that a state can contact for help.

RSN also provides collective technical assistance to RTT states through communities of practice. Communities of practice use a variety of mechanisms to support states in meeting their RTT goals, including the use of working groups, publications, and various forms of direct technical assistance, such as webinars and individualized technical assistance. RSN established a capacity-building community of practice designed to strengthen the organizational capacity of RTT states and a working group to help states assess the sustainability of their reform initiatives and take action if needed.

RTT Spurred Reform amid Different Types of State and District Capacity Challenges

RTT Both Accelerated Reforms Already Under Way and Spurred New Reform Efforts

RTT accelerated reforms under way or spurred new reforms in all 19 states and in an estimated 81 percent of districts that were awarded RTT grants, according to states and districts we surveyed (see fig. 2 for district survey responses). For example, several state officials reported in their survey comments that their states began implementing reform activities— such as developing standards, longitudinal data systems, and new teacher evaluation systems—before they received RTT funds. In addition, 16 states reported that RTT provided the opportunity to accelerate or enhance existing reform plans or existing priorities. For example, one state official reported that RTT allowed their state to increase courses in science, technology, engineering, and math for students and teachers and provide professional development opportunities for pre-kindergarten teachers.

In addition, RTT may have helped promote reforms not only within the 19 states that received RTT grants, but also in the states that applied but did not receive RTT funding. A 2014 Education study found that although RTT states implemented more reform activities in the four core reform areas than non-RTT states, many non-RTT states also adopted similar reforms. Specifically, many of the 47 states that applied for the grant had aligned their educational policies and actions to RTT’s four core education reform areas to develop a competitive application. For example, 43 states had adopted Common Core State Standards (Common Core) in both math and reading/English language arts in the 2010-11 school year. Adopting college- and career-ready standards was one of the 19 criteria peer reviewers used to select RTT grantees. Similarly, our prior work on RTT found that four states that applied for but were not awarded a RTT grant reported enacting new state legislation or making formal executive branch policy changes to be more competitive for RTT. Further, our 2011 report found that sharing information with all states carrying out initiatives similar to RTT initiatives can accelerate the pace and scope of reform efforts. Education developed RTT resources and subsequently made them available to all states on its website.

States and Districts Faced Moderate but Different Capacity Challenges Implementing RTT

In our survey of states and districts that received RTT funds, we asked officials to identify capacity challenges they faced in implementing and sustaining RTT and the level of difficulty associated with each challenge identified. In general, capacity issues posed a somewhat moderate level of challenge to states and currently participating districts implementing RTT, according to our survey of states and districts that received RTT funds. However, some states and districts described particular aspects of the four types of capacity—organizational, human capital, financial and stakeholder—as very or extremely challenging.

For example, RTT states rated stakeholder capacity as the greatest challenge faced while implementing RTT reform initiatives. Overall, they rated this challenge as moderate; however, about one-quarter to one-third of RTT states reported that obtaining support from state legislatures, organizations that represent teachers and/or administrators, and district leaders was very or extremely challenging. Further, in implementing changes in two of the four core reform areas—standards and assessments and effective teachers and leaders—more than one-third of RTT states found stakeholder capacity to be very or extremely challenging. Although states were encouraged to show in their grant applications that they had garnered support for reforms from stakeholders, some states said that they had difficulty maintaining that support throughout the grant period. One state official told us that the state’s teachers’ union was seeking to reverse elements of their evaluation system linking teacher performance to student achievement, and the legislature was seeking to reverse the adoption of the Common Core—key elements of the state’s RTT application.

RTT states rated organizational capacity as the second greatest challenge faced while implementing RTT. Although they rated this challenge as moderate overall, officials from 4 of the 19 states reported that consistency in leadership at the state educational agency was a specific aspect of organizational capacity that was very or extremely challenging. One state official we spoke with explained that frequent turnover at the superintendent level made implementing its teacher evaluation system difficult because they had to constantly educate new superintendents on how to use the evaluations to improve instruction.

School districts reported facing different types of capacity challenges than did states. For example, school districts currently participating in RTT reforms reported that financial capacity was the most challenging. In each of the four core reform areas, about one-third of currently participating districts reported that financial capacity was very or extremely challenging to implementing RTT initiatives (see appendix IV). District officials we surveyed stated in their written comments that decreased state funding, the effects of the 2008 recession, and increasing enrollments affected their financial capacity to fund reform at the local level. While RTT grant funding to currently participating districts represented an estimated 1 to 2 percent of their budgets during each school year of the grant period, district officials told us that RTT funds were crucial to their ability to implement reforms.

Districts also reported particular difficulties with human capital capacity— the second greatest challenge they faced implementing RTT. currently participating in RTT reported the most challenging aspect of human capital capacity was recruiting staff through competitive compensation, with an estimated 45 percent of districts reporting that doing so was very or extremely challenging. An estimated one-third of currently participating districts also cited retaining staff and having the appropriate number of staff among the most challenging aspects of human capital capacity, as well as issues related to Common Core implementation, such as having staff prepared to develop and/or implement curricula meeting the new standards.

Human capital capacity is the extent to which an organization has sufficient staff, knowledge, and technical skills to effectively meet its program goals.

Most States and Districts Increased Capacity During the Grant Period but Anticipate Challenges Sustaining Their Efforts

States and districts reported taking various actions to build and increase their capacity overall throughout the grant period (see fig. 3). However, both indicated that human capital and financial capacity would be the most challenging to sustain after the RTT grant period ends. State and district officials we spoke with explained that these issues were inter- related; that is, staff shortages and skill gaps required continued funds for professional development.

Throughout the grant period, more than half of the 19 states reported putting great or very great effort into building stakeholder capacity—the area that state officials cited as the most challenging—most frequently by consulting with organizations that represent teachers and/or administrators (17 states), consulting with district leadership (16 states), and building political relationships (15 states).

Similarly, most states reported building organizational capacity—another area that presented great challenges as they implemented reforms—by, for example, establishing an RTT point of contact or office (18 states) and establishing communication mechanisms for RTT staff, such as group email lists (17 states). To a lesser extent, states reported that reorganizing an existing office (12 states) and appointing new RTT leadership (13 states) were also helpful in building organizational capacity. According to one state official we spoke with, the state reorganized its entire state educational agency into departments aligned with its RTT reforms. The official noted that the RTT grant helped the state fund the reorganization which, in turn, helped them mitigate capacity challenges throughout implementation. Another state official explained that the state focused on reorganizing how staff conduct their work by fostering collaboration among program officers.

School districts—whose second greatest capacity challenge related to human capital—reported making great or very great effort to build human capital capacity for RTT reform by training existing staff (80 percent), expanding the responsibilities of current staff (74 percent), and shifting responsibilities among staff (64 percent). Similarly, all three district officials we spoke with in our follow-up interviews noted that efforts to build human capital capacity focused on training and shifting the roles of their current staff. One district official explained that they avoided funding new staff positions that they might not be able to retain after RTT funds ended. To build financial capacity, an estimated 23 percent of currently participating districts reported receiving supplemental funding from their state general fund. Additionally, an estimated 7 percent of districts reported receiving funds from foundations to build capacity.

Despite their efforts, state and district officials reported that capacity struggles would likely remain once the RTT grant period ends. For both states and districts, financial capacity and human capital capacity represented the greatest challenges to sustaining reforms (see fig. 4). However, states and districts also reported planning to take various actions to help sustain their capacity for reform. All 19 states, as well as an estimated 84 percent of currently participating districts, indicated that retaining staff with requisite knowledge and skills is part of their plan to sustain RTT reform efforts. For example, one district official explained that they used a large portion of their RTT funds on training for teachers and administrators. Using the RTT funds for this purpose—as opposed to hiring many new staff—helped them build capacity and institutional knowledge that would be easier to sustain once the RTT funding ends. Additionally, 17 states indicated that modifying existing staff roles and responsibilities was the second most planned action to sustain RTT reforms. An estimated 72 percent of districts indicated that building institutional knowledge was their second most planned action to sustain RTT reforms.

Rural Districts Faced Greater Capacity Challenges Than Urban and Suburban Districts in Many of the Reform Areas

Rural school districts reported facing significantly greater challenges than urban districts in the standards and assessments and data systems core reform areas when implementing RTT, according to our survey results (see fig. 5).

These survey results are consistent with our past work on the capacity challenges rural districts face. For example, in a 2013 report, we found that a rural district in New York faced unique difficulties implementing its teacher evaluation system because its small student population required some teachers to teach more than one subject, which made the evaluation process more complex and time-consuming. Similarly, our prior work on implementation of School Improvement Grants showed that rural districts faced difficulties because attracting and retaining high- quality teachers and implementing increased learning time requirements were difficult, in part due to higher transportation costs in rural areas.

In addition, in responding to our survey, rural districts reported anticipating more difficulty than urban districts in sustaining all four types of capacity after the RTT grant period ends; and anticipated more difficulty than suburban districts in sustaining three of the four capacity types. For example, according to our survey, an estimated 40 percent of rural districts anticipated that human capital capacity would be very or extremely challenging in sustaining RTT reform efforts compared to 26 percent for urban and suburban districts (see fig. 6). One expert participating on our panel agreed, noting that rural districts would also face challenges sustaining reforms because constrained budgets and a lack of human capital capacity are often particularly challenging for rural districts. In addition, a rural district official told us that they have a small number of employees, and attracting and retaining skilled employees who can perform multiple work functions can be more difficult for them. The official also noted that recruiting staff is a challenge because rural districts are often also among the poorer districts and do not have the resources to implement large-scale hiring efforts.

Although states and districts across the country likely face capacity challenges and resource limitations to some degree, research suggests that some rural districts—and states that have many rural districts—may be less likely to have the skills, knowledge, or expertise to overcome these challenges. For example, one 2013 report recommended that states may have to play a much more direct role in guiding school improvement in smaller, rural districts, where capacity is lacking.addition, a 2014 Education Office of Inspector General report indicated In this approach may be effective in reducing project delays and provided an example of a state that planned to help districts build capacity in order to better support low-performing schools in rural areas.

Our prior work and other research demonstrate that states with many rural districts need additional supports in this area. Given that rural districts reported that they faced challenges implementing and sustaining reforms that were statistically significantly greater than urban and suburban districts, a greater understanding of these challenges could help Education provide more targeted support to rural districts. According to Education’s Handbook for the Discretionary Grant Process, Education is to provide technical assistance to grantees to help them achieve successful project outcomes. also required to hold grantees accountable for meeting the commitments made in their approved RTT applications. Education has recognized and reported on challenges facing rural districts. In addition, Education officials stated that they have supported RTT grantees and their rural districts through a series of convenings, work groups, publications, webinars, and individual technical assistance, and provided examples of these activities. However, we reviewed RSN’s technical assistance documents and found that most of the activities were not provided in the manner that RTT states reported finding most helpful—as discussed later in this report—nor were they tailored to helping states address the unique capacity challenges that rural districts reported facing in the reform areas identified in our survey. Unless Education provides assistance specifically designed to help states support their rural districts in addressing their capacity challenges in implementing and sustaining high-quality reform, states may not be able to help the districts that need it the most.

U.S. Department of Education, Handbook for the Discretionary Grant Process, Handbook OS-01 (Washington, D.C.: January 2009).

Many RTT States Found Education’s Technical Assistance the Most Helpful Resource

States Found Technical Assistance Most Helpful in Building and Sustaining Capacity

According to our state survey, individualized technical assistance provided by Education program officers was the most helpful resource when building capacity to implement and sustain reform plans (see fig. 7). This was consistent with the views of officials we interviewed in four RTT states, who described very positive interactions with their Education program officer. For example, state officials explained that the program officers practiced collaborative problem-solving and provided a significant amount of support to the state as it implemented reform activities.

The next most helpful resources, according to our state survey, were technical assistance provided by other staff in the Implementation and Support Unit and RSN. One state official we spoke with noted that Implementation and Support Unit staff provided useful information on how other states were implementing their reform activities. An official from another state explained that the state is working closely with RSN to better understand how to work with its participating RTT districts to better leverage federal funding to improve student outcomes.

As shown in figure 7, RSN’s communities of practice ranked fourth in terms of helpfulness to build capacity to implement and sustain RTT reform. According to state officials and one expert participating on our panel, these communities of practice encouraged collaboration across states, which has helped them leverage knowledge, talent, and resources, as well as facilitate the sharing of promising practices. Education officials observed similar value in RSN’s communities of practice, noting that through them, states had a forum in which to learn from each other and discuss RTT implementation issues. It is worth noting that state officials we interviewed commented that communities of practice may have been more helpful to states that were in the early stages of implementing RTT reforms. For example, one official noted that their state was farther along in implementing its teacher and principal evaluation system and school turnaround efforts and therefore did not gain as much from those communities of practice.

State officials ranked RSN’s capacity-building community of practice and web-based resources from Education and RSN among the least helpful to states. Education officials similarly noted that while webinars were an easy way to disseminate information, they are likely not as valuable as other RTT resources because they are not as tailored to a particular state’s needs. Two experts participating on our panel noted that although an abundance of school reform-related information exists on websites, little is known about the effectiveness of the information.

In December 2013, RSN published the results of an evaluation of its technical assistance activities that generally aligned with the results of our state survey. For example, according to RSN’s evaluation report, participants indicated they were satisfied with the quality of the support, the format and content of the technical assistance activities provided by RSN. Individualized technical assistance had the highest ratings because, according to the evaluation report, it was designed to address a state’s specific implementation challenges. In addition, participants in the RSN evaluation indicated that on average, technical assistance activities had a moderate effect on states’ ability to build capacity overall. The results of the RSN evaluation also showed that while webinars were useful for disseminating information to larger audiences and convening states on a regular basis, they received lower ratings than other forms of assistance. Our body of work on performance measures and evaluations has shown that successful organizations conduct periodic or ad hoc program evaluations to examine how well a program is working. These types of evaluations allow agencies to more closely examine aspects of program operations, factors in the program environment that may impede or contribute to its success, and the extent to which the program is operating as intended. Information from periodic reviews of RSN’s technical assistance efforts are an important factor in determining if adjustments are needed to help grantees meet their goals for education reform.

State officials we surveyed also identified additional activities that Education could undertake that would better assist states with implementing RTT. Specifically, 10 of 19 states reported wanting ongoing professional development throughout the grant period, as opposed to during the early stages of the grant. Ten of 19 states reported wanting training to be provided in their respective states to make it more easily accessible, rather than having to travel to Washington, D.C. Further, 11 of 19 states reported wanting assistance identifying skilled contractors who could assist with reform efforts. Education officials stated that any assistance it provides to identify contractors cannot compromise the fairness and objectivity of the states’ procurement processes. Education officials also pointed out other legal challenges to identifying contractors, such as prohibitions against endorsements of private entities. However, Education officials stated they can assist grantees by, for example, helping them to develop objective criteria, analysis, or research regarding the qualifications of skilled contractors. They said they can also provide resource lists using objective criteria, as well as technical assistance in this area.

Education Established a New Office and Aims to Develop Policies that Reflect a Coordinated Approach to Providing Technical Assistance

In October 2014, Education created the Office of State Support to expand and sustain the collaborative approach to providing oversight and technical assistance that began under the Implementation and Support Unit. More specifically, the purpose of the Office of State Support is to design a coordinated approach across multiple Education programs to reduce redundancy and improve the efficiency and effectiveness of Education’s oversight efforts. The Office of State Support will provide states with one point of contact for multiple education programs that will provide support and technical assistance. The Office of State Support plans to establish advisory committees, involve staff from other education programs in decision making, and maintain close communication with staff from other education programs that have similar goals and activities as programs covered under the new office.

Officials from the Office of State Support stated that the lessons learned from the RTT monitoring and technical assistance processes will inform their work in the new office for programs they oversee—many of which are helping states to facilitate comprehensive education reforms similar to those started under RTT. However, officials stated that they will need to eventually transition to a longer-range plan for monitoring and reconsider how they provide technical assistance because Education’s contract with RSN ends on June 30, 2015. Education officials noted that it was unlikely that the department would receive such a large amount of funding ($43 million) for technical assistance again. They explained that the type and extent of technical assistance efforts to states after the end of the RSN contract will, in turn, be dependent upon the funding available for that purpose. Lastly, they said that they will look to leverage existing technical assistance funds, such as those provided for the Comprehensive Centers program, to help increase state capacity to assist districts and schools.

Education’s Handbook for the Discretionary Grant Process requires program offices to develop a monitoring and technical assistance plan for each grant program. In addition, according to Federal Standards for Internal Control, policies and procedures help ensure that necessary actions are taken to address risks to achieving the entity’s objectives. Education has a monitoring and technical assistance plan for RTT, which it has been using for the past four years and has continued to use during the transition from the Implementation and Support Unit to the Office of State Support. However, officials from the Office of State Support stated that they planned to establish coordinated technical assistance processes and procedures for all of the programs administered by the new office, while meeting the needs of the states and their particular initiatives. For example, they said they need to consider how to bring the various kinds of monitoring and technical assistance conducted by different program offices together to provide support for and make connections across programs, and be less burdensome for states. Officials stated that they formed a working group of staff from various Education program offices, including former Implementation and Support Unit staff, to help inform the new office’s coordinated technical assistance policies. However, officials noted that the working group was in the early stages of this process, and had not yet developed any draft policies or established a definitive deadline for accomplishing this task.

Given the valuable technical assistance that RSN provided to states, and that Education has not determined the type or amount of technical assistance to be provided, there could be a gap in the type of support that Education can provide to states when the contract expires. Until the Office of State Support develops and finalizes policies and procedures that include support activities states identified as most helpful, Education runs the risk of not providing the most effective assistance to its grantees to help them successfully implement and sustain reform efforts.

Experts Identified Lessons Learned to Help Sustain Reform and Made Observations to Consider in Future Efforts

Key Lessons Learned from RTT May Be Critical to Sustaining Reform

Our analysis of our expert panel transcript revealed key lessons that could help states and districts address their greatest capacity challenges and help sustain reforms after the RTT grant period ends.

To address challenges with financial capacity, five of the 10 experts participating on our panel noted that federal formula grants are better suited than competitive grants for building and sustaining capacity because they provide a more stable funding source. Three experts stated that there are several ways that states and districts can leverage the funds they receive annually in formula grants to help sustain reforms. The Title I formula grant—designed to improve schools with high concentrations of students from low-income families—gives districts and schools flexibility to use federal funds to support instructional strategies and methods that best meet local needs. For example, schools where at least 40 percent of students are from low-income families may operate “school-wide” Title I programs, which allow schools to combine Title I funds with other federal, state, and local funds to improve the overall instructional program for all children in a school. In the 2012–2013 school year, approximately 40,632 schools, or 74 percent of all Title I schools, operated school-wide programs.

Despite the large number of schools running a school-wide program, districts and schools may not be using the flexibilities to combine Title I funds with other federal funds to their fullest extent due, in part, to a lack of organizational capacity at the state and district levels. According to Education officials and two experts on our panel, states and districts are often uncertain about whether they are allowed to combine federal formula grants in new ways to support comprehensive reforms. For example, Education officials told us that historically, states and districts have used Title II funds—formula grants designed in part to increase student academic achievement through strategies such as improving teacher and principal quality—to reduce class size. However, according to Education’s guidance, states and districts could also choose to combine Title I and Title II funds to sustain reforms initiated under RTT, such as providing academic support coaches and financial incentives and rewards to attract and retain qualified and effective teachers to help low- performing schools. According to five experts on our panel, uncertainties about what is allowed may stem from lack of communication and coordination among the multiple federal education program and financial management offices, and because these offices are not always focused on helping states and districts better leverage their funds.

GAO/AIMD-00-21.3.1. programs to support the four core reform areas. Further, in 2013, the Council of Chief State School Officers developed a toolkit for states to help clarify how districts and schools may spend K-12 federal formula grants. This toolkit encourages states to improve collaboration among offices supported by federal grants to help ensure they effectively leverage federal funds. Currently, Education is working with RSN to develop another toolkit for states and districts on ways to leverage federal formula grants to sustain educational reforms. Education officials could not provide definitive time frames for the release and dissemination of the toolkit, but noted that they are hoping to release it sometime in 2015. This toolkit, when finalized, may help states and districts better understand how to leverage their formula grants to sustain reform activities and help raise student achievement—a primary objective of education reform.

Education officials and one expert participating on our panel also said that states and districts do not use funding flexibilities to their fullest extent because they have concerns about compliance with state audit requirements. Education officials explained that states and auditors may believe that federal law prohibits certain activities, even when the law and its implementation rules do not. Education officials told us they tried to address these uncertainties by issuing guidance to clarify how states and districts can leverage federal funds to support reforms. According to this guidance, states may use Title I funds to provide technical assistance to low-achieving schools, and districts may consolidate Title I, Title II, and IDEA funds in schools under the school-wide program to support comprehensive reforms by, for example, extending the school day or school year. However, Education officials said that there is still confusion about this issue, particularly among the audit community, and that it needs to provide new guidance to help auditors better understand allowable spending within federal formula grants, especially with Title I funds. However, it does not have a definitive plan for developing and implementing this guidance. Such guidance—when developed and fully implemented—may help auditors better understand funding flexibilities in existing formula grants and help states and districts fully leverage these flexibilities.

Further, the pending reauthorization of ESEA also provides an opportunity to address these capacity issues. Education told us that it is exploring new options to help states and districts build capacity to implement comprehensive reforms, including increasing the portion of Title I grant funds that can be set aside for administrative purposes. Currently, two of the set-asides in the Title I program limit the maximum percentage of funds that can be set aside to support state administrative functions and districts’ school improvement activities. Specifically, ESEA requires that a state generally spend no more than 1 percent (or $400,000, whichever is greater) of its Title I funds on state administration and 4 percent on district school improvement activities.

Education told us that the current portion of funds under the ESEA Title I grant that may be used for administrative functions may be inadequate given the range and complexity of state-level work in supporting effective implementation of local Title I projects. In its fiscal year 2016 budget proposal, the Administration proposed increasing the funds a state can spend on administration from 1 percent to 3 percent. According to Education officials, the trade-off, particularly in a tight fiscal environment, is that larger set-asides may reduce the portion of available funds that would transfer to districts and schools to implement programs. In the current Congress, the Student Success Act, which was reported out of the House Committee on Education and the Workforce, would make changes to both of these set-asides.

To help address human capital and stakeholder capacity challenges, five experts on our panel noted the importance of fostering partnerships between a state and its districts, among districts within a state, and with non-governmental entities by, for example, convening groups of experts across the state to share expertise, solve problems, and share lessons learned to help leverage knowledge and talent. They further noted the potential for such a strategy to solve common challenges, such as how to develop effective strategies for evaluating teachers who teach subjects that are not assessed using standardized tests (e.g. foreign language or art). Universities with research and professional development institutes are another potential resource to help states and districts build and sustain human capital capacity. For example, one expert noted that strong relationships with higher education institutions and teacher unions are needed to revamp teacher, principal and superintendent training programs and teacher licensure requirements. Lastly, three panelists said that to maintain key stakeholder support for reforms, states need to show progress in meeting their established time frames for RTT reform, or increase student achievement.

Observations from Experts May Help Education Address Capacity Concerns in Future Grant Competitions

Three experts on our panel noted that competitive grants may be better suited than formula grants for spurring reforms and innovative approaches, but varying levels of capacity among states and districts raises concerns about their ability to win competitive grants and successfully implement large-scale education reforms. Research suggests that states’ capacity was an important variable in helping to predict who applied for RTT funds and which states scored well during the competition. In particular, a 2011 study found that states with quality standards and accountability procedures, and that had achieved overall student gains, were more likely to receive higher scores during the RTT grant competition. When making competitive grant awards in the future, Education officials told us they expect to look at demonstrated capacity as evidenced by a state’s performance under previous grants and may offer a competitive priority for previous success. To help states and districts that may be struggling in these areas, experts participating on our panel made four observations that they believe could be incorporated into the design of future competitive grants to help level the playing field between high- and low-capacity states and districts. Education has incorporated some of the observations into its competitive grant programs to varying degrees and pointed out some advantages and disadvantages of each.

Observation 1: Allow joint applications so that states and districts with greater capacity can partner with those with less capacity. Education noted that it used this approach in recent grant competitions. Education encouraged states that opted to adopt a common set of college- and career-ready standards to form collaborative groups to apply for RTT assessment grants to develop assessments aligned with the new standards. A 2011 study proposed that such arrangements could help states with less capacity more easily benefit from the initiatives of ones with more capacity by helping them identify partners and providing them access to funds that may help valuable reforms gain traction. Education officials told us, however, that when they have allowed joint applications or consortia for some competitive grants, the complexity of implementing the grants increased because states have different procurement rules which take longer to navigate. Education officials also noted that these joint initiatives sometimes take longer to implement because states have to establish a framework for how they are going to coordinate.

Observation 2: Staggering or “phasing” competitive grant funding to allow for varying capacity needs of grantees. Education officials told us that they have had mixed success using planning grants to allow grantees additional time to build capacity to implement plans. For example, Education used a two-phase strategy for awarding competitive grants under its Promise Neighborhoods grant program, including 1-year planning grants to organizations to enhance the grantees’ capacity and a separate competition for a 5- year implementation grant to organizations that demonstrated they were ready to implement their plans. However, we recently reported that Education did not communicate clearly to grantees about its expectations for the planning grants and the likelihood of receiving implementation grants. Education officials told us that they do not always have the authority to offer this feature, but they consider it where it is possible. Education officials told us that they are considering adding a planning year to the School Improvement Grant, which is federal money awarded to states that states, in turn, award to districts using a competitive process. Education officials told us that they believe that low-capacity districts could benefit from this approach, but noted that it will be important to emphasize their expectation that grantees use the planning year to build capacity to implement their reform plans.

Observation 3: Allowing intermediary entities that often help coordinate or provide technical assistance to districts to apply for competitive grants. Education officials told us that they see a benefit to using partners such as nonprofit organizations to drive reform, noting, for example, that the Investing in Innovation program allows nonprofits to partner with school districts as part of the application process and throughout the grant period. Research supports such an approach as well. A 2011 RAND study examining the federal and state role in improving schools in 15 states found that although some states assumed primary responsibility for assisting low-performing schools, others relied on regional organizations, area education agencies, or intermediate school districts to fill this role.However, Education officials noted that applicant eligibility is generally defined in statute.

Observation 4: Streamlining Education’s grant application processes to make it easier for states and districts with less capacity to apply. Education officials told us that one example of streamlining the grant process was allowing states that did not win an award in the first phase of a competition to revise the same application and resubmit for subsequent phases. Education adopted this strategy in the RTT grant competition. Another way to streamline the grant application process is by encouraging shorter applications. Education officials said it used this approach in a grant competition for the Investing in Innovation program. Education officials noted that, in general, one disadvantage to shorter applications is that there may not be sufficient detail in the applications to hold grantees accountable for implementing their plans.

Conclusions

As Education’s technical assistance contract for RSN comes to a close, and it develops new processes for technical assistance under the new Office of State Support, it has an opportunity to apply the technical assistance that RTT states reported as most helpful, such as individualized technical assistance and professional development, to other grant programs that the office oversees. Such technical assistance could help states implement and sustain the comprehensive education reforms which will continue to be supported by other grant programs managed by the Office of State Support.

In addition, because rural districts face unique challenges implementing and sustaining RTT reforms, focusing efforts to enhance Education’s understanding of the types of additional supports they may need could help these districts successfully implement and sustain their reform efforts, and ultimately improve student achievement. Further, as the RTT grant period comes to an end, RTT states may need to better leverage their federal formula grants to continue to support comprehensive reform in the absence of RTT funds. Education officials and other experts have emphasized the importance of leveraging existing funding flexibilities in education formula grants to help states implement and sustain large-scale reform efforts. However, concerns about a lack of communication between states’ program and financial management offices, as well as concerns about non-compliance with state and federal requirements may be limiting states’ willingness to use the funding flexibilities present in current law to develop and implement strategies tailored to their unique local needs. By taking actions to address these issues, Education can help states and districts better use their federal funding in the most effective way to improve student achievement and to support comprehensive school reform.

Recommendations

To help ensure that states are better able to sustain RTT reforms and that Education can effectively support other grant programs managed by the Office of State Support, we recommend that the Secretary of Education direct the Office of State Support to fully implement and incorporate into its coordinated technical assistance policies and procedures the types of support that would be useful in sustaining RTT reforms and providing effective support to grantees in other programs supporting education reform that the Office of State Support oversees. These could include: providing individualized technical assistance to states, such as that currently provided by Education program officers; facilitating communities of practice to promote opportunities for collaboration across states; providing professional development (or training) throughout the grant period, as opposed to only during the early stages of the grant; making training more easily accessible by conducting training locally in their respective states, when possible; and to the extent permissible in the context of federal and state requirements and restrictions, exploring the possibility of assisting states in identifying skilled contractors to help implement reform efforts.

To help states address capacity challenges as they sustain comprehensive education reforms similar to RTT, we recommend that the Secretary of Education direct the Office of State Support to take steps, such as: providing ongoing individualized technical assistance to states to help them target assistance to rural districts, particularly in the reform areas that were most challenging for rural districts; finalizing and disseminating guidance to be included in Education’s toolkit to help states leverage federal formula grants to sustain education reforms; and clarifying and improving understanding of how funding flexibilities in existing formula grants could be used to support education reform efforts to help states and the audit community address impediments to using formula grants in different ways.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Education for comment. Education provided technical comments, which we incorporated into the report as appropriate. Education’s written comments are reproduced in appendix VI and summarized below. Education did not explicitly agree or disagree with our recommendations, but outlined steps to address many elements contained in them. It also provided additional information related to our findings and recommendations.

In response to our first recommendation, Education stated that it shares our interest in supporting states as they sustain RTT reforms and supporting other grant programs under the Office of State Support through performance management and technical assistance. To this end, Education described plans to build on its generally successful RTT monitoring strategy to develop a consolidated technical assistance strategy for all programs under the auspices of the Office of State Support. We have added clarifying language in the body of the report to better reflect existing elements of the RTT monitoring and technical assistance plan.

Education’s plan to provide coordinated policy development, performance management, technical assistance, and data analysis services through a structure intended to more effectively support the implementation of key reforms and provide individualized support is a positive step. These coordinated policies and procedures could continue to support RTT grantees as well as other grantees under other Office of State Support programs that have a role in helping states implement comprehensive education reforms. However, we continue to believe that until these policies are fully implemented, Education risks providing less effective support than it otherwise might. Further, as Education’s technical assistance contract for RSN comes to an end, we continue to believe that Education should take explicit steps to incorporate into its new consolidated assistance strategy for all programs under the Office of State Support the technical assistance activities that RTT grantees identified as being most helpful to them in sustaining their reforms. In addition, Education should incorporate those additional supports that states reported as desirable. We have clarified the intent of our recommendation accordingly.

In response to our second recommendation, Education agreed that it is important to identify ways to help states target assistance to rural districts. Education stated, however, that the draft report does not adequately recognize the actions it has taken to support RTT grantees in rural states and districts, and provided a list of 17 activities it has undertaken through RSN to support rural areas. We acknowledge Education’s efforts to provide support to rural areas and have incorporated additional information in the draft report, as appropriate, to reflect this. However, in further reviewing these 17 activities, we found significant limitations and believe our overall finding and corresponding recommendation is still warranted. Specifically:

Nearly all of the activities (16 of 17) were in the form of working groups, convenings, webinars, toolkits, and publications developed by the RSN, many of which were located on the RSN website. According to our survey of all 19 RTT states, web-based resources were among the least helpful to RTT states in building and sustaining the necessary capacity to implement reforms. Only one of the 17 activities provided individualized technical assistance which, according to our survey, was the most helpful form of assistance to RTT states. We realize that Education formed RSN to provide support in a variety of formats and agree that RSN has generally well supported RTT grantees. However, given the unique capacity challenges that rural districts face, we believe there is value in offering technical assistance tailored to the individual needs of rural areas.

According to our generalizable survey of districts that received RTT funds, rural districts faced statistically significantly greater challenges than urban districts in implementing reforms in two areas: standards and assessments and data systems. However, 14 of the 17 RSN activities focused on the other two reform areas (school turnaround and effective teachers and leaders). RSN’s efforts to focus resources on assisting states in implementing RTT reforms are important ones, and we believe that many states and districts may have benefitted from these efforts. However, in order to best support states that are working to implement and sustain reforms in their rural districts, Education should target future support in the reform areas in which rural districts most struggled: standards and assessments and data systems. Accordingly, we modified our recommendation to clarify that Education should take steps to provide targeted assistance to states in those reform areas that we have identified as statistically significantly more challenging for rural districts.

Many of the activities undertaken to support rural districts were conducted in 2012 and 2013 (6 of the 11 that included specific dates) when states and districts were fully engaged in implementing RTT reforms. However, our survey of districts that received RTT funds was deployed from June through September 2014, and the results indicated that rural districts continued to face challenges long after they would have availed themselves of these resources.

Some of the activities (6 of 17) provided support that was not specifically tailored for rural districts; rather, it could be applied in rural, suburban, and urban school settings alike. We continue to believe that opportunities exist to help states better target support to rural districts. Without a better understanding of the unique capacity challenges that rural districts face, and a more focused approach to providing support, Education may not be able to help the states and districts that need it the most.

Finally, Education recognized the importance of clarifying its guidance on the use of funding flexibilities and provided several examples of “Dear Colleague” letters it has provided to states. We referenced one of these letters in the draft of the report. We did not include the other two “Dear Colleague” letters (guidance related to leveraging federal funds to support school counselors and digital education) because they do not address the use of funding flexibilities in support of education reform initiatives, which was at the heart of our finding and corresponding recommendation. To address this apparent confusion we have clarified our recommendation accordingly. We noted in our report, and Education emphasized, that it is working with RSN to release new guidance in 2015 on ways to leverage federal grants to sustain educational reforms. However, as stated in our report, Education officials could not provide definitive time frames for the release and dissemination of the toolkit. We continue to believe that until this guidance is fully implemented, states and districts will continue to lack clarity on how to leverage their formula grants to sustain reform activities.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on GAO’s web site at http://www.gao.gov.

If you or your staff should have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

We framed our study of capacity challenges faced by states and districts implementing Race to the Top (RTT) reforms around three objectives: (1) What effect did RTT have on education reform, and what capacity challenges did states and districts face in implementing and sustaining RTT initiatives?; (2) How helpful was the assistance the U.S. Department of Education provided to states to build capacity to implement and sustain RTT reforms?; and (3) What lessons have been learned from RTT that could inform future education reform efforts?

In addressing these objectives, we incorporated elements of “grounded foresight,” a methodological approach developed by GAO to examine future implications by identifying key trends, emerging challenges, and opportunities to inform government’s future role and responsibilities. According to GAO’s internal grounded foresight methodology paper, the heart of the proposed approach consists of three elements of grounding, designed to support GAO’s core values of integrity and reliability: (1) a strong factual-conceptual base, (2) one or more methods for discussing or anticipating the future, and (3) transparent communication of the outcomes. We developed a strong factual-conceptual base to assure that relevant trends and occurrences related to capacity issues and competitive grants are documented, recognized, and understood as part of the study. We reviewed and analyzed existing literature on capacity issues and competitive grants in K-12 education using GAO’s prospective We examined the features of RTT, and evaluation synthesis approach.reviewed findings from published reports to identify capacity challenges. We also deployed two web-based surveys of state educational agency and district officials; reviewed relevant federal laws, regulations, and guidance; and conducted interviews with a variety of federal, state, and local officials. We then convened a panel of experts who were knowledgeable about capacity issues and federal grants to obtain their views on the implications of capacity challenges on the sustainability of RTT reform efforts and potential future competitive grants. We made the results of the two web-based surveys publicly available to help ensure transparent communication of the capacity challenges states and districts reported facing.

Surveys of RTT Grantees and Districts

To obtain information on capacity challenges states faced in implementing and sustaining RTT reforms we conducted a web-based survey of RTT points of contact at each state educational agency in all 19 We conducted the survey from May through July 2014. In grantee states.the survey, we asked RTT states about their capacity to implement RTT efforts, the support received to do so, and efforts to build and sustain capacity for RTT reform, among other things. We received responses from all 19 RTT states for a 100 percent response rate. We reviewed state responses and followed up by telephone and e-mail with selected states for additional clarification and context. We also published survey responses in an e-publication supplemental to this report, RACE TO THE TOP: Survey of State Educational Agencies’ Capacity to Implement Reform (GAO-15-316SP, April 2015).

To obtain information on capacity challenges districts faced in implementing and sustaining RTT reform efforts we conducted a web- based survey of a sample of district officials whose districts received RTT funds. We selected a stratified random sample of 643 from 3,251 school districts that received RTT funds from a population of 18,541 school districts in the 19 RTT states (see table 1). Although the focus was on districts that currently receive RTT funds, we also included districts that initially were participating in RTT but later decided to formally withdraw.

We obtained data from Education’s National Center for Education Statistics, which maintains the Common Core of Data for public school districts, for the 2011-12 school year. Our sample allowed us to make estimates to all RTT districts and to subpopulations by urban status of the district.

We conducted the school district survey from June through September 2014 and had a 76.7 percent final weighted response rate. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we expressed our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., plus or minus 6 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. Unless otherwise noted, all percentage estimates in this report have confidence intervals within plus or minus 6 percentage points. For other estimates, the confidence intervals are presented along with the estimates themselves. In the survey, we asked questions about school districts’ capacity to implement RTT efforts, the support received to do so, and efforts to build and sustain capacity for RTT reform, among other things. We reviewed survey responses and followed up by telephone and e-mail with selected districts, as needed for additional clarification and to determine that their responses were complete, reasonable, and sufficiently reliable for the purposes of this report. We also published survey responses in an e-publication supplement to this report, RACE TO THE TOP: Survey of School Districts’ Capacity to Implement Reform (GAO-15-317SP, April 2015).

The quality of the state and district survey data can be affected by nonsampling error, which includes variations in how respondents interpret questions, respondents’ willingness to offer accurate responses, and data collection and processing errors. To minimize such error, we included the following steps in developing the survey and in collecting and analyzing survey data. We pretested draft versions of the instrument with state educational agency officials in three states and officials in four districts to check the clarity of the questions and the flow and layout of the survey. On the basis of the pretests, we made revisions to both surveys. We contacted respondents to clarify any questions or responses where appropriate. Further, using a web-based survey and allowing state and district officials to enter their responses into an electronic instrument created an automatic record for each state and district and eliminated the errors associated with a manual data entry process. In addition, the programs used to analyze the survey data were independently verified to ensure the accuracy of this work.

Expert Panel

To obtain information on lessons learned from RTT that could inform future education reform efforts, we convened a group of knowledgeable individuals for an expert panel. In identifying the experts, we compiled a preliminary list of 15 individuals with research or professional experience related to RTT reforms, state and district capacity, federal grant making, and state or federal education policy. These experts represented the following entities: state educational agencies, school districts, education associations, academia, and education think tanks. They also included a former Education official and a representative from Education’s Office of Inspector General. We identified a state educational agency official based on participation in RTT and the state’s proximity to Washington, D.C. where the panel was convened.

To obtain a different local perspective, we selected a school district official from a different state. In addition, we selected the school district based on proximity to Washington, D.C. and the extent to which the district had completed questions in our district survey. An external expert who conducted extensive research on K-12 education and federal policy vetted our initial list of panelists. We used feedback from this expert, along with biographical information about the experts, to determine which experts would be invited to participate.

The resulting 10 experts participated in a 1-day panel focused on capacity challenges and their implications for RTT reforms and future competitive grants (see appendix V for list of participants). Each panelist completed a questionnaire to document any conflicts of interest. This information was not used to determine the qualification of the expert for the panel, but to ensure that we were aware of circumstances that could be viewed by others as affecting the expert’s point of view on these topics. We developed discussion topics and questions for the panelists based on information gathered from the surveys, interviews, and academic literature. A contractor recorded the panel and transcribed the discussion. We performed a content analysis of the transcript of the panel discussion to develop common themes among the experts on lessons learned from RTT that could help sustain reform efforts, inform the design or implementation of future education competitive grants, and inform future education reform efforts. We tallied responses for each panelist who commented on those themes. This analysis was independently verified to ensure the accuracy of this work.

Review of Laws, Regulations, and Guidance and Interviews with Officials

For all three objectives, we reviewed relevant federal laws, regulations, and guidance—including federal internal control standards and Education’s Handbook for the Discretionary Grant Process—and interviewed federal, state, and district officials and other experts regarding capacity to implement and sustain RTT reforms. We reviewed RTT applications to identify commitments states made to build capacity to implement RTT initiatives. To identify actions taken to build capacity, we compared the states’ commitments to information provided in their progress reports for school year 2012-2013. We also reviewed information on Education’s efforts to assist states with building capacity, such as guidance, technical assistance, webinars, and other information on the RTT website. We interviewed federal officials from the Implementation and Support Unit in Education’s Office of the Deputy Secretary and staff from the newly established Office of State Support. In addition, we conducted interviews with a variety of interested parties, such as educational organizations, researchers, and university professors. For example, we met with representatives from the American Association of School Administrators, the Council of Chief State School Officers, and the Center on Reinventing Public Education, among others. We also conducted follow-up interviews with officials in four state educational agencies and three districts to obtain more detailed information and illustrative examples. We selected these state and district officials based on their responses to our surveys and representation across award phase.

We conducted this performance audit from November 2013 to April 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Race to the Top (RTT) Grant Awards by Phase

Appendix III: Race to the Top State Selection Criteria

Appendix IV: Capacity Challenges by Race to the Top Reform Area and Type of Capacity, as Reported by States and Estimated by Districts

Appendix IV: Capacity Challenges by Race to the Top Reform Area and Type of Capacity, as Reported by States and Estimated by Districts STATES (Percent and Number)

DISTRICTS (Estimated Percentage)

11% (2) 17% (3) 11% (2) 39% (7) 24% (4) 18% (3)

Effective Teachers and Leaders Organizational 24% (4) 24% (4) 22% (4) 11% (2) 22%(4)

School Turnaround Organizational

44% (8)

22% 33% 16% 40% (6) 27% (4) 33% (5) 33% (5)

Appendix V: List of Participants on GAO’s Panel on Implications of Race to the Top Capacity Challenges

Appendix VI: Comments from the U.S. Department of Education

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Elizabeth Morrison (Assistant Director), Jamila Jones Kennedy (Analyst-in-Charge), Sheranda Campbell, Kathryn O’Dea Lamas, Amanda Parker, and Stacy Spence made significant contributions to this report. Assistance, expertise, and guidance were provided by David Chrisinger, Nancy Donovan, Alexander Galuten, Catherine Hurley, Jill Lacey, Jean McSween, Mark Ramage, Walter Vance, and Mimi Nguyen.