Race To The Top: States Implementing Teacher and Principal Evaluation Systems despite Challenges
Released: 2013-09-18T00:00:00

Letter

Background

RTT Overview

In accordance with Recovery Act requirements, Education established the RTT grant fund to encourage states to reform their K-12 education systems and to reward states for improving student outcomes, such as making substantial gains in student achievement and improving high school graduation rates. States competed for RTT grant funds based on reforms across four areas: adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global market; building data systems that measure student academic growth and success and inform teachers and principals about how they can improve instruction; recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most; and turning around the lowest-achieving schools.

Education awarded RTT grants in three phases. Twelve states received grants in 2010 in Phases 1 and 2 to support the design and implementation of their teacher and principal evaluation systems and other RTT reforms. Award amounts ranged from $75 million to $700 million (see table 1). States were required to subgrant at least 50 percent of their total grant award to districts that chose to participate in RTT. The 4-year grant period began on the date funds were awarded to the state. States must obligate all funds within that period, and they have 90 days following the end of their grant period to liquidate all obligated funds unless they receive a no-cost extension. Education may grant extensions for states beyond the 90 days on a case-by-case basis.and Phase 2 funds not obligated and liquidated by September 30, 2015, will revert to the U.S. Treasury.

Education identified 19 primary criteria to guide peer reviewers in the selection of states for RTT grants (see table 2).

The criterion—improving teacher and principal effectiveness based on performance— established the RTT guidelines for teacher and principal evaluation systems. Reviewers evaluated the state’s plan to ensure its participating RTT districts (1) measure student growth for each individual student; (2) design and implement evaluation systems, developed with teacher and principal involvement, that include multiple rating categories that take into account data on student growth as a significant factor; (3) evaluate teachers and principals annually and provide feedback, including student growth data; and (4) use these evaluations to inform decisions regarding professional development, compensation, promotion, retention, tenure, and certification.

Education defines student growth as the change in student achievement for an individual student between two or more points in time. For students in grades and subjects that are tested by state standardized tests, Education defines student achievement as the score received on the state’s assessments required under the ESEA. For students in grades and subjects that are not tested by state standardized tests, Education defines student achievement based on alternative measures of student learning and performance. These measures include student scores on pre-tests and end-of-course tests, student performance on English language proficiency assessments, and other measures of student achievement that are rigorous and comparable across classrooms. Student achievement for students in tested grades and subjects can also be assessed using other measures as appropriate, including the same measures as students in nontested grades and subjects.

Education provided background information in its notice of proposed priorities, requirements, definitions, and selection criteria for RTT on why it included student growth as a factor in its criteria for teacher and principal evaluation. Education noted the difficulty in predicting teacher quality based solely on the qualifications that teachers bring to the job. The department cited research on the limited predictive power of measures such as certification, education, and years of experience, and research on the value of measuring student growth to assess teacher quality. In response to public comments that expressed concern about the use of student growth data as the sole means to evaluate teachers and principals, Education revised its definitions of an effective teacher and effective principal to require that multiple measures be used to assess effectiveness, with student growth as a significant factor. Education also provided examples of these supplemental measures, such as multiple observation-based assessments of teacher performance and high school graduation rates as a measure for evaluating principals.

Education also established criteria for peer reviewers to consider a state’s capacity to sustain its reforms. The criterion—building strong statewide capacity to implement, scale up, and sustain proposed plans—required reviewers to assess the extent to which the state had a plan to ensure sufficient capacity and use stakeholder support to implement its plans. States were evaluated on, among other things, the extent to which they demonstrated that they would provide strong leadership and dedicated teams to implement the reforms and use their fiscal, political, and human capital resources to continue successful grant-funded reforms after RTT funds are no longer available.

Role of the Department of Education

Education is responsible for fiscal and programmatic oversight of all aspects of RTT, reviewing and responding to states’ requests to amend their RTT applications, and providing technical assistance. To monitor states’ progress, Education established a program review process that includes ongoing conversations with grantees, on-site program reviews, grantee self-evaluations, and meetings with Education officials. As we reported previously, Education uses a common set of questions to oversee state progress and to address specific needs and challenges of each grantee. Education also publishes annual reports to the public summarizing the progress of each state. To provide technical assistance, Education established the Reform Support Network (RSN), a 4-year, $43 million technical assistance contract with ICF International, which works with Education to support RTT states.

Education’s process for reviewing and approving changes to a state’s RTT plans includes reviewing the state’s approved application, budget, and scope of work. According to Education’s guidance, an RTT grantee must submit an amendment request for (1) a proposed revision that constitutes a change in activities from the approved grant project, regardless of budgetary impact; (2) budgetary changes, including transfers among categories or programs, that exceed $500,000 of the current approved budget; or (3) changes to the list of districts participating with the grantee’s RTT plan. Education will not approve amendment requests that would change the overall scope and objectives of the approved proposal, fail to comply with the terms of the award or the statutory and regulatory provisions of the program, or violate the general principles of the program.

Education’s Institute of Education Sciences’ National Center for Education Evaluation and Regional Assistance is conducting two studies that relate to RTT teacher and principal evaluation systems. One study will assess the RTT and School Improvement Grant programs and whether these programs are related to improvement in student outcomes. The results of this study, which will not specifically assess the impact of teacher and principal evaluations on student outcomes, are expected in 2014. In the second study, experimental teacher and principal evaluation systems will be implemented in schools in eight districts in order to study their effects on factors such as student achievement and teacher and principal mobility. A report on this study is expected in 2015.

Teacher and Principal Evaluation Systems in the RTT States

The RTT states provide districts with varying amounts of flexibility to develop their evaluation systems. For example, some RTT states developed evaluation systems for use by all districts, unless a district develops an alternate evaluation system that meets state requirements. In other states, districts can develop their own evaluation systems within guidelines provided by the state, and the state must approve each district’s system.state or the district, districts evaluate teachers and principals using multiple measures that assess professional practice and student academic growth (see fig.1).

Six of 12 RTT States Fully Implemented Their Evaluation Systems by the 2012-13 School Year

According to state officials, 6 of the 12 RTT states fully implemented both their teacher and principal evaluation systems by school year (SY) 2012- 13 (see fig. 2), though their success in meeting their original target date for implementation varied. The states that fully implemented their systems evaluated all teachers and principals in RTT districts, according to state officials. The six states that fully implemented both teacher and principal evaluation systems targeted SY 2011-12 for full implementation in their RTT applications. Three of the six states met that target and SY 2012-13 was their second school year of full implementation. The other three states did not meet the targets set in their applications, but did fully According to Education’s implement their systems in SY 2012-13.amendment approval letters, states shifted implementation time frames, in part, because they needed additional time to develop student academic growth measures. For example, Delaware required an additional year to develop measures for its student academic growth component, which state officials said resulted in a better evaluation system.

The six states that did not fully implement both their teacher and principal evaluation systems in SY 2012-13 either piloted or partially implemented evaluation systems, according to state officials (see fig. 2). Based on the targets set in their RTT applications, four of the six states originally planned to fully implement by SY 2012-13 but are instead piloting or partially implementing their systems. The proportion of teachers and principals participating in pilots varied. According to state officials, Hawaii’s teacher evaluation pilot covered about 30 percent of its teachers, and Maryland’s evaluation systems pilot covered about 14 percent of its teachers and principals in RTT districts. Among the four districts we visited in Maryland, district officials said the percentage of teachers who participated in the districts’ pilots ranged from about 4 percent to 100 State or district officials in four of the six states expressed percent.some concerns about their readiness for full implementation. For example, officials in one Maryland district that piloted with about 4 percent of teachers said they will move from learning about the system to full implementation without sufficient time to address issues that arose during the pilot. Similarly, officials in another Maryland district that piloted with about 5 percent of teachers and 19 percent of principals said the district did not have sufficient time to work with teachers and principals on the new evaluation systems and would have benefited from another pilot year. The Maryland district officials said that two individuals were responsible for all of the evaluation systems work. These officials added that they anticipate budget and staff reductions as they move from their pilot, in which about 100 teachers and 10 principals were evaluated, to full implementation that will cover more than 3,000 people.

Due in part to the difficulty of managing many changes simultaneously, including new curriculum and assessments in many states, in June 2013 Education offered states that have received ESEA waivers or RTT grants the option to request permission from Education to delay the use of their new evaluation systems to inform personnel determinations and consequences for up to 1 year. Education officials noted that many states are already successfully implementing these changes or have requirements in state law about implementation timeframes and thus may not need to request the waiver. teachers and principals, such as retention rewards and dismissal. According to Hawaii officials, the state plans to fully implement its teacher evaluation system in SY 2013-14, but all consequences related to evaluations will be added the following school year. In several states, RTT districts decide how to use evaluation results to determine consequences. Ohio officials said that RTT districts were required to use evaluation results to inform some personnel decisions—including professional development, retention, and pay for performance—and the state surveyed RTT districts to confirm that they did so. Tennessee officials said that RTT districts were required to use the results of evaluations to inform certain personnel decisions, such as employment, compensation, and dismissal, but that the state did not prescribe the consequences attached to different ratings.

Most RTT States Cited Challenges with Developing and Using Certain Evaluation Measures, Addressing Teacher Concerns, and Building Capacity and Sustainability

RTT States Struggled to Develop and Use Student Learning Objectives and Assess Teacher Professional Practice Consistently

State or district officials in most RTT states (8 of the 12) said they had difficulty developing and using student learning objectives (SLOs) to assess student academic growth for teachers. SLOs measure student academic growth for teachers in nontested grades and subjects, which represent 65 to 75 percent of teachers nationwide, according to an RSN report. SLOs are learning objectives for groups of students, such as students in a social studies class, that use a specific measure, such as a course exam, to track academic progress throughout a school year.However, some RTT state and district officials said it can be difficult to ensure that these learning objectives are rigorous and accurately measure student learning. Tennessee officials said that while SLOs are popular and promising in theory, they are difficult to reliably implement because some teachers set non-rigorous goals in order to get high scores. Tennessee officials further explained that some teachers selected a schoolwide social studies score for their SLO measure—despite having no connection to the subject—because students did well on that exam, rather than selecting learning objectives relevant to their own subject matter. Officials in three Maryland districts said determining how to measure student academic growth using learning objectives was a challenge because, for example, they may have difficulty assessing students’ abilities when they enter a class, not just when they leave. Officials in a New York district described the difficulty of implementing learning objectives in their small, rural district (see sidebar). Despite these challenges, RTT state and district officials said that SLOs improved their evaluation systems, in part by engaging teachers in the evaluation process and by leading to more in-depth discussions about teacher performance. To address some of their challenges, RTT states developed guidance, templates, or model learning objectives to help districts develop and use SLOs. In addition, states participated in an RSN-sponsored working group on developing SLOs. They could also access RSN guidance from Education’s website that outlined the benefits of learning objectives and provided information about the elements that comprise rigorous, high-quality learning objectives.

Some RTT state and district officials said it was difficult to ensure that principals assess teacher professional practice consistently. For example, officials said it was challenging to ensure consistency in how principals use classroom observations and other evidence, such as lesson plans, to State or district officials in 6 of assess a teacher’s instructional methods. the 12 RTT states expressed concerns that, for example, some principals may not be appropriately identifying teachers who were ineffective and rating them accordingly. Officials in a few of these states attributed this to principals lacking the skill to differentiate between effective and ineffective teachers, or the will to rate teachers in lower categories or to rate them lower than under the prior evaluation system. Officials in Tennessee and in two North Carolina districts said evaluation data have shown that some teachers with low scores on their student academic growth component received high professional practice ratings. They said this may indicate that some principals are inflating scores or not identifying lower-performing teachers and providing critical feedback. Officials in another North Carolina district described a different concern about the mismatch in professional practice ratings and student academic growth. They noted that student academic growth data are not available until the following year and might influence how some principals assess teachers in the year in which the data become available. For example, after receiving data that shows a teacher demonstrated good student academic growth the prior year, a principal might overlook poor classroom management when observing the teacher. Organizations representing teachers and principals also raised concerns about evaluation consistency (see sidebar).

State, district, or union officials in six RTT states described efforts to improve consistency in principals’ evaluations of teachers, generally through training. In New York, officials from a state teachers’ union said they provided training to more than 750 principals on ensuring consistency when conducting teacher evaluations. Tennessee officials said that during the first year of implementation, principals participated in 4 days of training and had to pass a test in order to perform classroom observations. During the second year, Tennessee identified principals who did not evaluate teachers appropriately and provided them with additional support and coaches. Officials also said that Tennessee plans to make its certification test more rigorous. Officials in one North Carolina district said that, in addition to providing state training and workshops on evaluation consistency, district administrators conduct informal classroom walk-throughs to observe teachers and then discuss rating consistency while comparing their notes with the principal’s observation ratings.

Concerns about Magnitude of Change Challenged States’ Efforts

State or district officials in 11 of the 12 RTT states discussed the difficulty of addressing teacher concerns about the scale of evaluation reform. According to these officials, teachers were concerned about some of the significant changes in the new systems, such as the use of student academic growth data in evaluations and using evaluation results to make personnel decisions (e.g., retention or compensation). For example, state, district, and union officials in Maryland said that teachers did not trust the validity of the state test scores used in some of the student academic growth measures. Officials in one New York district were concerned generally about the rise in annual testing and its use in evaluations to inform personnel decisions. District and union officials in New York said the release of teacher evaluation ratings to parents added to concerns about evaluation systems. Officials in one small district said their teachers were particularly concerned because protecting their anonymity might be difficult even if data are aggregated and not linked to individual teachers. Officials in three states and one district said they had difficulty convincing teachers that evaluation systems were focused on professional development, rather than consequences. Some RTT state and district officials said the simultaneous transition to new state assessments and the Common Core curriculum—a single set of educational standards in language arts and math—increased teacher concerns about consequences. For example, North Carolina officials said teachers were concerned about the fairness of measuring student academic growth while schools are implementing a new curriculum. In some RTT states, according to state and union officials, lengthy collective bargaining processes or lawsuits slowed implementation efforts.

State and district officials said they took steps to address teacher concerns, in part by involving teachers in the design and implementation of the evaluation systems and through ongoing communication with teachers. State or district officials in 10 of the 12 RTT states highlighted efforts such as teacher participation on committees that designed the systems, teacher involvement in national training workshops, and regular communication and feedback from teachers on implementation. In addition, officials from all three state organizations representing teachers said they helped develop the legal framework or overarching standards for their states’ evaluation systems and participated on committees or provided training to teachers and principals to support their state’s efforts. To reduce teacher concerns once reform efforts were under way, officials in Maryland regularly distributed a document to key stakeholders that, among other things, provided updates related to evaluation components. Officials in Georgia said they made presentations in the community, held focus groups in districts, and provided training to help manage the culture shift to the new evaluation system.

State or district officials in most RTT states (9 of the 12) said they faced fewer concerns related to principal evaluations due to greater principal support, the smaller scale of implementation, or because principals were used to being evaluated based on student performance. According to North Carolina officials, superintendents used student academic growth in principal evaluations prior to RTT, so principals did not have the same level of concern as teachers. In addition, North Carolina officials noted that the state had 2,600 administrators compared to 95,000 teachers, which made principal evaluation easier to implement. Hawaii officials said implementing principal evaluations was a generally collaborative and productive process for several reasons. For instance, they said principal associations were relatively easy to work with, administrator assessments already existed, and principals understood the need for a new evaluation system and contributed significantly to its design. In another state, officials from a principals’ association echoed the view that principals were accustomed to being evaluated on student academic growth and added that principals in their state were more concerned about teacher evaluations than their own evaluations.

States Took Steps to Address Insufficient Capacity

Insufficient state and district capacity challenged RTT states’ efforts to design and implement their evaluation systems (see fig. 3). State or district officials in most of these states said they lacked either sufficient staff or needed expertise when they began to reform their evaluation systems. Some state officials also said they faced capacity challenges related to supporting district efforts, such as reviewing and approving district evaluation systems and providing technical assistance. For example, Florida officials said that, because their RTT districts had the flexibility to design their own systems, it was difficult to develop solutions to challenges that would be applicable to all RTT districts. State and district officials said that at the local level, districts had difficulty managing principal workloads or prioritizing evaluation reform amid multiple educational initiatives. For example, officials in a New York district said that the time commitment required for observing and evaluating teachers prevented some principals from thoroughly reviewing evidence submitted for evaluations or providing meaningful feedback to teachers. District officials in New York and Maryland told us that their evaluation reform efforts took precedence over other initiatives, such as implementation of the Common Core curriculum. Building capacity to enact education reforms has been a recurring challenge for states and districts, as we have discussed in previous reports.

While RTT was designed to encourage education innovation and reform—rather than covering all costs of reform efforts—several state and district officials cited the high cost of designing and implementing evaluation systems as a challenge. For example, officials in Hawaii and Delaware (see sidebar) noted that they underestimated how much it would cost to develop these systems. Similarly, officials in 7 of the 12 districts we spoke with said their RTT funds did not cover the costs of reforming their evaluation systems. For example, one small, rural New York district spent about $62,400 on its teacher and principal evaluation systems in addition to the $22,856 it received in RTT funds. Other New York districts faced similar challenges. A 2011 survey conducted by the New York State Council of School Superintendents shows that 81 percent of responding superintendents were concerned that cost considerations might prevent their districts from soundly implementing new evaluation requirements. Cost may have been more of a challenge for some districts because they were responsible for a significant part of the design and implementation work. Six of the 7 districts in which officials raised cost as a capacity challenge were in Maryland and New York, both of which provide RTT districts with significant flexibility to design their own systems. Officials in Tennessee explained that some RTT districts in their state did not have funding concerns because they used the evaluation system and data system provided by the state.

States and districts responded to capacity challenges through different efforts to supplement their staff and resources (see fig. 4). Several RTT states also submitted amendment requests and received approval from Education to shift funds among RTT projects to provide additional funding for particular aspects of their evaluation systems. For example, Tennessee shifted approximately $1.1 million to support, among other things, additional training on evaluation systems because the state did not originally estimate sufficient funds for this purpose. Similarly, New York increased its budget for its evaluation systems by $11.9 million by shifting funds to develop its student academic growth model, pilot evaluation system software, and provide additional resources to districts.

State or district officials in most RTT states (10 of the 12) said that fewer staff or other resources after RTT grant funds are no longer available could affect their ability to sustain their evaluation systems. For example, Rhode Island officials said they will likely lose staff that they hired using RTT funds because the state may not be able to use other education funds to make these positions permanent. Officials in New York said that with the loss of RTT funding, the state will have fewer staff to review district evaluation plans every year and to provide technical assistance to districts, as well as to manage the analysis of statewide evaluation data. District of Columbia officials were concerned that without RTT funds, they would be unable to pay the contractor that operates the student academic growth model used by its charter school districts.

Officials from all 12 RTT states said they are considering how to sustain their evaluation systems after RTT grant funds are no longer available. Officials in a few of these states discussed some of the difficulties they have faced in preparing for sustainability, such as turnover in state leadership and uncertainty over future funding levels, and a few officials provided examples of how they might address sustainability. For example, Hawaii officials said they are considering how to reallocate funds to sustain the systems but are concerned about the availability of other federal and state funds to do so. Georgia officials said they are collaborating with stakeholders to develop a sustainability plan—to be completed in summer 2013. In addition, Florida officials said they were working to ensure that they have in-house expertise on all aspects of the evaluation systems. For example, contractors who assisted with the state’s student academic growth component will train state staff on how to run the models.

Officials in 5 of the 12 RTT states told us more information from Education could help address their concerns about sustaining their evaluation systems and other reforms after RTT grant funds are no longer available. Specifically, state and district officials from some of these states told us they were concerned about or would like guidance on how to use other federal funds to support their evaluation systems. For example, officials in one state said Education issued some guidance on acceptable uses of ESEA funding, but could provide more concrete information on how best to leverage those funds for RTT initiatives. In addition, Education officials told us a few states have requested technical assistance to support their sustainability planning. Officials from four states told us it was too soon for them to know whether they would need Education’s assistance with sustainability.

Education Supports RTT State Implementation Efforts through Monitoring and Technical Assistance

Education Monitors States Using a New Process to Hold Them Accountable for Quality of Implementation

Education developed a new process to monitor RTT states’ progress toward meeting their RTT goals, including those related to teacher and principal evaluation systems. Education officials said that the RTT monitoring process differs from the department’s other monitoring efforts in that Education has more frequent contact with the states in order to identify and address implementation challenges. In addition, the new process emphasizes states’ continuous improvement and quality of RTT reforms, rather than focusing solely on compliance with laws and regulations and the ability of states to meet their time frames.said the intensity of communication with RTT states and the quality standards are greater for RTT than for Education’s previous monitoring efforts. Education developed the new process to provide assistance to RTT states as they implement comprehensive reforms and to differentiate support based on individual state needs.

Education officials said they work to identify and address obstacles to the goals states established in their RTT plans through ongoing communication, including monthly monitoring calls, the amendment consideration process, and other contacts with RTT state officials. To assess the quality of implementation efforts, officials said they consider each state’s progress toward its goals and timelines, risk factors and strategies for addressing them, and the state’s own assessment of its quality of implementation, among other factors. For example, in addition to verifying that a state implemented an evaluation tool, such as a test or performance measure, Education officials work with the state to ensure that the tool is meeting the state’s needs. Instead of focusing solely on RTT compliance, program officers also help identify areas in which Education can assist states in meeting their goals, according to Education officials.

Officials from 8 of the 12 RTT states expressed generally positive views about Education’s RTT monitoring activities. Some said, for example, that Education officials were collaborative, well-informed, and that they generally provided useful feedback. For example, officials from one state said that Education staff were very detailed and thorough in monthly monitoring calls and that they usually provided actionable feedback. Officials from another state said they spoke almost daily with Education officials and received strong support. They noted that, as a result, monitoring reviews were not stressful, and they were not surprised by the results. Officials from another state said they appreciated the discussion with Education officials about the state’s amendment requests and how Education worked with them to ensure that the state maintained its original RTT goals.

While RTT state officials expressed generally positive views about the monitoring process overall, officials in nine states expressed concerns about specific aspects of the process, including delays in the amendment process, time-consuming monthly calls and related requirements, and slow feedback from Education after site visits. Officials from one state said monitoring requirements seemed more burdensome than those for other federal education programs. Education officials stated that they have revised some aspects of their monitoring process in response to state feedback. For example, they modified the monthly monitoring call and onsite review protocols, revised the amendment process and dollar threshold amounts that require approval, and worked to explain the rationale and use of the information Education requests.

To ensure that states are held accountable for meeting their RTT goals for teacher and principal evaluation systems, Education may take the following corrective actions for states that have not demonstrated adequate progress in implementing their systems:

Conditional amendment approval. If Education has concerns about a state’s requested amendments to its RTT plans, it may grant conditional approval, requiring the state to provide additional information over a period of time. For example, Rhode Island submitted a proposed amendment requesting a change related to its use of SLOs. Education approved the request on the condition that the state provide additional information, such as quarterly progress updates during SY 2012-13 and additional reports. In addition, Maryland received approval to decrease the percentage of the evaluation component that is based on student academic growth models on the condition that the state provide Education a plan for a statewide field test of its evaluation systems. Maryland was also required to commit to measuring student academic growth using common assessments of high school teachers and principals when those assessments are available, among other requirements. Education may also elect not to approve an amendment request.

High-risk status. Education placed 2 of the 12 RTT states—Georgia and Hawaii—on high-risk status because officials determined that the states required intensive attention and support in order to meet their RTT goals. In July 2012, Education placed the teacher and principal evaluation portion of Georgia’s RTT grant on high-risk status because officials were concerned about the overall strategic planning, evaluation, and project management of the evaluation system. Education officials also expressed concern that Georgia had requested two major amendments that seemed to constitute significant changes to the evaluation system in the state’s approved plan. As a result of the high-risk designation, Georgia was required to provide Education a revised work plan for its system, monthly updates in accordance with the work plan, and related information. As of July 2013, Education officials said the evaluation system portion of Georgia’s RTT grant remains on high-risk status because of Education’s continued concerns about the quality of implementation. In December 2011, Education designated Hawaii’s entire RTT grant as high-risk because the state experienced major delays and made inadequate progress on implementing its systems and because the scope and breadth of amendment requests indicated a potentially significant shift in the state’s approved plans. Education temporarily placed Hawaii on a cost-reimbursement basis, which required the state to submit receipts for expenditures to the department prior to drawing down grant funds. The state was also required to submit documentation prior to obligating funds and to submit a revised scope of work and budget. As of July 2013, Education had removed Hawaii’s high-risk designation based on the state’s demonstrated progress in implementing its RTT reforms, including its evaluation systems.

Additional information. Education has required certain reporting or follow-up information, other than that included in conditional approval of amendments or high-risk status, and has used other measures deemed appropriate. For example, according to Education officials, one state that experienced procurement problems is required to provide monthly procurement information to Education.

Withholding of funds. Although Education may withhold grant funds from states if they do not comply with the terms of the award,Education officials said they have not withheld funds from any RTT state. Officials added that states have always demonstrated progress toward addressing Education’s concerns.

Education Provides Technical Assistance to Help States Resolve Implementation Issues

Education helps states meet their RTT goals and implement high-quality reforms by providing technical assistance, including access to experts and information on options for evaluation systems. Education officials said technical assistance helps states resolve implementation issues, including those identified through the monitoring process. Most RTT federal assistance is provided by the contractor-supported RSN, and Education also provides some technical assistance to RTT states.officials said they work closely with Education staff to learn about the RSN types of technical assistance that might be useful to states on teacher and leader effectiveness, including teacher and principal evaluations, as well as other RTT topics.

From 2010 through March 2013, RSN provided technical assistance on teacher and leader effectiveness in group settings—such as webinars and in-person meetings—to RTT states, as well as individualized technical assistance (see fig. 5). RSN officials said they provide individualized assistance to states when requested, particularly for states in more advanced stages of implementation with needs that could not be met through larger group technical assistance activities. RSN also developed publications related to teacher and leader effectiveness, including case studies, tool kits, and lessons learned, and has provided them through the RTT grantee web portal. These publications included longer reports on school reform and shorter briefs, such as a paper that described rules governing classroom observations used in teacher evaluations in selected RTT states.

RSN has worked to strengthen the quality of its technical assistance and adapt to states’ changing needs, according to Education and RSN officials. They said that early in the contract, RSN revised its approach to better meet the needs of states. For example, in response to state feedback, RSN provided states with access to education practitioners who had worked in schools rather than experts without hands-on experience, as they had done in the initial stages of the contract. RSN officials also said issues for which states requested technical assistance changed as implementation progressed, and RSN adapted its technical assistance accordingly. For example, early in implementation, states often requested assistance with designing evaluation systems, communicating with stakeholders, working with unions, and measuring growth in nontested grades and subjects, according to RSN officials. As implementation progressed, states requested assistance with issues such as the consistency of observations and the sustainability of evaluation systems.

Officials from 10 of the 12 RTT states told us that Education’s technical assistance related to teacher and principal evaluation systems was generally helpful, and officials in several states said assistance had improved since the start of the contract. Officials from Hawaii said RSN had helped states by sharing existing knowledge and developing new information. Officials in Massachusetts told us that RSN’s in-person meetings had been especially helpful because they provided a platform for states to share best practices. Although most states were complimentary of RSN assistance, officials in some states said RSN and Education could improve technical assistance by providing additional information on specific topics, such as information about states that have successfully implemented evaluation systems and more opportunities to share lessons learned.

A recent survey by RSN also indicates that states are generally satisfied with the contractor’s technical assistance on teacher and principal evaluation systems. In March 2013, RSN surveyed and obtained responses from officials in all RTT states regarding their perception of technical assistance, including assistance provided through the teacher and leader effectiveness community of practice. Sixteen of the 18 states that participated in teacher and leader effectiveness assistance reported that they were satisfied or very satisfied with the assistance, and the remaining states were neutral. On multiple dimensions, the state officials rated assistance with teacher and leader effectiveness higher than other areas and higher than RSN activities overall. States also identified opportunities for strengthening technical assistance by ranking potential topics on the basis of impact and urgency. For teacher and leader effectiveness, states ranked continuous improvement of teacher evaluation as one of the top areas of interest, according to RSN. Officials in a few states mentioned that they would like more opportunities to collaborate and learn from one another.

Education Plans to Provide RTT States and Nongrantees Additional Information to Support State Efforts

Education plans to provide information to RTT states on sustaining teacher and principal evaluation systems and other reforms, but Education officials said they have not yet done so. Education planned to launch a new workgroup in the summer of 2013 to help states consider how to sustain their evaluation systems after the RTT grant ends. Draft plans for the work group included providing expert and peer-to-peer support and developing a sustainability rubric. The plans also included providing workshops on sustainability efforts, including ones on state capacity, performance management, and communication, and eventually developing and sharing case studies. In July 2013, Education officials said they had postponed work in this area until fall 2013. Although Education has not asked states to provide specific plans for addressing sustainability, department officials said they have learned about state plans through ongoing communication with states. For example, a few states discussed their sustainability planning during monthly monitoring calls. In addition, Education obtained information on state sustainability strategies through RTT applications. In general, states provided this information on RTT reforms as a whole. We did not identify any states as having provided a sustainability strategy specific to teacher and principal evaluation systems.

To allow states more time to accomplish the goals and deliverables they committed to in their RTT plans, Education officials will consider requests for no-cost extensions, but they have not determined how to provide technical assistance during the extension period. States may request extensions on a case-by-case basis for more time to spend awarded funds for those aspects of their RTT reforms that require additional work. If approved, a state could have until September 30, 2015 to obligate and liquidate its remaining RTT grant funds. According to Education officials, states that request no-cost extensions will be required to provide to Education their plans to address sustainability, among other information. As of July 2013, Education officials had approved two no-cost extensions related to teacher and principal evaluation systems, and officials in an additional state told us they had submitted an extension request related to their evaluation systems. Officials in 6 more of the 12 RTT states told us they are considering requesting extensions related to their evaluation systems. However, it is not clear what technical assistance would be available to states approved for no-cost extensions. The current contract for technical assistance ends in September 2014, and RSN officials said they do not have plans to sustain technical assistance beyond the duration of the current contract. Education officials said they were working to identify options for providing continued technical assistance.

GAO-11-658. student academic growth data. In addition to the materials currently available, Education officials told us that they were developing additional resources and materials on topics such as SLOs, observation rubrics, rating inflation, teacher engagement, data analytics, and leadership development. Education identified some products for targeted dissemination and presented them in conferences, cross-program meetings, and to organizations such as the National Governors Association, in order to promote awareness of the resources available. Education is also working to develop a more robust dissemination plan that includes ways to reach people other than state-level leaders, according to officials.

Concluding Observations

Education created the RTT grant program to encourage sweeping changes in K-12 education. RTT spurred changes to the way states and districts evaluate their teachers and principals, particularly with the addition of student academic growth data as a factor in assessing effectiveness. Education has been proactive in monitoring states’ progress in implementing their evaluation systems, and the department’s continued monitoring and assistance will be important to help RTT states overcome challenges and implement the reforms to which they committed. In addition, Education’s new monitoring process has resulted in a wealth of information on states’ efforts. As a result, Education is uniquely positioned to use the lessons learned from RTT states to inform other states’ efforts to improve teacher effectiveness and ultimately raise student academic achievement.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Education for review and comment. Their comments are reproduced in appendix I. Education also provided technical comments, which we incorporated into the report.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Education. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the U.S. Department of Education

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Elizabeth Morrison, Assistant Director, Nisha R. Hazra, Marissa Jones, and Michael Kniss made significant contributions to this report. Also contributing to this report were Deborah Bland, Sarah Cornetto, Jamila Jones Kennedy, Amy Moran- Lowe, Jean McSween, Mimi Nguyen, Jason Palmer, Kathleen van Gelder, and Rebecca Woiwode.

Related GAO Products

School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for Schools and Contractors. GAO-12-373. Washington, D.C.: April 11, 2012.

Race to the Top: Characteristics of Grantees’ Amended Programs and Education’s Review Process. GAO-12-228R. Washington, D.C.: December 8, 2011.

Race to the Top: Reform Efforts are Under Way and Information Sharing Could Be Improved. GAO-11-658. Washington, D.C.: June 30, 2011.

Department of Education: Improved Oversight and Controls Could Help Education Better Respond to Evolving Priorities. GAO-11-194. Washington, D.C.: February 10, 2011.

Grant Monitoring: Department of Education Could Improve Its Processes with Greater Focus on Assessing Risks, Acquiring Financial Skills, and Sharing Information. GAO-10-57. Washington, D.C.: November 19, 2009.

Student Achievement: Schools Use Multiple Strategies to Help Students Meet Academic Standards, Especially Schools with Higher Proportions of Low-Income and Minority Students. GAO-10-18. Washington, D.C.: November 16, 2009.

No Child Left Behind Act: Enhancements in the Department of Education’s Review Process Could Improve State Academic Assessments. GAO-09-911. Washington, D.C.: September 24, 2009.

Teacher Quality: Sustained Coordination among Key Federal Education Programs Could Enhance State Efforts to Improve Teacher Quality. GAO-09-593. Washington, D.C.: July 6, 2009.

No Child Left Behind Act: Improvements Needed in Education’s Process for Tracking States’ Implementation of Key Provisions. GAO-04-734. Washington D.C.: September 30, 2004.

Recovery Act

Recovery Act: Opportunities to Improve Management and Strengthen Accountability over States’ and Localities’ Uses of Funds. GAO-10-999. Washington, D.C.: September 20, 2010.

Recovery Act: One Year Later, States’ and Localities’ Uses of Funds and Opportunities to Strengthen Accountability. GAO-10-437. Washington, D.C.: March 3, 2010.

Recovery Act: Status of States’ and Localities’ Use of Funds and Efforts to Ensure Accountability. GAO-10-231. Washington, D.C.: December 10, 2009.