Race To The Top: States Implementing Teacher and Principal Evaluation Systems despite Challenges
Published: 2013-09-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Education created RTT under the American Recovery and Reinvestment Act of 2009 to provide incentives for states to reform K-12 education in areas such as improving the lowest performing schools and developing effective teachers and leaders. In 2010, Education awarded 12 states nearly $4 billion in RTT grant funds to spend over 4 years. A state's RTT application and scope of work included the state's plans for development and implementation of teacher and principal evaluation systems by participating school districts. These systems assess teacher and principal effectiveness based on student academic growth and other measures, such as observation of professional practice. Currently, additional states are designing and implementing similar evaluation systems.
GAO was asked to review RTT teacher and principal evaluation systems. This report examines (1) the extent to which the 2010 grantee states have implemented their teacher and principal evaluation systems, (2) the challenges the grantee states have faced in designing and implementing these systems, and (3) how Education has helped grantee states meet their RTT objectives for teacher and principal evaluation systems.
GAO reviewed relevant federal laws, regulations, and guidance; analyzed RTT applications and documentation on each state's guidelines for their evaluation systems; and interviewed officials from all 12 states, selected districts, and Education.
section_title: What GAO Found
paragraphs: By school year 2012-13, 6 of 12 Race to The Top (RTT) states fully implemented their evaluation systems (i.e., for all teachers and principals in all RTT districts). However, their success in fully implementing by the date targeted in their RTT applications varied. Three of these states met their target date while three did not for various reasons, such as needing more time to develop student academic growth measures. The six states that did not fully implement either piloted or partially implemented. The scope of pilots varied. One state piloted to about 14 percent of teachers and principals while another piloted to about 30 percent of teachers. State or district officials in four of the six states expressed some concerns about their readiness for full implementation.
Officials in most RTT states cited challenges related to developing and using evaluation measures, addressing teacher concerns, and building capacity and sustainability. State officials said it was difficult to design and implement rigorous student learning objectives--an alternate measure of student academic growth. In 6 states, officials said they had difficulty ensuring that principals conducted evaluations consistently. Officials in 11 states said teacher concerns about the scale of change, such as the use of student academic growth data and consequences attached to evaluations, challenged state efforts. State and district officials also discussed capacity challenges, such as too few staff or limited staff expertise and prioritizing evaluation reform amid multiple educational initiatives. Officials in 10 states had concerns about sustaining their evaluation systems.
Education helps RTT states meet their goals for teacher and principal evaluation systems through a new monitoring process and through technical assistance. Education officials said the RTT monitoring process differs from other monitoring efforts in the frequency of contact with the states and the emphasis on continuous improvement and quality of RTT reforms. Officials from 8 of the 12 RTT states expressed generally positive views about Education's monitoring. When states have not demonstrated adequate progress, Education has taken corrective action. For example, Education designated two states as high-risk, which resulted in additional monitoring. Education provides technical assistance through a contractor; officials from 10 RTT states told us assistance related to evaluation systems was generally helpful. Education officials said they plan to provide RTT and nongrantee states with more information to support their efforts.
section_title: What GAO Recommends
paragraphs: GAO is not making recommendations in this report.

section_title: Letter
paragraphs: The U.S. Department of Education (Education) created Race to the Top  (RTT) to provide incentives for states to implement large-scale, far- reaching reforms to improve student achievement, close achievement  gaps, and increase graduation and college enrollment rates. This  program is part of the current administration’s larger effort to ensure that  students are college and career ready. RTT is the largest competitive  grant fund ever administered by Education, and in 2010, Education  awarded nearly $4 billion in grant funds to 12 states. To be competitive  for RTT grants, states had to demonstrate a commitment to reforming K- 12 education across four interconnected areas: adopting standards and  assessments, building data systems, supporting teachers and principals,  and improving the lowest-achieving schools. Among the reforms related  to teachers and principals, Education called for states and their  participating school districts to improve teacher and principal  effectiveness by developing evaluation systems that, in part, take into  account data on student academic growth and inform personnel  decisions, such as compensation, promotion, and retention.also using other means to encourage reforms of teacher and principal  evaluation systems. Currently, many non-RTT states and districts are  designing and implementing evaluation systems to receive waivers from  certain requirements of the Elementary and Secondary Education Act   Education is  (ESEA) of 1965, as amended.evaluation system reform by setting a goal in its strategic plan to increase  the number of states and districts that have comprehensive teacher and  principal evaluation systems. More broadly, the current administration has  included evaluation systems as an area of reform in its blueprint for  revising ESEA.
In addition, Education has prioritized  In 2011, we found that some states faced challenges implementing their  RTT initiatives. We also found that states had made amendments to their  RTT plans—including their evaluation systems—such as changes to   You asked us to provide information about RTT  costs and timelines.grantees’ progress toward implementing their teacher and principal  evaluation systems. In this report, we examined (1) the extent to which  the 2010 grantee states have implemented their teacher and principal  evaluation systems; (2) the challenges the grantee states have faced in  designing and implementing these systems; and (3) how Education has  helped grantee states meet their RTT objectives for teacher and principal  evaluation systems.
In conducting this work, we focused on the 12 states that received RTT  grant funds in 2010. We interviewed state officials from all 12 grantee  states as well as school district officials in Maryland, New York, and North  Carolina, either in person or by phone. We chose states that differed in  terms of the flexibility they provided to districts to design their teacher and  principal evaluation systems (Maryland and New York offer districts more  flexibility, whereas North Carolina has a statewide evaluation system and  offers districts less flexibility). In selecting these states, we also  considered the RTT award amount and the percentage of students  served by school districts participating in RTT. We interviewed a total of  12 school districts in these three states. We asked state officials to  identify districts that varied in terms of the design of their evaluation  systems (if applicable) and the range of challenges they faced. Other  criteria included whether a district was urban, suburban, or rural and the  percentage of high-poverty or high-minority schools in the district. The  views of these 12 districts cannot be generalized to all districts that  participated in RTT, but they do provide important insights into some  districts’ experiences with RTT implementation. The state and district  officials we interviewed provided information on the key components of  their evaluation systems, the implementation status of these systems,  what they considered to be their design and implementation challenges,  and Education’s monitoring activities and technical assistance. We also  interviewed officials from unions or organizations representing teachers or  principals in Maryland, New York, and North Carolina to obtain their  perspectives on design and implementation challenges.
In addition, we reviewed documents to obtain information on states’  guidelines for the evaluation systems used by their districts, including  each state’s RTT application, teacher and principal evaluation system  manuals, and progress reports developed by the states and Education.  We also reviewed relevant federal laws, regulations, and guidance and  interviewed Education officials from the Implementation and Support Unit,  as well as officials from the contractor Education hired to provide  technical assistance. We obtained data from the contractor on the  number and types of technical assistance activities provided to RTT  states. We assessed the reliability of data by interviewing agency officials  knowledgeable about the data and determined that the data were  sufficiently reliable for the purposes of this report. The American  Recovery and Reinvestment Act of 2009 (Recovery Act) requires that we  conduct bimonthly reviews of how the act’s funds are used by selected  states and localities. This report fulfills this requirement in that we examined the use of Recovery Act funds by the 12 states that were  awarded RTT grants in 2010.
We conducted this performance audit from August 2012 through  September 2013 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: RTT Overview
paragraphs: In accordance with Recovery Act requirements, Education established the  RTT grant fund to encourage states to reform their K-12 education  systems and to reward states for improving student outcomes, such as  making substantial gains in student achievement and improving high  school graduation rates. States competed for RTT grant funds based on  reforms across four areas: adopting standards and assessments that prepare students to  succeed in college and the workplace and to compete in the global  market; building data systems that measure student academic growth and  success and inform teachers and principals about how they can  improve instruction;  recruiting, developing, rewarding, and retaining effective teachers and  principals, especially where they are needed most; and  turning around the lowest-achieving schools.
Education awarded RTT grants in three phases. Twelve states received  grants in 2010 in Phases 1 and 2 to support the design and  implementation of their teacher and principal evaluation systems and  other RTT reforms. Award amounts ranged from $75 million to $700  million (see table 1). States were required to subgrant at least 50 percent  of their total grant award to districts that chose to participate in RTT. The  4-year grant period began on the date funds were awarded to the state.  States must obligate all funds within that period, and they have 90 days  following the end of their grant period to liquidate all obligated funds  unless they receive a no-cost extension. Education may grant extensions  for states beyond the 90 days on a case-by-case basis.and Phase 2 funds not obligated and liquidated by September 30, 2015,  will revert to the U.S. Treasury.
Education identified 19 primary criteria to guide peer reviewers in the  selection of states for RTT grants (see table 2).
The criterion—improving teacher and principal effectiveness based on  performance— established the RTT guidelines for teacher and principal  evaluation systems. Reviewers evaluated the state’s plan to ensure its  participating RTT districts (1) measure student growth for each individual  student; (2) design and implement evaluation systems, developed with  teacher and principal involvement, that include multiple rating categories  that take into account data on student growth as a significant factor; (3)  evaluate teachers and principals annually and provide feedback, including  student growth data; and (4) use these evaluations to inform decisions  regarding professional development, compensation, promotion, retention,  tenure, and certification.
Education defines student growth as the change in student achievement  for an individual student between two or more points in time. For students  in grades and subjects that are tested by state standardized tests,  Education defines student achievement as the score received on the  state’s assessments required under the ESEA. For students in grades  and subjects that are not tested by state standardized tests, Education  defines student achievement based on alternative measures of student  learning and performance. These measures include student scores on  pre-tests and end-of-course tests, student performance on English  language proficiency assessments, and other measures of student  achievement that are rigorous and comparable across classrooms.  Student achievement for students in tested grades and subjects can also  be assessed using other measures as appropriate, including the same  measures as students in nontested grades and subjects.
Education provided background information in its notice of proposed  priorities, requirements, definitions, and selection criteria for RTT on why  it included student growth as a factor in its criteria for teacher and  principal evaluation. Education noted the difficulty in predicting teacher  quality based solely on the qualifications that teachers bring to the job.  The department cited research on the limited predictive power of  measures such as certification, education, and years of experience, and  research on the value of measuring student growth to assess teacher  quality. In response to public comments that expressed concern about  the use of student growth data as the sole means to evaluate teachers  and principals, Education revised its definitions of an effective teacher  and effective principal to require that multiple measures be used to  assess effectiveness, with student growth as a significant factor.  Education also provided examples of these supplemental measures, such  as multiple observation-based assessments of teacher performance and  high school graduation rates as a measure for evaluating principals.
Education also established criteria for peer reviewers to consider a state’s  capacity to sustain its reforms. The criterion—building strong statewide  capacity to implement, scale up, and sustain proposed plans—required  reviewers to assess the extent to which the state had a plan to ensure  sufficient capacity and use stakeholder support to implement its plans.  States were evaluated on, among other things, the extent to which they  demonstrated that they would provide strong leadership and dedicated  teams to implement the reforms and use their fiscal, political, and human  capital resources to continue successful grant-funded reforms after RTT  funds are no longer available.
section_title: Role of the Department of Education
paragraphs: Education is responsible for fiscal and programmatic oversight of all  aspects of RTT, reviewing and responding to states’ requests to amend  their RTT applications, and providing technical assistance. To monitor  states’ progress, Education established a program review process that  includes ongoing conversations with grantees, on-site program reviews,  grantee self-evaluations, and meetings with Education officials. As we  reported previously, Education uses a common set of questions to  oversee state progress and to address specific needs and challenges of  each grantee. Education also publishes annual reports to the public  summarizing the progress of each state. To provide technical assistance,  Education established the Reform Support Network (RSN), a 4-year, $43  million technical assistance contract with ICF International, which works  with Education to support RTT states.
Education’s process for reviewing and approving changes to a state’s  RTT plans includes reviewing the state’s approved application, budget,  and scope of work. According to Education’s guidance, an RTT grantee  must submit an amendment request for (1) a proposed revision that  constitutes a change in activities from the approved grant project,  regardless of budgetary impact; (2) budgetary changes, including  transfers among categories or programs, that exceed $500,000 of the  current approved budget; or (3) changes to the list of districts participating  with the grantee’s RTT plan. Education will not approve amendment  requests that would change the overall scope and objectives of the  approved proposal, fail to comply with the terms of the award or the  statutory and regulatory provisions of the program, or violate the general  principles of the program.
Education’s Institute of Education Sciences’ National Center for  Education Evaluation and Regional Assistance is conducting two studies  that relate to RTT teacher and principal evaluation systems. One study  will assess the RTT and School Improvement Grant programs and  whether these programs are related to improvement in student outcomes.  The results of this study, which will not specifically assess the impact of  teacher and principal evaluations on student outcomes, are expected in  2014. In the second study, experimental teacher and principal evaluation  systems will be implemented in schools in eight districts in order to study  their effects on factors such as student achievement and teacher and  principal mobility. A report on this study is expected in 2015.
section_title: Teacher and Principal Evaluation Systems in the RTT States
paragraphs: The RTT states provide districts with varying amounts of flexibility to  develop their evaluation systems. For example, some RTT states  developed evaluation systems for use by all districts, unless a district  develops an alternate evaluation system that meets state requirements.  In other states, districts can develop their own evaluation systems within  guidelines provided by the state, and the state must approve each  district’s system.state or the district, districts evaluate teachers and principals using  multiple measures that assess professional practice and student  academic growth (see fig.1).
section_title: Six of 12 RTT States Fully Implemented Their Evaluation Systems by the 2012-13 School Year
paragraphs: According to state officials, 6 of the 12 RTT states fully implemented both  their teacher and principal evaluation systems by school year (SY) 2012- 13 (see fig. 2), though their success in meeting their original target date  for implementation varied. The states that fully implemented their systems  evaluated all teachers and principals in RTT districts, according to state  officials. The six states that fully implemented both teacher and principal  evaluation systems targeted SY 2011-12 for full implementation in their  RTT applications. Three of the six states met that target and SY 2012-13  was their second school year of full implementation. The other three  states did not meet the targets set in their applications, but did fully   According to Education’s  implement their systems in SY 2012-13.amendment approval letters, states shifted implementation time frames,  in part, because they needed additional time to develop student academic  growth measures. For example, Delaware required an additional year to  develop measures for its student academic growth component, which  state officials said resulted in a better evaluation system.
The six states that did not fully implement both their teacher and principal  evaluation systems in SY 2012-13 either piloted or partially implemented  evaluation systems, according to state officials (see fig. 2). Based on  the targets set in their RTT applications, four of the six states originally  planned to fully implement by SY 2012-13 but are instead piloting or  partially implementing their systems. The proportion of teachers and  principals participating in pilots varied. According to state officials,  Hawaii’s teacher evaluation pilot covered about 30 percent of its teachers,  and Maryland’s evaluation systems pilot covered about 14 percent of its  teachers and principals in RTT districts. Among the four districts we  visited in Maryland, district officials said the percentage of teachers who  participated in the districts’ pilots ranged from about 4 percent to 100   State or district officials in four of the six states expressed  percent.some concerns about their readiness for full implementation. For  example, officials in one Maryland district that piloted with about 4 percent  of teachers said they will move from learning about the system to full  implementation without sufficient time to address issues that arose during  the pilot. Similarly, officials in another Maryland district that piloted with  about 5 percent of teachers and 19 percent of principals said the district  did not have sufficient time to work with teachers and principals on the  new evaluation systems and would have benefited from another pilot  year. The Maryland district officials said that two individuals were  responsible for all of the evaluation systems work. These officials added  that they anticipate budget and staff reductions as they move from their  pilot, in which about 100 teachers and 10 principals were evaluated, to full  implementation that will cover more than 3,000 people.
Due in part to the difficulty of managing many changes simultaneously, including new  curriculum and assessments in many states, in June 2013 Education offered states that  have received ESEA waivers or RTT grants the option to request permission from  Education to delay the use of their new evaluation systems to inform personnel  determinations and consequences for up to 1 year. Education officials noted that many  states are already successfully implementing these changes or have requirements in state  law about implementation timeframes and thus may not need to request the waiver. teachers and principals, such as retention rewards and dismissal.  According to Hawaii officials, the state plans to fully implement its teacher  evaluation system in SY 2013-14, but all consequences related to  evaluations will be added the following school year. In several states,  RTT districts decide how to use evaluation results to determine  consequences. Ohio officials said that RTT districts were required to use  evaluation results to inform some personnel decisions—including  professional development, retention, and pay for performance—and the  state surveyed RTT districts to confirm that they did so. Tennessee  officials said that RTT districts were required to use the results of  evaluations to inform certain personnel decisions, such as employment,  compensation, and dismissal, but that the state did not prescribe the  consequences attached to different ratings.
section_title: Most RTT States Cited Challenges with Developing and Using Certain Evaluation Measures, Addressing Teacher Concerns, and Building Capacity and Sustainability
subsections: section_title: RTT States Struggled to Develop and Use Student Learning Objectives and Assess Teacher Professional Practice Consistently
paragraphs: State or district officials in most RTT states (8 of the 12) said they had  difficulty developing and using student learning objectives (SLOs) to  assess student academic growth for teachers. SLOs measure student  academic growth for teachers in nontested grades and subjects, which  represent 65 to 75 percent of teachers nationwide, according to an RSN  report. SLOs are learning objectives for groups of students, such as  students in a social studies class, that use a specific measure, such as a  course exam, to track academic progress throughout a school year.However, some RTT state and district officials said it can be difficult to  ensure that these learning objectives are rigorous and accurately  measure student learning. Tennessee officials said that while SLOs are  popular and promising in theory, they are difficult to reliably implement  because some teachers set non-rigorous goals in order to get high  scores. Tennessee officials further explained that some teachers selected  a schoolwide social studies score for their SLO measure—despite having  no connection to the subject—because students did well on that exam,  rather than selecting learning objectives relevant to their own subject  matter. Officials in three Maryland districts said determining how to  measure student academic growth using learning objectives was a  challenge because, for example, they may have difficulty assessing  students’ abilities when they enter a class, not just when they leave.  Officials in a New York district described the difficulty of implementing  learning objectives in their small, rural district (see sidebar). Despite these  challenges, RTT state and district officials said that SLOs improved their  evaluation systems, in part by engaging teachers in the evaluation  process and by leading to more in-depth discussions about teacher  performance. To address some of their challenges, RTT states developed  guidance, templates, or model learning objectives to help districts develop  and use SLOs. In addition, states participated in an RSN-sponsored  working group on developing SLOs. They could also access RSN  guidance from Education’s website that outlined the benefits of learning  objectives and provided information about the elements that comprise  rigorous, high-quality learning objectives.
Some RTT state and district officials said it was difficult to ensure that  principals assess teacher professional practice consistently. For example,  officials said it was challenging to ensure consistency in how principals  use classroom observations and other evidence, such as lesson plans, to   State or district officials in 6 of  assess a teacher’s instructional methods. the 12 RTT states expressed concerns that, for example, some principals  may not be appropriately identifying teachers who were ineffective and  rating them accordingly. Officials in a few of these states attributed this  to principals lacking the skill to differentiate between effective and  ineffective teachers, or the will to rate teachers in lower categories or to  rate them lower than under the prior evaluation system. Officials in  Tennessee and in two North Carolina districts said evaluation data have  shown that some teachers with low scores on their student academic  growth component received high professional practice ratings. They said  this may indicate that some principals are inflating scores or not  identifying lower-performing teachers and providing critical feedback.  Officials in another North Carolina district described a different concern  about the mismatch in professional practice ratings and student academic  growth. They noted that student academic growth data are not available  until the following year and might influence how some principals assess  teachers in the year in which the data become available. For example,  after receiving data that shows a teacher demonstrated good student  academic growth the prior year, a principal might overlook poor  classroom management when observing the teacher. Organizations  representing teachers and principals also raised concerns about  evaluation consistency (see sidebar).
State, district, or union officials in six RTT states described efforts to  improve consistency in principals’ evaluations of teachers, generally  through training. In New York, officials from a state teachers’ union said  they provided training to more than 750 principals on ensuring  consistency when conducting teacher evaluations. Tennessee officials  said that during the first year of implementation, principals participated in  4 days of training and had to pass a test in order to perform classroom  observations. During the second year, Tennessee identified principals  who did not evaluate teachers appropriately and provided them with  additional support and coaches. Officials also said that Tennessee plans  to make its certification test more rigorous. Officials in one North Carolina  district said that, in addition to providing state training and workshops on  evaluation consistency, district administrators conduct informal classroom  walk-throughs to observe teachers and then discuss rating consistency  while comparing their notes with the principal’s observation ratings.
section_title: Concerns about Magnitude of Change Challenged States’ Efforts
paragraphs: State or district officials in 11 of the 12 RTT states discussed the difficulty  of addressing teacher concerns about the scale of evaluation reform.  According to these officials, teachers were concerned about some of the  significant changes in the new systems, such as the use of student  academic growth data in evaluations and using evaluation results to make  personnel decisions (e.g., retention or compensation). For example,  state, district, and union officials in Maryland said that teachers did not  trust the validity of the state test scores used in some of the student  academic growth measures. Officials in one New York district were  concerned generally about the rise in annual testing and its use in  evaluations to inform personnel decisions. District and union officials in  New York said the release of teacher evaluation ratings to parents added  to concerns about evaluation systems. Officials in one small district said  their teachers were particularly concerned because protecting their  anonymity might be difficult even if data are aggregated and not linked to  individual teachers. Officials in three states and one district said they had  difficulty convincing teachers that evaluation systems were focused on  professional development, rather than consequences. Some RTT state  and district officials said the simultaneous transition to new state  assessments and the Common Core curriculum—a single set of  educational standards in language arts and math—increased teacher  concerns about consequences. For example, North Carolina officials  said teachers were concerned about the fairness of measuring student  academic growth while schools are implementing a new curriculum. In  some RTT states, according to state and union officials, lengthy collective  bargaining processes or lawsuits slowed implementation efforts.
State and district officials said they took steps to address teacher  concerns, in part by involving teachers in the design and implementation  of the evaluation systems and through ongoing communication with  teachers. State or district officials in 10 of the 12 RTT states highlighted  efforts such as teacher participation on committees that designed the  systems, teacher involvement in national training workshops, and regular  communication and feedback from teachers on implementation. In  addition, officials from all three state organizations representing teachers  said they helped develop the legal framework or overarching standards  for their states’ evaluation systems and participated on committees or  provided training to teachers and principals to support their state’s efforts.  To reduce teacher concerns once reform efforts were under way, officials  in Maryland regularly distributed a document to key stakeholders that,  among other things, provided updates related to evaluation components.  Officials in Georgia said they made presentations in the community, held  focus groups in districts, and provided training to help manage the culture  shift to the new evaluation system.
State or district officials in most RTT states (9 of the 12) said they faced  fewer concerns related to principal evaluations due to greater principal  support, the smaller scale of implementation, or because principals were  used to being evaluated based on student performance. According to  North Carolina officials, superintendents used student academic growth in  principal evaluations prior to RTT, so principals did not have the same  level of concern as teachers. In addition, North Carolina officials noted  that the state had 2,600 administrators compared to 95,000 teachers,  which made principal evaluation easier to implement. Hawaii officials said  implementing principal evaluations was a generally collaborative and  productive process for several reasons. For instance, they said principal  associations were relatively easy to work with, administrator assessments  already existed, and principals understood the need for a new evaluation  system and contributed significantly to its design. In another state,  officials from a principals’ association echoed the view that principals  were accustomed to being evaluated on student academic growth and  added that principals in their state were more concerned about teacher  evaluations than their own evaluations.
section_title: States Took Steps to Address Insufficient Capacity
paragraphs: Insufficient state and district capacity challenged RTT states’ efforts to  design and implement their evaluation systems (see fig. 3). State or  district officials in most of these states said they lacked either sufficient  staff or needed expertise when they began to reform their evaluation  systems. Some state officials also said they faced capacity challenges  related to supporting district efforts, such as reviewing and approving  district evaluation systems and providing technical assistance. For  example, Florida officials said that, because their RTT districts had the  flexibility to design their own systems, it was difficult to develop solutions  to challenges that would be applicable to all RTT districts. State and  district officials said that at the local level, districts had difficulty managing  principal workloads or prioritizing evaluation reform amid multiple  educational initiatives. For example, officials in a New York district said  that the time commitment required for observing and evaluating teachers  prevented some principals from thoroughly reviewing evidence submitted  for evaluations or providing meaningful feedback to teachers. District  officials in New York and Maryland told us that their evaluation reform  efforts took precedence over other initiatives, such as implementation of  the Common Core curriculum. Building capacity to enact education  reforms has been a recurring challenge for states and districts, as we  have discussed in previous reports.
While RTT was designed to encourage education innovation and  reform—rather than covering all costs of reform efforts—several state and  district officials cited the high cost of designing and implementing  evaluation systems as a challenge. For example, officials in Hawaii and  Delaware (see sidebar) noted that they underestimated how much it  would cost to develop these systems. Similarly, officials in 7 of the 12  districts we spoke with said their RTT funds did not cover the costs of  reforming their evaluation systems. For example, one small, rural New  York district spent about $62,400 on its teacher and principal evaluation  systems in addition to the $22,856 it received in RTT funds. Other New  York districts faced similar challenges. A 2011 survey conducted by the  New York State Council of School Superintendents shows that 81 percent  of responding superintendents were concerned that cost considerations  might prevent their districts from soundly implementing new evaluation  requirements. Cost may have been more of a challenge for some districts  because they were responsible for a significant part of the design and  implementation work. Six of the 7 districts in which officials raised cost as  a capacity challenge were in Maryland and New York, both of which  provide RTT districts with significant flexibility to design their own  systems. Officials in Tennessee explained that some RTT districts in their  state did not have funding concerns because they used the evaluation  system and data system provided by the state.
States and districts responded to capacity challenges through different  efforts to supplement their staff and resources (see fig. 4). Several RTT  states also submitted amendment requests and received approval from  Education to shift funds among RTT projects to provide additional funding  for particular aspects of their evaluation systems. For example,  Tennessee shifted approximately $1.1 million to support, among other  things, additional training on evaluation systems because the state did not  originally estimate sufficient funds for this purpose. Similarly, New York  increased its budget for its evaluation systems by $11.9 million by shifting  funds to develop its student academic growth model, pilot evaluation  system software, and provide additional resources to districts.
State or district officials in most RTT states (10 of the 12) said that fewer  staff or other resources after RTT grant funds are no longer available  could affect their ability to sustain their evaluation systems. For example,  Rhode Island officials said they will likely lose staff that they hired using  RTT funds because the state may not be able to use other education  funds to make these positions permanent. Officials in New York said that  with the loss of RTT funding, the state will have fewer staff to review  district evaluation plans every year and to provide technical assistance to  districts, as well as to manage the analysis of statewide evaluation data.  District of Columbia officials were concerned that without RTT funds, they  would be unable to pay the contractor that operates the student academic  growth model used by its charter school districts.
Officials from all 12 RTT states said they are considering how to sustain  their evaluation systems after RTT grant funds are no longer available.  Officials in a few of these states discussed some of the difficulties they  have faced in preparing for sustainability, such as turnover in state  leadership and uncertainty over future funding levels, and a few officials  provided examples of how they might address sustainability. For  example, Hawaii officials said they are considering how to reallocate  funds to sustain the systems but are concerned about the availability of  other federal and state funds to do so. Georgia officials said they are  collaborating with stakeholders to develop a sustainability plan—to be  completed in summer 2013. In addition, Florida officials said they were  working to ensure that they have in-house expertise on all aspects of the  evaluation systems. For example, contractors who assisted with the  state’s student academic growth component will train state staff on how to  run the models.
Officials in 5 of the 12 RTT states told us more information from  Education could help address their concerns about sustaining their  evaluation systems and other reforms after RTT grant funds are no longer  available. Specifically, state and district officials from some of these  states told us they were concerned about or would like guidance on how  to use other federal funds to support their evaluation systems. For  example, officials in one state said Education issued some guidance on  acceptable uses of ESEA funding, but could provide more concrete  information on how best to leverage those funds for RTT initiatives. In  addition, Education officials told us a few states have requested technical  assistance to support their sustainability planning. Officials from four  states told us it was too soon for them to know whether they would need  Education’s assistance with sustainability.
section_title: Education Supports RTT State Implementation Efforts through Monitoring and Technical Assistance
subsections: section_title: Education Monitors States Using a New Process to Hold Them Accountable for Quality of Implementation
paragraphs: Education developed a new process to monitor RTT states’ progress  toward meeting their RTT goals, including those related to teacher and  principal evaluation systems. Education officials said that the RTT  monitoring process differs from the department’s other monitoring efforts  in that Education has more frequent contact with the states in order to  identify and address implementation challenges. In addition, the new  process emphasizes states’ continuous improvement and quality of RTT  reforms, rather than focusing solely on compliance with laws and  regulations and the ability of states to meet their time frames.said the intensity of communication with RTT states and the quality  standards are greater for RTT than for Education’s previous monitoring  efforts. Education developed the new process to provide assistance to  RTT states as they implement comprehensive reforms and to differentiate  support based on individual state needs.
Education officials said they work to identify and address obstacles to the  goals states established in their RTT plans through ongoing  communication, including monthly monitoring calls, the amendment  consideration process, and other contacts with RTT state officials. To  assess the quality of implementation efforts, officials said they consider  each state’s progress toward its goals and timelines, risk factors and  strategies for addressing them, and the state’s own assessment of its  quality of implementation, among other factors. For example, in addition  to verifying that a state implemented an evaluation tool, such as a test or  performance measure, Education officials work with the state to ensure  that the tool is meeting the state’s needs. Instead of focusing solely on  RTT compliance, program officers also help identify areas in which  Education can assist states in meeting their goals, according to Education  officials.
Officials from 8 of the 12 RTT states expressed generally positive views  about Education’s RTT monitoring activities. Some said, for example, that  Education officials were collaborative, well-informed, and that they  generally provided useful feedback. For example, officials from one state  said that Education staff were very detailed and thorough in monthly  monitoring calls and that they usually provided actionable feedback.  Officials from another state said they spoke almost daily with Education  officials and received strong support. They noted that, as a result,  monitoring reviews were not stressful, and they were not surprised by the  results. Officials from another state said they appreciated the discussion  with Education officials about the state’s amendment requests and how  Education worked with them to ensure that the state maintained its  original RTT goals.
While RTT state officials expressed generally positive views about the  monitoring process overall, officials in nine states expressed concerns  about specific aspects of the process, including delays in the amendment  process, time-consuming monthly calls and related requirements, and  slow feedback from Education after site visits. Officials from one state  said monitoring requirements seemed more burdensome than those for  other federal education programs. Education officials stated that they  have revised some aspects of their monitoring process in response to  state feedback. For example, they modified the monthly monitoring call  and onsite review protocols, revised the amendment process and dollar  threshold amounts that require approval, and worked to explain the  rationale and use of the information Education requests.
To ensure that states are held accountable for meeting their RTT goals  for teacher and principal evaluation systems, Education may take the  following corrective actions for states that have not demonstrated  adequate progress in implementing their systems:
Conditional amendment approval. If Education has concerns about  a state’s requested amendments to its RTT plans, it may grant  conditional approval, requiring the state to provide additional  information over a period of time. For example, Rhode Island  submitted a proposed amendment requesting a change related to its  use of SLOs. Education approved the request on the condition that  the state provide additional information, such as quarterly progress  updates during SY 2012-13 and additional reports. In addition,  Maryland received approval to decrease the percentage of the  evaluation component that is based on student academic growth  models on the condition that the state provide Education a plan for a  statewide field test of its evaluation systems. Maryland was also  required to commit to measuring student academic growth using  common assessments of high school teachers and principals when  those assessments are available, among other requirements.  Education may also elect not to approve an amendment request.
High-risk status. Education placed 2 of the 12 RTT states—Georgia  and Hawaii—on high-risk status because officials determined that the  states required intensive attention and support in order to meet their  RTT goals. In July 2012, Education placed the teacher and principal  evaluation portion of Georgia’s RTT grant on high-risk status because  officials were concerned about the overall strategic planning,  evaluation, and project management of the evaluation system.  Education officials also expressed concern that Georgia had  requested two major amendments that seemed to constitute  significant changes to the evaluation system in the state’s approved  plan. As a result of the high-risk designation, Georgia was required to  provide Education a revised work plan for its system, monthly updates  in accordance with the work plan, and related information. As of July  2013, Education officials said the evaluation system portion of  Georgia’s RTT grant remains on high-risk status because of  Education’s continued concerns about the quality of implementation.  In December 2011, Education designated Hawaii’s entire RTT grant  as high-risk because the state experienced major delays and made  inadequate progress on implementing its systems and because the  scope and breadth of amendment requests indicated a potentially  significant shift in the state’s approved plans. Education temporarily  placed Hawaii on a cost-reimbursement basis, which required the  state to submit receipts for expenditures to the department prior to  drawing down grant funds. The state was also required to submit  documentation prior to obligating funds and to submit a revised scope  of work and budget. As of July 2013, Education had removed Hawaii’s  high-risk designation based on the state’s demonstrated progress in  implementing its RTT reforms, including its evaluation systems.
Additional information. Education has required certain reporting or  follow-up information, other than that included in conditional approval  of amendments or high-risk status, and has used other measures  deemed appropriate. For example, according to Education officials,  one state that experienced procurement problems is required to  provide monthly procurement information to Education.
Withholding of funds. Although Education may withhold grant funds  from states if they do not comply with the terms of the award,Education officials said they have not withheld funds from any RTT  state. Officials added that states have always demonstrated progress  toward addressing Education’s concerns.
section_title: Education Provides Technical Assistance to Help States Resolve Implementation Issues
paragraphs: Education helps states meet their RTT goals and implement high-quality  reforms by providing technical assistance, including access to experts  and information on options for evaluation systems. Education officials  said technical assistance helps states resolve implementation issues,  including those identified through the monitoring process. Most RTT  federal assistance is provided by the contractor-supported RSN, and  Education also provides some technical assistance to RTT states.officials said they work closely with Education staff to learn about the   RSN  types of technical assistance that might be useful to states on teacher  and leader effectiveness, including teacher and principal evaluations, as  well as other RTT topics.
From 2010 through March 2013, RSN provided technical assistance on  teacher and leader effectiveness in group settings—such as webinars  and in-person meetings—to RTT states, as well as individualized  technical assistance (see fig. 5). RSN officials said they provide  individualized assistance to states when requested, particularly for states  in more advanced stages of implementation with needs that could not be  met through larger group technical assistance activities. RSN also  developed publications related to teacher and leader effectiveness,  including case studies, tool kits, and lessons learned, and has provided  them through the RTT grantee web portal. These publications included  longer reports on school reform and shorter briefs, such as a paper that  described rules governing classroom observations used in teacher  evaluations in selected RTT states.
RSN has worked to strengthen the quality of its technical assistance and  adapt to states’ changing needs, according to Education and RSN  officials. They said that early in the contract, RSN revised its approach to  better meet the needs of states. For example, in response to state  feedback, RSN provided states with access to education practitioners  who had worked in schools rather than experts without hands-on  experience, as they had done in the initial stages of the contract. RSN  officials also said issues for which states requested technical assistance  changed as implementation progressed, and RSN adapted its technical  assistance accordingly. For example, early in implementation, states  often requested assistance with designing evaluation systems,  communicating with stakeholders, working with unions, and measuring  growth in nontested grades and subjects, according to RSN officials. As  implementation progressed, states requested assistance with issues such  as the consistency of observations and the sustainability of evaluation  systems.
Officials from 10 of the 12 RTT states told us that Education’s technical  assistance related to teacher and principal evaluation systems was  generally helpful, and officials in several states said assistance had  improved since the start of the contract. Officials from Hawaii said RSN  had helped states by sharing existing knowledge and developing new  information. Officials in Massachusetts told us that RSN’s in-person  meetings had been especially helpful because they provided a platform  for states to share best practices. Although most states were  complimentary of RSN assistance, officials in some states said RSN and  Education could improve technical assistance by providing additional  information on specific topics, such as information about states that have  successfully implemented evaluation systems and more opportunities to  share lessons learned.
A recent survey by RSN also indicates that states are generally satisfied  with the contractor’s technical assistance on teacher and principal  evaluation systems. In March 2013, RSN surveyed and obtained  responses from officials in all RTT states regarding their perception of  technical assistance, including assistance provided through the teacher  and leader effectiveness community of practice. Sixteen of the 18 states  that participated in teacher and leader effectiveness assistance reported  that they were satisfied or very satisfied with the assistance, and the  remaining states were neutral. On multiple dimensions, the state officials  rated assistance with teacher and leader effectiveness higher than other  areas and higher than RSN activities overall. States also identified  opportunities for strengthening technical assistance by ranking potential  topics on the basis of impact and urgency. For teacher and leader  effectiveness, states ranked continuous improvement of teacher  evaluation as one of the top areas of interest, according to RSN. Officials  in a few states mentioned that they would like more opportunities to  collaborate and learn from one another.
section_title: Education Plans to Provide RTT States and Nongrantees Additional Information to Support State Efforts
paragraphs: Education plans to provide information to RTT states on sustaining  teacher and principal evaluation systems and other reforms, but  Education officials said they have not yet done so. Education planned to  launch a new workgroup in the summer of 2013 to help states consider  how to sustain their evaluation systems after the RTT grant ends. Draft  plans for the work group included providing expert and peer-to-peer  support and developing a sustainability rubric. The plans also included  providing workshops on sustainability efforts, including ones on state  capacity, performance management, and communication, and eventually  developing and sharing case studies. In July 2013, Education officials  said they had postponed work in this area until fall 2013. Although  Education has not asked states to provide specific plans for addressing  sustainability, department officials said they have learned about state  plans through ongoing communication with states. For example, a few  states discussed their sustainability planning during monthly monitoring  calls. In addition, Education obtained information on state sustainability  strategies through RTT applications. In general, states provided this  information on RTT reforms as a whole. We did not identify any states as  having provided a sustainability strategy specific to teacher and principal  evaluation systems.
To allow states more time to accomplish the goals and deliverables they  committed to in their RTT plans, Education officials will consider requests  for no-cost extensions, but they have not determined how to provide  technical assistance during the extension period. States may request  extensions on a case-by-case basis for more time to spend awarded  funds for those aspects of their RTT reforms that require additional work.  If approved, a state could have until September 30, 2015 to obligate and  liquidate its remaining RTT grant funds. According to Education officials,  states that request no-cost extensions will be required to provide to  Education their plans to address sustainability, among other information.  As of July 2013, Education officials had approved two no-cost extensions  related to teacher and principal evaluation systems, and officials in an  additional state told us they had submitted an extension request related to  their evaluation systems. Officials in 6 more of the 12 RTT states told us  they are considering requesting extensions related to their evaluation  systems. However, it is not clear what technical assistance would be  available to states approved for no-cost extensions. The current contract  for technical assistance ends in September 2014, and RSN officials said  they do not have plans to sustain technical assistance beyond the  duration of the current contract. Education officials said they were working  to identify options for providing continued technical assistance.
GAO-11-658. student academic growth data. In addition to the materials currently  available, Education officials told us that they were developing additional  resources and materials on topics such as SLOs, observation rubrics,  rating inflation, teacher engagement, data analytics, and leadership  development. Education identified some products for targeted  dissemination and presented them in conferences, cross-program  meetings, and to organizations such as the National Governors  Association, in order to promote awareness of the resources available.  Education is also working to develop a more robust dissemination plan  that includes ways to reach people other than state-level leaders,  according to officials.
section_title: Concluding Observations
paragraphs: Education created the RTT grant program to encourage sweeping  changes in K-12 education. RTT spurred changes to the way states and  districts evaluate their teachers and principals, particularly with the  addition of student academic growth data as a factor in assessing  effectiveness. Education has been proactive in monitoring states’  progress in implementing their evaluation systems, and the department’s  continued monitoring and assistance will be important to help RTT states  overcome challenges and implement the reforms to which they  committed. In addition, Education’s new monitoring process has resulted  in a wealth of information on states’ efforts. As a result, Education is  uniquely positioned to use the lessons learned from RTT states to inform  other states’ efforts to improve teacher effectiveness and ultimately raise  student academic achievement.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Education for  review and comment. Their comments are reproduced in appendix I.  Education also provided technical comments, which we incorporated into  the report.
We are sending copies of this report to the appropriate congressional  committees and the Secretary of Education. In addition, the report is  available at no charge on GAO’s website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix II.
section_title: Appendix I: Comments from the U.S. Department of Education
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Morrison, Assistant  Director, Nisha R. Hazra, Marissa Jones, and Michael Kniss made  significant contributions to this report. Also contributing to this report were  Deborah Bland, Sarah Cornetto, Jamila Jones Kennedy, Amy Moran- Lowe, Jean McSween, Mimi Nguyen, Jason Palmer, Kathleen van  Gelder, and Rebecca Woiwode.
section_title: Related GAO Products
paragraphs: School Improvement Grants: Education Should Take Additional Steps to  Enhance Accountability for Schools and Contractors. GAO-12-373.  Washington, D.C.: April 11, 2012.
Race to the Top: Characteristics of Grantees’ Amended Programs and  Education’s Review Process. GAO-12-228R. Washington, D.C.:  December 8, 2011.
Race to the Top: Reform Efforts are Under Way and Information Sharing  Could Be Improved. GAO-11-658. Washington, D.C.: June 30, 2011.
Department of Education: Improved Oversight and Controls Could Help  Education Better Respond to Evolving Priorities. GAO-11-194.  Washington, D.C.: February 10, 2011.
Grant Monitoring: Department of Education Could Improve Its Processes  with Greater Focus on Assessing Risks, Acquiring Financial Skills, and  Sharing Information. GAO-10-57. Washington, D.C.: November 19, 2009.
Student Achievement: Schools Use Multiple Strategies to Help Students  Meet Academic Standards, Especially Schools with Higher Proportions of  Low-Income and Minority Students. GAO-10-18. Washington, D.C.:  November 16, 2009.
No Child Left Behind Act: Enhancements in the Department of  Education’s Review Process Could Improve State Academic  Assessments. GAO-09-911. Washington, D.C.: September 24, 2009.
Teacher Quality: Sustained Coordination among Key Federal Education  Programs Could Enhance State Efforts to Improve Teacher Quality.  GAO-09-593. Washington, D.C.: July 6, 2009.
No Child Left Behind Act: Improvements Needed in Education’s Process  for Tracking States’ Implementation of Key Provisions. GAO-04-734.  Washington D.C.: September 30, 2004.
subsections: section_title: Recovery Act
paragraphs: Recovery Act: Opportunities to Improve Management and Strengthen  Accountability over States’ and Localities’ Uses of Funds. GAO-10-999.  Washington, D.C.: September 20, 2010.
Recovery Act: One Year Later, States’ and Localities’ Uses of Funds and  Opportunities to Strengthen Accountability. GAO-10-437. Washington,  D.C.: March 3, 2010.
Recovery Act: Status of States’ and Localities’ Use of Funds and Efforts  to Ensure Accountability. GAO-10-231. Washington, D.C.: December 10,  2009.