In Search of High-Quality Evaluation Feedback: An Administrator Training Field Experiment

Starting in 2011, Boston Public Schools (BPS) implemented major reforms to its teacher evaluation system with a focus on promoting teacher development. We administered independent district-wide surveys in 2014 and 2015 to capture BPS teachers’ perceptions of the evaluation feedback they receive. Teachers generally reported that evaluators were fair and accurate, but that they struggled to provide high-quality feedback. We conduct a randomized controlled trial to evaluate the district’s efforts to improve this feedback through an intensive training program for evaluators. We find little evidence the program affected evaluators’ feedback, teacher retention, or student achievement. Our results suggest that improving the quality of evaluation feedback may require more fundamental changes to the design and implementation of teacher evaluation systems. IN SEARCH OF HIGH-QUALITY EVALUATION FEEDBACK 2 In Search of High-Quality Evaluation Feedback: An Administrator Training Field Experiment Over the last decade, nearly every state in the U.S. has implemented major reforms to their teacher evaluation systems (Donaldson & Papay, 2015; Steinberg & Donaldson, 2016). A twofold theory of action motivated these reforms: differentiating teacher performance for accountability (Hanushek, 2009; Thomas, Wingert, Conant, & Register, 2010) and promoting professional development through classroom observations and feedback (Almy, 2011; Curtis & Wiener, 2012; Papay, 2012). Most states and districts have emphasized the latter goal of improving teachers’ instruction (Center on Great Teachers and Leaders, 2014). Several years into this national policy experiment, we still know little about the quality of feedback teachers receive as part of these new high-stakes evaluation systems. A growing body of evidence affirms the potential of frequent teacher feedback to drive instructional improvement and student achievement. Extensive feedback is a core organizational practice of both effective urban charter schools (Angrist, Pathak, & Walters, 2013; Dobbie & Fryer, 2013) and high-performing traditional public schools (Reinhorn, Johnson, & Simon, 2017). Randomized controlled trials demonstrate that teacher coaching programs centered around individualized, context-specific feedback have sizable effects on teachers’ classroom practices and students’ academic performance (Kraft, Blazar, & Hogan, 2018). Several studies have also found that observation and feedback cycles that leverage data from evaluation systems in low-stakes settings can increase student achievement (Garet et al., 2017; Papay, Taylor, Tyler, & Laski, 2016; Steinberg & Sartain, 2015). Although most state plans have emphasized using evaluation reforms to drive teacher development, many of these systems have been designed and implemented in ways that prioritize teacher accountability. Investments in new systems have skewed heavily towards efforts to IN SEARCH OF HIGH-QUALITY EVALUATION FEEDBACK 3 develop and use new evaluation measures to better differentiate teachers’ performance (Institute for Education Science, 2014). Most new evaluation systems require only two formal and two informal observations and do not consistently mandate post-observation conferences (Steinberg & Donaldson, 2016). The vast majority of districts have simply added the responsibility of conducting regular teacher observations to administrators’ existing tasks (Kraft & Gilmour, 2016; Neumersiki et al., 2018). Efforts to support administrators have focused on familiarizing them with new observation instruments and improving their reliability as evaluators, rather than on their ability to provide high-quality instructional feedback (Herlihy et al., 2014). In this paper, we examine teachers’ perceptions of the feedback they receive as part of Boston Public Schools’ (BPS) new teacher evaluation system and evaluate the district’s efforts to strengthen the quality of this feedback. In the 2011-12 academic year, BPS implemented major reforms to its evaluation system, with a focus on using the evaluation process as a tool for teacher development. The following year, BPS convened a group of experienced administrators to develop and pilot a multi-day evaluator training program intended to improve the quality of feedback administrators provided to teachers. We evaluate the implementation and effects of this intensive 15-hour evaluator training series by exploiting the staggered rollout of the program across two academic years. Working with BPS, we randomly assigned school-based evaluation teams to attend the training program in one of the four semesters across the 2013-14 and 2014-15 school years. This semester-specific randomization design allows us to test a range of treatment-control contrasts. We also administered an independent, district-wide survey to teachers to learn about their experiences with the evaluation process at the end of these two school years. Teachers’ responses provide a IN SEARCH OF HIGH-QUALITY EVALUATION FEEDBACK 4 range of outcomes for the randomized controlled trial (RCT) and allow us to extend prior research on teachers’ experiences with evaluation feedback. Analyzing the quality of teacher feedback generated as part of high-stakes evaluation systems, the correlates of high-quality feedback, and the potential to improve this feedback advances our understanding of a key mechanism in the theory of action for how evaluation reforms were intended to promote teacher development. We examine teachers’ perceptions of the feedback they receive because both pedagogical theory and prior empirical research suggest that teachers are unlikely to respond to feedback in constructive ways if they don’t believe it to be of high quality (Garubo & Rothstein, 1998; Feeney, 2007; Cherasaro et al., 2016). We find that BPS teachers generally thought that evaluators were fair and accurate raters, but viewed the quality of feedback they received less favorably. Ultimately, just over a quarter of teachers felt that their instruction improved because of this feedback. We next explore what teacher, evaluator, and school characteristics are correlated with feedback that teachers perceived to be of high quality. Holding constant teachers’ overall evaluation ratings, we find that less-experienced teachers are more likely to rate the feedback they receive as higher-quality, and that evaluators with longer tenures in their current schools are perceived to provide better feedback. We also find that teachers of color who are evaluated by administrators of the same race report receiving substantially higher-quality feedback than others. These results are consistent with prior research that document the positive benefits of teacher and student racial congruence for student outcomes (Dee, 2004, 2007; Egalite, Kisida, & Winters, 2015; Lindsay & Hart, 2017; Egalite & Kisida, 2018; Holt & Gershenson, 2015; Gershenson, Hart, Hyman, Lindsay, & Papageorge, 2018). 1 See Donaldson, 2012; Donaldson, 2016; Neumerski et al., 2018; Jiang, Sporte, & Luppescu 2015; Donaldson et al., 2014; Cherasaro, Brodersen, Reale, & Yanoski, 2016; Firestone, Nordin, Shcherbakov, Kirova, & Blitz, 2014. IN SEARCH OF HIGH-QUALITY EVALUATION FEEDBACK 5 The evaluator training program we evaluate was reasonably well attended by administrators and implemented successfully. Administrators reported high levels of satisfaction with the training and increases on a range of self-evaluated skills from baseline to end-of-course surveys. However, we find little evidence to suggest that the intensive training program was successful at improving the perceived quality of evaluation feedback or affected the frequency or duration of observation and feedback cycles in the short or medium-term. Together, our descriptive and causal evidence shed new light on the potential and limitations of promoting professional development through the teacher evaluation process. These findings can inform states’ and districts’ ongoing efforts to redesign their teacher evaluation systems under the increased flexibility provided by the Every Student Succeeds Act (ESSA).