## Teacher and Leader E **va luation Systems**  **Impact E valuation of** 

**OMB Clearance Request For Data Collection Instruments**  **Introduction and Part**  **A**

**U.S. Department of Education**  **Contract No. ED**

**Am erican Institutes for Research**  

## Impact E **valuation of Teacher and Leader E**  **va luation Systems** 

**OMB Clearance Request For Data Collection Instruments**  **, Part**  **A**

**October 1**  **9 , 2012**  **Prepared for: Department of Education**  **Contract No. ED -**  **IES -**  **11 -**  **C -**  **0066**

**Prepared by :**  **Am erican Institutes for Research** 

## Contents **Page**

Introduction

................................ ................................ ................................ ................................ ...... 1 Description of the Impact Evaluation of Teacher and L eader Evaluation Systems

....................... 2 Purpose

................................ ................................ ................................ ................................ ....... 2 Research Questions and Theory of Action

................................ ................................ ................ 2 Int ervention Selection and Characteristics

................................ ................................ ................. 5 Data Collection

................................ ................................ ................................ .......................... 8 Supporting Statement for Paperwork Reduction Act Submission

................................ ................. 13 Justification (Part A)

................................ ................................ ................................ ................ 13 Reference s ................................ ................................ ................................ ................................ ...... 2 3 

### Exhibits Exhibit 1. Simplified Theory of Action

................................ ................................ ........................... 4 Exhibit 2. Domains and Constructs for FFT and CLASS

................................ ................................ 7 Exhibit 3. Summary

of Data Collection Instruments and Schedule

................................ ................ 9 Exhibit 4 . Expert Reviewers

................................ ................................ ................................ .......... 16 Exhibit 5 . Hour Burden for Respondents

................................ ................................ ....................... 20 Exhibit 6 . Schedule for Dissemination of Study Results

................................ ............................... 21Impact Evaluation of Teacher and Leader Evaluation Systems

This request is the second of a two for the Impact Evaluation of Teacher and Leader Evaluation Sy stems purpose of this study

is to examine the implementation sy stem components that reflects current In May 2012 , the Office of Management and Budget ( (OMB 1850

- 0890) , which described the study, the design, and r ecruitment activities included contacting a sample of for the study and further contact s In this second request, the Institute of Education Sciences (IES) of the U.S. Department of Education requests clearance for the study ’s data collection instruments, specifically the survey , the principal survey , the protocol

. Information about o ther data collections (e.g., intervention fidelity data study is included in this request to provide a complete picture of the stud y. This request contains three major sections. 1. Description of the Impact Evaluation of Teacher and L eader Evaluation Systems

 Purpose

 Research questions and theory of action

 Treatment selection and

 Data collection 2. Supporting Statement for the Paperwork

 Justification (Part A)

 Description of s tatistical 3. Instruments for wh ich we are requesting clearance

 Teacher survey
 Principal survey
 District interview

 District archival records collection protocol

Impact Evaluation of Teacher and Leader Evaluation Systems

— 1 

## Introduction is the second of a two

- stage clearance request to carry out data collection activities for the Impact Evaluation of Teacher and Leader Evaluation Sy stems (TLES) Study purpose of this study

is to examine the implementation and impacts of a package of reflects current federal policy

. the Office of Management and Budget ( OMB ) approved the first clearance request , which described the study, the design, and the recruitment activities. ecruitment activities included contacting a sample of school districts to establish their eligibility s and visits with the eligi ble school districts

. In this second request, the Institute of Education Sciences (IES) of the U.S. Department of Education requests clearance for the study ’s data collection instruments, specifically the the district interview , and the district archival records collection ther data collections (e.g., intervention fidelity data to provide a complete picture of the stud y. contains three major sections. Description of the Impact Evaluation of Teacher and L eader Evaluation Systems and theory

of action and characteristics Supporting Statement for the Paperwork Reduction Act Submission

tatistical m ethods (Part B) ich we are requesting clearance

archival records collection protocol

stage clearance request to carry out data collection activities

Study . The package of evaluation approved the first clearance request recruitment activities. The districts to establish their eligibility In this second request, the Institute of Education Sciences (IES) of the U.S. Department of Education requests clearance for the study ’s data collection instruments, specifically the teacher district archival records collection ther data collections (e.g., intervention fidelity data ) related to this

Description of the Impact Evaluation of Teacher and L eader Evaluation SystemsImpact Evaluation of Teacher and Leader Evaluation Systems 

## Description of t **of Teacher and Leader E**  **va luation S**  **ys tem**  **Purpose** The TLES Study is designed to examine the implementation and impact evaluation sy stem components that practices , the package of sy stem measures, including student growth, to meaningfully differentiate teachers and principals

. It will draw leadership that provide timely feedback to guide the efforts of educators and those who supervise and support them. To measure impacts on student achievement and other outcomes, a purposive sampl districts with tradit ional evaluation systems school districts were selected on the basis of infrastructure , and interest in using achievement. A pool of eligible elementary and middle school district and randomly assigned to numbers of treatment and control schools in each study ’s teacher and leader evaluation s ys tem will be implemented during the 2012 Institutes for Research ( AIR ) . In the control schools, the district’s current evaluation system practices will continue to be used. collec ted to support analy ses of the implementation with a final collection of data for impact analys es in fall 2014. To ensure objectivity , the study will be carried out by

two independent teams. implementation team will design and implement the intervention at the sites recruited for the study , and the evaluation team will develop the study instruments and collect and analyz e data. 

### Research Questions **and** The study has six research question components provided b y the study and **RQ1.** How was the intervention implemented (e.g., district decisions about implementation, the cost of the intervention, and the participation of key actors) was it implemented ? **RQ2.** Did the intervention produce a contrast between the treatment and in tea chers’ and principals’ experiences how frequently

teachers reported being observed)

Impact Evaluation of Teacher and Leader Evaluation Systems

—2 

## Description of t **he Impact E**  **va luation**  **Teacher and Leader E va**  **luation S ys**  **tem**

is designed to examine the implementation and impact s of a package of evaluation sy stem components that reflects current federal policy . In contrast to traditional sy stem components provided through the study will use multiple, valid measures, including student growth, to meaningfully differentiate the performance levels draw on multiple assessments of teacher practice and principal leadership that provide timely feedback to guide the efforts of educators and those who supervise impacts on student achievement and other outcomes, a purposive sampl ional evaluation systems has been recruited to pilot the study ’s s ys tem selected on the basis of their current evaluation s ys tem practices, using educator pe rformance information to improve student elementary

and middle schools was identified in each district and randomly

assigned to either treatment or control conditions, with equal numbers of treatment and control schools in each school district. In the treatment schools, the study ’s teacher and leader evaluation s ys tem components were introduced in summer 2012 and implemented during the 2012

– 1 3 and 2013

– 14 school years, with support from

. In the control schools, the district’s current evaluation system practices will continue to be used. During the two

- ye ar implementation period, d the implementation and the impact of the pilot evaluation system, with a final collection of data for impact analys es in fall 2014. To ensure objectivity , the study will be carried out by

two independent teams. The team will design and implement the intervention at the sites recruited for the

will develop the study

instruments and collect and analyz e 

### and **Theory of Action** The study has six research question s designed to assess the implementation of the evaluation provided b y the study

and its impacts

. How was the intervention implemented (e.g., district decisions about implementation, cost of the intervention, the fidelity with which the intervention was delivered, participation of key

actors) , and in what context regarding district policies Did the intervention produce a contrast between the treatment and the chers’ and principals’ experiences with performance evaluation s ys tems (e.g., how frequently

teachers reported being observed) ? 

## Teacher and Leader E **va luation S**  **ys tem**  **s**
a package of . In contrast to traditional will use multiple, valid performance levels of on multiple assessments of teacher practice and principal leadership that provide timely feedback to guide the efforts of educators and those who supervise impacts on student achievement and other outcomes, a purposive sampl e of school to pilot the study ’s s ys tem . The se current evaluation s ys tem practices, data rformance information to improve student

identified in each selected control conditions, with equal district. In the treatment schools, the

summer 2012 and with support from American . In the control schools, the district’s current evaluation system

ye ar implementation period, d ata will b e impact of the pilot evaluation system,

The team will design and implement the intervention at the sites recruited for the

will develop the study

instruments and collect and analyz e the

s designed to assess the implementation of the evaluation How was the intervention implemented (e.g., district decisions about implementation, fidelity with which the intervention was delivered, , and in what context regarding district policies the control schools with performance evaluation s ys tems (e.g.,Impact Evaluation of Teacher and Leader Evaluation Systems

**RQ3.** What impacts did the decisions to try

new techniques, wo **RQ4.** What impacts did the and high

- value

- added teachers? **RQ5.** What impacts did the practice that are the focus of the intervention and on principal instructional leadership? **RQ6.** What impacts did the Because the study intervention is (RQ1) in reference to Year 1 and RQ3, RQ4, RQ5, and RQ6) separately at the end of data pertaining to the end of Y ear intervention. The theory

of action underlying the study ’s teacher depicted in simplified form in Exhibit 1. The implementation of the intervention in the schools is expected to cause a contrast between the treatment and educator s’ experience s of the evaluation feedback beyond what traditional systems provide, the evaluation influence the decisions of key actors. For example, teachers may decide to try

to acquire knowledge of new techniques, refine certain skills, or work differentl y. The evaluation system components may

also influence teacher career mobi teachers to exit their positions and encouraging high differential mobility

). These decisions of key

actors are expected to influence teachers’ instructional practice and princip achievement.

1 Because observations of instructional practice will be conducted only during instructional practice ( within RQ5 ) wi ll be examined Impact Evaluation of Teacher and Leader Evaluation Systems

— 3 What impacts did the intervention have on the decisions of key actors (e.g., decisions to try

new techniques, wo rk differentl y, or pursue learning experiences) What impacts did the intervention have on the mobility of low

- value

added teachers? What impacts did the intervention have on the dimensions of teacher instructional practice that are the focus of the intervention and on principal instructional leadership? What impacts did the intervention have on student achievement?

is two years in length , we will address the descriptive question and Y ear 2 separately . We will examine the impact questions (RQ2, RQ3, RQ4, RQ5, and RQ6) separately at the end of Year 1 and Y ear 2

. 1 Impact anal ys es using ear 2 will measure the cumulative impact of the two years of the The theory

of action underlying the study ’s teacher and leader evaluation sy stem depicted in simplified form in Exhibit 1. The implementation of the intervention in the schools is expected to cause a contrast between the treatment and the control schools in evaluation sy stems . By

clarifying expectations and providing feedback beyond what traditional systems provide, the evaluation sy stem components influence the decisions of key actors. For example, teachers may decide to try

to acquire knowledge of new techniques, refine certain skills, or work differentl y. The evaluation system may also influence teacher career mobi lity , potentiall y leading low value teachers to exit their positions and encouraging high

- value

- added teachers to stay (i.e., differential mobility

). These decisions of key

actors are expected to influence teachers’ instructional practice and princip al leadership, which in turn are hypothesized to improve student

Because observations of instructional practice will be conducted only during Y ear 2 of implementation, impact on wi ll be examined only at the end of Y ear 2

.

have on the decisions of key actors (e.g., teachers’ rk differentl y, or pursue learning experiences) ?

- added teachers have on the dimensions of teacher instructional practice that are the focus of the intervention and on principal instructional leadership?
, we will address the descriptive question

impact questions (RQ2, Impact anal ys es using two years of the evaluation sy stem components is depicted in simplified form in Exhibit 1. The implementation of the intervention in the treatment control schools in . By clarifying expectations and providing

components may influence the decisions of key actors. For example, teachers may decide to try

to acquire knowledge of new techniques, refine certain skills, or work differentl y. The evaluation system lity , potentiall y leading low value

- added added teachers to stay

(i.e., differential mobility

). These decisions of key

actors are expected to influence teachers’ al leadership, which in turn are hypothesized to improve student

of implementation, impact onImpact Evaluation of Teacher and Leader Evaluation Systems

**Evaluation S**  **ystem** Measures that differentiate performance levels
 Measure of student growth
 Measure of teacher practice/principal leadership District support for use of performance information **RQ1: Implementation**  **of the intervention** 

Impact Evaluation of Teacher and Leader Evaluation Systems

—4 **Exhibit 1. Simplified Theory of A** **ction**

**RQ3: Decisions**  **of key actors**  **RQ5: Teacher**  **practice/ principal**  **leadership RQ2:**  **Educator experience**  **of evaluation RQ4:**  **Differential mobility**  Mobility of low
value
- added teachers Mobility of high
value
- added teachers **Improved teacher/leader**  **knowledge, s**  **kills, and**  **effort RQ5:** 

**Teacher practice/**  **principal leadership**  **RQ6:**

**Student achievement** Impact Evaluation of Teacher and Leader Evaluation Systems



### Intervention **Selection and Characteristics** The study ’s intervention will consist of teacher evaluation components
. M easures of student growth the teacher evaluation component evaluation component will measure principal leadership. This package of evaluation sy stem repeatedly

measure performance, and produce actionable reports. In addition, the study districts are expected to support the use of performance information. these objectives in the following subsections **Performance Exp**  **ectations** To define performance expectations, our study treatment will use objective, transparent measures for teacher practice, principal leadership, and student growth. Teacher practice will be measured using observation protocols grounded in frameworks video libraries that illustrate superior performance. Principal leadership will be measured survey - based assessment focused on instructional leadership. Student growth will be measured using value

- added mo deling of achievement data in reading and mathematics Grades 4

– 8. **Repea ted**  **Measurement of**  **Performance** Repeated measurement provides a basis for both formative and summative uses of the performance information. The study’s evaluation system teachers’ instructional practice (two to four observations per teacher per year), p leadership (two assessments per principal per y ear), and student growth (annually, using value added methods). **Ac tionable**  **Performance Reports** In

- person feedback and online reports will provide performance reports tailored to the needs of the educators being evaluated (i.e., evaluees), their supervisors, and others designated b y the central office who are expected to use the performance information. For evaluees, performance reports will indicate the individual performance level in the context of expectations, which will guide their efforts to perform and encourage skill development. Each measurement instance will include opportunities to discuss the measurement results. Observers and assessors will conduct these discussions usin evaluee’s performance and the performance expectations Croft, 2011). For supervisors and others designated by the central office, additional reporting formats will be available to illuminate patterns

Impact Evaluation of Teacher and Leader Evaluation Systems

— 5 

### Selection and Characteristics will consist of teacher evaluation components and leader evaluation easures of student growth will be used for both teacher and leaders component will measure teachers’ instructional practice, and the leader will measure principal leadership. evaluation sy stem components is intended to define performance expectati repeatedly

measure performance, and produce actionable reports. In addition, the study districts are expected to support the use of performance information. W e elaborate on each of

in the following subsections

.
To define performance expectations, our study treatment will use objective, transparent measures for teacher practice, principal leadership, and student growth. Teacher practice will be measured using observation protocols grounded in frameworks for effective instruction and supported b y video libraries that illustrate superior performance. Principal leadership will be measured based assessment focused on instructional leadership. Student growth will be measured

deling of achievement data in reading and mathematics for teachers in
**Performance** Repeated measurement provides a basis for both formative and summative uses of the performance information. The study’s evaluation system components will repeatedly measure teachers’ instructional practice (two to four observations per teacher per year), p rincipal leadership (two assessments per principal per y ear), and student growth (annually, using value **Performance Reports** person feedback and online reports will provide performance reports tailored to the needs of ucators being evaluated (i.e., evaluees), their supervisors, and others designated b y the central office who are expected to use the performance information. For evaluees, performance reports will indicate the individual performance level in the context of clear performance guide their efforts to perform and encourage skill development. Each measurement instance will include opportunities to discuss the measurement results. Observers and assessors will conduct these discussions usin g protocols that focus discussion evaluee’s performance and the performance expectations (Danielson, 2010; Goe, Bigger Croft, 2011). For supervisors and others designated by the central office, additional reporting illuminate patterns in the ratings

. and leader evaluation

will be used for both teacher and leaders . In addition, will measure teachers’ instructional practice, and the leader

is intended to define performance expectati ons, repeatedly

measure performance, and produce actionable reports. In addition, the study school e elaborate on each of

To define performance expectations, our study treatment will use objective, transparent measures for teacher practice, principal leadership, and student growth. Teacher practice will be measured for effective instruction and supported b y video libraries that illustrate superior performance. Principal leadership will be measured with a based assessment focused on instructional leadership. Student growth will be measured for teachers in

Repeated measurement provides a basis for both formative and summative uses of the will repeatedly

measure rincipal leadership (two assessments per principal per y ear), and student growth (annually, using value

-

person feedback and online reports will provide performance reports tailored to the needs of ucators being evaluated (i.e., evaluees), their supervisors, and others designated b y the central office who are expected to use the performance information. For evaluees, performance clear performance guide their efforts to perform and encourage skill development. Each measurement instance will include opportunities to discuss the measurement results. Observers focus discussion on the (Danielson, 2010; Goe, Bigger s , & Croft, 2011). For supervisors and others designated by the central office, additional reportingImpact Evaluation of Teacher and Leader Evaluation Systems

**District Support**  **for Using**  **the** To promote the use of performance information that may improve student achievement, facilitate planning meetings in each evaluation sy stem components will be used in the treatment schools in their district, how this use will be supported. For example

 School districts may

choose to use the performance information to support teache efforts to engage in professional development opportunities that are tailored to their needs. To do so, a school district development offerings to assess which activities are aligned to the stud y’s obse frameworks and then facilitate teachers’ access to those offerings.

 School districts may

choose to performance evaluation system, as was done in the schools participating in the Excellence in Teachin g Program in Chicago Public Schools. I n that program, principals incorporated ratings from taking advantage of a policy that criteria with “local school requirements” at the beginning of the school y ear. In what follows , we describe the three main components of the planned performance measurement s ys tem: a component instructional practice, and a component **1. Component**  **for Feedback on Student Grow th**  Within the first intervention year and a will generate ev aluation sy stem reports for teachers, principals, and district leaders that summarize teacher and school performance as measured teachers will receive an online webinar on how to interpret these reports with existing school processes, principals will be encouraged to integrate these sessions into existing meetings focused on the interpretation and use of student achievement data. **2. Component**  **for Feedback on Instructional Practice** A number of frameworks for instructional practice have been developed for research purposes, including frameworks for specific content areas most suitable for use in teacher evaluation systems and grades and because of evidence of their validity and connection to student achievement: the Framework for Teaching (FFT) and the Classroom Assessment Scoring Sy stem (CLASS Bell, & Li ttle, 2008). The study districts were divided four school districts will receive feedback on instructional practice using FFT, and those in the treatment schools in the other 5 school one of the two frameworks was nonrandom; district preferences for one or the other framework were taken into account.
Impact Evaluation of Teacher and Leader Evaluation Systems

— 6 **the Performance Information**  use of performance information that may improve student achievement, each school district to facilitate districts’ decisions will be used in the treatment schools in their district, how this use will be supported. For example ,

may choose to use the performance information to support teache efforts to engage in professional development opportunities that are tailored to their needs.

district may , for example, review its menu of professional development offerings to assess which activities are aligned to the stud y’s obse facilitate teachers’ access to those offerings. may choose to use the performance information in the ir current performance evaluation system, as was done in the schools participating in the

g Program in Chicago Public Schools. I n that program, principals incorporated ratings from a pilot observation measure in teachers ’ official evaluations by taking advantage of a policy that allowed principals to supplement standard performance h “local school requirements” at the beginning of the school y ear. , we describe the three main components of the planned performance component for feedback on student growth, a component component for feedback on principal leadership. **for Feedback on Student Grow th**  the first intervention year and a t the end of each of the two intervention y ears, the study aluation sy stem reports for teachers, principals, and district leaders that

performance as measured by student growth

. Principals and teachers will receive an online webinar on how to interpret these reports . To ensure coherence with existing school processes, principals will be encouraged to integrate these sessions into existing meetings focused on the interpretation and use of student achievement data.

**for Feedback on Instructional Practice** number of frameworks for instructional practice have been developed for research purposes, including frameworks for specific content areas . Among these frameworks , two have emerged as most suitable for use in teacher evaluation systems because of their applicability across subjects evidence of their validity and connection to student achievement: the Framework for Teaching (FFT) and the Classroom Assessment Scoring Sy stem (CLASS into two groups , such that teachers in the treatment schools in districts will receive feedback on instructional practice using FFT, and those in the

school dis tricts will do so using CL ASS. Distric t assignment to nonrandom; district preferences for one or the other framework use of performance information that may improve student achievement, AI R will districts’ decisions about how the will be used in the treatment schools in their district, including
may choose to use the performance information to support teache rs’ efforts to engage in professional development opportunities that are tailored to their needs. , for example, review its menu of professional development offerings to assess which activities are aligned to the stud y’s obse rvation
current performance evaluation system, as was done in the schools participating in the

g Program in Chicago Public Schools. I n that program, principals official evaluations by standard performance h “local school requirements” at the beginning of the school y ear. , we describe the three main components of the planned performance component for feedback on

t the end of each of the two intervention y ears, the study aluation sy stem reports for teachers, principals, and district leaders that Principals and . To ensure coherence with existing school processes, principals will be encouraged to integrate these sessions into existing meetings focused on the interpretation and use of student achievement data.
number of frameworks for instructional practice have been developed for research purposes, , two have emerged as applicability

across subjects evidence of their validity and connection to student achievement: the Framework for Teaching (FFT) and the Classroom Assessment Scoring Sy stem (CLASS ; Goe, , such that teachers in the treatment schools in districts will receive feedback on instructional practice using FFT, and those in the t assignment to nonrandom; district preferences for one or the other frameworkImpact Evaluation of Teacher and Leader Evaluation Systems

With this design , the main impact findings, frameworks for instructional practice developments, rather than one particular despite their different origins and theoretical underpinnings (Goe shows , the frameworks have man y similar construct the ir rating scales are somewhat different: seven - level scale for CLASS.

**Exhibit 2**  **. Domains Framework for Teaching**  **Domain 2: Classroom E**  **nvironment**  Creating an Environment of Respect and Rapport
 Establishing a Culture for Learning
 Managing Classroom Procedures
 Managing Student Behavior
 Organizing Physical Space **Domain 3: Instruction**
 Communicating With Students
 Using Questioning and Discussion Techniques
 Engaging Students in Learning
 Using Assessment in Instruction  Demonstrating Flexibility and Responsiveness

Note. FFT includes two other domains that are not amenable to measurement through observation and are not included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4, called Professional Responsibilities. The implementation of FFT and CLASS parameters. For example, for teachers in reading/English language arts (ELA) observation and feedback, and both will use peer observers for three cycles and principals for fourth cycle. In addition, b oth frameworks illustrate superior performance. **3. Component**  **for Feedback on Principal Leadership** To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of Leadership in Education (VAL

- ED) in treatment schools in all study districts. VAL for principal evalua tion with established Clifford, 2010 ) . The researchers properties in peer

- reviewed journals and websites (www.valed.com/research continuously

improv ing the tool. leadership ( Goldring et al. , 2009 ).

Impact Evaluation of Teacher and Leader Evaluation Systems

— 7 main impact findings, pooled across all 9 districts, will pertain to frameworks for instructional practice having the general features called for in recent policy developments, rather than one particular approach . FFT and CLASS have many similarities different origins and theoretical underpinnings (Goe et al. , 2008). As

man y similar construct s and measurement approaches, although rating scales are somewhat different: Items are rated on a four

- level scale for FFT but a
**. Domains and Constructs for FFT and CLA**  **SS Framework for Teaching**  **Classroom Assessment Scoring System (CLASS**  **- U**  **pper E**  **lementary and CLASS**  Creating an Environment of Respect and Rapport

Using Questioning and Discussion Techniques

Flexibility and Responsiveness **Domain 1: Emotional Support**
 Positive Climate
 Negative Climate
 Teacher Sensitivity
 Regard for Adolescent Perspectives **Domain 2: Classroom Organization**
 Behavior Management  Productivity  Instructional Learning Formats **Domain 3: Instructional Support** 
 Content Understanding
 Analysis and Problem Solving
 Quality of Feedback
 Instructional Dialogue FFT includes two other domains that are not amenable to measurement through observation and are not included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4,
and CLASS in the study

districts will follow the same general for teachers in Grades 4

– 8 who are responsible for mathematics or (ELA) instruction , both frameworks will provide four rounds of observation and feedback, and both will use peer observers for three cycles and principals for frameworks will provide teachers with video libraries that **for Feedback on Principal Leadership** To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of ED) in treatment schools in all study districts. VAL with established validity and implementation fidelity (Condon & . The researchers who develop ed VAL

- ED have published its psychometric reviewed journals and websites (www.valed.com/research .html the tool. VAL

- ED also is aligned with national standards ). districts, will pertain to having the general features called for in recent policy many similarities 2008). As Exhibit 2 approaches, although for FFT but a

**Classroom Assessment Scoring System**

**and CLASS -**  **S econdary**  **)**

FFT includes two other domains that are not amenable to measurement through observation and are not included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4, will follow the same general 8 who are responsible for mathematics or

will provide four rounds of observation and feedback, and both will use peer observers for three cycles and principals for a video libraries that

To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of ED) in treatment schools in all study districts. VAL

- ED is a tool (Condon & psychometric .html ) and have been ED also is aligned with national standards for principalImpact Evaluation of Teacher and Leader Evaluation Systems

VAL

- ED focuses on leadership as it relates to teacher and student learning same broad purpose as the stu dy ’s on principal behaviors from principals, supervisors, and teachers (an approach degree assessment ” ) and provides both criterion principal s can understand their performance ratings relative to other principals evaluated using 

### Data Collection An overview of the study’s instruments and the schedule for their use is provided in Exhibit The bolded instruments involve burden and are therefore the basis for seeking clearance. The unbolded instruments are listed to provide a complete picture of th include district screening and recruitment data collection activities because clearance has alread y been received for these activities

Impact Evaluation of Teacher and Leader Evaluation Systems

— 8 ED focuses on leadership as it relates to teacher and student learning and thus shares the

dy ’s teacher and leader evaluation sy stem. VAL

- ED gathers data on principal behaviors from principals, supervisors, and teachers (an approach called

) and provides both criterion

- referenced and norm

- referenced scores. Thus,

performance ratings both in terms of specified criteria and evaluated using VAL

- ED in the United States.

An overview of the study’s instruments and the schedule for their use is provided in Exhibit The bolded instruments involve burden and are therefore the basis for seeking clearance. The unbolded instruments are listed to provide a complete picture of th e study

. T he exhibit include district screening and recruitment data collection activities because clearance has alread y for these activities

. and thus shares the ED gathers data called “ 360

- referenced scores. Thus, both in terms of specified criteria and

An overview of the study’s instruments and the schedule for their use is provided in Exhibit 3

. The bolded instruments involve burden and are therefore the basis for seeking clearance. The exhibit does not include district screening and recruitment data collection activities because clearance has alread yImpact Evaluation of Teacher and Leader Evaluation Systems **Exhibit 3. Summary of Data Collection Instruments and Schedule** 
**Treatment Control**  **Jul** 1. Classroom observations All Grade 4

– 8 teachers of

mathematics and/or reading/ELA (approx. 1 , 310 teachers) **2. Teacher**  **survey** All K

– 8 teachers of mathematics and/or reading/ELA (approx. 2, 940 teachers) **3. Principal**  **survey** 70 principals 70 principals **4. District**  **interviews** 9 districts 5. Intervention fidelity data 9 districts

**6. District**  **archival record data**  9 districts

7. Intervention cost record s 9 districts

Impact Evaluation of Teacher and Leader Evaluation Systems

— 9 **Summary of Data Collection Instruments and Schedule**

**2012 –**  **13 2013**  **– 14**  **Jul –**  **Sep Oct** **– Dec**  **Jan –**  **Mar Apr**  **– Jun**  **Jul –**  **Sep Oct** **– Dec**  **Jan –** 

End of year End of year End of year Multiple times throughout the year Student records and rosters of study schools Archival

employee and student records R osters

of study schools

Cost records **2014**

**– Mar**  **Apr**

**– Jun**  **Jul –**  **Sep** End of year End of year End of year End of year

osters of study schools Archival

employee and student records
Cost recordsImpact Evaluation of Teacher and Leader Evaluation Systems

**1. Classroom Observations**  Independent classroom observations will be conducted in addition to the classroom observations that are part of the evaluation system. During onl y Grade 4

– 8 teachers of mathematics and reading/ELA within recorded for the duration of one class session up to two times be used to conduct anal yse s of the impact of the evaluation system on tea To enable the anal ys es of impact on teacher practice to complement the analy ses of impact on student achievement, teachers in grade level s and subjects for which student impacts will be analyzed (i.e., mathematics and reading /ELA )

. Because teachers will be video recorded, be coded using both CLASS and FFT by CLASS and FFT

. ) **2. Teacher**  **S urveys**  T eacher survey s will be administered in spring 2013 and mathematics and reading/EL A within determine whether the intervention produced a contrast between the treatment and schools, particularl y with respect to number and the duration of classroo form of observation

- specific feedback related to instructional improvement and mobility, as well as their attitudes and beliefs about their ratings and their capacity to change practice in the desired ways will be used to address whether the evaluation s ys tem had an impact on principal leadership practices (RQ5)

. **3. Principal Survey** 

**s** P rincipal survey s will be adminis study school. The survey will be used to of calculating intervention costs (e.g., observing teachers, preparing written feedback, and providing individual feedback during meetings; RQ1). The survey will be used to whether the intervention produced a contrast between the treatment and particularl y with respect to principals’ experiences with performance evaluation (e.g., number of times they

received a rating of some typ e for their performance as a principal by a the current school year and how those evaluations were conducted produced different ty

pes of teacher performance information will focus on principals’ decisions related to improv providing vary

ing levels of support to teachers of differing performance levels) well as their attitudes and beliefs about their ratings and their capacit y to change practice in the desired way s (RQ 3)

.

Impact Evaluation of Teacher and Leader Evaluation Systems

— 10 **Classroom Observations** ndependent classroom observations will be conducted in addition to the classroom observations that are part of the evaluation system. During onl y Y ear 2 of the study i n spring 2014, 8 teachers of mathematics and reading/ELA within all 140 study schools for the duration of one class session up to two times . Data from these observations will

s of the impact of the evaluation system on tea cher practice To enable the anal ys es of impact on teacher practice to complement the analy ses of impact on the independent sample of observations will be and subjects for which student impacts will be analyzed (i.e., G rades 4

Because teachers will be video recorded, each observation and FFT . (See Exhibit 2 for the domains and constructs

will be administered in spring 2013 and spring 2014 to all K

– 8 teachers of within all the 140 study

schools . The survey will be used to determine whether the intervention produced a contrast between the treatment and

with respect to teachers’ experiences with performance evaluation (e.g., the duration of classroo m observations through which they received a rating or other feedback ; RQ2) . The survey will focus on teachers’ decisions improvement and mobility, as well as their attitudes and beliefs about ratings and their capacity to change practice in the desired ways (RQ3) . Finally , will be used to address whether the evaluation s ys tem had an impact on principal leadership

will be adminis tered i n spring 2013 and spring 2014 to the principal study school. The survey will be used to capture principal time which is an important component of calculating intervention costs (e.g., observing teachers, preparing written feedback, and providing individual feedback during meetings; RQ1). The survey will be used to vention produced a contrast between the treatment and the control schools,

principals’ experiences with performance evaluation (e.g., number of times they

received a rating of some typ e for their performance as a principal by a the current school year and how those evaluations were conducted ) and whether the intervention produced different ty

pes of teacher performance information for principals ( RQ2) will focus on principals’ decisions related to improv ement in instructional leadership providing vary

ing levels of support to teachers of differing performance levels) and mobility , as well as their attitudes and beliefs about their ratings and their capacit y to change practice in the ndependent classroom observations will be conducted in addition to the classroom observations n spring 2014, all schools will be video . Data from these observations will

cher practice (RQ5)

. To enable the anal ys es of impact on teacher practice to complement the analy ses of impact on from among the rades 4

– 8 in each observation can . (See Exhibit 2 for the domains and constructs measured

8 teachers of . The survey will be used to determine whether the intervention produced a contrast between the treatment and the control teachers’ experiences with performance evaluation (e.g., the m observations through which they received a rating or other . The survey

will focus on teachers’ decisions improvement and mobility, as well as their attitudes and beliefs about . Finally , the survey will be used to address whether the evaluation s ys tem had an impact on principal leadership

he principal in each capture principal time which is an important component of calculating intervention costs (e.g., observing teachers, preparing written feedback, and providing individual feedback during meetings; RQ1). The survey will be used to determine control schools, principals’ experiences with performance evaluation (e.g., number of times they

received a rating of some typ e for their performance as a principal by a supervisor in ) and whether the intervention RQ2) . The survey in instructional leadership (e.g.,

and mobility , as well as their attitudes and beliefs about their ratings and their capacit y to change practice in theImpact Evaluation of Teacher and Leader Evaluation Systems

**4. District**  **I nterview**  **s** To answer RQ1, i nterviews will be conducted in each school district who are responsible for teacher and leader evaluation sy stems. The interview s will be used to collect information regarding the extent to which followed through on their plans to implement the intervention, as well as barriers they encountered to implementing their plans and how they overcame those barriers. In be collected regarding the integration of the stud y’s evaluation s ys tem with existing district processes. The interview s will also be used to collect contextual information regarding the districts’ human resources policies in the control sch teacher and principal evaluation system used. **5. Intervention Fidelity Data**  V arious intervention fidelity records will be collected capture the fidelity

of delivery

and participation in key

intervention events through in visits and the monitoring of online webinars ( agenda s /schedule s )

. For each district about their credentials , prior experience ( Teachstone/Teachscape ) training materials and tools, we will capture performance information as well as administrative records. For each observation session and feedback session, the s ys tem will provide the session dates part icipant list. Additionally , in the course of completing logs t will record the focus of the feedback session and session

. O nline sy stem records from the VAL performance information as well as administrative records regarding the number of teachers and district staff who were asked to complete VAL rates, the dates when principals received their assessment feedback sessions occurred. Finall y, teachers and principa ls in the treatment **6. District**  **Ar chiv**  **al Record** Finally , various district archival records will be collected. A brief description of each follows. **Student Records for V**  **alue -**  **A dded**  demographic records will be collected added modeling for the evaluation system reports Student Growth” section ). These data also will be used study reports
. To complete the evaluation s ys tem data from the 2008

– 09 through the evaluation team will require data from **Teacher and Principal Records** request administrative data (e.g., e
Impact Evaluation of Teacher and Leader Evaluation Systems

— 11 nterviews will be conducted in spring 2013 and spring 2014 with the officials district who are responsible for teacher and leader evaluation sy stems. The will be used to collect information regarding the extent to which the school followed through on their plans to implement the intervention, as well as barriers they encountered to implementing their plans and how they overcame those barriers. In be collected regarding the integration of the stud y’s evaluation s ys tem with existing district will also be used to collect contextual information regarding the districts’ human resources policies in the control sch ools during the study, focusing on their teacher and principal evaluation system policies and the way s in which performance **Intervention Fidelity Data** arious intervention fidelity records will be collected to answer RQ1

. For instance, fidelity of delivery

and participation in key

intervention events through in monitoring of online webinars ( i.e. , collecting of attendance sheets

each district

- selected observer and pr incipal , we will obtain prior experience , and certification test results from the providers’ training materials and online sy stem. Through the providers’ tools, we will capture performance information as well as administrative records. For each observation session and feedback session, the s ys tem will provide the session dates in the course of completing logs t o track their work, the observers will record the focus of the feedback session and use of the video library during the feedback from the VAL

- ED provider (Discovery Education) performance information as well as administrative records regarding the number of teachers and district staff who were asked to complete VAL

- ED assessments, the VAL

- ED survey response rates, the dates when principals received their assessment results, and the dates when principal Finall y, AI R’s online sy stem will report value

- added scores for all ls in the treatment schools

. **Ar chiv**  **al Record D**  **ata Collection**  various district archival records will be collected. A brief description of each follows.

**dded M**  **odeling and**  **Impact Analyses .**  S tudent assessment and will be collected from districts . These data will be used to conduct value added modeling for the evaluation system reports noted above (see “Component for Feedback on

). These data also will be used to conduct impact analy ses To complete the evaluation s ys tem reports , the implementation team

the 201 3

– 1 4 school years ; to complete the study reports data from the 2011

– 12 through the 2013

– 14 school years. **Teacher and Principal Records for Survey Administration and Mobility Analyses**  (e.g., e

- mail addresses and grade/subject assignments) with the officials district who are responsible for teacher and leader evaluation sy stems. The the school districts followed through on their plans to implement the intervention, as well as barriers they encountered to implementing their plans and how they overcame those barriers. In formation will be collected regarding the integration of the stud y’s evaluation s ys tem with existing district will also be used to collect contextual information regarding the ools during the study, focusing on their

performance data are

For instance, w e plan to fidelity of delivery

and participation in key

intervention events through in

- person attendance sheets and

will obtain information providers’ rough the providers’ online tools, we will capture performance information as well as administrative records. For each observation session and feedback session, the s ys tem will provide the session dates and the o track their work, the observers use of the video library

during the feedback ED provider (Discovery

Education) will provide performance information as well as administrative records regarding the number of teachers and ED survey

response results, and the dates when principal

added scores for all

various district archival records will be collected. A brief description of each follows. assessment and conduct value

- “Component for Feedback on analy ses (RQ6) for the implementation team will request ; to complete the study

reports , the school years. **for Survey Administration and Mobility Analyses .**  We will

grade/subject assignments) for teacherImpact Evaluation of Teacher and Leader Evaluation Systems

survey administration in January (As noted earlier , the survey s will be used to answer RQ1, will request administrative data (e.g., school entry /exit dates all teachers and principals in the participating between the 2011

– 12 school year principals , we will request teacher placement information for an y teacher who worked in a participating school at any time during this fall 2013 ( for the Year 1 report ), 12 records (prior to randomization) 13 and 201 3

– 14 school years. In fall 2014 they correspond to the start of the 2014 **Performance Evaluation System** the performance ratings given to teachers under the local evaluation sy stem summer 2013, we will request local ratings correspond 2014, we will request local ratings correspond linkages to schools rather than teachers; of the teachers in the study

. These data will be used intervention on local ratings given that principals may be more willing to give low ratings in their local s ys tems
. **7. Intervention**  **Cost Record**  Within RQ1, one aspect of the intervention to be examined is its cost. AI R will use its internal project

- cost tracking s ys tem to compute the component for feedback on student growth, the component for feedback on instructional practice, and the component for feedback on principal leadership AIR assessment costs ( i.e ., to produce Teachscape, and Teachstone; e.g., to provide training and support to districts), and conduct observations of teacher instructional practice

2 These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptive information on mobility as context f or interpreting the study results. Impact Evaluation of Teacher and Leader Evaluation Systems

— 12 January 2013 and January 2014 for each school included in the study

. , the survey s will be used to answer RQ1, RQ 2, RQ 3, and RQ 5.)

(e.g., school entry

/exit dates and demographic characteristics) all teachers and principals in the participating school districts for anal ys es to track mobility

12 school year and the 2014

– 15 school year (RQ4)

. 2 For these teachers , we will request teacher placement information for an y teacher who worked in a participating school at any time during this time period. These requests wi ll be made twice. In ), we will request data corresponding to final, end 12 records (prior to randomization) and final rosters as they correspond to the start of the 2012 14 school years. In fall 2014 ( for the Year 2 report ), we will request final rosters as they correspond to the start of the 2014

– 15 school year. **Performance Evaluation System Ratings**  **.** Insummer 2013 and summer 2014, AI R will request ratings given to teachers under the local evaluation sy stem to answer RQ3 local ratings correspond ing to 2011

– 12 and 2012

– local ratings correspond ing to 2013

– 14. We will collect these rat linkages to schools rather than teachers; we will request for each school a list of the local ratings These data will be used to answer RQ3, to estimate the impact of the given that principals may be more willing to give low ratings in
**Record s**  Within RQ1, one aspect of the intervention to be examined is its cost. AI R will use its internal compute the costs of each component of the intervention (i.e., component for feedback on student growth, the component for feedback on instructional practice, and the component for feedback on principal leadership )

. Using this sy stem , we will produce value

- added reports ) , sy stem providers’ costs Teachstone; e.g., to provide training and support to districts), and conduct observations of teacher instructional practice and to p rovide feedback.

These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptive or interpreting the study results.

2014 for each school included in the study

. 5.) In addition, we demographic characteristics) for anal ys es to track mobility For these teachers and , we will request teacher placement information for an y teacher who worked in a

ll be made twice. In will request data corresponding to final, end

- of

- ye ar 2011

– final rosters as they

correspond to the start of the 2012

–

uest final rosters as

Insummer 2013 and summer 2014, AI R will request to answer RQ3

. In

– 13. In summer We will collect these rat ings with

of the local ratings estimate the impact of the given that principals may be more willing to give low ratings in

Within RQ1, one aspect of the intervention to be examined is its cost. AI R will use its internal costs of each component of the intervention (i.e., the component for feedback on student growth, the component for feedback on instructional practice, we will combine costs ( Discovery , Teachstone; e.g., to provide training and support to districts), and study costs to

These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptiveImpact Evaluation of Teacher and Leader Evaluation Systems



## Supporting Statement for Paperw **ork Reduction A ct** 
**A. Justification 1. Circumstances Making Collection of Information Necessary**  Improving student achievement is a core priorit y for federal education policy 34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth graders and 35 percent of eighth graders attained the profic Center for Education Statistics, 2010a). assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek, Peterson, & Woessmann, 2010). In an effort to improve student outcomes, federal education policy reauthorization of the Elementary and Secondary

Education Act (ESEA), the American Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program targets teacher quality

as a potential lever for improving student achievement. Several studies support this focus, with recent work examining individual teachers for several years to estimate the degree of variation in achievement gains that can be attrib measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above average in effectiveness at raising studen increase in student achievement. If such a teacher effect accumulates as students move from one grade to the next, it would translate into a substantial difference in achievement by the time students leave high school. Similar studies suggest that some variation in achievement gains can be attributed to principals, which indicates that principal quality may be another lever for education policy

makers (Hallinger & Heck, 1998; Leithwood, Leithwood & Jantzi, 2005; Leithwood, Louis, Anderson, & Wahlstrom, 2004; & McNulty, 2003). The federal government has invested considerable financial resources into supporting efforts to develop s ys tems to measure educator effectiveness and use that information in human resource policies. The Teacher Incentive Fund (T IF) was established in 2006 of approximately

$100 million to identify

and reward effective educators in high based in part on student growth. The goal of the program is to improve increasing the effectiveness of edu ARRA provided additional funding for TIF and established competitive grants to help states build their pool s of effective teachers and principals through the Race to the Top (RTT) program, in which a core priority area is the effectiveness of teachers and leaders. and districts are developing new teacher and leader evaluation sys tems, wi improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a continued emphasis on the development of teacher and leader eva Impact Evaluation of Teacher and Leader Evaluation Systems

— 13 

## Supporting Statement for Paperw **ork Reduction A ct** 

**Submission**
**1. Circumstances Making Collection of Information Necessary** Improving student achievement is a core priorit y for federal education policy . In 2011, onl y 34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth graders and 35 percent of eighth graders attained the profic ient level in mathematics (National Center for Education Statistics, 2010a). The performance of U.S. students on international assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek, Peterson, & Woessmann, 2010). effort to improve student outcomes, federal education policy

— as manifested in the 2002 reauthorization of the Elementary and Secondary

Education Act (ESEA), the American Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program targets teacher quality

as a potential lever for improving student achievement. Several studies support this focus, with recent work examining individual teachers for several years to estimate the degree of variation in achievement gains that can be attrib uted to teachers rather than measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above average in effectiveness at raising studen t achievement may lead to a 0.1 standard deviation increase in student achievement. If such a teacher effect accumulates as students move from one grade to the next, it would translate into a substantial difference in achievement by the time

high school. Similar studies suggest that some variation in achievement gains can be attributed to principals, which indicates that principal quality may be another lever for

policy makers (Hallinger & Heck, 1998; Leithwood, Harris, & Strauss, Leithwood & Jantzi, 2005; Leithwood, Louis, Anderson, & Wahlstrom, 2004; Waters, Marzano, The federal government has invested considerable financial resources into supporting efforts to develop s ys tems to measure educator effectiveness and use that information in human resource policies. The Teacher Incentive Fund (T IF) was established in 2006 with an initial funding base of approximately

$100 million to identify

and reward effective educators in high based in part on student growth. The goal of the program is to improve student achievement b y increasing the effectiveness of edu cators in high

- poverty schools. ARRA provided additional funding for TIF and established competitive grants to help states

of effective teachers and principals through the Race to the Top (RTT) program, in which a core priority area is the effectiveness of teachers and leaders. Using RTT grants, states and districts are developing new teacher and leader evaluation sys tems, wi th the goal of improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a continued emphasis on the development of teacher and leader eva luation systems. Through the 

## Supporting Statement for Paperw **ork Reduction A ct** 

. In 2011, onl y 34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth ient level in mathematics (National

on international assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek,
as manifested in the 2002 reauthorization of the Elementary and Secondary

Education Act (ESEA), the American Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program

— targets teacher quality

as a potential lever for improving student achievement. Several studies support this focus, with recent work examining individual teachers for several years to estimate uted to teachers rather than measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above t achievement may

lead to a 0.1 standard deviation increase in student achievement. If such a teacher effect accumulates as students move from one grade to the next, it would translate into a substantial difference in achievement by the time

high school. Similar studies suggest that some variation in achievement gains can be attributed to principals, which indicates that principal quality may be another lever for Harris, & Strauss, 20 10;

Waters, Marzano, The federal government has invested considerable financial resources into supporting efforts to develop s ys tems to measure educator effectiveness and use that information in human resource with an initial funding base of approximately

$100 million to identify

and reward effective educators in high

- poverty schools, student achievement b y ARRA provided additional funding for TIF and established competitive grants to help states

of effective teachers and principals through the Race to the Top (RTT) program, Using RTT grants, states th the goal of improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a luation systems. Through theImpact Evaluation of Teacher and Leader Evaluation Systems

Flexibility

program, ED relieves states from selected ESEA requirements in exchange for compliance with new requirements, including evaluation sy stems. Although states and dist ricts across the country have been developing and implementing new evaluation sy stems to identify and reward highly

effective educators, no large been carried out to confirm the efficacy of these new s ys tems. The TLES Study will address need. This study

is funded by TI F This is the second of a two

- stage clearance request to carry out data collection activities for the TLES Study

. In May

2012 , OMB approved the first clearance request described the study, the design, and contacting a sample of districts to establish their eligibility for the study and visiting eligible districts. I n this second request, the Institut of Education requests clearance for the study’s data collection instruments, specificall y the teacher survey , the principal survey , collection pr otocol. **2. Purposes and Uses of the Data** The purpose of the TLES Study is to test a package of teacher and leader evaluation sy stem components that is consistent with current federal policy. In contrast to traditional sy stems, the evaluation sy stem components provided b y the study will use multiple, valid measures, including measures of student growth, teacher practice, and principal leadership, to meaningfully differentiate the performance levels. addressing such evaluation systems implementation and impacts (e.g., Data collected b y the TLES Study will be of immediate researchers, and practitioners. **3. Use of Technology to Reduce Burden** The data collection plan is designed to obtain reliable information in an efficient way that minimizes respondent burden, and technology will be us collections for which we are seeking clearance. The teacher and principal surveys will be administered online using a allow respondents to complete the survey at a time and place that is conveni respondent. District interviews will be conducted personnel. This mode of data collection is appropriate for the conversational exchange necessary to obtain answers to the open

- ended qu administered survey

can provide.

Impact Evaluation of Teacher and Leader Evaluation Systems

— 14 Flexibility

program, ED relieves states from selected ESEA requirements in exchange for compliance with new requirements, including the implementation of reformed teacher and leader

ricts across the country

have been developing and implementing new evaluation sy stems to identify and reward highly

effective educators, no large

- scale study has yet been carried out to confirm the efficacy of these new s ys tems. The TLES Study will address TI F

. stage clearance request to carry out data collection activities for the , OMB approved the first clearance request (OMB 1850 described the study, the design, and the recruitment activities. The recruitment activities included contacting a sample of districts to establish their eligibility for the study and visiting eligible districts. I n this second request, the Institut e of Education Sciences (IES) of the U.S. Department of Education requests clearance for the study’s data collection instruments, specificall y the principal survey , the district interview, and the district archival records **2. Purposes and Uses of the Data** The purpose of the TLES Study is to test a package of teacher and leader evaluation sy stem components that is consistent with current federal policy. In contrast to traditional sy stems, the provided b y the study will use multiple, valid measures, including student growth, teacher practice, and principal leadership, to meaningfully performance levels. This study will be the first large

- scale randomized addressing such evaluation systems . Using an experimental design, the study will examine both and impacts (e.g., impacts on teacher mobility , impacts on student achievement Data collected b y the TLES Study will be of immediate interest and import for policy makers, **3. Use of Technology to Reduce Burden** The data collection plan is designed to obtain reliable information in an efficient way that minimizes respondent burden, and technology will be us ed to reduce burden for all the collections for which we are seeking clearance. The teacher and principal surveys will be administered online using a Web platform that will allow respondents to complete the survey at a time and place that is conveni ent for
District interviews will be conducted via tele phone to reduce the writing burden for district personnel. This mode of data collection is appropriate for the conversational exchange necessary ended qu estions and allow s probing for more detail than a self administered survey

can provide. Flexibility

program, ED relieves states from selected ESEA requirements in exchange for implementation of reformed teacher and leader ricts across the country

have been developing and implementing new scale study

has yet been carried out to confirm the efficacy of these new s ys tems. The TLES Study will address this stage clearance request to carry out data collection activities for the

(OMB 1850

- 0890) , which activities included contacting a sample of districts to establish their eligibility for the study and visiting eligible e of Education Sciences (IES) of the U.S. Department of Education requests clearance for the study’s data collection instruments, specificall y the district archival records

The purpose of the TLES Study is to test a package of teacher and leader evaluation sy stem components that is consistent with current federal policy. In contrast to traditional sy stems, the provided b y the study will use multiple, valid measures, including student growth, teacher practice, and principal leadership, to meaningfully scale randomized field trial study will examine both student achievement )

. interest and import for policy makers,

The data collection plan is designed to obtain reliable information in an efficient way that ed to reduce burden for all the data

platform that will ent for each writing burden for district personnel. This mode of data collection is appropriate for the conversational exchange necessary probing for more detail than a self

-Impact Evaluation of Teacher and Leader Evaluation Systems

For the archival data, we will reduce burden b y gathering the data electronically rather than in hard cop y. We will provide clear instructions on the data requested a the data securel y. A toll

- free number and an e - mail address will be available during the data collection process to permit respondents to contact AI R and the e - mail address will be included in all communication with respondents. **4. Efforts to Identify**

**Duplication** Throughout the evaluation, efforts will be made to minimize and reduce the burden on respondents. Wherever possible, we rely on secondary data sources to reduce burden on district and school personnel. By collecting mobility information), we will eliminate the need for several items on the teacher and principal survey s and thereb y avoid redundancy. In addition, we will make sure to coordinate efforts with other studies going on in the field. Curr ently , there is an IES impact evaluation of of performance

- based compensation on student achievement and teachers and principals. The national TIF evaluation measures the impact compensation specifically, while our purpose is to test making the focus of the two studies very different. **5. Methods to Minimize Burden on Small Entities** To be considered a small entity b y OMB, a school district would need to have a population of fewer than 50,000 students. Our first step in minimizing burden to small entities was some of the smallest school districts requirements with certain recruitment criteria. Specifically, our criteria required that districts have (1) at least 10 qualifying schools, (2) data s ys tems that support value (3) not announced a recent implement leader evaluation s ys tem to take effect before all districts, we seek to minimize burden to what is necessary

. partic ipate in the study d ata collections to provide the data necessary for the study’s anal ys es. Thus, we cannot reduce burden on small entities by changing t placing extra burden on small entities associated with travel, the implementa conducted separately

in each district. the most efficient processes are established for data collection to minimize burden on the part of district personnel. **6. Consequences of Not Collecting the Data**  States and school districts are increasingl y reshaping teacher and leader evaluation systems, in part due to the federal initiatives described above (see A.1).Without this study, states and districts will have a limited understanding o educators because there have been no rigorous, large
Impact Evaluation of Teacher and Leader Evaluation Systems

— 15 For the archival data, we will reduce burden b y gathering the data electronically rather than in hard cop y. We will provide clear instructions on the data requested a nd methods of transmitting mail address will be available during the data collection process to AI R with questions or requests for assistance. The toll address will be included in all communication with respondents. **4. Efforts to Identify**

**Duplication** Throughout the evaluation, efforts will be made to minimize and reduce the burden on respondents. Wherever possible, we rely on secondary data sources to reduce burden on district and school personnel. By collecting archival records on teachers and princip als (e.g., salary mobility information), we will eliminate the need for several items on the teacher and principal survey s and thereb y avoid redundancy. In addition, we will make sure to coordinate efforts with other studies going on in the field. ently , there is an IES impact evaluation of the TI F program that seeks to measure the impact

based compensation on student achievement and the mobility and retention of teachers and principals. The national TIF evaluation measures the impact of performance compensation specifically, while our purpose is to test a teacher and leader evaluation sy stem, making the focus of the two studies very different. **5. Methods to Minimize Burden on Small Entities** To be considered a small entity b y OMB, a school district would need to have a population of fewer than 50,000 students. Our first step in minimizing burden to small entities was some of the smallest school districts that might have been the mos t burdened by the study with certain recruitment criteria. Specifically, our criteria required that districts have (1) at least 10 qualifying schools, (2) data s ys tems that support value

- added modeling, and (3) not announced a recent implement ation (or planned implementation) of a new teacher or leader evaluation s ys tem to take effect before the 2014

– 15 school year . Second, in working with all districts, we seek to minimize burden to what is necessary

. All participating districts need to ipate in the study d ata collections to provide the data necessary for the study’s Thus, we cannot reduce burden on small entities by changing t he collections. placing extra burden on small entities associated with travel, the implementa tion events are conducted separately

in each district. The study team will work closel y with districts to ensure the most efficient processes are established for data collection to minimize burden on the part of **Collecting the Data** districts are increasingl y reshaping teacher and leader evaluation systems, in part due to the federal initiatives described above (see A.1).Without this study, states and districts will have a limited understanding o f how these sy stems affect their students and

there have been no rigorous, large

- scale studies of the impact of such s ys tems. For the archival data, we will reduce burden b y gathering the data electronically rather than in nd methods of transmitting mail address will be available during the data collection process to with questions or requests for assistance. The toll

- free number

Throughout the evaluation, efforts will be made to minimize and reduce the burden on respondents. Wherever possible, we rely on secondary data sources to reduce burden on district

als (e.g., salary and mobility information), we will eliminate the need for several items on the teacher and principal In addition, we will make sure to coordinate efforts with other studies going on in the field. seeks to measure the impact mobility and retention of of performance

- based evaluation sy stem,

To be considered a small entity b y OMB, a school district would need to have a population of fewer than 50,000 students. Our first step in minimizing burden to small entities was to exclude t burdened by

the study with certain recruitment criteria. Specifically, our criteria required that districts added modeling, and ation (or planned implementation) of a new teacher or Second, in working with All participating districts need to ipate in the study d ata collections to provide the data necessary for the study’s he collections. To avoid tion events are will work closel y with districts to ensure the most efficient processes are established for data collection to minimize burden on the part of

districts are increasingl y reshaping teacher and leader evaluation systems, in part due to the federal initiatives described above (see A.1).Without this study, states and f how these sy stems affect their students and scale studies of the impact of such s ys tems.Impact Evaluation of Teacher and Leader Evaluation Systems

**7. Special Circumstances** No special circumstances apply to this study

. **8. Federal Register**  **Comments and Persons Consulted Outside the A gency**  The 60

- day

n otice to solicit public comments was published in Volume 7 46748 of the Federal Register on To assist with the dev elopment of the study , project staff will draw on the experience and expertise of a network of outside experts who will serve as our technical working group members. The consultants and their affiliations are listed in Exhibit

**Proposed Expert Reviewer** Laura Goe Thomas Dee Daniel McCaffrey Catherine McClellan Jonah Rockoff Patrick Schuermann Carla Stevens John Tyler Judy Wurtzel To date, the project advisor and the TWG members have convened twice once via teleconference) to discuss consult outside experts individually on an as **9. Pay ment or Gifts**  In centives have been proposed for the teacher survey effort in completing the survey . We propose offering a $2 she completes a questionnaire to acknowledge the questionnaire. In centives are also proposed because high response rates are needed to make the survey findings reliable, and we are aware that teachers are the targets of numerous requests to complete surveys on a wide variety

of topics from state and district offices, i Although some districts will have solicited buy our recent experience with teacher surveys supports our view that monetary incentives needed to ensure adequate response rates.

Impact Evaluation of Teacher and Leader Evaluation Systems

— 16 No special circumstances apply to this study

. **Comments and Persons Consulted Outside the A gency**  day n otice to solicit public comments was published in Volume 7 7 , Number of the Federal Register on August 6, 2012

. No comments were received from the public

elopment of the study , project staff will draw on the experience and expertise of a network of outside experts who will serve as our technical working group members. The consultants and their affiliations are listed in Exhibit 4

. **Exhibit 4** 
**. Expert Reviewers**  **Proposed Expert Reviewer Professional Affiliation**  Educational Testing Service University of Virginia Rand Corporation Catherine McClellan Clowder Consulting Columbia University Patrick Schuermann Vanderbilt University Houston Independent School District Brown University Independent consultant To date, the project advisor and the TWG members have convened twice (once in person and to discuss the study

design and data collection. Project staff outside experts individually on an as - needed basis.

In centives have been proposed for the teacher survey to partially offset respondents’ time and effort in completing the survey . We propose offering a $2 5 incentive to a teacher each time he or she completes a questionnaire to acknowledge the 30 minutes required to complete each

In centives are also proposed because high response rates are needed to make the survey findings reliable, and we are aware that teachers are the targets of numerous requests to complete surveys on a wide variety

of topics from state and district offices, i ndependent researchers, and Although some districts will have solicited buy

- in from teachers to participate in the evaluation, our recent experience with teacher surveys supports our view that monetary incentives

ensure adequate response rates.

**Comments and Persons Consulted Outside the A gency** 

, Number 151 , page received from the public

. elopment of the study , project staff will draw on the experience and the expertise of a network of outside experts who will serve as our technical working group (TWG)

(once in person and study design and data collection. Project staff will also

to partially

offset respondents’ time and incentive to a teacher each time he or minutes required to complete each

In centives are also proposed because high response rates are needed to make the survey findings reliable, and we are aware that teachers are the targets of numerous requests to complete surveys ndependent researchers, and ED

. in from teachers to participate in the evaluation, our recent experience with teacher surveys supports our view that monetary incentives areImpact Evaluation of Teacher and Leader Evaluation Systems

N o incentives will be given to principals for for completing the district interview or **10. A ssurances of Confidentiality**  A consistent and cautious approach will be taken to protect all information collected during data collection. This approach will be in accordance with all relevant regulations and requirements. These include the Education Sciences Institute Reform Act of which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the Institute...to conform with the requirements of section 552 of Title 5, United States Code, the confidentiality

standards of subsect General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family

Education Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In a ddition, for student information, the project director will ensure that all individually

identifiable information about students, their academic achievements and families, and information with respect to individual schools shall remain confidential in acco section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and sections 444 and 445 of the General Educations Provision Act. Subsection (c) of Section 183 requires the designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting disclosure of individually identifiable information as inappropriate communication of individually identifiable information by employees or staff a felony

. Except with respect to the information provided to educators and their supervisors as part of the study ’s package of teac her and leader evaluation system components, AIR will use all information collected in the study for research purposes only . When reporting the study results, data will be presented in aggregate form identified. A statement to this effect will be included with all requests for study data. All members of the study team with access to the study data will be trained and certified on the importance of privacy

and data security . All study data will be kept identifiers will be destroyed as soon as they are no longer required. We will request unique identifiers reading/ELA teacher(s) associated with each student. To minimize identifiable information, we will request that districts send a separate crosswalk file that links teacher names to IDs. Both the implementation and security b y allowing access to shared folders only to team members who need it. We also will create a process whereb y the project director must approve all requests for access to these folders (and maintain a list of who has acces To enable us to conduct anal ys es that link individual value responses and classroom observation scores, the implementation team will attach scrambled

Impact Evaluation of Teacher and Leader Evaluation Systems

— 17 will be given to principals for completing the principal survey or to the district interview or the collection of district archival records

. **10. A ssurances of Confidentiality**  consistent and cautious approach will be taken to protect all information collected during data collection. This approach will be in accordance with all relevant regulations and requirements. These include the Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183, which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the Institute...to conform with the requirements of section 552 of Title 5, United States Code, the confidentiality

standards of subsect ions (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family

Education Rights and Privacy Act, and the Protection of Pupil Rights ddition, for student information, the project director will ensure that all individually

identifiable information about students, their academic achievements and families, and information with respect to individual schools shall remain confidential in acco section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and sections 444 and 445 of the General Educations Provision Act. requires the IES director to “develop and enforce standar designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a Except with respect to the information provided to educators and their supervisors as part of the

her and leader evaluation system components, AIR will use all information collected in the study for research purposes only . When reporting the study results, data will be presented in aggregate form onl y , such that individuals and institutions will not be identified. A statement to this effect will be included with all requests for study data. All members of the study team with access to the study data will be trained and certified on the importance of privacy

and data security . All study data will be kept in secured locations identifiers will be destroyed as soon as they are no longer required. We will request unique identifiers (IDs) for each student, as well IDs for the mathematics and reading/ELA teacher(s) associated with each student. To minimize access to personall y identifiable information, we will request that districts send a separate crosswalk file that links teacher names to IDs. Both the implementation and the evaluation teams will maintain data security b y allowing access to shared folders only to team members who need it. We also will create a process whereb y the project director must approve all requests for access to these folders (and maintain a list of who has acces s). To enable us to conduct anal ys es that link individual value

- added scores to both survey responses and classroom observation scores, the implementation team will attach scrambled or to district staff

.
consistent and cautious approach will be taken to protect all information collected during data collection. This approach will be in accordance with all relevant regulations and requirements. 2002, Title I, Part E, Section 183, which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the Institute...to conform with the requirements of section 552 of Title 5, United States Code, the ions (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family

Education Rights and Privacy Act, and the Protection of Pupil Rights ddition, for student information, the project director will ensure that all individually

identifiable information about students, their academic achievements and families, and information with respect to individual schools shall remain confidential in acco rdance with section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and director to “develop and enforce standar ds designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting the well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a Except with respect to the information provided to educators and their supervisors as part of the

her and leader evaluation system components, AIR will use all information collected in the study for research purposes only . When reporting the study results,

, such that individuals and institutions will not be identified. A statement to this effect will be included with all requests for study data. All members of the study team with access to the study data will be trained and certified on the in secured locations , and
for the mathematics and access to personall y identifiable information, we will request that districts send a separate crosswalk file that links teams will maintain data security b y allowing access to shared folders only to team members who need it. We also will create a process whereb y the project director must approve all requests for access to these folders added scores to both survey responses and classroom observation scores, the implementation team will attach scrambledImpact Evaluation of Teacher and Leader Evaluation Systems

teacher IDs to the survey and observation data before the data are made a evaluation team. In addition to the se safeguards, AI R routinel y employs assurances with respect to study data:

 All AIR employees sign a privacy pledge emphasizing its importance and describing their obligation.

 Id entifying information is maintained on separate forms and files, which are linked only by sample identification number.

 Access to hard

- cop y documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are

 Computer data files are protected with passwords

 Especiall y sensitive data are maintained on removable storage devices that are kept phy sicall y secure when not in use. **11. Justification of Sensitive Ques** No questions of a sensitive nature will be included in the teacher and principal survey s and the district interview. For the district archival records collection protocol, we will request performance ratings given to teachers under the local will request local ratings corresponding to 2011 request local ratings corresponding to 2013 schools rather than teachers; we w teachers in the study

. These data will be used to estimate the impact of the intervention on local ratings

. Whether or not the intervention has an impact on local ratings an important aspect of RQ3. In response to the intervention, give low ratings in their local systems. clustered in the highest performance category ( **12. Estimates of Hour Burden** The total estimated hour burden for the data collections for the TLES Study is Based on average hourly

wages for participants ye arl y salaries) , this amounts to an estimated monetary cost of $ the estimates of respondent burden for survey includes ti me for 85 percent of a sample of 2, districts to respond to a 3 0

- minute survey in the spring in each year of the study . The burden estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and control) in the 9 districts to respond to a 3 The burden estimate for the district interview includes contacts to respond to a 90

- minute interview in the spring in each y ear of the study . The archival records requests require an estimated 20 the requested data in each district. I n records request: student records from 2008 Impact Evaluation of Teacher and Leader Evaluation Systems

— 18 teacher IDs to the survey and observation data before the data are made a vailable to the
AI R routinel y employs the following to carry out privacy assurances with respect to study data: All AIR employees sign a privacy pledge emphasizing its importance and describing their Id entifying information is maintained on separate forms and files, which are linked only by sample identification number.

cop y documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded. Computer data files are protected with passwords , and access is limited to specific users. Especiall y sensitive data are maintained on removable storage devices that are kept phy sicall y secure when not in use. **11. Justification of Sensitive Ques tions**  No questions of a sensitive nature will be included in the teacher and principal survey s and the For the district archival records collection protocol, we will request performance ratings given to teachers under the local evaluation sy stem
. In summer 2013, we will request local ratings corresponding to 2011

– 12 and 2012

– 13. In summer 2014, we will request local ratings corresponding to 2013

– 14. We will collect these ratings with linkages to schools rather than teachers; we w ill request for each school a list of the local ratings of the

These data will be used to estimate the impact of the intervention on local the intervention has an impact on local ratings , and in what direction aspect of RQ3. In response to the intervention, principals may be more willing to give low ratings in their local systems. Currentl y, local rating s ys tems tend to produce ratings clustered in the highest performance category ( Weisberg, Se xton, Mulhern, & Keeling, 2009) **12. Estimates of Hour Burden** The total estimated hour burden for the data collections for the TLES Study is 4,265 Based on average hourly

wages for participants (which were calculated based on estimates of , this amounts to an estimated monetary cost of $ 1 24 , 2 85

. Exhibit the estimates of respondent burden for the study activities. The burden estimate for the teacher me for 85 percent of a sample of 2, 940 teachers (treatment and control) in the minute survey

in the spring in each year of the study . The burden estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and districts to respond to a 3 0

- minute survey in the spring in each ye ar o The burden estimate for the district interview includes time for 100 percent of all minute interview in the spring in each y ear of the study . The requests require an estimated 20 hours of burden for 1 district data person data in each district. I

n Year 1 of implementation, there will be four from 2008

– 09 through 2011

– 12, student records from 2012 vailable to the
to carry out privacy
All AIR employees sign a privacy pledge emphasizing its importance and describing their Id entifying information is maintained on separate forms and files, which are linked only

cop y documents is strictly limited. Documents are stored in locked files and access is limited to specific users. Especiall y sensitive data are maintained on removable storage devices that are kept

No questions of a sensitive nature will be included in the teacher and principal survey s and the For the district archival records collection protocol, we will request the In summer 2013, we 13. In summer 2014, we will We will collect these ratings with linkages to ill request for each school a list of the local ratings of the These data will be used to estimate the impact of the intervention on local , and in what direction , is be more willing to Currentl y, local rating s ys tems tend to produce ratings

xton, Mulhern, & Keeling, 2009)

.

4,265 hours. (which were calculated based on estimates of Exhibit 5 summarizes study activities. The burden estimate for the teacher teachers (treatment and control) in the 9 minute survey

in the spring in each year of the study . The burden estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and ye ar o f the study

. of all 9 district minute interview in the spring in each y ear of the study . The district district data person to pull four archival 12, student records from 2012

– 13 ,Impact Evaluation of Teacher and Leader Evaluation Systems

teacher and principal records for sampling purposes ratings . In Year 2 of implementation student records request s ) and will add an additional request in the fall of principal records for the purposes of tracking retention and mobility in our study

schools. Finally , in the fall of the 2014

– 15 school year, we will be collecting the final round of teacher and principal records for the purposes of tracki

Impact Evaluation of Teacher and Leader Evaluation Systems

— 19 principal records for sampling purposes , and educator performance evaluation 2 of implementation , we will have the same request as in Year 1 and will add an additional request in the fall of Year 2 for teacher and purposes of tracking retention and mobility in our study

schools. Finally , 15 school year, we will be collecting the final round of teacher and purposes of tracki ng retention and mobility in our study

schools. educator performance evaluation

1 (with one fewer 2 for teacher and purposes of tracking retention and mobility in our study

schools. Finally , 15 school year, we will be collecting the final round of teacher and

ng retention and mobility in our study

schools.Impact Evaluation of Teacher and Leader Evaluation Systems **Exhibit**
**Data Collection Total** 

**Sample Size**  **Estimated Response**  **Rate**

School Year 2012

– 13 Teacher survey,

spring 2, 940 85% Principal survey, spring 140 85% District interview, spring 9 100% District archival records collection 9 100% **Subtotal**

School Year 2013

– 14 Teacher survey,

spring 2, 940 85% Principal survey, spring 140 85% District interview, spring 9 100% District archival records collection 9 100% **Subtotal** School Year 2014

– 15 District archival

records collection 9 100% **Subtotal** Total across all school years

Impact Evaluation of Teacher and Leader Evaluation Systems

—20 **Exhibit 5**  **. Hour Burden for Respondents Estimated**  **Response Number of** 

**Respondents Number of** 

**Administrations Total** 

**Number of Responses**  **Time**

**Estimate (in hours)**  **Total**

**Hours** 2, 499 1 2, 499 0.5 1, 119 1 119 0.5 9 1 9 1.5 9 4 36 20 720 **2,043** 2, 499 1 2, 499 0.5 1, 119 1 119 0.5 9 1 9 1.5 9 4 36 20 720 **2,043** 9 1 9 20 180 **180** 4,265 **Total Hours**  **Hourly**

**Rate Estimated** 

**Monetary Cost**  **of Burden**  1, 250 $25 $ 31,238 60 $35 $2,083 14 $35 $473 720 $35 $ 25,200 **2,043 $**  **58,993** 1, 250 $25 $ 31,238 60 $35 $2,083 14 $35 $473 720 $35 $ 25,200 **2,043 $**  **58,993** 180 $35 $6,300 **180 $6,300**  4,265 $ 124,285Impact Evaluation of Teacher and Leader Evaluation Systems

**13. Estimate of Cost Burden to** There are no additional respondent costs associated with than the hour burden accounted for in item 12. **14. Estimate of A nnual Cost to the Federal Government**  The estimated cost for all aspects of the stud y is $16,783,022 over annual cost to the federal government Year 2 cost is $5,998,265; the Year 3 cost is $5,224,346; Year 5 cost is $127,071. **15. Program Changes or A djustments**  This is the addendum package to a clearance request that was approved in May, 2012, to conduct recruitment activities (see OMB 1840 described in this addendum package requires **16. Plans for Tabulation and Publication of Results** AIR will report the findings from the TLES Study to IES in two substantive reports. The timeline for the dissemination of the study results is summarized in Exhibit

**Exhibit 6**  **. Schedule for Dissemination of Study Results**

**Deliverable** First report Final report The first report will present the results of the analyses conducted on the data collected for the 2012

– 13 school year, including mobility as measured in fall 2013. This report will include a description of the study d esign and anal ys es. More specifically , the descriptive analyses will include the following:

● Descriptive anal ys es of the characteristics of the study sample

● Analy ses of the equivalence of the treatment and characteristics before intervention implementation

● Descriptive anal ys es of how about implementation, the intervention was delivered, context it was implemented In addition, the report will provide results on service contrast (RQ2) and the impacts of the intervention on educators and students. following during the first treatment y ear
 Intermediate outcomes: decisions of key actors (e.g., Impact Evaluation of Teacher and Leader Evaluation Systems

—21 **13. Estimate of Cost Burden to Respondents**  There are no additional respondent costs associated with the data collection s for this stud than the hour burden accounted for in item 12. **14. Estimate of A nnual Cost to the Federal Government**  The estimated cost for all aspects of the stud y is $16,783,022 over 5 ye ars, with an average annual cost to the federal government of $3,356,604. T he t otal Year 1 cost is $4,668,189; Year 3 cost is $5,224,346; the Year 4 cost is $765,151; and the **15. Program Changes or A djustments**  This is the addendum package to a clearance request that was approved in May, 2012, to conduct recruitment activities (see OMB 1840

- 0890). Completion of the data colle ction activities described in this addendum package requires additional burden totaling 3 , 37 4 hours. **16. Plans for Tabulation and Publication of Results** from the TLES Study

to IES in two substantive reports. The timeline for the dissemination of the study results is summarized in Exhibit 6

. **. Schedule for Dissemination of Study Results Deliverable**  **Anticipated Release Date** eport April 2015 eport July 2016 The first report will present the results of the analyses conducted on the data collected for the 13 school year, including mobility as measured in fall 2013. This report will include a description of the study d esign and the results from both descriptive analyses and impact anal ys es. More specifically , the descriptive analyses will include the following: Descriptive anal ys es of the characteristics of the study sample Analy ses of the equivalence of the treatment and the control groups in their background

before intervention implementation how the intervention was implemented (e.g., district decisions the cost of the intervention, the fidelity with which the vention was delivered, and the participation of key actors) and in what district policy context it was implemented (RQ1) In addition, the report will provide results on service contrast (RQ2) and the impacts of the intervention on educators and students. Specifically , the impact anal ys es will include the following during the first treatment y ear ( 2012

– 13 ) : Intermediate outcomes: decisions of key actors (e.g., teachers’ decisions to try new s for this stud y other

with an average otal Year 1 cost is $4,668,189; the cost is $765,151; and the

This is the addendum package to a clearance request that was approved in May, 2012, to conduct ction activities hours.
from the TLES Study

to IES in two substantive reports. The timeline

The first report will present the results of the analyses conducted on the data collected for the 13 school year, including mobility as measured in fall 2013. This report will include a results from both descriptive analyses and impact

trol groups in their background the intervention was implemented (e.g., district decisions fidelity with which the and in what district policy In addition, the report will provide results on service contrast (RQ2) and the impacts of the Specifically , the impact anal ys es will include the

teachers’ decisions to try

newImpact Evaluation of Teacher and Leader Evaluation Systems

techniques, work differently, or pursue learning experiences

 Outcomes: mobility

of teachers and principals (RQ4); teacher instructional practice and principal instructional leadership (RQ5); These analy ses will be carried out using hierarchical linear modeling where appropriate to take into account nesting (e.g., the nesting of students and teachers within schools) and will incorporate covariates measured at baseline to maximize precision. To bias, the impact anal ys es will employ an “intent teachers , and principals in all randomly assigned schools during the 2012 included in the anal ys es, whether or not the intended teacher and leader evaluation system or participated to the full extent expected. This approach is explained further in Part The final report will be a capstone report, focusing mainly on the cumulative impacts of providing the intervention for two y ears (i.e., 2012 those in the first report and will also Additionally , using AIR’s internal project each component of the intervention: the component for feedback on student growth, the component for feedback on instructional practice, leadership

. However, the AIR project principals; therefore, we will compute principal costs using items from the principal survey

. **17. A pproval to Not**  **Display OMB Expiration Date**  Approval is not being requested; all data collection instruments will include the OMB expiration date. **18. Explanation of Exceptions** No exceptions are requested.

Impact Evaluation of Teacher and Leader Evaluation Systems

—22 techniques, work differently, or pursue learning experiences ; RQ3 )

mobility of teachers and principals (RQ4); teacher instructional practice and instructional leadership (RQ5); and student achievement ( RQ6) These analy ses will be carried out using hierarchical linear modeling where appropriate to take into account nesting (e.g., the nesting of students and teachers within schools) and will incorporate covariates measured at baseline to maximize precision. To avoid potential selection bias, the impact anal ys es will employ an “intent

- to
- treat” approach, in which all and principals in all randomly assigned schools during the 2012

– 13 school year are included in the anal ys es, whether or not the teachers or principals actually participated in the intended teacher and leader evaluation system or participated to the full extent expected. This Part B

. The final report will be a capstone report, focusing mainly on the cumulative impacts of providing the intervention for two y ears (i.e., 2012

– 13 and 2013

– 14). The anal ys es will parallel also have impact results for teache r instructional practice. using AIR’s internal project

- cost tracking s ys tem, we will document each component of the intervention: the component for feedback on student growth, the component for feedback on instructional practice, and the component for feedback on principal

However, the AIR project

- cost tracking s ys tem will not capture hours spent by school principals; therefore, we will compute principal costs using items from the principal survey

. **Display OMB Expiration Date**  Approval is not being requested; all data collection instruments will include the OMB expiration **18. Explanation of Exceptions**

mobility of teachers and principals (RQ4); teacher instructional practice and RQ6) These analy ses will be carried out using hierarchical linear modeling where appropriate to take into account nesting (e.g., the nesting of students and teachers within schools) and will avoid potential selection treat” approach, in which all students,

13 school year are teachers or principals actually participated in the intended teacher and leader evaluation system or participated to the full extent expected. This The final report will be a capstone report, focusing mainly on the cumulative impacts of 14). The anal ys es will parallel r instructional practice. we will document the costs of each component of the intervention: the component for feedback on student growth, the and the component for feedback on principal cost tracking s ys tem will not capture hours spent by school principals; therefore, we will compute principal costs using items from the principal survey

.
Approval is not being requested; all data collection instruments will include the OMB expirationImpact Evaluation of Teacher and Leader Evaluation Systems

Aud, S., Hussar, W., Kena, G., Bianco, K., Frohlich, L ., Kemp, J., et al. (2011). education 2011 (NCES 2011 National Center for Education Statistics. Retrieved from http://nces.ed.gov/pro grams/coe/pdf/coe_msl.pdf Bill & Melinda Gates Foundation. (2011). measures of effective teaching project. http://www.metproject.org/downloads/Preliminary_Findings Condon, C., & Clifford, M. (2010). commonly used principal performance assessment instruments? Point Associates. Danielson, C. (2010). Evaluations that help teache Goe, L., Bell, C., & Little, O. (2008). Washington, DC: National Comprehensive Center for Teacher Qualit y. Goe, L., Biggers, K., & Croft, A. (2011, May ). development: Focusing on improving teaching and learning. Comprehensive Center for Teacher Quality Conference, Washington, DC. Goldring, E., Carvens, X., Murphy, J., Porter, A., Ellio of principals: What and how do states and urban districts assess leadership? School Journal, 110 (1), 19 Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school effec tiveness: 1980

– 1995. Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2010). perspective: How well does each state do at producing high Report No. 10

- 19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State and Lo cal Government, Program on Education Policy and Governance & Education Next. Retrieved from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG10 19_HanushekPetersonWoess Leithwood, K., Harris, A., & Strauss, T. (2010). leaders transform low

-

performing public schools. Leithwood, K., & Jantzi, D. (2005). A review of transformational school lead 1996

–2005. Leadership and Policy in Schools, 4

Impact Evaluation of Teacher and Leader Evaluation Systems

—23 

## References Aud, S., Hussar, W., Kena, G., Bianco, K., Frohlich, L ., Kemp, J., et al. (2011). The condition of

(NCES 2011

- 033). Washington, DC: U.S. Department of Education, National Center for Education Statistics. Retrieved from

grams/coe/pdf/coe_msl.pdf Bill & Melinda Gates Foundation. (2011). Learning about teaching: Initial findings from the measures of effective teaching project. Seattle, WA: Author. Retrieved from http://www.metproject.org/downloads/Preliminary_Findings

- Res earch_Paper.pdf Condon, C., & Clifford, M. (2010). Measuring principal performance: How rigorous are commonly used principal performance assessment instruments? Naperville, IL: Learning Danielson, C. (2010). Evaluations that help teache rs learn. Educational Leadership, 68 Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness. Washington, DC: National Comprehensive Center for Teacher Qualit y. Goe, L., Biggers, K., & Croft, A. (2011, May ). Linking teacher evaluation to professional development: Focusing on improving teaching and learning. Presentation at the National Comprehensive Center for Teacher Quality Conference, Washington, DC. Goldring, E., Carvens, X., Murphy, J., Porter, A., Ellio tt, S., & Carson, B. (2009). The evaluation of principals: What and how do states and urban districts assess leadership?

(1), 19

– 39. Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school 1995. School Effectiveness and School Improvement, 9 Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2010). U.S. math performance in global perspective: How well does each state do at producing high

- achieving students?

19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State and Lo cal Government, Program on Education Policy and Governance & Education Next. http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG10

- 19_HanushekPetersonWoess mann.pdf Leithwood, K., Harris, A., & Strauss, T. (2010). Leading school turnarounds: How successful performing public schools. San Francisco: Jossey

Leithwood, K., & Jantzi, D. (2005). A review of transformational school lead ership research

Leadership and Policy in Schools, 4 (3), 177

– 199. The condition of

033). Washington, DC: U.S. Department of Education,
Learning about teaching: Initial findings from the Seattle, WA: Author. Retrieved from earch_Paper.pdf Measuring principal performance: How rigorous are Naperville, IL: Learning Educational Leadership, 68 (4), 35

– 39. Approaches to evaluating teacher effectiveness. Linking teacher evaluation to professional

Presentation at the National Comprehensive Center for Teacher Quality Conference, Washington, DC. tt, S., & Carson, B. (2009). The evaluation of principals: What and how do states and urban districts assess leadership? Elementary Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school School Effectiveness and School Improvement, 9 (2), 157

– 191. U.S. math performance in global achieving students? (PEPG 19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State and Lo cal Government, Program on Education Policy and Governance & Education Next.
Leading school turnarounds: How successful

- Bass. ership researchImpact Evaluation of Teacher and Leader Evaluation Systems

Leithwood, K., Louis, K. S., Anderson, S., & Wahlstrom, K. (2004). student learning . St. Paul: University of Minnesota, Center for Applied Research and E ducational Improvement. National Center for Education Statistics. (2010a). (NCES 2012

–457). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. http://nces.ed.gov/nationsreportcard/pdf/main2011/2012457.pdf National Center for Education Statistics. (2010b). (NCES 2012

–458). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012458.pdf Schochet, P. Z., & Chiang, H. S. (2010) performance based on student test score gains Department of Education, I nstitute of Education Sciences, National Center for Education Evaluation and Regional Assistance. R http://eric.ed.gov/PDFS/ED511026.pdf U.S. Department of Education. (2010). Elementary and Secondary Education Act http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf Waters, T., Marzano, R. J., & McNulty, B. (2003). research tells us about the effect of leadership on student achie continent Research for Education and Learning. Retrieved from http://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade rship.pdf Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). failure to acknowledge and act on differences in teacher effectiveness. Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf

Impact Evaluation of Teacher and Leader Evaluation Systems

—24 Leithwood, K., Louis, K. S., Anderson, S., & Wahlstrom, K. (2004). How leadership influences

. St. Paul: University

of Minnesota, Center for Applied Research and ducational Improvement. National Center for Education Statistics. (2010a). The nation’s report card: Mathematics 2011

457). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012457.pdf National Center for Education Statistics. (2010b). The nation’s report card: Reading 20011

458). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012458.pdf Schochet, P. Z., & Chiang, H. S. (2010)

. Error rates in measuring teacher and school performance based on student test score gains (NCEE 2010

- 4004). Washington, DC: U.S. Department of Education, I nstitute of Education Sciences, National Center for Education Evaluation and Regional Assistance. R etrieved from http://eric.ed.gov/PDFS/ED511026.pdf U.S. Department of Education. (2010). A blueprint for reform: The reauthorization of the Elementary and Secondary Education Act . Washington, DC: Author. Retrieved from http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf Waters, T., Marzano, R. J., & McNulty, B. (2003). Balanced leadership: What 30 years of research tells us about the effect of leadership on student achie vement. Aurora, CO: Mid continent Research for Education and Learning. Retrieved from http://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. The New Teacher Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf How leadership influences . St. Paul: University

of Minnesota, Center for Applied Research and The nation’s report card: Mathematics 2011 457). Washington, DC: U.S. Department of Education, Institute of Retrieved from The nation’s report card: Reading 20011 458). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from Error rates in measuring teacher and school 4004). Washington, DC: U.S. Department of Education, I nstitute of Education Sciences, National Center for Education
A blueprint for reform: The reauthorization of the

. Washington, DC: Author. Retrieved from Balanced leadership: What 30 years of Aurora, CO: Mid

http://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade
Our national The New Teacher Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf