Teacher and Leader Evaluation Systems

Impact Evaluation of

OMB Clearance Request

For Data Collection Instruments

Introduction and Part A

U.S. Department of Education

Contract No. ED

American Institutes for Research

Impact Evaluation of

Teacher and Leader Evaluation Systems

OMB Clearance Request

For Data Collection Instruments, 

Part A

October 19, 2012

Prepared for:

Department of Education

Contract No. ED-IES-11-C-0066

Prepared by:

American Institutes for Research
Contents

Page

Introduction......................................................................................................................................1

Description of the Impact Evaluation of Teacher and Leader Evaluation Systems .......................2

Purpose.......................................................................................................................................2

Research Questions and Theory of Action ................................................................................2

Intervention Selection and Characteristics.................................................................................5

Data Collection ..........................................................................................................................8

Supporting Statement for Paperwork Reduction Act Submission.................................................13

Justification (Part A)................................................................................................................13

References......................................................................................................................................23

Exhibits

Exhibit 1. Simplified Theory of Action ...........................................................................................4

Exhibit 2. Domains and Constructs for FFT and CLASS................................................................7

Exhibit 3. Summary of Data Collection Instruments and Schedule ................................................9

Exhibit 4. Expert Reviewers..........................................................................................................16

Exhibit 5. Hour Burden for Respondents.......................................................................................20

Exhibit 6. Schedule for Dissemination of Study Results...............................................................21
Impact Evaluation of Teacher and Leader Evaluation Systems

This request is the second of a two

for the Impact Evaluation of Teacher and Leader Evaluation Systems 

purpose of this study is to examine the implementation 

system components that reflects current

In May 2012, the Office of Management and Budget (

(OMB 1850-0890), which described the study, the design, and 

recruitment activities included contacting a sample of 

for the study and further contacts

In this second request, the Institute of Education Sciences (IES) of the U.S. Department of 

Education requests clearance for the study’s data collection instruments, specifically the 

survey, the principal survey, the 

protocol. Information about other data collections (e.g., intervention fidelity data

study is included in this request to provide a complete picture of the study.

This request contains three major sections. 

1. Description of the Impact Evaluation of Teacher and Leader Evaluation Systems

 Purpose 

 Research questions and theory of action

 Treatment selection and 

 Data collection

2. Supporting Statement for the Paperwork 

 Justification (Part A)

 Description of statistical 

3. Instruments for which we are requesting clearance

 Teacher survey

 Principal survey

 District interview

 District archival records collection protocol

Impact Evaluation of Teacher and Leader Evaluation Systems—1

Introduction

is the second of a two-stage clearance request to carry out data collection activities 

for the Impact Evaluation of Teacher and Leader Evaluation Systems (TLES) Study

purpose of this study is to examine the implementation and impacts of a package of

reflects current federal policy.

the Office of Management and Budget (OMB) approved the first clearance request

, which described the study, the design, and the recruitment activities. 

ecruitment activities included contacting a sample of school districts to establish their eligibility

s and visits with the eligible school districts.

In this second request, the Institute of Education Sciences (IES) of the U.S. Department of 

Education requests clearance for the study’s data collection instruments, specifically the 

the district interview, and the district archival records collection 

ther data collections (e.g., intervention fidelity data

to provide a complete picture of the study.

contains three major sections. 

Description of the Impact Evaluation of Teacher and Leader Evaluation Systems

and theory of action

and characteristics

Supporting Statement for the Paperwork Reduction Act Submission

tatistical methods (Part B)

ich we are requesting clearance

archival records collection protocol

stage clearance request to carry out data collection activities 

Study. The 

package of evaluation 

approved the first clearance request

recruitment activities. The 

districts to establish their eligibility

In this second request, the Institute of Education Sciences (IES) of the U.S. Department of 

Education requests clearance for the study’s data collection instruments, specifically the teacher 

district archival records collection 

ther data collections (e.g., intervention fidelity data) related to this 

Description of the Impact Evaluation of Teacher and Leader Evaluation Systems
Impact Evaluation of Teacher and Leader Evaluation Systems

Description of t

of Teacher and Leader Evaluation System

Purpose

The TLES Study is designed to examine the implementation and impact

evaluation system components that 

practices, the package of system 

measures, including student growth, to meaningfully differentiate 

teachers and principals. It will draw

leadership that provide timely feedback to guide the efforts of educators and those who supervise 

and support them. 

To measure impacts on student achievement and other outcomes, a purposive sampl

districts with traditional evaluation systems 

school districts were selected on the basis of 

infrastructure, and interest in using

achievement. A pool of eligible elementary and middle 

school district and randomly assigned to 

numbers of treatment and control schools in each 

study’s teacher and leader evaluation system 

will be implemented during the 2012

Institutes for Research (AIR). In the control schools, the district’s current evaluation system 

practices will continue to be used.

collected to support analyses of the implementation

with a final collection of data for impact analyses in fall 2014.

To ensure objectivity, the study will be carried out by two independent teams. 

implementation team will design and implement the intervention at the sites recruited for the 

study, and the evaluation team will develop the study instruments and collect and analyze 

data. 

Research Questions and 

The study has six research question

components provided by the study and 

RQ1. How was the intervention implemented (e.g., district decisions about implementation, 

the cost of the intervention, 

and the participation of key actors)

was it implemented?

RQ2. Did the intervention produce a contrast between the treatment and 

in teachers’ and principals’ experiences 

how frequently teachers reported being observed)

Impact Evaluation of Teacher and Leader Evaluation Systems—2

Description of the Impact Evaluation 

Teacher and Leader Evaluation System

is designed to examine the implementation and impacts of a package of 

evaluation system components that reflects current federal policy. In contrast to traditional 

system components provided through the study will use multiple, valid 

measures, including student growth, to meaningfully differentiate the performance levels

draw on multiple assessments of teacher practice and principal 

leadership that provide timely feedback to guide the efforts of educators and those who supervise 

impacts on student achievement and other outcomes, a purposive sampl

ional evaluation systems has been recruited to pilot the study’s system

selected on the basis of their current evaluation system practices, 

using educator performance information to improve student 

elementary and middle schools was identified in each 

district and randomly assigned to either treatment or control conditions, with equal 

numbers of treatment and control schools in each school district. In the treatment schools, the 

study’s teacher and leader evaluation system components were introduced in summer 2012 and 

implemented during the 2012–13 and 2013–14 school years, with support from 

. In the control schools, the district’s current evaluation system 

practices will continue to be used. During the two-year implementation period, d

the implementation and the impact of the pilot evaluation system, 

with a final collection of data for impact analyses in fall 2014.

To ensure objectivity, the study will be carried out by two independent teams. The 

team will design and implement the intervention at the sites recruited for the 

will develop the study instruments and collect and analyze 

and Theory of Action

The study has six research questions designed to assess the implementation of the evaluation 

provided by the study and its impacts.

How was the intervention implemented (e.g., district decisions about implementation, 

cost of the intervention, the fidelity with which the intervention was delivered, 

participation of key actors), and in what context regarding district policies 

Did the intervention produce a contrast between the treatment and the 

chers’ and principals’ experiences with performance evaluation systems (e.g., 

how frequently teachers reported being observed)?

Teacher and Leader Evaluation Systems

a package of 

. In contrast to traditional 

will use multiple, valid 

performance levels of 

on multiple assessments of teacher practice and principal 

leadership that provide timely feedback to guide the efforts of educators and those who supervise 

impacts on student achievement and other outcomes, a purposive sample of school 

to pilot the study’s system. These

current evaluation system practices, data 

rformance information to improve student 

identified in each selected 

control conditions, with equal 

district. In the treatment schools, the 

summer 2012 and 

with support from American 

. In the control schools, the district’s current evaluation system 

year implementation period, data will be 

impact of the pilot evaluation system, 

The 

team will design and implement the intervention at the sites recruited for the 

will develop the study instruments and collect and analyze the 

s designed to assess the implementation of the evaluation 

How was the intervention implemented (e.g., district decisions about implementation, 

fidelity with which the intervention was delivered, 

, and in what context regarding district policies 

the control schools 

with performance evaluation systems (e.g., 
Impact Evaluation of Teacher and Leader Evaluation Systems

RQ3. What impacts did the 

decisions to try new techniques, wo

RQ4. What impacts did the 

and high-value-added teachers?

RQ5. What impacts did the 

practice that are the focus of the intervention and on principal instructional leadership?

RQ6. What impacts did the 

Because the study intervention is

(RQ1) in reference to Year 1 and 

RQ3, RQ4, RQ5, and RQ6) separately at the end of 

data pertaining to the end of Year 

intervention.

The theory of action underlying the study’s teacher 

depicted in simplified form in Exhibit 1. The implementation of the intervention in the

schools is expected to cause a contrast between the treatment and 

educators’ experiences of the evaluation

feedback beyond what traditional systems provide, the evaluation 

influence the decisions of key actors. For example, teachers may decide to try to acquire 

knowledge of new techniques, refine certain skills, or work differently. The evaluation system 

components may also influence teacher career mobi

teachers to exit their positions and encouraging high

differential mobility). These decisions of key actors are expected to influence teachers’ 

instructional practice and princip

achievement. 

 

1 Because observations of instructional practice will be conducted only during 

instructional practice (within RQ5) will be examined

Impact Evaluation of Teacher and Leader Evaluation Systems—3

What impacts did the intervention have on the decisions of key actors (e.g.,

decisions to try new techniques, work differently, or pursue learning experiences)

What impacts did the intervention have on the mobility of low-valueadded teachers?

What impacts did the intervention have on the dimensions of teacher instructional 

practice that are the focus of the intervention and on principal instructional leadership?

What impacts did the intervention have on student achievement?

is two years in length, we will address the descriptive question 

and Year 2 separately. We will examine the impact questions (RQ2, 

RQ3, RQ4, RQ5, and RQ6) separately at the end of Year 1 and Year 2.

1

Impact analyses using 

ear 2 will measure the cumulative impact of the two years of the 

The theory of action underlying the study’s teacher and leader evaluation system 

depicted in simplified form in Exhibit 1. The implementation of the intervention in the

schools is expected to cause a contrast between the treatment and the control schools in 

evaluation systems. By clarifying expectations and providing 

feedback beyond what traditional systems provide, the evaluation system components 

influence the decisions of key actors. For example, teachers may decide to try to acquire 

knowledge of new techniques, refine certain skills, or work differently. The evaluation system 

may also influence teacher career mobility, potentially leading low value

teachers to exit their positions and encouraging high-value-added teachers to stay (i.e., 

differential mobility). These decisions of key actors are expected to influence teachers’ 

instructional practice and principal leadership, which in turn are hypothesized to improve student 

Because observations of instructional practice will be conducted only during Year 2 of implementation, impact on 

will be examined only at the end of Year 2.

have on the decisions of key actors (e.g., teachers’ 

rk differently, or pursue learning experiences)?

-added teachers 

have on the dimensions of teacher instructional 

practice that are the focus of the intervention and on principal instructional leadership?

, we will address the descriptive question 

impact questions (RQ2, 

Impact analyses using 

two years of the 

evaluation system components is 

depicted in simplified form in Exhibit 1. The implementation of the intervention in the treatment 

control schools in 

. By clarifying expectations and providing 

components may

influence the decisions of key actors. For example, teachers may decide to try to acquire 

knowledge of new techniques, refine certain skills, or work differently. The evaluation system 

lity, potentially leading low value-added 

added teachers to stay (i.e., 

differential mobility). These decisions of key actors are expected to influence teachers’ 

al leadership, which in turn are hypothesized to improve student 

of implementation, impact on 
Impact Evaluation of Teacher and Leader Evaluation Systems

Evaluation System

Measures that differentiate 

performance levels

 Measure of student growth

 Measure of teacher 

practice/principal 

leadership

District support for use of 

performance information

RQ1: 

Implementation 

of the 

intervention

Impact Evaluation of Teacher and Leader Evaluation Systems—4

Exhibit 1. Simplified Theory of Action

RQ3: 

Decisions 

of key 

actors

RQ5: 

Teacher 

practice/

principal 

leadership

RQ2: 

Educator 

experience 

of evaluation

RQ4: Differential 

mobility

Mobility of lowvalue-added 

teachers

Mobility of highvalue-added 

teachers

Improved 

teacher/leader 

knowledge, 

skills, and effort

RQ5: 

Teacher 

practice/

principal 

leadership

RQ6: 

Student 

achievement
Impact Evaluation of Teacher and Leader Evaluation Systems

Intervention Selection and Characteristics

The study’s intervention will consist of teacher evaluation 

components. Measures of student growth

the teacher evaluation component

evaluation component will measure principal leadership. 

This package of evaluation system 

repeatedly measure performance, and produce actionable reports. In addition, the study

districts are expected to support the use of performance information. 

these objectives in the following subsections

Performance Expectations

To define performance expectations, our study treatment will use objective, transparent measures 

for teacher practice, principal leadership, and student growth. Teacher practice will be measured 

using observation protocols grounded in frameworks

video libraries that illustrate superior performance. Principal leadership will be measured 

survey-based assessment focused on instructional leadership. Student growth will be measured 

using value-added modeling of achievement data in reading and mathematics

Grades 4–8.

Repeated Measurement of Performance

Repeated measurement provides a basis for both formative and summative uses of the 

performance information. The study’s evaluation system 

teachers’ instructional practice (two to four observations per teacher per year), p

leadership (two assessments per principal per year), and student growth (annually, using value

added methods). 

Actionable Performance Reports

In-person feedback and online reports will provide performance reports tailored to the needs of 

the educators being evaluated (i.e., evaluees), their supervisors, and others designated by the 

central office who are expected to use the performance information. For evaluees, performance 

reports will indicate the individual performance level in the context of

expectations, which will guide their efforts to perform and encourage skill development. Each 

measurement instance will include opportunities to discuss the measurement results. Observers 

and assessors will conduct these discussions usin

evaluee’s performance and the performance expectations

Croft, 2011). For supervisors and others designated by the central office, additional reporting 

formats will be available to illuminate patterns

Impact Evaluation of Teacher and Leader Evaluation Systems—5

Selection and Characteristics

will consist of teacher evaluation components and leader evaluation 

easures of student growth will be used for both teacher and leaders

component will measure teachers’ instructional practice, and the leader 

will measure principal leadership. 

evaluation system components is intended to define performance expectati

repeatedly measure performance, and produce actionable reports. In addition, the study

districts are expected to support the use of performance information. We elaborate on each of 

in the following subsections.

To define performance expectations, our study treatment will use objective, transparent measures 

for teacher practice, principal leadership, and student growth. Teacher practice will be measured 

using observation protocols grounded in frameworks for effective instruction and supported by 

video libraries that illustrate superior performance. Principal leadership will be measured 

based assessment focused on instructional leadership. Student growth will be measured 

deling of achievement data in reading and mathematics for teachers in 

Performance

Repeated measurement provides a basis for both formative and summative uses of the 

performance information. The study’s evaluation system components will repeatedly measure 

teachers’ instructional practice (two to four observations per teacher per year), principal 

leadership (two assessments per principal per year), and student growth (annually, using value

Performance Reports

person feedback and online reports will provide performance reports tailored to the needs of 

ucators being evaluated (i.e., evaluees), their supervisors, and others designated by the 

central office who are expected to use the performance information. For evaluees, performance 

reports will indicate the individual performance level in the context of clear performance 

guide their efforts to perform and encourage skill development. Each 

measurement instance will include opportunities to discuss the measurement results. Observers 

and assessors will conduct these discussions using protocols that focus discussion

evaluee’s performance and the performance expectations (Danielson, 2010; Goe, Bigger

Croft, 2011). For supervisors and others designated by the central office, additional reporting 

illuminate patterns in the ratings.

and leader evaluation 

will be used for both teacher and leaders. In addition, 

will measure teachers’ instructional practice, and the leader 

is intended to define performance expectations, 

repeatedly measure performance, and produce actionable reports. In addition, the study school 

e elaborate on each of 

To define performance expectations, our study treatment will use objective, transparent measures 

for teacher practice, principal leadership, and student growth. Teacher practice will be measured 

for effective instruction and supported by 

video libraries that illustrate superior performance. Principal leadership will be measured with a 

based assessment focused on instructional leadership. Student growth will be measured 

for teachers in 

Repeated measurement provides a basis for both formative and summative uses of the 

will repeatedly measure 

rincipal 

leadership (two assessments per principal per year), and student growth (annually, using value -

person feedback and online reports will provide performance reports tailored to the needs of 

ucators being evaluated (i.e., evaluees), their supervisors, and others designated by the 

central office who are expected to use the performance information. For evaluees, performance 

clear performance 

guide their efforts to perform and encourage skill development. Each 

measurement instance will include opportunities to discuss the measurement results. Observers 

focus discussion on the 

(Danielson, 2010; Goe, Biggers, & 

Croft, 2011). For supervisors and others designated by the central office, additional reporting 
Impact Evaluation of Teacher and Leader Evaluation Systems

District Support for Using the 

To promote the use of performance information that may improve student achievement, 

facilitate planning meetings in each 

evaluation system components will be used in the treatment schools in their district, 

how this use will be supported. For example

 School districts may choose to use the performance information to support teache

efforts to engage in professional development opportunities that are tailored to their needs. 

To do so, a school district 

development offerings to assess which activities are aligned to the study’s obse

frameworks and then facilitate teachers’ access to those offerings.

 School districts may choose to 

performance evaluation system, as was done in the schools participating in the 

Excellence in Teaching Program in Chicago Public Schools. In that program, principals 

incorporated ratings from 

taking advantage of a policy that 

criteria with “local school requirements” at the beginning of the school year.

In what follows, we describe the three main components of the planned performance 

measurement system: a component 

instructional practice, and a component 

1. Component for Feedback on Student Growth

Within the first intervention year and a

will generate evaluation system reports for teachers, principals, and district leaders that 

summarize teacher and school performance as measured 

teachers will receive an online webinar on how to interpret these reports

with existing school processes, principals will be encouraged to integrate these sessions into 

existing meetings focused on the interpretation and use of student achievement data. 

2. Component for Feedback on Instructional Practice

A number of frameworks for instructional practice have been developed for research purposes, 

including frameworks for specific content areas

most suitable for use in teacher evaluation systems 

and grades and because of evidence of their validity and connection to student achievement: the 

Framework for Teaching (FFT) and the Classroom Assessment Scoring System (CLASS

Bell, & Little, 2008).

The study districts were divided

four school districts will receive feedback on instructional practice using FFT, and those in the 

treatment schools in the other 5 school 

one of the two frameworks was nonrandom; district preferences for one or the other framework 

were taken into account. 

Impact Evaluation of Teacher and Leader Evaluation Systems—6

the Performance Information

use of performance information that may improve student achievement, 

each school district to facilitate districts’ decisions

will be used in the treatment schools in their district, 

how this use will be supported. For example,

may choose to use the performance information to support teache

efforts to engage in professional development opportunities that are tailored to their needs. 

district may, for example, review its menu of professional 

development offerings to assess which activities are aligned to the study’s obse

facilitate teachers’ access to those offerings.

may choose to use the performance information in their current 

performance evaluation system, as was done in the schools participating in the 

g Program in Chicago Public Schools. In that program, principals 

incorporated ratings from a pilot observation measure in teachers’ official evaluations by

taking advantage of a policy that allowed principals to supplement standard performance 

h “local school requirements” at the beginning of the school year.

, we describe the three main components of the planned performance 

component for feedback on student growth, a component 

component for feedback on principal leadership. 

for Feedback on Student Growth

the first intervention year and at the end of each of the two intervention years, the study 

aluation system reports for teachers, principals, and district leaders that 

performance as measured by student growth. Principals and 

teachers will receive an online webinar on how to interpret these reports. To ensure coherence 

with existing school processes, principals will be encouraged to integrate these sessions into 

existing meetings focused on the interpretation and use of student achievement data. 

for Feedback on Instructional Practice

number of frameworks for instructional practice have been developed for research purposes, 

including frameworks for specific content areas. Among these frameworks, two have emerged as 

most suitable for use in teacher evaluation systems because of their applicability across subjects 

evidence of their validity and connection to student achievement: the 

Framework for Teaching (FFT) and the Classroom Assessment Scoring System (CLASS

into two groups, such that teachers in the treatment schools in

districts will receive feedback on instructional practice using FFT, and those in the 

school districts will do so using CLASS. District assignment to 

nonrandom; district preferences for one or the other framework 

use of performance information that may improve student achievement, AIR will 

districts’ decisions about how the 

will be used in the treatment schools in their district, including 

may choose to use the performance information to support teachers’ 

efforts to engage in professional development opportunities that are tailored to their needs. 

, for example, review its menu of professional 

development offerings to assess which activities are aligned to the study’s obse rvation 

current 

performance evaluation system, as was done in the schools participating in the 

g Program in Chicago Public Schools. In that program, principals 

official evaluations by

standard performance 

h “local school requirements” at the beginning of the school year.

, we describe the three main components of the planned performance 

component for feedback on 

t the end of each of the two intervention years, the study 

aluation system reports for teachers, principals, and district leaders that 

Principals and 

. To ensure coherence 

with existing school processes, principals will be encouraged to integrate these sessions into 

existing meetings focused on the interpretation and use of student achievement data. 

number of frameworks for instructional practice have been developed for research purposes, 

, two have emerged as 

applicability across subjects 

evidence of their validity and connection to student achievement: the 

Framework for Teaching (FFT) and the Classroom Assessment Scoring System (CLASS; Goe, 

, such that teachers in the treatment schools in

districts will receive feedback on instructional practice using FFT, and those in the 

t assignment to 

nonrandom; district preferences for one or the other framework 
Impact Evaluation of Teacher and Leader Evaluation Systems

With this design, the main impact findings, 

frameworks for instructional practice 

developments, rather than one particular 

despite their different origins and theoretical underpinnings (Goe

shows, the frameworks have many similar construct

their rating scales are somewhat different: 

seven-level scale for CLASS.

Exhibit 2. Domains 

Framework for Teaching

Domain 2: Classroom Environment

 Creating an Environment of Respect and Rapport 

 Establishing a Culture for Learning

 Managing Classroom Procedures

 Managing Student Behavior

 Organizing Physical Space

Domain 3: Instruction

 Communicating With Students

 Using Questioning and Discussion Techniques

 Engaging Students in Learning

 Using Assessment in Instruction

 Demonstrating Flexibility and Responsiveness

Note. FFT includes two other domains that are not amenable to measurement through observation and are not 

included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4, 

called Professional Responsibilities.

The implementation of FFT and CLASS 

parameters. For example, for teachers in 

reading/English language arts (ELA)

observation and feedback, and both will use peer observers for three cycles and principals for 

fourth cycle. In addition, both frameworks

illustrate superior performance. 

3. Component for Feedback on Principal Leadership

To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of 

Leadership in Education (VAL-ED) in treatment schools in all study districts. VAL

for principal evaluation with established

Clifford, 2010). The researchers 

properties in peer-reviewed journals and websites (www.valed.com/research

continuously improving the tool. 

leadership (Goldring et al., 2009). 

Impact Evaluation of Teacher and Leader Evaluation Systems—7

main impact findings, pooled across all 9 districts, will pertain to 

frameworks for instructional practice having the general features called for in recent policy

developments, rather than one particular approach. FFT and CLASS have many similarities 

different origins and theoretical underpinnings (Goe et al., 2008). As 

many similar constructs and measurement approaches, although 

rating scales are somewhat different: Items are rated on a four-level scale for FFT but a 

. Domains and Constructs for FFT and CLASS

Framework for Teaching Classroom Assessment Scoring System

(CLASS-Upper Elementary and CLASS

Creating an Environment of Respect and Rapport 

Using Questioning and Discussion Techniques

Flexibility and Responsiveness

Domain 1: Emotional Support

 Positive Climate

 Negative Climate

 Teacher Sensitivity

 Regard for Adolescent Perspectives

Domain 2: Classroom Organization

 Behavior Management

 Productivity

 Instructional Learning Formats

Domain 3: Instructional Support

 Content Understanding

 Analysis and Problem Solving

 Quality of Feedback

 Instructional Dialogue

FFT includes two other domains that are not amenable to measurement through observation and are not 

included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4, 

and CLASS in the study districts will follow the same general 

for teachers in Grades 4–8 who are responsible for mathematics or 

(ELA) instruction, both frameworks will provide four rounds of 

observation and feedback, and both will use peer observers for three cycles and principals for 

frameworks will provide teachers with video libraries that 

for Feedback on Principal Leadership

To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of 

ED) in treatment schools in all study districts. VAL

with established validity and implementation fidelity (Condon & 

. The researchers who developed VAL-ED have published its psychometric 

reviewed journals and websites (www.valed.com/research.html

the tool. VAL-ED also is aligned with national standards 

). 

districts, will pertain to 

having the general features called for in recent policy

many similarities 

2008). As Exhibit 2

approaches, although 

for FFT but a 

Classroom Assessment Scoring System

and CLASS-Secondary)

FFT includes two other domains that are not amenable to measurement through observation and are not 

included in the study’s measure of teacher practice: Domain 1, called Planning and Preparation, and Domain 4, 

will follow the same general 

8 who are responsible for mathematics or 

will provide four rounds of 

observation and feedback, and both will use peer observers for three cycles and principals for a

video libraries that 

To define expectations for principal leadership, we will deploy the Vanderbilt Assessment of 

ED) in treatment schools in all study districts. VAL-ED is a tool 

(Condon & 

psychometric 

.html) and have been 

ED also is aligned with national standards for principal 
Impact Evaluation of Teacher and Leader Evaluation Systems

VAL-ED focuses on leadership as it relates to teacher and student learning 

same broad purpose as the study’s 

on principal behaviors from principals, supervisors, and teachers (an approach 

degree assessment”) and provides both criterion

principals can understand their performance ratings 

relative to other principals evaluated using 

Data Collection

An overview of the study’s instruments and the schedule for their use is provided in Exhibit 

The bolded instruments involve burden and are therefore the basis for seeking clearance. The 

unbolded instruments are listed to provide a complete picture of th

include district screening and recruitment data collection activities because clearance has already 

been received for these activities

Impact Evaluation of Teacher and Leader Evaluation Systems—8

ED focuses on leadership as it relates to teacher and student learning and thus shares the 

dy’s teacher and leader evaluation system. VAL-ED gathers data 

on principal behaviors from principals, supervisors, and teachers (an approach called 

) and provides both criterion-referenced and norm-referenced scores. Thus, 

performance ratings both in terms of specified criteria and

evaluated using VAL-ED in the United States.

An overview of the study’s instruments and the schedule for their use is provided in Exhibit 

The bolded instruments involve burden and are therefore the basis for seeking clearance. The 

unbolded instruments are listed to provide a complete picture of the study. The exhibit 

include district screening and recruitment data collection activities because clearance has already 

for these activities.

and thus shares the 

ED gathers data 

called “360-

referenced scores. Thus, 

both in terms of specified criteria and

An overview of the study’s instruments and the schedule for their use is provided in Exhibit 3.

The bolded instruments involve burden and are therefore the basis for seeking clearance. The 

exhibit does not 

include district screening and recruitment data collection activities because clearance has already 
Impact Evaluation of Teacher and Leader Evaluation Systems

Exhibit 3. Summary of Data Collection Instruments and Schedule

Treatment Control

Jul

1. Classroom 

observations

All Grade 4–8 teachers of 

mathematics and/or reading/ELA 

(approx. 1,310 teachers)

2. Teacher 

survey

All K–8 teachers of mathematics 

and/or reading/ELA (approx. 

2,940 teachers)

3. Principal 

survey

70 principals 70 principals

4. District 

interviews 9 districts

5. Intervention 

fidelity data 9 districts

6. District 

archival 

record data

9 districts

7. Intervention

cost records

9 districts

Impact Evaluation of Teacher and Leader Evaluation Systems—9

Summary of Data Collection Instruments and Schedule

2012–13 2013–14

Jul–Sep Oct–Dec Jan–Mar Apr–Jun Jul–Sep Oct–Dec Jan–

End of 

year

End of 

year

End of 

year

Multiple times throughout the year

Student 

records 

and 

rosters of 

study 

schools

Archival 

employee 

and 

student 

records

Rosters 

of study 

schools

Cost 

records

2014

–Mar Apr–Jun Jul–Sep

End of 

year

End of 

year

End of 

year

End of 

year

osters 

of study 

schools

Archival 

employee 

and 

student 

records

Cost 

records
Impact Evaluation of Teacher and Leader Evaluation Systems

1. Classroom Observations

Independent classroom observations will be conducted in addition to the classroom observations 

that are part of the evaluation system. During only 

Grade 4–8 teachers of mathematics and reading/ELA within 

recorded for the duration of one class session up to two times

be used to conduct analyses of the impact of the evaluation system on tea

To enable the analyses of impact on teacher practice to complement the analyses of impact on 

student achievement, teachers in

grade levels and subjects for which student impacts will be analyzed (i.e., 

mathematics and reading/ELA). Because teachers will be video recorded, 

be coded using both CLASS and FFT

by CLASS and FFT.)

2. Teacher Surveys

Teacher surveys will be administered in spring 2013 and 

mathematics and reading/ELA within 

determine whether the intervention produced a contrast between the treatment and 

schools, particularly with respect to

number and the duration of classroo

form of observation-specific feedback

related to instructional improvement and mobility, as well as their attitudes and beliefs about 

their ratings and their capacity to change practice in the desired ways

will be used to address whether the evaluation system had an impact on principal leadership 

practices (RQ5). 

3. Principal Surveys

Principal surveys will be adminis

study school. The survey will be used to

of calculating intervention costs (e.g., observing teachers, preparing written feedback, and 

providing individual feedback during meetings; RQ1). The survey will be used to

whether the intervention produced a contrast between the treatment and 

particularly with respect to principals’ experiences with performance evaluation (e.g., number of 

times they received a rating of some type for their performance as a principal by a

the current school year and how those evaluations were conducted

produced different types of teacher performance information

will focus on principals’ decisions related to improv

providing varying levels of support to teachers of differing performance levels)

well as their attitudes and beliefs about their ratings and their capacity to change practice in the 

desired ways (RQ3). 

Impact Evaluation of Teacher and Leader Evaluation Systems—10

Classroom Observations

ndependent classroom observations will be conducted in addition to the classroom observations 

that are part of the evaluation system. During only Year 2 of the study in spring 2014, 

8 teachers of mathematics and reading/ELA within all 140 study schools 

for the duration of one class session up to two times. Data from these observations will 

s of the impact of the evaluation system on teacher practice

To enable the analyses of impact on teacher practice to complement the analyses of impact on 

the independent sample of observations will be

and subjects for which student impacts will be analyzed (i.e., Grades 4

Because teachers will be video recorded, each observation 

and FFT. (See Exhibit 2 for the domains and constructs 

will be administered in spring 2013 and spring 2014 to all K–8 teachers of 

within all the 140 study schools. The survey will be used to 

determine whether the intervention produced a contrast between the treatment and 

with respect to teachers’ experiences with performance evaluation (e.g., the 

duration of classroom observations through which they received a rating or other 

feedback; RQ2). The survey will focus on teachers’ decisions 

improvement and mobility, as well as their attitudes and beliefs about 

ratings and their capacity to change practice in the desired ways (RQ3). Finally, 

will be used to address whether the evaluation system had an impact on principal leadership 

will be administered in spring 2013 and spring 2014 to the principal 

study school. The survey will be used to capture principal time which is an important component 

of calculating intervention costs (e.g., observing teachers, preparing written feedback, and 

providing individual feedback during meetings; RQ1). The survey will be used to

vention produced a contrast between the treatment and the control schools, 

principals’ experiences with performance evaluation (e.g., number of 

times they received a rating of some type for their performance as a principal by a

the current school year and how those evaluations were conducted) and whether the intervention 

produced different types of teacher performance information for principals (RQ2)

will focus on principals’ decisions related to improvement in instructional leadership

providing varying levels of support to teachers of differing performance levels) and mobility, as 

well as their attitudes and beliefs about their ratings and their capacity to change practice in the 

ndependent classroom observations will be conducted in addition to the classroom observations 

n spring 2014, all 

schools will be video 

. Data from these observations will 

cher practice (RQ5). 

To enable the analyses of impact on teacher practice to complement the analyses of impact on 

from among the 

rades 4–8 in

each observation can 

. (See Exhibit 2 for the domains and constructs measured 

8 teachers of 

. The survey will be used to 

determine whether the intervention produced a contrast between the treatment and the control 

teachers’ experiences with performance evaluation (e.g., the 

m observations through which they received a rating or other 

. The survey will focus on teachers’ decisions 

improvement and mobility, as well as their attitudes and beliefs about 

. Finally, the survey

will be used to address whether the evaluation system had an impact on principal leadership 

he principal in each 

capture principal time which is an important component 

of calculating intervention costs (e.g., observing teachers, preparing written feedback, and 

providing individual feedback during meetings; RQ1). The survey will be used to determine 

control schools, 

principals’ experiences with performance evaluation (e.g., number of 

times they received a rating of some type for their performance as a principal by a supervisor in 

) and whether the intervention 

RQ2). The survey

in instructional leadership (e.g., 

and mobility, as 

well as their attitudes and beliefs about their ratings and their capacity to change practice in the 
Impact Evaluation of Teacher and Leader Evaluation Systems

4. District Interviews

To answer RQ1, interviews will be conducted 

in each school district who are responsible for teacher and leader evaluation systems. The 

interviews will be used to collect information regarding the extent to which 

followed through on their plans to implement the intervention, as well as barriers they 

encountered to implementing their plans and how they overcame those barriers. In

be collected regarding the integration of the study’s evaluation system with existing district 

processes. The interviews will also be used to collect contextual information regarding the 

districts’ human resources policies in the control sch

teacher and principal evaluation system

used. 

5. Intervention Fidelity Data 

Various intervention fidelity records will be collected

capture the fidelity of delivery and participation in key intervention events through in

visits and the monitoring of online webinars (

agendas/schedules). For each district

about their credentials, prior experience

(Teachstone/Teachscape) training materials and

tools, we will capture performance information as well as administrative records. For each 

observation session and feedback session, the system will provide the session dates 

participant list. Additionally, in the course of completing logs t

will record the focus of the feedback session and 

session. Online system records from the VAL

performance information as well as administrative records regarding the number of teachers and 

district staff who were asked to complete VAL

rates, the dates when principals received their assessment 

feedback sessions occurred. Finally, 

teachers and principals in the treatment 

6. District Archival Record 

Finally, various district archival records will be collected. A brief description of each follows.

Student Records for Value-Added 

demographic records will be collected

added modeling for the evaluation system reports 

Student Growth” section). These data also will be used 

study reports. To complete the evaluation system 

data from the 2008–09 through the 

evaluation team will require data from 

Teacher and Principal Records

request administrative data (e.g., e

Impact Evaluation of Teacher and Leader Evaluation Systems—11

nterviews will be conducted in spring 2013 and spring 2014 with the officials 

district who are responsible for teacher and leader evaluation systems. The 

will be used to collect information regarding the extent to which the school 

followed through on their plans to implement the intervention, as well as barriers they 

encountered to implementing their plans and how they overcame those barriers. In

be collected regarding the integration of the study’s evaluation system with existing district 

will also be used to collect contextual information regarding the 

districts’ human resources policies in the control schools during the study, focusing on their 

teacher and principal evaluation system policies and the ways in which performance 

Intervention Fidelity Data 

arious intervention fidelity records will be collected to answer RQ1. For instance, 

fidelity of delivery and participation in key intervention events through in

monitoring of online webinars (i.e., collecting of attendance sheets

each district-selected observer and principal, we will obtain

prior experience, and certification test results from the providers’

training materials and online system. Through the providers’ 

tools, we will capture performance information as well as administrative records. For each 

observation session and feedback session, the system will provide the session dates 

in the course of completing logs to track their work, the observers 

will record the focus of the feedback session and use of the video library during the feedback 

from the VAL-ED provider (Discovery Education) 

performance information as well as administrative records regarding the number of teachers and 

district staff who were asked to complete VAL-ED assessments, the VAL-ED survey response 

rates, the dates when principals received their assessment results, and the dates when principal 

Finally, AIR’s online system will report value-added scores for all 

ls in the treatment schools. 

Archival Record Data Collection

various district archival records will be collected. A brief description of each follows.

dded Modeling and Impact Analyses. Student assessment and 

will be collected from districts. These data will be used to conduct value

added modeling for the evaluation system reports noted above (see “Component for Feedback on 

). These data also will be used to conduct impact analyses

To complete the evaluation system reports, the implementation team

the 2013–14 school years; to complete the study reports

data from the 2011–12 through the 2013–14 school years. 

Teacher and Principal Records for Survey Administration and Mobility Analyses

(e.g., e-mail addresses and grade/subject assignments)

with the officials 

district who are responsible for teacher and leader evaluation systems. The 

the school districts 

followed through on their plans to implement the intervention, as well as barriers they 

encountered to implementing their plans and how they overcame those barriers. Information will 

be collected regarding the integration of the study’s evaluation system with existing district 

will also be used to collect contextual information regarding the 

ools during the study, focusing on their 

performance data are 

For instance, we plan to 

fidelity of delivery and participation in key intervention events through in-person 

attendance sheets and 

will obtain information 

providers’

rough the providers’ online 

tools, we will capture performance information as well as administrative records. For each 

observation session and feedback session, the system will provide the session dates and the 

o track their work, the observers 

use of the video library during the feedback 

ED provider (Discovery Education) will provide 

performance information as well as administrative records regarding the number of teachers and 

ED survey response 

results, and the dates when principal 

added scores for all 

various district archival records will be collected. A brief description of each follows.

assessment and 

conduct value-

“Component for Feedback on 

analyses (RQ6) for the 

implementation team will request 

; to complete the study reports, the 

school years. 

for Survey Administration and Mobility Analyses. We will 

grade/subject assignments) for teacher 
Impact Evaluation of Teacher and Leader Evaluation Systems

survey administration in January

(As noted earlier, the surveys will be used to answer RQ1,

will request administrative data (e.g., school entry/exit dates

all teachers and principals in the participating 

between the 2011–12 school year 

principals, we will request teacher placement information for any teacher who worked in a 

participating school at any time during this 

fall 2013 (for the Year 1 report), 

12 records (prior to randomization) 

13 and 2013–14 school years. In fall 2014

they correspond to the start of the 2014

Performance Evaluation System 

the performance ratings given to teachers under the local evaluation system

summer 2013, we will request local ratings correspond

2014, we will request local ratings correspond

linkages to schools rather than teachers; 

of the teachers in the study. These data will be used 

intervention on local ratings given that principals may be more willing to give low ratings in 

their local systems. 

7. Intervention Cost Record

Within RQ1, one aspect of the intervention to be examined is its cost. AIR will use its internal 

project-cost tracking system to compute the

component for feedback on student growth, the component for feedback on instructional practice, 

and the component for feedback on principal leadership

AIR assessment costs (i.e., to produce

Teachscape, and Teachstone; e.g., to provide training and support to districts), and

conduct observations of teacher instructional practice

 

2 These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptive 

information on mobility as context for interpreting the study results. 

Impact Evaluation of Teacher and Leader Evaluation Systems—12

January 2013 and January 2014 for each school included in the study.

, the surveys will be used to answer RQ1, RQ2, RQ3, and RQ5.)

(e.g., school entry/exit dates and demographic characteristics) 

all teachers and principals in the participating school districts for analyses to track mobility

12 school year and the 2014–15 school year (RQ4).

2

For these teachers

, we will request teacher placement information for any teacher who worked in a 

participating school at any time during this time period. These requests will be made twice. In 

), we will request data corresponding to final, end

12 records (prior to randomization) and final rosters as they correspond to the start of the 2012

14 school years. In fall 2014 (for the Year 2 report), we will request final rosters as 

they correspond to the start of the 2014–15 school year.

Performance Evaluation System Ratings. In summer 2013 and summer 2014, AIR will request 

ratings given to teachers under the local evaluation system to answer RQ3

local ratings corresponding to 2011–12 and 2012–

local ratings corresponding to 2013–14. We will collect these rat

linkages to schools rather than teachers; we will request for each school a list of the local ratings 

These data will be used to answer RQ3, to estimate the impact of the 

given that principals may be more willing to give low ratings in 

Records

Within RQ1, one aspect of the intervention to be examined is its cost. AIR will use its internal 

compute the costs of each component of the intervention (i.e., 

component for feedback on student growth, the component for feedback on instructional practice, 

and the component for feedback on principal leadership). Using this system, we will 

produce value-added reports), system providers’ costs 

Teachstone; e.g., to provide training and support to districts), and

conduct observations of teacher instructional practice and to provide feedback.

These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptive 

or interpreting the study results. 

2014 for each school included in the study.

5.) In addition, we 

demographic characteristics) for 

analyses to track mobility

For these teachers and 

, we will request teacher placement information for any teacher who worked in a 

ll be made twice. In 

will request data corresponding to final, end-of-year 2011–

final rosters as they correspond to the start of the 2012–

uest final rosters as 

In summer 2013 and summer 2014, AIR will request 

to answer RQ3. In 

–13. In summer 

We will collect these ratings with 

of the local ratings 

estimate the impact of the 

given that principals may be more willing to give low ratings in 

Within RQ1, one aspect of the intervention to be examined is its cost. AIR will use its internal 

costs of each component of the intervention (i.e., the 

component for feedback on student growth, the component for feedback on instructional practice, 

we will combine 

costs (Discovery, 

Teachstone; e.g., to provide training and support to districts), and study costs to

These data will provide the basis for estimating impacts on teacher mobility (RQ4) and providing descriptive 
Impact Evaluation of Teacher and Leader Evaluation Systems

Supporting Statement for Paperwork Reduction Act 

A. Justification

1. Circumstances Making Collection of Information Necessary

Improving student achievement is a core priority for federal education policy

34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency 

level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth 

graders and 35 percent of eighth graders attained the profic

Center for Education Statistics, 2010a). 

assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek, 

Peterson, & Woessmann, 2010). 

In an effort to improve student outcomes, federal education policy

reauthorization of the Elementary and Secondary Education Act (ESEA), the American 

Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program

targets teacher quality as a potential lever for improving student achievement. Several studies 

support this focus, with recent work examining individual teachers for several years to estimate 

the degree of variation in achievement gains that can be attrib

measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On 

balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above 

average in effectiveness at raising studen

increase in student achievement. If such a teacher effect accumulates as students move from one 

grade to the next, it would translate into a substantial difference in achievement by the time 

students leave high school. Similar studies suggest that some variation in achievement gains can 

be attributed to principals, which indicates that principal quality may be another lever for 

education policymakers (Hallinger & Heck, 1998; Leithwood, 

Leithwood & Jantzi, 2005; Leithwood, Louis, Anderson, & Wahlstrom, 2004; 

& McNulty, 2003).

The federal government has invested considerable financial resources into supporting efforts to 

develop systems to measure educator effectiveness and use that information in human resource 

policies. The Teacher Incentive Fund (TIF) was established in 2006 

of approximately $100 million to identify and reward effective educators in high

based in part on student growth. The goal of the program is to improve 

increasing the effectiveness of edu

ARRA provided additional funding for TIF and established competitive grants to help states 

build their pools of effective teachers and principals through the Race to the Top (RTT) program, 

in which a core priority area is the effectiveness of teachers and leaders. 

and districts are developing new teacher and leader evaluation systems, wi

improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for 

RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a 

continued emphasis on the development of teacher and leader eva

Impact Evaluation of Teacher and Leader Evaluation Systems—13

Supporting Statement for Paperwork Reduction Act 

Submission

1. Circumstances Making Collection of Information Necessary

Improving student achievement is a core priority for federal education policy. In 2011, only 

34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency 

level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth 

graders and 35 percent of eighth graders attained the proficient level in mathematics (National 

Center for Education Statistics, 2010a). The performance of U.S. students on international 

assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek, 

Peterson, & Woessmann, 2010). 

effort to improve student outcomes, federal education policy—as manifested in the 2002 

reauthorization of the Elementary and Secondary Education Act (ESEA), the American 

Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program

targets teacher quality as a potential lever for improving student achievement. Several studies 

support this focus, with recent work examining individual teachers for several years to estimate 

the degree of variation in achievement gains that can be attributed to teachers rather than 

measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On 

balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above 

average in effectiveness at raising student achievement may lead to a 0.1 standard deviation 

increase in student achievement. If such a teacher effect accumulates as students move from one 

grade to the next, it would translate into a substantial difference in achievement by the time 

high school. Similar studies suggest that some variation in achievement gains can 

be attributed to principals, which indicates that principal quality may be another lever for 

policymakers (Hallinger & Heck, 1998; Leithwood, Harris, & Strauss, 

Leithwood & Jantzi, 2005; Leithwood, Louis, Anderson, & Wahlstrom, 2004; Waters, Marzano, 

The federal government has invested considerable financial resources into supporting efforts to 

develop systems to measure educator effectiveness and use that information in human resource 

policies. The Teacher Incentive Fund (TIF) was established in 2006 with an initial funding base 

of approximately $100 million to identify and reward effective educators in high

based in part on student growth. The goal of the program is to improve student achievement by 

increasing the effectiveness of educators in high-poverty schools. 

ARRA provided additional funding for TIF and established competitive grants to help states 

of effective teachers and principals through the Race to the Top (RTT) program, 

in which a core priority area is the effectiveness of teachers and leaders. Using RTT grants, states 

and districts are developing new teacher and leader evaluation systems, with the goal of 

improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for 

RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a 

continued emphasis on the development of teacher and leader evaluation systems. Through the 

Supporting Statement for Paperwork Reduction Act 

. In 2011, only 

34 percent of fourth graders and 34 percent of eighth graders performed at or above proficiency 

level in reading (National Center for Education Statistics, 2010b). Similarly, 40 percent of fourth 

ient level in mathematics (National 

on international 

assessments lags far behind that of their international peers (Aud et al., 2011; Hanushek, 

as manifested in the 2002 

reauthorization of the Elementary and Secondary Education Act (ESEA), the American 

Reinvestment and Recovery Act (ARRA) of 2009, and the ESEA Flexibility waiver program—

targets teacher quality as a potential lever for improving student achievement. Several studies 

support this focus, with recent work examining individual teachers for several years to estimate 

uted to teachers rather than 

measurement error (Bill & Melinda Gates Foundation, 2011; Schochet & Chiang, 2010). On 

balance, these studies suggest that being assigned to a teacher who is 1 standard deviation above 

t achievement may lead to a 0.1 standard deviation 

increase in student achievement. If such a teacher effect accumulates as students move from one 

grade to the next, it would translate into a substantial difference in achievement by the time 

high school. Similar studies suggest that some variation in achievement gains can 

be attributed to principals, which indicates that principal quality may be another lever for 

Harris, & Strauss, 2010; 

Waters, Marzano, 

The federal government has invested considerable financial resources into supporting efforts to 

develop systems to measure educator effectiveness and use that information in human resource 

with an initial funding base 

of approximately $100 million to identify and reward effective educators in high-poverty schools, 

student achievement by 

ARRA provided additional funding for TIF and established competitive grants to help states 

of effective teachers and principals through the Race to the Top (RTT) program, 

Using RTT grants, states 

th the goal of 

improving teacher effectiveness and student outcomes. ARRA included $4.35 billion in funds for 

RTT. The U.S. Department of Education’s (ED’s) ESEA Flexibility program provides a 

luation systems. Through the 
Impact Evaluation of Teacher and Leader Evaluation Systems

Flexibility program, ED relieves states from selected ESEA requirements in exchange for 

compliance with new requirements, including 

evaluation systems.

Although states and districts across the country have been developing and implementing new 

evaluation systems to identify and reward highly effective educators, no large

been carried out to confirm the efficacy of these new systems. The TLES Study will address

need. This study is funded by TIF

This is the second of a two-stage clearance request to carry out data collection activities for the 

TLES Study. In May 2012, OMB approved the first clearance request

described the study, the design, and 

contacting a sample of districts to establish their eligibility for the study and visiting eligible 

districts. In this second request, the Institut

of Education requests clearance for the study’s data collection instruments, specifically the 

teacher survey, the principal survey, 

collection protocol. 

2. Purposes and Uses of the Data

The purpose of the TLES Study is to test a package of teacher and leader evaluation system 

components that is consistent with current federal policy. In contrast to traditional systems, the 

evaluation system components provided by the study will use multiple, valid measures, including 

measures of student growth, teacher practice, and principal leadership, to meaningfully

differentiate the performance levels. 

addressing such evaluation systems

implementation and impacts (e.g., 

Data collected by the TLES Study will be of immediate 

researchers, and practitioners.

3. Use of Technology to Reduce Burden

The data collection plan is designed to obtain reliable information in an efficient way that 

minimizes respondent burden, and technology will be us

collections for which we are seeking clearance. 

The teacher and principal surveys will be administered online using a 

allow respondents to complete the survey at a time and place that is conveni

respondent. 

District interviews will be conducted 

personnel. This mode of data collection is appropriate for the conversational exchange necessary 

to obtain answers to the open-ended qu

administered survey can provide. 

Impact Evaluation of Teacher and Leader Evaluation Systems—14

Flexibility program, ED relieves states from selected ESEA requirements in exchange for 

compliance with new requirements, including the implementation of reformed teacher and leader 

ricts across the country have been developing and implementing new 

evaluation systems to identify and reward highly effective educators, no large-scale study has yet 

been carried out to confirm the efficacy of these new systems. The TLES Study will address

TIF.

stage clearance request to carry out data collection activities for the 

, OMB approved the first clearance request (OMB 1850

described the study, the design, and the recruitment activities. The recruitment activities included 

contacting a sample of districts to establish their eligibility for the study and visiting eligible 

districts. In this second request, the Institute of Education Sciences (IES) of the U.S. Department 

of Education requests clearance for the study’s data collection instruments, specifically the 

principal survey, the district interview, and the district archival records 

2. Purposes and Uses of the Data

The purpose of the TLES Study is to test a package of teacher and leader evaluation system 

components that is consistent with current federal policy. In contrast to traditional systems, the 

provided by the study will use multiple, valid measures, including 

student growth, teacher practice, and principal leadership, to meaningfully

performance levels. This study will be the first large-scale randomized 

addressing such evaluation systems. Using an experimental design, the study will examine both 

and impacts (e.g., impacts on teacher mobility, impacts on student achievement

Data collected by the TLES Study will be of immediate interest and import for policymakers, 

3. Use of Technology to Reduce Burden

The data collection plan is designed to obtain reliable information in an efficient way that 

minimizes respondent burden, and technology will be used to reduce burden for all the 

collections for which we are seeking clearance. 

The teacher and principal surveys will be administered online using a Web platform that will 

allow respondents to complete the survey at a time and place that is convenient for 

District interviews will be conducted via telephone to reduce the writing burden for district 

personnel. This mode of data collection is appropriate for the conversational exchange necessary 

ended questions and allows probing for more detail than a self

administered survey can provide. 

Flexibility program, ED relieves states from selected ESEA requirements in exchange for 

implementation of reformed teacher and leader 

ricts across the country have been developing and implementing new 

scale study has yet 

been carried out to confirm the efficacy of these new systems. The TLES Study will address this 

stage clearance request to carry out data collection activities for the 

(OMB 1850-0890), which 

activities included 

contacting a sample of districts to establish their eligibility for the study and visiting eligible 

e of Education Sciences (IES) of the U.S. Department 

of Education requests clearance for the study’s data collection instruments, specifically the 

district archival records 

The purpose of the TLES Study is to test a package of teacher and leader evaluation system 

components that is consistent with current federal policy. In contrast to traditional systems, the 

provided by the study will use multiple, valid measures, including 

student growth, teacher practice, and principal leadership, to meaningfully

scale randomized field trial 

study will examine both 

student achievement). 

interest and import for policymakers, 

The data collection plan is designed to obtain reliable information in an efficient way that 

ed to reduce burden for all the data 

platform that will 

ent for each 

writing burden for district 

personnel. This mode of data collection is appropriate for the conversational exchange necessary 

probing for more detail than a self-
Impact Evaluation of Teacher and Leader Evaluation Systems

For the archival data, we will reduce burden by gathering the data electronically rather than in 

hard copy. We will provide clear instructions on the data requested a

the data securely. 

A toll-free number and an e-mail address will be available during the data collection process to 

permit respondents to contact AIR

and the e-mail address will be included in all communication with respondents.

4. Efforts to Identify Duplication

Throughout the evaluation, efforts will be made to minimize and reduce the burden on 

respondents. Wherever possible, we rely on secondary data sources to reduce burden on district 

and school personnel. By collecting 

mobility information), we will eliminate the need for several items on the teacher and principal 

surveys and thereby avoid redundancy.

In addition, we will make sure to coordinate efforts with other studies going on in the field. 

Currently, there is an IES impact evaluation of 

of performance-based compensation on student achievement and 

teachers and principals. The national TIF evaluation measures the impact

compensation specifically, while our purpose is to test 

making the focus of the two studies very different. 

5. Methods to Minimize Burden on Small Entities

To be considered a small entity by OMB, a school district would need to have a population of 

fewer than 50,000 students. Our first step in minimizing burden to small entities was 

some of the smallest school districts 

requirements with certain recruitment criteria. Specifically, our criteria required that districts 

have (1) at least 10 qualifying schools, (2) data systems that support value

(3) not announced a recent implement

leader evaluation system to take effect before 

all districts, we seek to minimize burden to what is necessary. 

participate in the study data collections to provide the data necessary for the study’s 

analyses. Thus, we cannot reduce burden on small entities by changing t

placing extra burden on small entities associated with travel, the implementa

conducted separately in each district.

the most efficient processes are established for data collection to minimize burden on the part of 

district personnel. 

6. Consequences of Not Collecting the Data

States and school districts are increasingly reshaping teacher and leader evaluation systems, in 

part due to the federal initiatives described above (see A.1).Without this study, states and 

districts will have a limited understanding o

educators because there have been no rigorous, large

Impact Evaluation of Teacher and Leader Evaluation Systems—15

For the archival data, we will reduce burden by gathering the data electronically rather than in 

hard copy. We will provide clear instructions on the data requested and methods of transmitting 

mail address will be available during the data collection process to 

AIR with questions or requests for assistance. The toll

address will be included in all communication with respondents.

4. Efforts to Identify Duplication

Throughout the evaluation, efforts will be made to minimize and reduce the burden on 

respondents. Wherever possible, we rely on secondary data sources to reduce burden on district 

and school personnel. By collecting archival records on teachers and principals (e.g., salary

mobility information), we will eliminate the need for several items on the teacher and principal 

surveys and thereby avoid redundancy.

In addition, we will make sure to coordinate efforts with other studies going on in the field. 

ently, there is an IES impact evaluation of the TIF program that seeks to measure the impact 

based compensation on student achievement and the mobility and retention of 

teachers and principals. The national TIF evaluation measures the impact of performance

compensation specifically, while our purpose is to test a teacher and leader evaluation system, 

making the focus of the two studies very different. 

5. Methods to Minimize Burden on Small Entities

To be considered a small entity by OMB, a school district would need to have a population of 

fewer than 50,000 students. Our first step in minimizing burden to small entities was 

some of the smallest school districts that might have been the most burdened by the study 

with certain recruitment criteria. Specifically, our criteria required that districts 

have (1) at least 10 qualifying schools, (2) data systems that support value-added modeling, and 

(3) not announced a recent implementation (or planned implementation) of a new teacher or 

leader evaluation system to take effect before the 2014–15 school year. Second, in working with 

all districts, we seek to minimize burden to what is necessary. All participating districts need to 

ipate in the study data collections to provide the data necessary for the study’s 

Thus, we cannot reduce burden on small entities by changing the collections. 

placing extra burden on small entities associated with travel, the implementation events are 

conducted separately in each district. The study team will work closely with districts to ensure 

the most efficient processes are established for data collection to minimize burden on the part of 

Collecting the Data

districts are increasingly reshaping teacher and leader evaluation systems, in 

part due to the federal initiatives described above (see A.1).Without this study, states and 

districts will have a limited understanding of how these systems affect their students and 

there have been no rigorous, large-scale studies of the impact of such systems. 

For the archival data, we will reduce burden by gathering the data electronically rather than in 

nd methods of transmitting 

mail address will be available during the data collection process to 

with questions or requests for assistance. The toll-free number 

Throughout the evaluation, efforts will be made to minimize and reduce the burden on 

respondents. Wherever possible, we rely on secondary data sources to reduce burden on district 

als (e.g., salary and

mobility information), we will eliminate the need for several items on the teacher and principal 

In addition, we will make sure to coordinate efforts with other studies going on in the field. 

seeks to measure the impact 

mobility and retention of 

of performance-based 

evaluation system, 

To be considered a small entity by OMB, a school district would need to have a population of 

fewer than 50,000 students. Our first step in minimizing burden to small entities was to exclude 

t burdened by the study 

with certain recruitment criteria. Specifically, our criteria required that districts 

added modeling, and 

ation (or planned implementation) of a new teacher or 

Second, in working with 

All participating districts need to 

ipate in the study data collections to provide the data necessary for the study’s 

he collections. To avoid 

tion events are 

will work closely with districts to ensure 

the most efficient processes are established for data collection to minimize burden on the part of 

districts are increasingly reshaping teacher and leader evaluation systems, in 

part due to the federal initiatives described above (see A.1).Without this study, states and 

f how these systems affect their students and 

scale studies of the impact of such systems. 
Impact Evaluation of Teacher and Leader Evaluation Systems

7. Special Circumstances

No special circumstances apply to this study. 

8. Federal Register Comments and Persons Consulted Outside the Agency

The 60-day notice to solicit public comments was published in Volume 7

46748 of the Federal Register on 

To assist with the development of the study, project staff will draw on the experience and 

expertise of a network of outside experts who will serve as our technical working group 

members. The consultants and their affiliations are listed in Exhibit 

Proposed Expert Reviewer

Laura Goe

Thomas Dee

Daniel McCaffrey

Catherine McClellan

Jonah Rockoff

Patrick Schuermann

Carla Stevens

John Tyler

Judy Wurtzel

To date, the project advisor and the TWG members have convened twice

once via teleconference) to discuss 

consult outside experts individually on an as

9. Payment or Gifts

Incentives have been proposed for the teacher survey

effort in completing the survey. We propose offering a $2

she completes a questionnaire to acknowledge the 

questionnaire. 

Incentives are also proposed because high response rates are needed to make the survey findings 

reliable, and we are aware that teachers are the targets of numerous requests to complete surveys 

on a wide variety of topics from state and district offices, i

Although some districts will have solicited buy

our recent experience with teacher surveys supports our view that monetary incentives 

needed to ensure adequate response rates. 

Impact Evaluation of Teacher and Leader Evaluation Systems—16

No special circumstances apply to this study. 

Comments and Persons Consulted Outside the Agency

day notice to solicit public comments was published in Volume 77, Number 

of the Federal Register on August 6, 2012. No comments were received from the public

elopment of the study, project staff will draw on the experience and 

expertise of a network of outside experts who will serve as our technical working group 

members. The consultants and their affiliations are listed in Exhibit 4. 

Exhibit 4. Expert Reviewers

Proposed Expert Reviewer Professional Affiliation

Educational Testing Service

University of Virginia

Rand Corporation

Catherine McClellan Clowder Consulting

Columbia University

Patrick Schuermann Vanderbilt University

Houston Independent School District

Brown University

Independent consultant

To date, the project advisor and the TWG members have convened twice (once in person and 

to discuss the study design and data collection. Project staff 

outside experts individually on an as-needed basis. 

Incentives have been proposed for the teacher survey to partially offset respondents’ time and 

effort in completing the survey. We propose offering a $25 incentive to a teacher each time he or 

she completes a questionnaire to acknowledge the 30 minutes required to complete each 

Incentives are also proposed because high response rates are needed to make the survey findings 

reliable, and we are aware that teachers are the targets of numerous requests to complete surveys 

on a wide variety of topics from state and district offices, independent researchers, and 

Although some districts will have solicited buy-in from teachers to participate in the evaluation, 

our recent experience with teacher surveys supports our view that monetary incentives 

ensure adequate response rates. 

Comments and Persons Consulted Outside the Agency

, Number 151, page 

received from the public.

elopment of the study, project staff will draw on the experience and the 

expertise of a network of outside experts who will serve as our technical working group (TWG) 

(once in person and 

study design and data collection. Project staff will also 

to partially offset respondents’ time and 

incentive to a teacher each time he or 

minutes required to complete each 

Incentives are also proposed because high response rates are needed to make the survey findings 

reliable, and we are aware that teachers are the targets of numerous requests to complete surveys 

ndependent researchers, and ED. 

in from teachers to participate in the evaluation, 

our recent experience with teacher surveys supports our view that monetary incentives are 
Impact Evaluation of Teacher and Leader Evaluation Systems

No incentives will be given to principals for 

for completing the district interview or 

10. Assurances of Confidentiality

A consistent and cautious approach will be taken to protect all information collected during data 

collection. This approach will be in accordance with all relevant regulations and requirements. 

These include the Education Sciences Institute Reform Act of 

which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the 

Institute...to conform with the requirements of section 552 of Title 5, United States Code, the 

confidentiality standards of subsect

General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the 

Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights 

Amendment. In addition, for student information, the project director will ensure that all 

individually identifiable information about students, their academic achievements and families, 

and information with respect to individual schools shall remain confidential in acco

section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and 

sections 444 and 445 of the General Educations Provision Act.

Subsection (c) of Section 183 requires the 

designed to protect the confidentiality of persons in the collection, reporting, and publication of 

data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting 

disclosure of individually identifiable information as

inappropriate communication of individually identifiable information by employees or staff a 

felony.

Except with respect to the information provided to educators and their supervisors as part of the 

study’s package of teacher and leader evaluation system components, AIR will use all 

information collected in the study for research purposes only. When reporting the study results, 

data will be presented in aggregate form

identified. A statement to this effect will be included with all requests for study data. All 

members of the study team with access to the study data will be trained and certified on the 

importance of privacy and data security. All study data will be kept

identifiers will be destroyed as soon as they are no longer required.

We will request unique identifiers 

reading/ELA teacher(s) associated with each student. To minimize

identifiable information, we will request that districts send a separate crosswalk file that links 

teacher names to IDs. Both the implementation and 

security by allowing access to shared folders only to team members who need it. We also will 

create a process whereby the project director must approve all requests for access to these folders 

(and maintain a list of who has acces

To enable us to conduct analyses that link individual value

responses and classroom observation scores, the implementation team will attach scrambled 

Impact Evaluation of Teacher and Leader Evaluation Systems—17

will be given to principals for completing the principal survey or to

the district interview or the collection of district archival records.

10. Assurances of Confidentiality

consistent and cautious approach will be taken to protect all information collected during data 

collection. This approach will be in accordance with all relevant regulations and requirements. 

These include the Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183, 

which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the 

Institute...to conform with the requirements of section 552 of Title 5, United States Code, the 

confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the 

General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the 

Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights 

ddition, for student information, the project director will ensure that all 

individually identifiable information about students, their academic achievements and families, 

and information with respect to individual schools shall remain confidential in acco

section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and 

sections 444 and 445 of the General Educations Provision Act.

requires the IES director to “develop and enforce standar

designed to protect the confidentiality of persons in the collection, reporting, and publication of 

data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting 

disclosure of individually identifiable information as well as making the publishing or 

inappropriate communication of individually identifiable information by employees or staff a 

Except with respect to the information provided to educators and their supervisors as part of the 

her and leader evaluation system components, AIR will use all 

information collected in the study for research purposes only. When reporting the study results, 

data will be presented in aggregate form only, such that individuals and institutions will not be

identified. A statement to this effect will be included with all requests for study data. All 

members of the study team with access to the study data will be trained and certified on the 

importance of privacy and data security. All study data will be kept in secured locations

identifiers will be destroyed as soon as they are no longer required.

We will request unique identifiers (IDs) for each student, as well IDs for the mathematics and 

reading/ELA teacher(s) associated with each student. To minimize access to personally 

identifiable information, we will request that districts send a separate crosswalk file that links 

teacher names to IDs. Both the implementation and the evaluation teams will maintain data 

security by allowing access to shared folders only to team members who need it. We also will 

create a process whereby the project director must approve all requests for access to these folders 

(and maintain a list of who has access).

To enable us to conduct analyses that link individual value-added scores to both survey

responses and classroom observation scores, the implementation team will attach scrambled 

or to district staff 

.

consistent and cautious approach will be taken to protect all information collected during data 

collection. This approach will be in accordance with all relevant regulations and requirements. 

2002, Title I, Part E, Section 183, 

which requires “[a]ll collection, maintenance, use, and wide dissemination of data by the 

Institute...to conform with the requirements of section 552 of Title 5, United States Code, the 

ions (c) of this section, and sections 444 and 445 of the 

General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the 

Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights 

ddition, for student information, the project director will ensure that all 

individually identifiable information about students, their academic achievements and families, 

and information with respect to individual schools shall remain confidential in accordance with 

section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and 

director to “develop and enforce standards 

designed to protect the confidentiality of persons in the collection, reporting, and publication of 

data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting the 

well as making the publishing or 

inappropriate communication of individually identifiable information by employees or staff a 

Except with respect to the information provided to educators and their supervisors as part of the 

her and leader evaluation system components, AIR will use all 

information collected in the study for research purposes only. When reporting the study results, 

, such that individuals and institutions will not be

identified. A statement to this effect will be included with all requests for study data. All 

members of the study team with access to the study data will be trained and certified on the 

in secured locations, and 

for the mathematics and 

access to personally 

identifiable information, we will request that districts send a separate crosswalk file that links 

teams will maintain data 

security by allowing access to shared folders only to team members who need it. We also will 

create a process whereby the project director must approve all requests for access to these folders 

added scores to both survey

responses and classroom observation scores, the implementation team will attach scrambled 
Impact Evaluation of Teacher and Leader Evaluation Systems

teacher IDs to the survey and observation data before the data are made a

evaluation team.

In addition to these safeguards, AIR routinely employs 

assurances with respect to study data:

 All AIR employees sign a privacy pledge emphasizing its importance and describing their 

obligation.

 Identifying information is maintained on separate forms and files, which are linked only

by sample identification number.

 Access to hard-copy documents is strictly limited. Documents are stored in locked files 

and cabinets. Discarded materials are 

 Computer data files are protected with passwords

 Especially sensitive data are maintained on removable storage devices that are kept 

physically secure when not in use.

11. Justification of Sensitive Ques

No questions of a sensitive nature will be included in the teacher and principal surveys and the 

district interview. For the district archival records collection protocol, we will request

performance ratings given to teachers under the local 

will request local ratings corresponding to 2011

request local ratings corresponding to 2013

schools rather than teachers; we w

teachers in the study. These data will be used to estimate the impact of the intervention on local 

ratings. Whether or not the intervention has an impact on local ratings

an important aspect of RQ3. In response to the intervention, 

give low ratings in their local systems.

clustered in the highest performance category (

12. Estimates of Hour Burden

The total estimated hour burden for the data collections for the TLES Study is 

Based on average hourly wages for participants

yearly salaries), this amounts to an estimated monetary cost of $

the estimates of respondent burden for 

survey includes time for 85 percent of a sample of 2,

districts to respond to a 30-minute survey in the spring in each year of the study. The burden 

estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and 

control) in the 9 districts to respond to a 3

The burden estimate for the district interview includes 

contacts to respond to a 90-minute interview in the spring in each year of the study. The 

archival records requests require an estimated 20

the requested data in each district. In 

records request: student records from 2008

Impact Evaluation of Teacher and Leader Evaluation Systems—18

teacher IDs to the survey and observation data before the data are made available to the 

AIR routinely employs the following to carry out privacy

assurances with respect to study data:

All AIR employees sign a privacy pledge emphasizing its importance and describing their 

Identifying information is maintained on separate forms and files, which are linked only

by sample identification number.

copy documents is strictly limited. Documents are stored in locked files 

and cabinets. Discarded materials are shredded.

Computer data files are protected with passwords, and access is limited to specific users.

Especially sensitive data are maintained on removable storage devices that are kept 

physically secure when not in use.

11. Justification of Sensitive Questions

No questions of a sensitive nature will be included in the teacher and principal surveys and the 

For the district archival records collection protocol, we will request

performance ratings given to teachers under the local evaluation system. In summer 2013, we 

will request local ratings corresponding to 2011–12 and 2012–13. In summer 2014, we will 

request local ratings corresponding to 2013–14. We will collect these ratings with linkages to 

schools rather than teachers; we will request for each school a list of the local ratings of the 

These data will be used to estimate the impact of the intervention on local 

the intervention has an impact on local ratings, and in what direction

aspect of RQ3. In response to the intervention, principals may be more willing to 

give low ratings in their local systems. Currently, local rating systems tend to produce ratings 

clustered in the highest performance category (Weisberg, Sexton, Mulhern, & Keeling, 2009)

12. Estimates of Hour Burden

The total estimated hour burden for the data collections for the TLES Study is 4,265

Based on average hourly wages for participants (which were calculated based on estimates of 

, this amounts to an estimated monetary cost of $124,285. Exhibit 

the estimates of respondent burden for the study activities. The burden estimate for the teacher 

me for 85 percent of a sample of 2,940 teachers (treatment and control) in the 

minute survey in the spring in each year of the study. The burden 

estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and 

districts to respond to a 30-minute survey in the spring in each year o

The burden estimate for the district interview includes time for 100 percent of all 

minute interview in the spring in each year of the study. The 

requests require an estimated 20 hours of burden for 1 district data person 

data in each district. In Year 1 of implementation, there will be four

from 2008–09 through 2011–12, student records from 2012

vailable to the 

to carry out privacy

All AIR employees sign a privacy pledge emphasizing its importance and describing their 

Identifying information is maintained on separate forms and files, which are linked only

copy documents is strictly limited. Documents are stored in locked files 

and access is limited to specific users.

Especially sensitive data are maintained on removable storage devices that are kept 

No questions of a sensitive nature will be included in the teacher and principal surveys and the 

For the district archival records collection protocol, we will request the 

In summer 2013, we 

13. In summer 2014, we will 

We will collect these ratings with linkages to 

ill request for each school a list of the local ratings of the 

These data will be used to estimate the impact of the intervention on local 

, and in what direction, is

be more willing to 

Currently, local rating systems tend to produce ratings 

xton, Mulhern, & Keeling, 2009).

4,265 hours. 

(which were calculated based on estimates of 

Exhibit 5 summarizes 

study activities. The burden estimate for the teacher 

teachers (treatment and control) in the 9 

minute survey in the spring in each year of the study. The burden 

estimate for the principal survey includes time for 85 percent of all 140 principals (treatment and 

year of the study. 

of all 9 district 

minute interview in the spring in each year of the study. The district 

district data person to pull 

four archival 

12, student records from 2012–13, 
Impact Evaluation of Teacher and Leader Evaluation Systems

teacher and principal records for sampling purposes

ratings. In Year 2 of implementation

student records requests) and will add an additional request in the fall of 

principal records for the purposes of tracking retention and mobility in our study schools. Finally, 

in the fall of the 2014–15 school year, we will be collecting the final round of teacher and 

principal records for the purposes of tracki

Impact Evaluation of Teacher and Leader Evaluation Systems—19

principal records for sampling purposes, and educator performance evaluation 

2 of implementation, we will have the same request as in Year 1

and will add an additional request in the fall of Year 2 for teacher and 

purposes of tracking retention and mobility in our study schools. Finally, 

15 school year, we will be collecting the final round of teacher and 

purposes of tracking retention and mobility in our study schools. 

educator performance evaluation 

1 (with one fewer 

2 for teacher and 

purposes of tracking retention and mobility in our study schools. Finally, 

15 school year, we will be collecting the final round of teacher and 

ng retention and mobility in our study schools. 
Impact Evaluation of Teacher and Leader Evaluation Systems

Exhibit 

Data Collection

Total 

Sample 

Size

Estimated 

Response 

Rate

School Year 

2012–13

Teacher survey, 

spring 2,940 85%

Principal survey, 

spring 140 85%

District interview, 

spring 9 100%

District archival 

records collection 9 100%

Subtotal

School Year 

2013–14

Teacher survey, 

spring 2,940 85%

Principal survey, 

spring 140 85%

District interview, 

spring 9 100%

District archival 

records collection 9 100%

Subtotal

School Year 

2014–15

District archival 

records collection 9 100%

Subtotal

Total across all 

school years

Impact Evaluation of Teacher and Leader Evaluation Systems—20

Exhibit 5. Hour Burden for Respondents

Estimated 

Response Number of 

Respondents

Number of 

Administrations

Total 

Number of 

Responses

Time 

Estimate 

(in hours)

Total 

Hours

2,499 1 2,499 0.5 1,

119 1 119 0.5

9 1 9 1.5

9 4 36 20 720

2,043

2,499 1 2,499 0.5 1,

119 1 119 0.5

9 1 9 1.5

9 4 36 20 720

2,043

9 1 9 20 180

180

4,265

Total 

Hours

Hourly 

Rate

Estimated 

Monetary 

Cost of 

Burden

1,250 $25 $31,238

60 $35 $2,083

14 $35 $473

720 $35 $25,200

2,043 $58,993

1,250 $25 $31,238

60 $35 $2,083

14 $35 $473

720 $35 $25,200

2,043 $58,993

180 $35 $6,300

180 $6,300

4,265 $124,285
Impact Evaluation of Teacher and Leader Evaluation Systems

13. Estimate of Cost Burden to 

There are no additional respondent costs associated with 

than the hour burden accounted for in item 12. 

14. Estimate of Annual Cost to the Federal Government

The estimated cost for all aspects of the study is $16,783,022 over 

annual cost to the federal government 

Year 2 cost is $5,998,265; the Year 3 cost is $5,224,346; 

Year 5 cost is $127,071.

15. Program Changes or Adjustments

This is the addendum package to a clearance request that was approved in May, 2012, to conduct 

recruitment activities (see OMB 1840

described in this addendum package requires 

16. Plans for Tabulation and Publication of Results

AIR will report the findings from the TLES Study to IES in two substantive reports. The timeline 

for the dissemination of the study results is summarized in Exhibit 

Exhibit 6. Schedule for Dissemination of Study Results

Deliverable

First report

Final report

The first report will present the results of the analyses conducted on the data collected for the 

2012–13 school year, including mobility as measured in fall 2013. This report will include a 

description of the study design and 

analyses. More specifically, the descriptive analyses will include the following:

● Descriptive analyses of the characteristics of the study sample

● Analyses of the equivalence of the treatment and 

characteristics before intervention implementation 

● Descriptive analyses of how

about implementation, the 

intervention was delivered, 

context it was implemented

In addition, the report will provide results on service contrast (RQ2) and the impacts of the 

intervention on educators and students. 

following during the first treatment year 

 Intermediate outcomes: decisions of key actors (e.g.,

Impact Evaluation of Teacher and Leader Evaluation Systems—21

13. Estimate of Cost Burden to Respondents

There are no additional respondent costs associated with the data collections for this stud

than the hour burden accounted for in item 12. 

14. Estimate of Annual Cost to the Federal Government

The estimated cost for all aspects of the study is $16,783,022 over 5 years, with an average 

annual cost to the federal government of $3,356,604. The total Year 1 cost is $4,668,189; 

Year 3 cost is $5,224,346; the Year 4 cost is $765,151; and the 

15. Program Changes or Adjustments

This is the addendum package to a clearance request that was approved in May, 2012, to conduct 

recruitment activities (see OMB 1840-0890). Completion of the data collection activities 

described in this addendum package requires additional burden totaling 3,374 hours. 

16. Plans for Tabulation and Publication of Results

from the TLES Study to IES in two substantive reports. The timeline 

for the dissemination of the study results is summarized in Exhibit 6.

. Schedule for Dissemination of Study Results

Deliverable Anticipated Release Date

eport April 2015

eport July 2016

The first report will present the results of the analyses conducted on the data collected for the 

13 school year, including mobility as measured in fall 2013. This report will include a 

description of the study design and the results from both descriptive analyses and impact 

analyses. More specifically, the descriptive analyses will include the following:

Descriptive analyses of the characteristics of the study sample

Analyses of the equivalence of the treatment and the control groups in their background 

before intervention implementation 

how the intervention was implemented (e.g., district decisions 

the cost of the intervention, the fidelity with which the 

vention was delivered, and the participation of key actors) and in what district policy

context it was implemented (RQ1)

In addition, the report will provide results on service contrast (RQ2) and the impacts of the 

intervention on educators and students. Specifically, the impact analyses will include the 

following during the first treatment year (2012–13):

Intermediate outcomes: decisions of key actors (e.g., teachers’ decisions to try new 

s for this study other 

with an average 

otal Year 1 cost is $4,668,189; the 

cost is $765,151; and the 

This is the addendum package to a clearance request that was approved in May, 2012, to conduct 

ction activities 

hours. 

from the TLES Study to IES in two substantive reports. The timeline 

The first report will present the results of the analyses conducted on the data collected for the 

13 school year, including mobility as measured in fall 2013. This report will include a 

results from both descriptive analyses and impact 

trol groups in their background 

the intervention was implemented (e.g., district decisions 

fidelity with which the 

and in what district policy

In addition, the report will provide results on service contrast (RQ2) and the impacts of the 

Specifically, the impact analyses will include the 

teachers’ decisions to try new 
Impact Evaluation of Teacher and Leader Evaluation Systems

techniques, work differently, or pursue learning experiences

 Outcomes: mobility of teachers and principals (RQ4); teacher instructional practice and 

principal instructional leadership (RQ5);

These analyses will be carried out using hierarchical linear modeling where appropriate to take 

into account nesting (e.g., the nesting of students and teachers within schools) and will 

incorporate covariates measured at baseline to maximize precision. To 

bias, the impact analyses will employ an “intent

teachers, and principals in all randomly assigned schools during the 2012

included in the analyses, whether or not the 

intended teacher and leader evaluation system or participated to the full extent expected. This 

approach is explained further in Part

The final report will be a capstone report, focusing mainly on the cumulative impacts of 

providing the intervention for two years (i.e., 2012

those in the first report and will also 

Additionally, using AIR’s internal project

each component of the intervention: the component for feedback on student growth, the 

component for feedback on instructional practice,

leadership. However, the AIR project

principals; therefore, we will compute principal costs using items from the principal survey.

17. Approval to Not Display OMB Expiration Date

Approval is not being requested; all data collection instruments will include the OMB expiration 

date. 

18. Explanation of Exceptions

No exceptions are requested. 

Impact Evaluation of Teacher and Leader Evaluation Systems—22

techniques, work differently, or pursue learning experiences; RQ3) 

mobility of teachers and principals (RQ4); teacher instructional practice and 

instructional leadership (RQ5); and student achievement (RQ6)

These analyses will be carried out using hierarchical linear modeling where appropriate to take 

into account nesting (e.g., the nesting of students and teachers within schools) and will 

incorporate covariates measured at baseline to maximize precision. To avoid potential selection 

bias, the impact analyses will employ an “intent-to-treat” approach, in which all 

and principals in all randomly assigned schools during the 2012–13 school year are 

included in the analyses, whether or not the teachers or principals actually participated in the 

intended teacher and leader evaluation system or participated to the full extent expected. This 

Part B.

The final report will be a capstone report, focusing mainly on the cumulative impacts of 

providing the intervention for two years (i.e., 2012–13 and 2013–14). The analyses will parallel 

also have impact results for teacher instructional practice. 

using AIR’s internal project-cost tracking system, we will document 

each component of the intervention: the component for feedback on student growth, the 

component for feedback on instructional practice, and the component for feedback on principal 

However, the AIR project-cost tracking system will not capture hours spent by school 

principals; therefore, we will compute principal costs using items from the principal survey.

Display OMB Expiration Date

Approval is not being requested; all data collection instruments will include the OMB expiration 

18. Explanation of Exceptions

mobility of teachers and principals (RQ4); teacher instructional practice and 

RQ6)

These analyses will be carried out using hierarchical linear modeling where appropriate to take 

into account nesting (e.g., the nesting of students and teachers within schools) and will 

avoid potential selection 

treat” approach, in which all students, 

13 school year are 

teachers or principals actually participated in the 

intended teacher and leader evaluation system or participated to the full extent expected. This 

The final report will be a capstone report, focusing mainly on the cumulative impacts of 

14). The analyses will parallel 

r instructional practice. 

we will document the costs of 

each component of the intervention: the component for feedback on student growth, the 

and the component for feedback on principal 

cost tracking system will not capture hours spent by school 

principals; therefore, we will compute principal costs using items from the principal survey.

Approval is not being requested; all data collection instruments will include the OMB expiration 
Impact Evaluation of Teacher and Leader Evaluation Systems

Aud, S., Hussar, W., Kena, G., Bianco, K., Frohlich, L., Kemp, J., et al. (2011). 

education 2011 (NCES 2011

National Center for Education Statistics. Retrieved from 

http://nces.ed.gov/programs/coe/pdf/coe_msl.pdf 

Bill & Melinda Gates Foundation. (2011). 

measures of effective teaching project. 

http://www.metproject.org/downloads/Preliminary_Findings

Condon, C., & Clifford, M. (2010). 

commonly used principal performance assessment instruments?

Point Associates.

Danielson, C. (2010). Evaluations that help teache

Goe, L., Bell, C., & Little, O. (2008). 

Washington, DC: National Comprehensive Center for Teacher Quality.

Goe, L., Biggers, K., & Croft, A. (2011, May). 

development: Focusing on improving teaching and learning. 

Comprehensive Center for Teacher Quality Conference, Washington, DC.

Goldring, E., Carvens, X., Murphy, J., Porter, A., Ellio

of principals: What and how do states and urban districts assess leadership? 

School Journal, 110(1), 19

Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school 

effectiveness: 1980–1995. 

Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2010). 

perspective: How well does each state do at producing high

Report No. 10-19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State 

and Local Government, Program on Education Policy and Governance & Education Next. 

Retrieved from http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG10

19_HanushekPetersonWoess

Leithwood, K., Harris, A., & Strauss, T. (2010). 

leaders transform low-performing public schools.

Leithwood, K., & Jantzi, D. (2005). A review of transformational school lead

1996–2005. Leadership and Policy in Schools, 4

Impact Evaluation of Teacher and Leader Evaluation Systems—23

References

Aud, S., Hussar, W., Kena, G., Bianco, K., Frohlich, L., Kemp, J., et al. (2011). The condition of 

(NCES 2011-033). Washington, DC: U.S. Department of Education, 

National Center for Education Statistics. Retrieved from 

grams/coe/pdf/coe_msl.pdf 

Bill & Melinda Gates Foundation. (2011). Learning about teaching: Initial findings from the 

measures of effective teaching project. Seattle, WA: Author. Retrieved from 

http://www.metproject.org/downloads/Preliminary_Findings-Research_Paper.pdf

Condon, C., & Clifford, M. (2010). Measuring principal performance: How rigorous are 

commonly used principal performance assessment instruments? Naperville, IL: Learning 

Danielson, C. (2010). Evaluations that help teachers learn. Educational Leadership, 68

Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness. 

Washington, DC: National Comprehensive Center for Teacher Quality.

Goe, L., Biggers, K., & Croft, A. (2011, May). Linking teacher evaluation to professional 

development: Focusing on improving teaching and learning. Presentation at the National 

Comprehensive Center for Teacher Quality Conference, Washington, DC.

Goldring, E., Carvens, X., Murphy, J., Porter, A., Elliott, S., & Carson, B. (2009). The evaluation 

of principals: What and how do states and urban districts assess leadership? 

(1), 19–39. 

Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school 

1995. School Effectiveness and School Improvement, 9

Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2010). U.S. math performance in global 

perspective: How well does each state do at producing high-achieving students?

19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State 

and Local Government, Program on Education Policy and Governance & Education Next. 

http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG10-

19_HanushekPetersonWoessmann.pdf

Leithwood, K., Harris, A., & Strauss, T. (2010). Leading school turnarounds: How successful 

performing public schools. San Francisco: JosseyLeithwood, K., & Jantzi, D. (2005). A review of transformational school leadership research 

Leadership and Policy in Schools, 4(3), 177–199.

The condition of 

033). Washington, DC: U.S. Department of Education, 

Learning about teaching: Initial findings from the 

Seattle, WA: Author. Retrieved from 

earch_Paper.pdf

Measuring principal performance: How rigorous are 

Naperville, IL: Learning 

Educational Leadership, 68(4), 35–39.

Approaches to evaluating teacher effectiveness. 

Linking teacher evaluation to professional 

Presentation at the National 

Comprehensive Center for Teacher Quality Conference, Washington, DC.

tt, S., & Carson, B. (2009). The evaluation 

of principals: What and how do states and urban districts assess leadership? Elementary 

Hallinger, P., & Heck, R. H. (1998). Exploring the principal’s contribution to school 

School Effectiveness and School Improvement, 9(2), 157–191.

U.S. math performance in global 

achieving students? (PEPG

19). Cambridge, MA: Harvard Kennedy School, Taubman Center for State 

and Local Government, Program on Education Policy and Governance & Education Next. 

Leading school turnarounds: How successful 

-Bass.

ership research 
Impact Evaluation of Teacher and Leader Evaluation Systems

Leithwood, K., Louis, K. S., Anderson, S., & Wahlstrom, K. (2004). 

student learning. St. Paul: University of Minnesota, Center for Applied Research and 

Educational Improvement.

National Center for Education Statistics. (2010a). 

(NCES 2012–457). Washington, DC: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics.

http://nces.ed.gov/nationsreportcard/pdf/main2011/2012457.pdf

National Center for Education Statistics. (2010b). 

(NCES 2012–458). Washington, DC: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics. Retrieved from 

http://nces.ed.gov/nationsreportcard/pdf/main2011/2012458.pdf

Schochet, P. Z., & Chiang, H. S. (2010)

performance based on student test score gains 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance. R

http://eric.ed.gov/PDFS/ED511026.pdf

U.S. Department of Education. (2010). 

Elementary and Secondary Education Act

http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf 

Waters, T., Marzano, R. J., & McNulty, B. (2003). 

research tells us about the effect of leadership on student achie

continent Research for Education and Learning. Retrieved from 

http://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade

rship.pdf

Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). 

failure to acknowledge and act on differences in teacher effectiveness.

Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf

Impact Evaluation of Teacher and Leader Evaluation Systems—24

Leithwood, K., Louis, K. S., Anderson, S., & Wahlstrom, K. (2004). How leadership influences 

. St. Paul: University of Minnesota, Center for Applied Research and 

ducational Improvement.

National Center for Education Statistics. (2010a). The nation’s report card: Mathematics 2011

457). Washington, DC: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics. Retrieved from 

http://nces.ed.gov/nationsreportcard/pdf/main2011/2012457.pdf

National Center for Education Statistics. (2010b). The nation’s report card: Reading 20011

458). Washington, DC: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics. Retrieved from 

http://nces.ed.gov/nationsreportcard/pdf/main2011/2012458.pdf

Schochet, P. Z., & Chiang, H. S. (2010). Error rates in measuring teacher and school 

performance based on student test score gains (NCEE 2010-4004). Washington, DC: U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance. Retrieved from 

http://eric.ed.gov/PDFS/ED511026.pdf

U.S. Department of Education. (2010). A blueprint for reform: The reauthorization of the 

Elementary and Secondary Education Act. Washington, DC: Author. Retrieved from 

http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf 

Waters, T., Marzano, R. J., & McNulty, B. (2003). Balanced leadership: What 30 years of 

research tells us about the effect of leadership on student achievement. Aurora, CO: Mid

continent Research for Education and Learning. Retrieved from 

http://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade

Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: Our national 

failure to acknowledge and act on differences in teacher effectiveness. The New Teacher 

Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf

How leadership influences 

. St. Paul: University of Minnesota, Center for Applied Research and 

The nation’s report card: Mathematics 2011

457). Washington, DC: U.S. Department of Education, Institute of 

Retrieved from 

The nation’s report card: Reading 20011

458). Washington, DC: U.S. Department of Education, Institute of 

Education Sciences, National Center for Education Statistics. Retrieved from 

Error rates in measuring teacher and school 

4004). Washington, DC: U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

A blueprint for reform: The reauthorization of the 

. Washington, DC: Author. Retrieved from 

Balanced leadership: What 30 years of 

Aurora, CO: Midhttp://www.mcrel.org/pdf/LeadershipOrganizationDevelopment/5031RR_BalancedLeade

Our national 

The New Teacher 

Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf