Submitted by:

Institute of Education Sciences

IES/NCEE

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

Project Officer: Elizabeth Warner

An Impact Evaluation of the 

Teacher Incentive Fund (TIF)

Part A

June 10, 2014
ii

CONTENTS

PART A. JUSTIFICATION ................................................................................................2

1. Circumstances Necessitating the Collection of Information......................2

2. Purposes and Uses of Data......................................................................7

3. Use of Technology to Reduce Burden.......................................................9

4. Efforts to Avoid Duplication of Effort.....................................................10

5. Methods to Minimize Burden on Small Entities.......................................10

6. Consequences of Not Collecting Data....................................................10

7. Special Circumstances...........................................................................11

8. Federal Register Announcement and Consultation.................................11

9. Payments or Gifts..................................................................................12

10. Assurances of Confidentiality ................................................................12

11. Justification for Sensitive Questions.......................................................14

12. Estimates of Hours Burden ....................................................................14

13. Estimates of Cost Burden to Respondents..............................................15

14. Annualized Costs to the Federal Government ........................................15

15. Reasons for Program Changes or Adjustments ......................................16

16. Plans for Tabulation and Publication of Results .....................................16

17. Approval Not to Display the OMB Expiration Date..................................19

18. Explanation of Exceptions .....................................................................19

REFERENCES ...............................................................................................................20

APPENDIX A: DISTRICT SURVEY

1. District Letter

2. District Questionnaire

APPENDIX B: PRINCIPAL SURVEY

1. Principal Letter

2. Principal Questionnaire

APPENDIX C: TEACHER SURVEY

1. Teacher Letter

2. Teacher Questionnaire

APPENDIX D: PRINCIPAL AND TEACHER ADMINISTRATIVE DATA REQUEST LETTER

APPENDIX E: STUDENT RECORDS DATA COLLECTION

1. Cover letter

2. Instructions for Providing Student Records

APPENDIX F: DISTRICT INTERVIEW PROTOCOL

APPENDIX G: CONFIDENTIALITY PLEDGE
iii

TABLES

Table 1. Data Collection Needs.................................................................................... 4

Table 2. Schedule of Major Study Activities.................................................................. 7

Table 3. Research Questions and Data Collection Methods .......................................... 8

Table 4. Technical Working Group Members.............................................................. 11

Table 5. Estimated Response Time for Data Collection .............................................. 15
1

SUPPORTING STATEMENT FOR PAPERWORK 

REDUCTION ACT SUBMISSION

This package requests clearance from the Office of Management and Budget (OMB) to 

continue data collection activities for a rigorous evaluation of the Teacher Incentive Fund (TIF). 

This evaluation includes TIF grantees who were awarded funds from the American Recovery and 

Reinvestment Act (ARRA) of 2009 and the U.S. Department of Education’s (ED) fiscal year (FY) 

2010 appropriation. The Institute of Education Sciences (IES), within ED, began the evaluation 

under contract with Mathematica Policy Research and its partners Chesapeake Research Associates 

and faculty and staff at the Peabody College of Education at Vanderbilt University to conduct the 

evaluation. A follow-up contract is scheduled to be awaded by September 2014 to complete the 

evaluation effort.

The main objective of the evaluation is to estimate the impact of differentiated performancebased incentive pay (DPBIP)1

on student achievement and the mobility and retention of teachers 

and principals. The evaluation design is an experiment in which researchers randomly assigned

schools within a district to either a treatment or control group. The treatment schools implemented

educator DPBIP as part of a performance-based compensation system (PBCS). Control schools 

implemented the same non-differentiated components of the PBCS program and a one percent 

across-the-board bonus, but did not implement any type of DPBIP for the duration of the TIF 

grant. We will compare student achievement and other outcomes between the treatment and control 

schools to estimate the impact of DPBIP compared to the one percent bonus.

The Notice of Final Priorities (NFP) for the TIF 2010 grants, published in the Federal Register on 

May 21, 2010, announced two competitions for grants to be awarded in 2010—the TIF main 

competition and the TIF evaluation competition; applicants applied to one or the other competition. 

The evaluation competition resulted in 12 districts participating in the random assignment 

evaluation.

This submission requests clearance to continue data collection for a fourth year (during the fifth 

and final year of the TIF cohort 3 grant) using instruments (with minor additions) previously cleared 

under OMB # 1850-0876. This package requests clearance to collect data that will support

completion of the full-scale study during the fifth and final year of the TIF 2010 grants. The minor 

additions includes seven new questions in section D of the district survey, four new questions in the 

principal survey (question D13, and questions 1 through 3 in section F), three new questions in the 

teacher survey (question E11 and questions 1 and 2 in section F), and five new questions in the 

district interview (question 3, 21, and questions 36-38 in section G). 

 

1 For this document, DPBIP refers to the differentiated incentive pay portion of a grantee’s PBCS. DPBIP 

programs provide bonuses for highly effective teachers and principals, where effectiveness is based on student 

achievement growth, observations, and any other criteria included in the district’s PBCS.
2

Part A. Justification

1. Circumstances Necessitating the Collection of Information

a. Statement of Need for a Rigorous Evaluation of TIF

The specific legislation necessitating this data collection is the ARRA, Division A, Title VIII, 

Pub. L. 111–5 and Departments of Labor, Health and Human Services, and Education, and Related 

Agencies Appropriations Act, 2010, Division D, Title III, Pub. L. 111–117. The ARRA requires that 

ED, to the extent possible, conduct a rigorous national evaluation to assess the impact of PBCS, 

supported by ARRA funds, on student achievement and educator recruitment and retention in highneed schools and hard-to-staff subjects. This evaluation would meet this requirement. 

Local educational agencies (LEAs) use TIF grants to implement performance-based teacher and 

principal compensation systems in high-need schools. ARRA requires that the funding be used to 

promote effective school reform in several priority areas. These priorities include increasing teacher 

effectiveness, achieving equity in the distribution of high-quality teachers, and turning around the 

lowest performing schools. TIF requirements address these priorities. 

Teacher quality is a critical input to student learning, but little is known about how to develop a 

strong teacher workforce (Rivkin et al. 2005; Rockoff 2004). Researchers have examined strategies to 

identify, attract, retain, and develop good teachers, including alternative preparation (Decker et al. 

2004; Constantine et al. 2009), certification (Tuttle et al. 2009), and in-service training and 

professional development (Glazerman et al. 2006, Garet et al. 2008, Yoon et al. 2007). However, 

little is known about incentive compensation programs that tie teacher pay to student performance. 

Do these programs boost student achievement by attracting and retaining effective teachers and 

motivating all teachers to improve performance? Which types—for example, school- or individualbased programs or mixed programs (a combination of the two)—are most effective? And what 

challenges do districts face in implementing these programs?

To assess the overall effectiveness of TIF projects and the effectiveness of particular program 

models and features, ED has contracted for an evaluation of DPBIP that will be implemented by the 

2010 grant recipients. This evaluation will provide important evidence on how changes to the 

traditional compensation systems for educators may be able to (1) improve student performance in 

high-need schools and/or (2) bring about desirable changes, such as the presence of more highly 

effective educators in high-need schools. Results of this evaluation will provide educators, 

policymakers, and researchers with critical information on educator compensation reform, the effect 

of performance-based educator compensation on student achievement, and other aspects of PBCSs 

associated with student achievement. 

b. Research Questions

The study’s research questions are:

 What is the effect on student achievement of a performance based bonus compared to an 

across-the-board 1% annual bonus? 

 Are there differences in the composition and effectiveness of teachers and principals 

between these two methods of paying teachers and principals? Are there any differential 

effects on recruitment and retention of teachers and principals? 
3

 Is a particular type of performance based bonus model—for example, school- or individualbased or mixed programs—associated with greater gains in student achievement? Are other 

key program features correlated with student and educator outcomes? 

 What are the experiences and challenges of districts when implementing these programs? 

c. Study Design

To answer the first research question, this study uses an experimental design—study schools 

within a district were randomly assigned to either a treatment or control group. Random assignment 

is considered the “gold standard” for social policy evaluations. More than any other approach, it 

minimizes the chance that any observed differences in outcomes between the study groups are due 

to unmeasured, pre-existing differences between members of these groups. In the random 

assignment design, the simple difference between outcomes in treatment and control schools within 

each district is an unbiased estimate of the impact of the district’s DPBIP component. 

Both treatment and control schools implemented the same non-DPBIP components of their 

program. However, only treatment schools include a DPBIP component, while control schools 

provide an across-the-board one percent educator bonus. Control schools are not permitted to 

implement a DPBIP component for the duration of the TIF grant. 

Treatment schools must implement both teacher and principal DPBIP components that 

measure effectiveness using gains in student academic achievement and classroom evaluations 

conducted multiple times during each school year. Teacher incentive models may be individualbased, group-based, or mixed models. 

Since we will not randomly assign schools to specific program features (program features differ 

among grantees), the study will use nonexperimental analyses to address the other research 

questions. To the extent possible, the study will examine the correlation between different types of 

DPBIP models and student and educator outcomes. The ability to separately analyze different 

DPBIP models will depend on the number and type of model(s) implemented by the grantees. The 

study will also examine the association of other key program features, such as how heavily the 

DPBIP model weights growth in student achievement with student achievement and educator 

outcomes. 

The study includes approximately 175 schools and their students. It is designed to detect 

student achievement gains of 0.11 of a standard deviation. Although this may be a larger effect than 

can be obtained in the first year or two of the program, if DPBIP is effective in retaining and 

attracting effective teachers as well as improving performance among all teachers, improvement in

student achievement should increase over time as educators observe bonuses received by colleagues. 

In addition, relatively small gains could be realized each year, contributing to larger effects after 

three or four years of implementation. 

As part of the continued evaluation, and to address the research questions, the contractor will:

 Collect principal and teacher contact information for the study team to contact 

respondents who may change schools during the course of the study.

 Collect student records data to estimate the impact of DPBIP on student achievement.
4

 Collect administrative data on principals and teachers to track their mobility and 

recruitment. 

 Use principal and teacher surveys to describe their understanding of and experiences 

with DPBIP, supplement district mobility data, and obtain background information.

 Use district surveys and interviews to describe experiences and challenges of districts 

when implementing the incentive programs.

d. Data Collection Needs

This study includes several data collection efforts, described below and summarized in Table 1 

below. Data will be collected from the districts and schools participating in the evaluation.

District survey. We will administer a survey in winter 2015 to all 2010 TIF main and evaluation 

districts (Appendix A). The survey will focus on districts’ experiences over the longer period and 

their plans for sustaining the incentive policies. The survey seeks to contrast how the districts’ 

programs were planned, implemented, and sustained. This survey has been informed by data 

collection efforts collected under the prior OMB approved data collection (OMB # 1850-0876).

We will mail the 30-minute hard copy questionnaire to each district representative. The mailing 

will contain a cover letter and district questionnaire. The letter, which will be on ED stationary and 

signed by an ED official, will describe the study and its objectives and the need for districts’

participation, address issues of confidentiality, and provide a senior study member’s contact 

information for questions or concerns. Districts will be asked to complete a hard copy questionnaire 

and mail it to the contractor selected for ED-IES-14-R-0019 in a postage-paid envelope.

Table 1. Data Collection Needs

Instrument Data Need Respondent Mode Schedule

District 

questionnaire

Specific program features,

changes made to program, and 

how district obtained 

buy-in 

District staff Hard copy, phone 

follow-up

Winter 2015

Principal 

questionnaire

Background characteristics,

mobility, and knowledge and 

perceptions of incentives

Principals Web with email,

hard copy and 

phone follow-up

Spring 2015

Teacher 

questionnaire

Background characteristics,

mobility, and knowledge and 

perceptions of incentives

Teachers Web with email,

hard copy and 

phone follow-up

Spring 2015

Principal and teacher 

administrative data 

letter

Educator retention, school 

assignment, background 

characteristics, standardized test 

scores

District staff Electronic or hard 

copy

Summer/fall 2015 

Student 

administrative 

records letter

Reading and math standardized 

test score data for current and 

prior school year

Demographic and socioeconomic 

characteristics

District staff Electronic or hard 

copy

Summer/fall 2015 

District interview 

protocol

Detailed information on program, 

implementation experiences, and 

other school improvement efforts 

District staff Telephone semistructured 

interviews

Spring 2015 
5

Principal survey. A 30-minute web-based survey will be administered to all principals in spring 

2015 (Appendix B). The principal survey will ask about their background characteristics, mobility, 

the school’s hiring practices, and knowledge and perceptions of incentives. 

Teacher survey. Administered to a sample of teachers, the teacher survey (Appendix C) will be 

similar to the principal survey regarding mode of administration and length of questionnaire. As with 

principals, we will administer surveys to the same teachers as in prior data collection efforts even if 

they have left the school, as well as new teachers in study schools. The survey will collect 

information on teachers’ educational and professional background, professional development 

experiences, teaching and leadership responsibilities, satisfaction with various aspects of their 

schools, salary and other sources of compensation, and understanding of their school’s PBCS. 

For both principal and teacher surveys, we will first contact the sample members by email or 

cover letter (if email is not available or invalid). The initial correspondence will include a description 

of the study and survey, a link to the website address and instructions on accessing the survey, and a 

unique username and password. The email will explain the importance of participation, address 

confidentiality, and provide a toll-free telephone number and email address for questions or 

concerns. Nonrespondents, whom we will contact by email, telephone, or a remailing, will have the 

additional option of providing answers either over the telephone or by completing a hard copy 

version of the questionnaire. 

Principal and teacher administrative data. In fall 2015, we will collect data from districts on 

the hiring, movement between schools, and attrition of principals and teachers participating in the 

study. We will also attempt to obtain information about the start and end dates of school 

assignments for these staff, as well as any available background characteristics such as age, sex, 

race/ethnicity, certifications, degrees, years of teaching experience, and scores on licensure or 

certification tests. In addition, we will collect several indicators of teacher and principal effectiveness 

and data on the actual payouts received by staff in recognition of their accomplishments. We will 

collect these data by the following means:

 Annual listings of principals and teachers (with personnel ID code, school, and grade if 

applicable) who are eligible for performance pay and the maximum amounts for which 

they are eligible.

 Annual listings of principals and teachers (with personnel ID code, school, and grade if 

applicable) who actually receive performance pay and the amounts that they receive.

 Annual data on performance measures received by principals and teachers in treatment 

and control schools (with personnel ID code, school, and grade if applicable). To the 

extent possible, performance measures should be separated into those based on 

observations of classroom or school practices, student achievement and growth, and 

other performance criteria.

Although we prefer to receive the data in an electronic format, we will use data in whatever 

form is most convenient for each district. We will send letters to the districts, specifying the specific 

data elements requested (Appendix D).

Student records data. We will request standardized math and reading test scores for all 

students in study schools in Summer/Fall 2015. We will also request scores from the year prior to 

the current study year if those scores have not been previously obtained. In addition to test scores, 
6

we will request that the district data on student characteristics such as sex, race/ethnicity, date of 

birth, grade, whether they are repeating a grade, eligibility for free- or reduced-price lunch, English 

language learner status, and mobility within the district. Where possible, we will also request student 

achievement scores in math and reading, linked to the appropriate teacher. We will send the district 

a letter specifying the data requested (Appendix E).

District interviews. In spring 2015, we will conduct semi-structured telephone interviews with 

a district official who is familiar with the TIF evaluation grant program. The interview protocol is 

designed to collect detailed information on each district in a format that will allow for standardized 

follow-up questions depending on the response given to a specific item. The interview will address 

topics such as program implementation experiences, other ongoing school improvement efforts, and 

plans for performance-based pay. The protocol for the administration in 2015 is included in

Appendix F.

e. Study Activities and Data Collection Timeline

This clearance request pertains to the administration of the district survey (Appendix A), 

principal survey (Appendix B), and teacher survey (Appendix C); collection of the district 

administrative records on principals, teachers, and students in the study (Appendices D and E); and 

administration of a district interview (Appendix F). The evaluation will be completed in eight years. 

Table 2 shows the schedule of data collection activities and the overall evaluation timeline.
7

Table 2. Schedule of Major Study Activities (including those previously conducted under OMB # 

1850-0876)

Activity Fall 2011

Spring 2012Winter 2013Spring 2013Fall 2013

Winter 2014Spring 2014

Fall 2014

Winter 2015

Spring 2015Fall 2015Spring 2016Spring 2017

Provide 

technical 

assistance to 

grantees

X X X X X X X X

X X

Collect 

principal and 

teacher 

contact 

information

X X X X

Conduct 

district survey

X X X X

Conduct 

principal 

survey

X X X X

Conduct 

teacher survey

X X X

Collect 

principal and 

teacher 

records data 

from districts

X X X X X

Collect student 

records data 

from districts

X X X X

Conduct 

district

interviews 

X X X

Prepare first 

report

X

Prepare 

second report

X

Prepare third 

report 

 X

Prepare fourth 

report 

X

NOTE: Bolded X signals data collection for this OMB request.

2. Purposes and Uses of Data

Data for the evaluation of TIF programs will be collected and analyzed by the contractor 

selected under ED-IES-14-R-0019. The data to be collected will be obtained from participants’ 

contact information, district administrative records, TIF district interviews, and surveys of teachers, 

principals and districts. The data will be used to address the research questions as shown in Table 3.
8

Table 3. Research Questions and Data Collection Methods

Research Question Data Sources

1. What is the effect on student achievement 

of a performance based bonus compared to an 

across-the-board 1% annual bonus? 

 District administrative records

 Principal survey

 Teacher survey

2. Are there differences in the composition 

and effectiveness of teachers and principals 

between these two methods of paying teachers 

and principals? Are there any differential effects 

on recruitment and retention of teachers and 

principals? 

 District administrative records

 Principal survey

 Teacher survey

 District survey

 District interviews

3. Is a particular type of performance based 

bonus model—for example, school- or 

individual-based or mixed programs—associated 

with greater gains in student achievement? Are 

other key program features correlated with 

student and educator outcomes? 

 District administrative records

 Principal survey

 Teacher survey

 District survey

 District interviews

4. What are the experiences and challenges 

of districts when implementing these programs?  District survey 

 Principal survey

 Teacher survey

 District interviews

 District survey. We will use the data from the district survey to examine the association 

between impacts and key program features. Data from the survey will be used to 

describe districts’ experiences since implementing the TIF program and ascertain their 

plans for sustaining the program. Data from the district surveys will be used to answer 

research questions 2, 3, and 4.

 Principal survey. The principal survey will be used to assess hiring practices, classroom 

assignments, knowledge and perceptions of the TIF program in the study schools, how 

this may change over time, and to supplement administrative data to be obtained from 

district records. The principal survey can also provide important insight on their 

motivation for remaining, leaving, or entering a study school. Data from the principal 

survey will be used to answer research questions 1, 2, 3, and 4.

 Teacher survey. The teacher survey will be used to assess knowledge and perceptions 

of the PBCS in the study schools and how this may change over time, and to supplement 

administrative data to be obtained from district records. The teacher survey can also 

provide important insight on teachers’ motivation for remaining, leaving, or entering a 

study school. Data from the teacher survey will be used to answer research questions 1, 

2, 3, and 4.
9

 Principal and teacher administrative data. These data will be used to estimate the 

impacts of DPBIP on educator mobility and recruitment. The data will also allow us to 

examine the association between educator characteristics and student and educator

outcomes, and to describe the educator sample. These data will be used to answer 

research questions 1, 2 and 3.

 Student records data. We will use existing state or district test score data to estimate 

the impact of DPBIP on student achievement, the key outcome of interest. Information 

on students’ demographic and socioeconomic characteristics and their achievement test 

scores prior to the study school year will be used to describe the students in the study 

and to develop more precise impact estimates. To the extent possible, we will use 

student-teacher linked data to estimate teachers’ value-added score to better understand 

mobility of high- and low-performing educators. Data obtained from student records 

will be used to address research questions 1, 2 and 3.

 District interview. The semi-structured district interviews will allow us to collect more 

in-depth information than that collected from the survey, and to follow up for 

clarification if necessary. We will use this detailed information to more thoroughly 

understand each program’s context, implementation strategy, and challenges. Data from 

the district interviews will be used to answer research questions 2, 3, and 4.

The overall purpose of this evaluation is to estimate the impacts of DPBIP on student 

achievement and educator mobility and recruitment in high-need schools. The findings from this 

study will provide important evidence for school districts and policymakers on the impacts of 

DPBIP on students, teachers, and school principals. If possible, this evaluation may provide 

policymakers and school districts with valuable information on the relative effectiveness of 

individual-based versus group-based compensation systems. The study will also provide important 

insight into the impacts of other key program aspects of DPBIP models, as well as how districts may 

overcome common implementation challenges. Study findings will be presented in four annual 

reports, beginning fall 2014. In addition, the data collected by the evaluation will be available as 

restricted-use data files that will serve as a valuable resource for other researchers.

3. Use of Technology to Reduce Burden

The data collection plan is designed to obtain reliable information in an efficient way that 

minimizes respondent burden. We will set up a toll-free telephone number and email address 

specific to the study so that participants with questions can easily contact the research team. As 

much information as possible will be gathered from existing data sources, such as TIF grant 

application packets submitted by awardees and electronic files provided by districts. If it is too 

burdensome or not possible for a district to provide data in electronic format, we will provide clear 

instructions on how to submit copies of the relevant information in hard copy form, to be coded by 

the study team. Some data, however, can only be obtained directly from principals, teachers, and 

districts.

A web-based survey will be the primary mode of data collection for teachers and principals in 

the study. Respondents will also have the option of completing a self-administered hard copy 

questionnaire or providing answers to a trained interviewer over the telephone. The web-based 

survey will enable respondents to complete the survey at a location and time of their choice, and its 

automatic editing system will reduce the number of response errors.
10

For participants who do not return contact forms, or those whose email addresses are invalid, 

we will search school or district websites to obtain email addresses. Using email to follow up with 

nonrespondents will also offer an additional convenient option for respondents. Email reminders 

will include a link to the survey website and a username-password combination, as well as an 

attached PDF of the survey if respondents choose to complete a hard copy version.

A district representative familiar with the TIF program will complete questionnaires in hard 

copy form. For nonresponse follow-up, we will also offer respondents the opportunity to complete 

the survey over the telephone with a trained telephone interviewer. The study team considered other 

modes of administering the district survey, such as computer-assisted telephone interview (CATI) or 

a web-based survey. However, because of the relatively small sample size, the predicted cost of 

developing these methods outweighed the expected benefits.

We will conduct the district interviews by telephone. This mode of data collection is 

appropriate for the conversational exchange necessary to obtain answers to the open-ended 

questions, and to allow probing for more detail than a self-administered survey can provide. 

4. Efforts to Avoid Duplication of Effort

The data collection plan avoids unnecessary collection of information by building off of work 

being conducted under ED-04-CO-0112/0012.

5. Methods to Minimize Burden on Small Entities

The primary entities for the study are TIF school districts, schools, principals, and teachers. We 

will minimize burden for all respondents by requesting only the minimum data required to meet 

study objectives. Burden on respondents will be further minimized through the careful specification 

of information needs. We will also keep our data collection instruments short and focused on the 

data of most interest, and we will speak with relatively few respondents in person. Sample sizes and 

data requirements for each respondent group were determined by careful consideration of the 

information needed to meet the study objectives, and were reviewed by the study’s technical 

working group (TWG).

6. Consequences of Not Collecting Data

The data collection plan described in this submission is necessary for ED to conduct a rigorous 

national evaluation of the TIF and to understand the effectiveness of this education reform strategy. 

Collecting these data will allow us to finish examining the range of performance-based 

compensation systems and to answer pressing policy questions about how DPBIP affects student 

achievement and how grant recipients design, communicate, and implement TIF programs over the 

full course of the grants.

The consequences of not collecting specific data are outlined below.

 Each wave of the district survey targets different aspects of the program: specific 

features of districts’ PBCS, if and how these features changed over time, how districts 

obtained buy-in, their experiences, and plans to continue their incentive policies. 

Without administering the district survey, and in multiple waves, we will not be able to 
11

capture these key program features and their impact on student achievement and 

educator mobility.

 Without the principal and teacher surveys, we will not know if educators understood 

the incentive policies, if their choice to stay in, move to, or move from a school was 

motivated by the incentives. We will also be unable to examine schools’ hiring practices 

and classroom assignments, two factors that may be influenced by the TIF program. 

Impacts in the second and subsequent years of the implementation of the DPBIP may 

be larger than those in the first year. Administering the surveys in multiple waves will 

allow us to examine educators’ experiences and perceptions of the programs over time. 

 Without principal and teacher records data, it will be more difficult to verify 

educators’ school assignment and track their mobility. Furthermore, without this data we 

will not be able to compare characteristics between principals and teachers in the 

treatment and control schools, or to examine whether staff characteristics are associated 

with student achievement growth or eductors’ mobility decisions. 

 Without student records data, we will have to administer assessments to students in 

place of using their district or state math and reading test scores to measure student 

achievement. Without the data on student characteristics, we will not be able to fully 

describe the study sample and verify the effectiveness of the random assignment. 

 Without the district interviews, we will not be able to follow up on information 

obtained from the surveys to obtain a more thorough understanding of the districts’

programs and experiences, or to fully understand any other related school reform 

initiatives within the district that may affect the impact of DBPIP in the study schools. 

Multiple waves are necessary as a detailed follow-up to each district survey.

7. Special Circumstances

There are no special circumstances associated with this data collection.

8. Federal Register Announcement and Consultation

a. Federal Register Announcement

The 60-day notice to solicit public comments will be published. The 30-day notice will be 

published to solicit additional public comments.

b. Consultation Outside the Agency

In executing the evaluation design, the study team has sought input from the technical working 

group (TWG), which includes some of the nation’s experts in teacher compensation, evaluation 

methodology, and education policy. We will continue to consult with the TWG throughout the 

study on other issues that would benefit from their input. Table 4 lists the TWG members.

Table 4. Technical Working Group Members

Name Title and Affiliation Expertise

Anthony Milanowski Assistant Scientist, University of Wisconsin Teacher compensation

Richard Murnane Professor of Education, Harvard Graduate School Teacher compensation and teacher 
12

of Education quality

Jacob Vigdor Professor of Public Policy and Economics, Duke 

University

Teacher compensation, teacher quality,

and evaluation methodology

Dan McCaffrey Senior Statistician, RAND Corporation Value added and evaluation 

methodology

Robert Meyer Research Professor, University of Wisconsin Value added

Jeffrey Smith Professor of Economics, University of Michigan Teacher quality/methodology

James Kemple Director of Research Alliance for NY City Schools, 

Research Professor, New York University

Teacher quality/methodology

David Heistad Executive Director of Research, Evaluation and 

Assessment, Minneapolis Public Schools

Program evaluation, value-added in 

teacher compensation systems

Carla Stevens Assistant Superintendent, Research and 

Accountability, Houston Independent School 

District

Accountability, student assessment, 

program evaluation, and performance 

pay models 

c. Unresolved Issues

There are no unresolved issues.

9. Payments or Gifts 

Incentives for principals and teachers. Incentives proposed here are the same as those 

cleared under OMB # 1850-0876. Specifically we propose incentives for the principal and teacher 

surveys to partially offset respondents’ time and effort in completing the surveys. We propose 

offering a $20 incentive to an educator each time he or she completes a questionnaire so as to 

acknowledge the 30 minutes required to complete each questionnaire. This proposed amount is 

within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for 

NCEE Evaluation Studies,” prepared for OMB. 

Incentives are also proposed because high response rates are needed to make the survey 

findings reliable, and we are aware that teachers and principals are the targets of numerous requests 

to complete surveys on a wide variety of topics from state and district offices, independent 

researchers, and the Department of Education. Although some districts will have solicited buy-in 

from teachers to participate in the evaluation, our recent experience with numerous teacher surveys 

supports our view that obtaining teacher buy-in does not guarantee teachers will devote the time it 

takes to complete a survey, and monetary incentives increase the likelihood of cooperation of school 

staff.

The study will not give incentives to districts for completing an interview or a survey, or for 

providing administrative records data.

10. Assurances of Confidentiality

The contractor will be required to conduct all data collection activities for this study in 

accordance with relevant regulations and requirements, which are:

 The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a).
13

 The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR 

Part 99).

 The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98).

 The Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183

The research team will be required to protect the confidentiality of all data collected for the 

study and will use it for research purposes only. The contractor’s project director will be required ot 

ensure that all individually identifiable information about respondents will remain confidential. All 

data will be kept in secured locations and identifiers will be destroyed as soon as they are no longer 

required. All members of the study team having access to the data will be trained and certified on 

the importance of confidentiality and data security. When reporting the results, data will be 

presented only in aggregate form, such that individuals and institutions will not be identified. 

Included in all voluntary requests for data will be the following statement: 

Responses to this data collection will be used only for statistical purposes. The reports 

prepared for this study will summarize findings across the sample and will not associate 

responses with a specific district or individual. We will not provide information that 

identifies you or your district to anyone outside the study team, except as required by law. 

Additionally, no one at your school or in your district will see your responses. While your 

participation in this study is voluntary, it is very important that you complete the 

questionnaire.

For those instruments where data collection is required as a condition of their evaluation grant, 

all grant required requests for data will include the following statement:

Responses to this data collection will be used only for statistical purposes. The reports prepared 

for this study will summarize findings across the sample and will not associate responses with a 

specific district or individual. We will not provide information that identifies you or your district 

to anyone outside the study team, except as required by law. Additionally, no one at your school 

or in your district will see your responses. Participation or cooperation with this activity is a 

condition of your grant (EDGAR: part 75.591, Authority: 20 U.S.C. 1221e–3 and 3474).

The following safeguards are routinely required of contractors for IES to carry out 

confidentiality assurances, and they will be consistently applied to this study: 

 All contractor employees will be required to sign a confidentiality pledge (such as 

Appendix G) that emphasizes the importance of confidentiality and describes employees’ 

obligations to maintain it.

 Personally identifiable information (PII) is maintained on separate forms and files, which 

are linked only by sample identification numbers.

 Access to hard copy documents is strictly limited. Documents are stored in locked files 

and cabinets. Discarded materials are shredded.

 Access to computer data files is protected by secure usernames and passwords, which are 

only available to specific users. 

 Sensitive data is encrypted and stored on removable storage devices that are kept 

physically secure when not in use.
14

The contractor will be required to maintain confidentiality including personnel training 

regarding the meaning of confidentiality, particularly as it relates to handling requests for 

information, and providing assurance to respondents about the protection of their responses. The 

contractor will also be required to build-in safeguards concerning status monitoring and receipt 

control systems.

11. Justification for Sensitive Questions

Some respondents may consider their contact information to be sensitive. This information is 

necessary in order to limit possible sample attrition that could result from respondents changing 

schools or professions. 

The principal and teacher surveys will ask for demographic information (ethnicity, race, year of 

birth) and information about respondents’ educational and professional background. Data on these 

topics are important to help us understand if there is an association between student achievement, 

educator outcomes, and educator characteristics. Questions used to obtain personal background 

information have been asked frequently in other surveys and were pretested for this study under 

OMB # 1850-0876, with the pretest sample of teachers and principals reporting no concerns. 

To address concerns about disclosing personal information, all cover letters and questionnaires 

will clearly state that all responses will be treated as confidential, that participation is voluntary, and 

that failure to provide some or all requested information will not affect the respondent’s 

professional status in any way. The questions will also be worded in a sensitive, nonjudgmental 

manner. 

Some demographic information about the students (for example, qualification for free- or 

reduced-price lunch or special education status) or their test scores may be sensitive. Demographic 

information is important to control for any differences in the characteristics of students in the 

classes that may have arisen by chance. Test score data is essential for this evaluation because 

student achievement is the primary outcome of interest. These scores will be linked to the data file 

by each respondent’s unique, study-generated identification number. After this linking process, 

personal identifiers, such as a student’s name, school identification number, and date of birth, will be 

removed. 

There are no questions of a sensitive nature in the district survey or interview.

12. Estimates of Hours Burden

Table 5 provides an estimate of time burden for the data collections, broken down by 

instrument and respondent. These estimates are based on our experience collecting administrative 

data from districts, administering surveys to school principals and teachers, and conducting 

telephone interviews with district representatives. 
15

Table 5. Estimated Response Time for Data Collection

Respondent/

Data Request

Number of 

Targeted 

Respondents

Expected 

Response 

Rate (%)

Number of 

Respondents

Unit 

Response 

Time 

(Hours)

Total 

Response 

Time 

(Hours/Year)

Total 

Burden 

Time 

(Hours)

Districtsa

Student records 

data 13 100 13 8.0 104 104

Principal and 

teacher records 

data 13 100 13 8.0 104 104

Principals

Principal surveys 175 90 158 0.5 79 79

Teachers

Teacher surveys 2000 85 1700 0.5 850 850

Districts

Surveys 153 80 122 0.5 61 61

Interviews 13 100 13 0.75 10 10

Total 2019 1208

Annual number of respondents and responses for the 3 years of this collection are 673 and the total

annual burden hours for this collection are 403 burden hours.

aDepending on the grantee, administrative records data may be provided by another source, for instance 

the state or grantee.

The total of 1208 hours covers the data collection conducted during the final grant year of the 

evaluation, and includes the following efforts: up to 16 hours for each of the 13 districts to collect 

and assemble administrative records on students, principals, and teachers participating in the 

evaluation; 30 minutes for 158 principals (90 percent of the anticipated 175 principals in the sample) 

to complete the principal survey; 30 minutes for 1,700 teachers (85 percent of the anticipated sample 

of 2000 teachers in the sample) to complete the teacher survey; 30 minutes for 122 district 

representatives (80 percent of the 153 districts participating in the study) to complete a district 

survey; and 45 minutes for the 13 districts participating in the evaluation to complete a telephone 

interview. Annual number of respondents and responses for the 3 years of this collection are 673

(the total number of respondents, 2019, divided by 3) and the total annual burden hours for this 

collection are 403 burden hours (the total annual burden hours, 1208, divided by 3).

13. Estimates of Cost Burden to Respondents

There are no direct costs for respondents.

14. Annualized Costs to the Federal Government

The estimated annual cost of the study to the federal government is $1,714,286. The total cost 

of the seven-year study is $12 million, which includes recruiting grantees, districts, and schools; 

designing and administering data collection instruments; processing and analyzing data; and 

preparing reports.
16

15. Reasons for Program Changes or Adjustments 

There is an overall program change decrease of 727 burden hours. This program change is a 

result of the burden hours from this data collection (403) being added and the burden hours (1130) 

from a prior phase of data collection during the first three years being eliminated since they will be 

completed by the time this phase of data collection is approved.

16. Plans for Tabulation and Publication of Results

a. Tabulation Plans

Our tabulation plans are the same as those cleared under OMB # 1850-0876. Specifically, our 

tabulation plans include four sets of analyses aligned to the research questions. Random assignment 

of schools within a district to a treatment group that will implement DPBIP or to a control group 

not allowed to do so for the duration of the TIF grant is an ideal design for assessing overall 

effectiveness. Our primary impact analysis will exploit this experimental design to provide rigorous 

estimates of the impact of DPBIP on student achievement and teacher/principal mobility and 

recruitment. Additional nonexperimental analyses are designed to estimate the relative effectiveness 

of individual-based versus group-based or mixed incentive programs, explore the association of 

other key program features with student achievement and teacher/principal outcomes, and to learn 

about districts’ implementation experiences and challenges.

Estimating the overall impact of DPBIP. What follows is the same as those cleared under 

OMB # 1850-0876. With this experimental design, the simple differences between mean outcomes 

in the treatment and control schools should yield unbiased estimates of the impacts of DPBIP. 

However, the precision of the estimates can be improved by using regression procedures to control 

for student, teacher, or school baseline characteristics that may explain some of the variation in 

outcomes not related to the treatment itself. These characteristics may include student controls, such 

as test scores from the year before TIF implementation; gender, race/ethnicity, free- or reducedprice lunch eligibility, special education status, and English learner status; teacher controls, such as 

demographic characteristics, age, experience, and educational background; and school-level averages 

of the student or teacher characteristics. Regression procedures also enable us to adjust for any 

differences between treatment and control groups in these baseline characteristics that happen to 

arise due to chance or sample attrition. The regression model must be flexible enough to include the 

full range of programs and generate estimates of district-specific impacts, which can then be 

aggregated to produce an overall estimate. We will therefore estimate variations of the following 

model for the outcome yijk of individual (student or teacher) i in school j within district k:

(1) 

1

( )

K

ijk k jk k jk ijk

k

y T   G u



  Rjkα X    ijkδ Z  jk γ 

where Rjk is a vector of indicators for combinations of grade levels and randomization strata; α is 

a vector of grade-by-strata fixed effects; Tjk is a treatment indicator; Gkis a dummy variable for 

district k; kis the impact of DPBIP in district k; Xijk is a vector of baseline individual 

characteristics with coefficient vector δ ; Zjk is a vector of baseline school-level characteristics with 

coefficient vector γ ; jk u is a random school effect; and ijk  is a random individual error term. The 
17

district-specific impacts of performance pay, k, are the key coefficients of interest in equation (1). 

We will estimate equation (1) with ordinary least squares (OLS) using Huber-White (“sandwich”) 

standard errors that account for school-level clustering.

Our primary interest is in the overall, average impact of DPBIP in the full study sample. To 

estimate the average impact of DPBIP on schools in the study, we will take a weighted average of 

the estimated district-specific effects, ˆk, with weights equal to the number of treatment and 

control schools within each district. The standard error of the average impact estimate can be 

calculated from the estimated variances and covariances among the district-specific impacts from 

equation (1).

The evaluation includes four years of analyses, and the following describes the analyses 

previously outlined in the previously cleared OMB submission. Impacts in the second and 

subsequent years of the implementation of the DPBIP may be larger than those in the first year for 

several reasons. First, changes in educator effort and the composition of the teaching staff at 

treatment schools may be more pronounced after educators observe the payments from earlier 

years. Also, if educators improve their performance over time, in years 2 through 5 of the grant, 

some students will have had multiple years of exposure to the treatment. For these reasons, equation 

(1) will be estimated separately for assessing impacts for each year of implementation, as well as 

cumulative impacts.

The impact of DPBIP on the outcomes of interest—student achievement and educator 

mobility and recruitment—will be estimated with a variant of equation (1). Student achievement 

outcomes are math and reading scores from spring 2012, 2013, 2014, and 2015 state or district 

assessments. Because tests will differ across states, grade levels, and subjects, we will convert raw 

scale scores to z-scores (raw scores minus the mean score divided by the standard deviation of 

scores on that test among students in that grade and state) in order to scale the outcome variable 

comparably across all students in the sample. Using district records, we will measure teacher 

retention as a dichotomous outcome for whether or not the teacher returns to work in the grantee 

site and/or in his or her initial school in fall of 2011 and continue to do so annually through 2015. 

Because the retention outcome is dichotomous, we will estimate the probit model analog of 

equation (1). Annual school-level teacher data from study schools in fall, 2011 through fall, 2015 

(from district records) and spring 2012, 2013, 2014, and 2015 (from the principal and teacher 

surveys) will be analyzed as outcomes to examine impacts on the composition of the teaching staff. 

If available from administrative records, the quality of applicants who apply to teach in study schools 

for school years 2012–2013, 2013–2014, 2014–2015, and 2015–2016 will also be analyzed, including 

the total number of applicants, average experience level, percentage of applicants who have teaching 

experience, and the selectivity of the college from which they graduated. Equation (1) can be 

aggregated to the school level for the analysis of composition outcomes.

To better understand mobility of high- and low-performing principals and teachers, for grantees 

where we can obtain or calculate a measure of staff effectiveness, we will also estimate a model of 

transitions that includes a teacher or school measure of effectiveness, and interactions of this 

measure with treatment indicators in the set of independent variables. The coefficients on the 

effectiveness measure by treatment interactions provide an estimate of whether differences in 

retention between highly effective and less effective principals or teachers are more or less 

pronounced in treatment versus control schools. Since high- and low-performing teachers are not 

being randomly assigned to treatment and control schools, and estimates of their effectiveness may 
18

be endogenous if DPBIP induces greater teacher effort, these estimates are nonexperimental and 

will need to be interpreted with caution. Wherever possible, we will obtain or calculate value-added 

estimates based on student achievement to measure teacher effectiveness. In addition, if possible, we 

will also use districts’ measures of effectiveness.

Estimating the effectiveness of key program features. What follows is the same as those 

cleared under OMB # 1850-0876. We will conduct exploratory analyses to assess whether particular 

features of DPBIP are associated with impacts on student achievement. These analyses will, in 

particular, examine the relative effectiveness of DPBIP models that place different weights on 

individual versus group performance in the determination of incentive payouts. Other programmatic 

features of interest include the average and maximum size of the incentive payouts and the degree to 

which the payouts vary across educators.

Since we do not expect that districts will randomly assign specific components of their DPBIP 

to schools, we will not be able to experimentally assess the relative effectiveness of different DPBIP 

program features. Instead, we will examine the association between impacts and key program 

features in a regression framework. We will be careful to note that an observed association between 

impacts and programmatic features may not necessarily have a causal interpretation.

For these analyses, we will rely on findings from the implementation analysis to examine how 

the variation in programmatic features is related to the impact. Our basic approach is to regress the 

estimated district-specific impacts from equation (1) on a measure of a specific programmatic 

feature. For the estimated impact ˆkfrom district k, we estimate:

(2) 0

ˆ  k k    W  k

where π0is an intercept, Wkis a measure of a specific programmatic feature with associated 

coefficient λ, and ωkis an error term that includes random error in estimating the true impact βk. 

Because impacts might be more precisely estimated in some districts than in others, we will weight 

grantees by the precision of the estimated impacts when estimating equation (2) to account for this 

source of heteroskedasticity in the error term. For each of the programmatic features described 

earlier, we will estimate equation (2) with the specified program feature as the only covariate, given 

the limited number of grantees in the sample.

Understanding the implementation experiences of TIF districts. What follows is the same 

as those cleared under OMB # 1850-0876 with minor revisions. Understanding the implementation 

experiences and challenges of TIF grantees will provide essential information for improving the 

implementation of future incentive programs and is crucial for the interpretation of the impact 

findings. We will analyze the implementation data collected from grantee, district, and school 

documents; district, principal, and teacher surveys; and telephone interviews with districts to report 

on their incentive policies and experiences. Since the evaluation districts were purposively selected, 

and the impact estimates cannot necessarily be generalized beyond this sample, we will use the 

district surveys to construct tables on their incentive policies, comparing the evaluation districts to 

all recent awardees. We also will use the district surveys and information from telephone interviews 

to document and analyze implementation challenges. The principal and teacher surveys will provide 

critical context to determine if they understood the incentive compensation policy and program in 

their district and school and adjusted their behavior accordingly. After the initial survey, for each 
19

subsequent wave of the principal and teacher surveys, we will construct tables to assess any changes 

in educators’ understanding and behavior.

b. Publication Plans 

We will prepare a final report presenting the results of these tabulations. This report will be the 

final of four reports prepared under this evaluation. The final report, with a projected release date 

of Spring 2017, will describe districts’ implementation strategies and challenges and examine impacts

through the fifth and final year of the TIF cohort 3 grants. Reports will be written in a style and 

format accessible to policymakers and research-savvy practitioners and will comply fully with the 

standards set by the National Center for Education Statistics.

17. Approval Not to Display the OMB Expiration Date

The study will display the OMB expiration date.

18. Explanation of Exceptions

No exceptions are being sought.
REFERENCES

Anderman, C., A. Cheadle, S. Curry, P. Diehr, L. Shultz, and E. Wagner. “Selection Bias Related to 

Parental Consent in School-Based Survey Research.” Evaluation Review, vol. 19, no. 6, 1995, pp.

663–674.

Constantine, Jill, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider, and John Deke. “An 

Evaluation of Teachers Trained Through Different Routes to Certification.” Princeton, NJ: 

Mathematica Policy Research, February 2009.

Decker, Paul, Steven Glazerman, and Daniel Mayer. “The Effects of Teach For America on 

Students: Findings from a National Evaluation.” Princeton, NJ: Mathematica Policy Research, 

June 9, 2004.

Eaton, Danice K., Richard Lowry, Nancy D. Brener, Jo Anne Grunbaum, and Laura Kann. “Passive 

Versus Active Parental Permission in School-Based Survey Research: Does the Type of 

Permission Affect Prevalence Estimates of Risk Behaviors?” Evaluation Review, vol. 28, no. 6, 

2004, pp. 564–577.

Garet, Michael S., Stephanie Cronen, Marian Eaton, Anja Kurki, Meredith Ludwig, Wehmah Jones, 

Kazuaki Uekawa, Audrey Falk, Howard Bloom, Fred Doolittle, Pei Zhu, and Laura Sztejnberg. 

“The Impact of Two Professional Development Interventions on Early Reading Instruction 

and Achievement.” Washington, DC: U.S. Department of Education, National Center for 

Education Evaluation and Regional Assistance, Institute of Education Sciences, September 

2008.

Glazerman, Steven, Paul Decker, and Daniel Mayer. “Alternative Routes to Teaching: The Impacts 

of Teach For America on Student Achievement and Other Outcomes.” Journal of Policy Analysis 

and Management, vol. 25, no. 1, 2006, pp. 75–96.

Hanushek, Eric A. “Efficient Estimators for Regressing Regression Coefficients.” American 

Statistician, vol. 28, no. 2, 1974, pp. 66–67.

Hanushek, Eric A., and S. Rivkin. “Generalizations About Using Value-Added Measures of Teacher 

Quality.” American Economics Review, vol. 100, no. 2, 2010, pp. 267-71. 

Rivkin, S., E. Hanushek, and J. Kain. “Teachers, Schools, and Academic Achievement.” 

Econometrica, vol. 73, no. 2, 2005, pp. 417–458.

Rockoff, J. “The Impact of Individual Teachers on Student Achievement: Evidence from Panel 

Data.” American Economic Review (AEA Papers and Proceedings), vol. 94, no. 2, 2004, pp. 247–

252.

Tuttle, Christina, Steven Glazerman, and Tara Anderson. “ABCTE Teachers in Florida and Their 

Effect on Student Performance.” Washington, DC: Mathematica Policy Research, April 27, 

2009.

Yoon, K. S., T. Duncan, S.W.Y. Lee, B. Scarloss, and K. Shapley. “Reviewing the Evidence on How 

Teacher Professional Development Affects Student Achievement.” Washington, DC: U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance, 2007.