P A R T A

Impact Evaluation of Support for 

Principals: OMB Data Collection 

Package

April 1, 2015

Submitted to:

Institute of Education Sciences 

555 New Jersey Ave NW, Suite 502A 

Washington, DC 20208

Project Officer: Elizabeth Warner

Contract Number: ED-IES-14-R-0008

Submitted by:

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Susanne James-Burdumy

Reference Number: 40412.492

CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

iii

CONTENTS

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION................. 1

A. Justification................................................................................................................................ 1

1. Circumstances necessitating the collection of information ................................................. 1

2. Purposes and uses of the data ........................................................................................... 5

3. Use of technology to reduce burden ................................................................................... 8

7. Special circumstances ...................................................................................................... 10

8. Federal Register announcement and consultation ........................................................... 10

9. Payments or gifts .............................................................................................................. 11

10. Assurances of confidentiality ............................................................................................ 12

11. Additional justification for sensitive questions................................................................... 13

12. Estimates of hours burden ................................................................................................ 13

13. Estimates of cost burden to respondents ......................................................................... 13

14. Estimates of annual costs to the federal government....................................................... 13

15. Reasons for program changes or adjustments................................................................. 13

16. Plan for tabulation and publication of results .................................................................... 14

17. Approval not to display the OMB expiration date.............................................................. 17

18. Explanation of exceptions ................................................................................................. 17

REFERENCES............................................................................................................................................ 18

APPENDIX A: PRINCIPAL SURVEY

APPENDIX B: PRINCIPAL DAILY LOG

APPENDIX C: TEACHER SURVEY

APPENDIX D: DISTRICT-LEVEL DATA REQUEST MEMO

APPENDIX E: ADVANCE LETTERS

APPENDIX F: CONFIDENTIALITY AGREEMENT
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

iv

TABLES

A.1 Study data collection activities, by data source ............................................................................... 4

A.2 Time line for data collection activities .............................................................................................. 5

A.3 Research questions and data sources............................................................................................. 5

A.4 Source, mode, and timing ................................................................................................................ 9

A.5 Technical Working Group members .............................................................................................. 11

A.6 Annual Reporting and recordkeeping hour burden........................................................................ 14

A.7 Principal, teacher, and student outcomes for the impact analysis................................................. 15

A.8 Timetable for project publications .................................................................................................. 17

EXHIBITS

A.1 Logic model for the CEL Program.................................................................................................... 3
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

1

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 

SUBMISSION

This Office of Management and Budget (OMB) package requests clearance for data 

collection activities to support the Impact Evaluation of Support for Principals. The evaluation 

will provide important information on the implementation and impacts of intensive professional 

development for principals. The Institute of Education Sciences (IES) at the U.S. Department of 

Education (ED) has contracted with Mathematica Policy Research and its subcontractors, the 

American Institutes for Research (AIR), Social Policy Research Associates (SPR), and 

Pemberton Research to conduct this evaluation.

The evaluation will include implementation and impact analyses. The implementation 

analysis will draw on information on the scope and sequence of the professional development 

and data from principal surveys to describe principals’ professional development experiences.1

The impact analysis will be based on a random assignment design in which participating schools 

in each district are randomly assigned to a treatment group whose principals are offered intensive 

professional development or to a control group whose principals are not. The impact analysis 

will draw on data from principal and teacher surveys, principal daily logs, and district 

administrative records to estimate the impacts of the professional development on principal 

performance, school climate, teacher retention and performance, and student achievement.

A. Justification

1. Circumstances necessitating the collection of information

a. Statement of need for a rigorous evaluation of principal professional development

Principals play an important role in the academic performance of students in their schools

(Hallinger and Heck 1998; Harris et al. 2010; Knapp et al. 2006; Leithwood et al. 2004), and 

there is widespread interest in the potential of intensive principal professional development 

programs to improve principals’ performance. However, little is known about the effectiveness 

of these programs and their ability to improve principals’ leadership skills and school quality

(Huff et al. 2013). The data collection described in this request will provide essential information 

for describing implementation of a principal professional development and providing rigorous 

estimates of its effects on principal performance, school climate, teacher retention and 

performance, and student achievement. 

Legislative authorization for this evaluation is found in Title II, Part A of the Elementary 

and Secondary Education Act (ESEA), Section 2121-2123, as amended by the No Child Left 

Behind Act (NCLB) (20 USC 6621-6623). Title II, Part A of ESEA provides funding to states to 

carry out professional development activities designed to improve the quality of principals.

Part F, Section 9601 of ESEA permits program funds to be used to evaluate activities authorized 

under the act.

 

1 We are not requesting OMB approval for collection of data on the scope and sequence of the professional 

development because this information will be collected from the professional development provider and will not 

impose any burden on study participants.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

2

b. Study design and research questions

To learn about the effectiveness of intensive principal professional development, the study 

team will evaluate an intensive professional development program using a rigorous random 

assignment design. In fall 2014 we held a competition to select a promising principal 

professional development program, and, with input from a panel of experts, selected the Center 

for Educational Leadership (CEL) at the University of Washington. In the coming months we 

plan to recruit 100 elementary schools from 10 school districts across the country and randomly 

assign them to a treatment group whose principals will be offered CEL’s principal professional 

development program or to a control group whose principals will not be offered this program. 

We will support and monitor the program for two school years to ensure high quality 

implementation, and will collect comprehensive data on program implementation and school, 

principal, teacher, and student outcomes. We will use these data to compare outcomes between 

treatment and control group schools to obtain rigorous evidence on the effectiveness of the 

program. 

The study will address the following research questions about the implementation and 

impacts of the CEL principal professional development program:

 Implementation. What are the professional development experiences of principals in the 

study? 

 Impacts. What are the impacts of intensive principal professional development on (a) school 

climate and educator behaviors, (b) teacher retention and effectiveness, and (c) student 

achievement and behavior?

c. The CEL principal professional development program

CEL’s professional development program is based on a theory of action that principals’ 

instructional-focused leadership boosts teaching quality and raises achievement. According to 

the program’s logic model (Exhibit A.1), if principals have a clear understanding of what quality 

instruction looks like, as well as a common language to describe and promote elements of quality 

instruction, and if they know how to lead instructional improvement, instructional practice will 

become more effective and student achievement will improve. 

CEL seeks to develop principals’ knowledge of high-quality instruction and to equip them to 

lead their teachers to deliver quality instruction, by honing their skills in three content areas:

1. Improving teacher instruction through observation, analysis of data (including 

information from district evaluation systems), and implementation of “inquiry 

cycles” that generate useful feedback for teachers (instructional leadership);

2. Guiding staffing plans and staff development (human capital management); and

3. Creating a school-wide culture of learning to facilitate improved academic success 

for all students (organizational leadership).
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

3

Exhibit A.1. Logic model for the CEL Program

CEL’s professional development sessions aim to sharpen principals’ skills in observing 

classrooms and identifying areas in which teachers need to improve. The professional 

development sessions are tailored to each district’s instructional framework and teacher 

evaluation process, using crosswalks between the different frameworks. CEL also provides 

individual coaching and professional learning communities (PLCs) aligned with the group 

sessions.

While the length of CEL’s program varies across districts, for the purposes of this study

CEL will provide its program for 24 months, beginning with a summer institute in 2015.2 During 

the 4-day summer institute, CEL will introduce principals to core program content; establish 

relationships between principals and coaches; analyze school data; and help principals plan 

specific leadership actions. After the summer institute, CEL will support principals in treatment 

schools throughout the 2015–2016 school year, to help them apply lessons from the summer 

institute in their schools. This support will include 8 one-day professional development sessions

in each district; 10 half-day one-on-one coaching sessions (primarily in-person but some may be 

virtual); and quarterly online PLCs. In this way, principals will interact at least twice a month 

with the professional development provider.

The second year of the program will begin with a summer institute in 2016 for any 

principals who are new to one of the 50 treatment schools in the study. Principal coaching will 

continue for all principals in treatment group schools throughout the 2016–2017 school year.

 

2 The institute will be offered on two dates, to help ensure all treatment group principals can attend.

Professional 

development

Improved leadership 

practices

Instructional 

leadership

Human capital 

management

Organizational 

leadership

Teacher 

changes

Schoolwide 

changes

Student 

learning

• Summer 

institute

• Formal, 

in-person 

trainings

• One-on-one 

coaching

• Online PLCs

• Increase the observation and 

analysis of instruction

• Use evidence to identify areas 

for growth

• Provide more frequent, bettertargeted, and actionable 

teacher feedback

• Use evidence to identify PD

and supports that are better 

targeted to staff needs

• Use evidence to develop 

school and staff growth plans

• Regularly communicate plans 

for, and progress toward, 

school improvement

• Improved 

instruction

• Improved/ 

targeted PD

and support 

for teachers

• Improved 

school 

climate and 

culture

• All students, 

regardless of 

race, class, 

language, and 

disability, 

achieve at the 

highest levels
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

4

d. Data collection needs

This study includes several data collection efforts that are summarized in Table A.1. The 

purposes and uses of these data are described in Section A.2 below.

Table A.1. Study data collection activities, by data source

Data 

source Mode, timing, and respondent Key constructs and outcomes

Principal survey (treatment and control groups)

Annual surveys 30-minute web-based survey, with telephone option 

and in-person follow-up, administered spring 2016 and 

spring 2017 to treatment and control group principals

Amount and usefulness of professional development received by 

principal during the current school year and the preceding summer

Principals’ leadership practices during the school year (human 

capital management, instructional leadership, and organizational 

leadership)

District and school context (resource availability, degree of 

principal autonomy, and district policies affecting principal practice)

Principal background characteristics (demographic traits, 

educational degrees, certification/licensure, and professional 

experience)

Principal log (treatment and control groups)

Daily logs 15-minute web-based logs completed on 5 

consecutive days by treatment and control group 

principals, during four weeklong periods of the 2015–

2016 and 2016–2017 school years

Time spent by principal on professional development activities 

during the school year

Time spent by principal on various leadership practices (human 

capital management, instructional leadership, organizational 

leadership, or other) during the school year

Teacher survey (treatment and control groups)

Annual surveys 30-minute web-based survey, with telephone and 

hard-copy options and in-person follow-up, 

administered to treatment and control group teachers 

in spring 2016 and spring 2017

Principals’ use of leadership practices and quality of principals’ 

instructional and organizational leadership during the school year

Amount and usefulness of formal and informal professional 

development received by teachers during the school year

Amount and usefulness of instructional feedback teachers received 

from principal or other school leaders during the school year

Teachers’ use of student achievement data and changes to 

instructional practice (teaching methods, student work, student 

assessment, and topics or materials) during the school year

School climate and culture (professional climate, staff 

collaboration, student engagement and belonging, family 

engagement, and school safety) 

Teacher background characteristics (demographic traits, 

educational degrees, certification/licensure, and professional 

experience)

District records (treatment and control groups)

Staff records Electronic grade and school assignment data for prior 

and current school years, and background 

characteristics for prior school year, requested from all 

districts in fall 2015, fall 2016, and fall 2017

Principal and teacher mobility, teacher retention, teacher hiring, 

and principal and teacher background characteristics 

Staff

performance 

records

Electronic performance data for prior school years, 

requested from all districts in fall 2016 and fall 2017b

Teachers’ instructional effectiveness and principals’ leadership 

effectiveness

Student 

records

Electronic student records data for prior school years, 

requested from all districts in fall 2016 and fall 2017 b

Student achievement (test scores), behavioral outcomes 

(attendance and disciplinary records), and background 

characteristics (demographic traits, grade level, English language 

learner status, and special education status)

bWe will request teacher and principal performance data and student records for the 2014–2015 and 2015–2016 school years in fall 2016 

and for the 2016–2017 school year in fall 2017.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

5

e. Time line for data collection activities 

The evaluation is expected to be completed in five years, with 2.5 years of data collection.

Table A.2 shows the schedule of data collection activities. 

2. Purposes and uses of the data

To address the study’s research questions, the evaluation will collect and analyze data from 

several sources. IES will use these data to better understand principals’ professional development 

experiences, and to estimate the impact of the professional development on school, teacher, and 

student outcomes. Table A.3 lists research questions and specific data sources that will be used 

to answer them—in this table we have broken Research Question 2 into three parts, to 

correspond to three sets of outcomes we will examine. We describe how the study will use each 

data source below. Information will be collected by Mathematica Policy Research and its 

partners AIR, SPR, and Pemberton Research, under contract with ED [contract number ED-IES14-R-0008].

Table A.2. Time line for data collection activities

Activity Date

Data for 2015-2016 school year

Collect principal daily logs 9/2015 through 5/2016

Collect principal and teacher survey data 3/2016 through 5/2016

Collect district recordsa 8/2015 through 12/2015; 8/2016 through 12/2016

Data for 2016-2017 school year

Collect principal daily logs 9/2016 through 5/2017

Collect principal and teacher survey data 3/2017 through 5/2018

Collect district recordsb 8/2017 through 12/2017

a

In fall 2015, we will only collect data on teacher assignments for fall 2015. In fall 2016 we will collect district records 

data for the prior (2014–2015) school year, as well as teacher assignment data for the 2015–2016 school year.

b

In this data collection, we will also collect district records data on teacher and principal assignments for the 2017–

2018 school year.

Table A.3. Research questions and data sources

Research question Data source(s)

1. Implementation

What are the professional development 

experiences of principals in the study?

Principal surveys

2. Impacts

a. Impacts on intermediate outcomes. What are the 

impacts of intensive principal professional 

development on school climate and educator 

behaviors?

Principal surveys

Principal daily logs

Teacher surveys

b. Impacts on teacher outcomes. What are the 

impacts of intensive principal professional 

development on teacher retention and 

effectiveness?

District records

Staff records

Staff performance records

c. Impacts on student outcomes. What are the 

impacts of intensive principal professional 

development on student achievement and 

behavior?

District records 

Student records
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

6

Principal Surveys. Surveys administered to treatment- and control-group principals in years 

1 and 2 will collect information on their leadership practices and perceptions of school climate. 

We will use this information to examine the effect of professional development on leadership 

practices and school climate (research question 2a). We will also ask principals to report the 

amount, content, and usefulness of their professional development experiences. We will use this 

information to document treatment and control group principals’ perceptions of their 

professional development experiences (research question 1). In addition, principals will provide 

data on their backgrounds, professional experience and educational attainment, and the district 

context in which they operate (including, for example, degree of perceived autonomy and 

barriers to school improvement). We will use this information to document the context of

principals’ professional development experiences (research question 1) and to measure whether 

impacts of intensive professional development vary across districts with high and low levels of 

principal autonomy (research question 2).3

Principal Daily Logs. Daily logs will collect information on how principals allocate their 

time across different categories of leadership practice. Logs will be administered to treatment 

and control groups on five consecutive days during four weeks of school years 1 and 2. We will 

use this information to examine the effect of professional development on daily time-use in 

different domains of leadership (research question 2a). As compared to annual surveys, daily 

logs have been shown to more accurately capture frequently occurring behaviors, such as day-today instructional leadership and managerial responsibilities, because they allow for measurement 

of daily fluctuations and are less susceptible to recall error (Camburn et al. 2010a, 2010b). 

Principal logs will complement annual principal surveys by ensuring that the study team can 

examine impacts on how principals are actually spending their time on leadership behaviors on a 

given day, rather than relying only on annual recall; and allowing detection of changes in 

principals’ leadership behaviors within a year.

Teacher Surveys. Surveys administered to teachers in years 1 and 2 will collect information 

on teachers’ experiences during the school year, including the amount and perceived usefulness 

of professional development and instructional support they received, instructional improvement 

and data-use practices, and perceptions of their principal’s leadership practices and school 

improvement efforts. We will use this information to examine the effect of principal professional 

development on teacher practices and teacher perceptions of principals’ leadership practices. 

Teachers will also be asked about their schools’ working conditions, staff collaboration and 

support, and academic culture during the school year. We will use this information to examine 

the effect of principal professional development on school climate (research question 2a). 

Finally, teachers will provide data on their backgrounds, educational attainment, and professional 

experience. We will use this information to provide context for understanding principal’s 

professional development experiences (research question 1) and to examine how principal 

 

3 To obtain a district-level measure of principal autonomy, we plan to average reported levels of principal autonomy 

across treatment and control group principals in each district. However, if we find significant differences in reported 

autonomy between treatment and control group principals, suggesting that reported autonomy may have been 

influenced by the intervention, we will instead construct the district-level measure of autonomy based only on the 

reports of control group principals.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

7

professional development affects the mix of teachers hired and retained in the study schools 

(research question 2b).

District Records. We will collect data on principals, teachers, and students from the district 

records listed below. We will use this information to estimate effects of principal professional 

development on staff mobility, retention, and performance (research question 2b) and on student 

achievement and behavior (research question 2c), and we will also use district data to draw a 

sample of teachers to take the teacher survey in years 1 and 2.

 Staff records. We will collect data on principals’ school assignments and teachers’ school 

assignments and the grades and subjects taught. We will use this information to examine 

effects of principal professional development on principal turnover and teacher retention 

(research question 2b). We will also use it to draw a random sample of teachers to take the 

survey in years 1 and 2 of the study. We will collect available information on characteristics 

of the study sample, such as principal and teacher demographics (age, sex, race, and 

ethnicity), educational attainment (certifications, degrees, and scores on licensure or 

certification exams), and years of teaching experience. We will use this information to 

describe the study context in the implementation analysis (research question 1), and to 

examine whether impacts of principal professional development vary depending on baseline 

values of these characteristics, such as principal experience (research question 2).

 Staff performance records. We will collect principal and teacher performance ratings from 

district evaluation systems. These may include composite ratings, observation ratings, 

teacher value-added scores, school value-added scores, or scores on student learning 

objectives. We will use this information to examine the effect of principal professional 

development on teacher and principal effectiveness as assessed by their district systems 

(research question 2b). In conjunction with staff records described above, we will also use 

this information to examine effects on hiring and retention of effective teachers, dismissal of 

ineffective teachers, and ways that principals assign teachers of different performance levels 

to classes (research question 2b).

 Student records. We will collect information from student records for years 1 and 2, as well 

as the school year before the program started. We will obtain student standardized test 

scores in math and English/language arts for all tested grades, which we will use to examine 

the effect of principal professional development on student achievement (research question 

2c). We will also obtain data on student attendance and discipline, which we will use to 

examine effects on student behavior. Finally, we will collect data on student demographics 

(age, sex, race, and ethnicity) and participation in other programs (free or reduced-price 

lunch eligibility, English Language Learner status, and individualized education plan status). 

We will use this information to describe students in the study sample and use baseline values 

of these measures and student test scores to improve precision of estimates in the impact 

analysis (research question 2).

Data collection in year 1. We will use information collected during year 1 to examine 

principals’ professional development experiences (research question 1) and impacts of principal 

professional development during the initial year of program implementation (research 

question 2). We will use information on year 1 educator behaviors and teacher and student 

outcomes to inform research question 2, by examining whether principal professional 
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

8

development resulted in differences between the treatment and control groups after the first year 

of the intervention’s implementation.

Data collection in year 2. We will use the information collected during year 2 to examine 

the impacts of intensive principal professional development one year later, after two years of the 

intervention’s implementation (research question 2). More specifically, we will use the data 

collected in year 2 to examine whether principal professional development resulted in differences

in school climate, teacher and principal practices, and teacher, principal, and student outcomes 

after two years of professional development.

3. Use of technology to reduce burden

The data collection plan is designed to obtain information in an efficient way that minimizes 

respondent burden. When feasible, we will gather information from existing data sources, using 

the most efficient methods available. Table A.4 provides information on the source, mode, and 

timing for each data collection activity.

A web-based survey will be the primary mode of data collection for principals and teachers 

in the study. The web-based surveys and log will enable respondents to complete the data 

collection instrument at a location and time of their choice, and its built-in editing checks and 

programmed skips will reduce the level of response errors. We will also offer two additional 

modes (telephone and in-person) of interviewing for educators who prefer these methods to 

reduce burden on them.

District records we will collect will be computer files provided by districts. Whenever 

possible, districts will upload the files to a secure Mathematica FTP site. While we will specify 

the required data elements, we will accept any format in which the data are provided, to reduce 

burden for the district. Regardless of the form in which it is received, these data will be 

converted to a consistent format so that they can be combined with data submitted by other 

districts and will be suitable for analysis. 

4. Efforts to avoid duplication of effort

Whenever possible, the study team will use existing data from districts, such as from their 

records for staff and students. This will reduce respondent burden and minimize duplication of 

data collection efforts. 

5. Methods to minimize burden on small entities

No small businesses or entities will be involved as respondents.

6. Consequences of not collecting data

These data are needed to evaluate intensive principal professional development. Little is 

known about the ability of principal professional development programs to improve principals’ 

leadership skills and school quality. Failing to collect these data would miss an opportunity to 

inform the field with rigorous evidence on the effectiveness of intensive principal professional 

development. 
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

9

Table A.4. Source, mode, and timing

Data source Mode and timing Respondent group

Principal survey

Annual Survey 30-minute web-based survey, with telephone 

option and in-person follow-up, administered in

spring 2016 and spring 2017

Treatment and control

Principal log

Daily logs 15-minute web-based logs completed on 5 

consecutive days, during four weeklong periods 

of the 2015–2016 and 2016–2017 school years

Treatment and control

Teacher survey

Annual Survey 30-minute web-based survey, with telephone 

and hard-copy options and in-person follow-up, 

administered in spring 2016 and spring 2017

Treatment and control

District records

Staff records Electronic grade and school assignment data 

and background characteristics for prior and 

current school years, requested from districts in 

fall 2016 and fall 2017

Treatment and control

Staff performance records Electronic performance data for prior school 

years, requested from districts in fall 2016 and 

fall 2017

Treatment and control

Student records Electronic student records data for prior school 

years, requested from districts in fall 2016 and 

fall 2017

Treatment and control

Consequences of not collecting data from specific sources. Each of the four data sources 

provides information needed for the evaluation.

Without information from principal surveys, we will be unable to examine the professional 

development experiences of principals in the study, nor to compare the experiences of controlgroup principals to those of treatment-group principals. In addition, we will be unable to examine 

intermediate impacts of intensive principal professional development on principals’ leadership 

practices and school-wide safety. Finally, without information on principals’ demographic 

background, professional experience, and district context, we will be unable to describe the 

sample of principals participating in the study or capture principal and district characteristics that 

may influence the implementation or effectiveness of the principal professional development.

Without the information from principal daily logs, we will be unable to examine how 

professional development impacts principals’ daily time-use in different domains of school 

leadership. Without collecting this information on five consecutive days per observation period, 

we will be unable to reliably measure principal time-use. Without collecting this information 

during four weeklong observation periods per school year, we will be unable to assess whether 

principal professional development induces changes in principals’ time-use over the school year.

Without teacher surveys, we will be unable to examine how intensive principal professional 

development impacts teachers’ behaviors and their perceptions of professional development 

opportunities, principals’ leadership practices, and school improvement efforts. Without the 

information on teacher perceptions of school climate, we will be unable to assess school-wide 

changes in working conditions, staff collaboration and support, and academic culture. Finally, 
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

10

without the information on teachers’ demographic backgrounds, educational attainment, and 

professional experience, we will be unable to capture teacher characteristics that may influence 

the implementation or effectiveness of the principal professional development.

Without district records, we will not be able to analyze the ultimate impact of the principal 

professional development on teacher or student outcomes. Without the following information, 

we will not be able to assess changes in principals’ ability to hire and retain effective teachers, 

determine effective teaching assignments, improve teachers’ performance, or improve student 

achievement and behavior:

 Without information from staff records, we will be unable to examine impacts of the 

principal professional development on staff mobility (principal turnover and teacher 

turnover), hiring and retention outcomes (hiring and retention of effective teachers and 

dismissal of ineffective teachers), and teacher assignments to grade levels or subjects. We 

will also be unable to draw a random sample of teachers for the teacher surveys, preventing 

us from capturing teacher perceptions about principal practices.

 Without information from staff performance records, we will be unable to examine 

impacts of the principal professional development on principal and teacher effectiveness or 

teacher-effectiveness-based staffing outcomes (e.g., hiring or retaining effective teachers).

 Without information from student records, we will be unable to examine impacts of the 

principal professional development on student outcomes (achievement, attendance, and 

disciplinary problems). Moreover, without information from prior-year student records, we 

will be unable to assess the validity of random assignment (i.e., establish baseline 

equivalence between treatment and control groups).

Consequences of not collecting data at the end of years 1 and 2. If we do not collect data 

at the end of year 1, we will be unable to examine program impacts—such as changes in 

principal or teacher practices—after the initial year of program implementation. In addition, we 

will not be able to examine differences across the treatment and control groups in principals’ 

professional development experiences—such as the amount, focus, or quality of professional 

development received—during the first year of program implementation. If we do not collect this 

data at the end of year 2, we will be unable to examine effects of the principal professional 

development after the second year of the program. It may take time for principals to implement 

leadership improvements learned through professional development. Consequently, school 

improvements and changes in teacher behaviors, teacher outcomes, or student outcomes may 

require time to become apparent. Without the year 2 information, we will be unable to detect any 

such effects that require time to develop.

7. Special circumstances

There are no special circumstances involved with this data collection.

8. Federal Register announcement and consultation

a. Federal Register announcement

The 60-day notice to solicit public comments was published in [insert volume, number, and 

page] of the Federal Register on [insert date]. [Insert text regarding comments]. The 30-day 
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

11

notice to solicit public comments was published in [insert volume, number, and page] of the 

Federal Register on [insert date].

b. Consultations outside the agency

In formulating the evaluation design, the study team sought input from the technical 

working group (TWG), which includes practitioners and experts in evaluation methods and data 

analysis, state assessment programs, and education reform. We will continue to consult with the 

TWG throughout the study on other issues that would benefit from their input. Table A.5 lists the 

TWG members. 

Table A.5. Technical Working Group members

Name Title and affiliation

Eric Camburn Associate Professor and Senior Researcher, Consortium for Policy Research in Education 

(CPRE), University of Wisconsin-Madison

Roger Goddard Professor and Fawcett Chair, Department of Educational Studies, the Ohio State University

Jason Grissom Assistant Professor and Faculty Affiliate, Center for the Study of Democratic Institutions, 

Vanderbilt University

Jason Huff Educational Consultant, Seattle Public Schools

Carolyn Kelley Professor, Educational Leadership and Policy Analysis, University of Wisconsin-Madison

Jim Kemple Executive Director of the Research Alliance for New York City Schools; Research Professor at 

the Steinhardt School of Culture, Education, and Human Development, New York University

Susanna Loeb Barnett Family Professor of Education, Stanford University

John Nunnery Executive Director, Center for Educational Partnerships, Old Dominion University

Jeff Smith Professor, Economics, University of Michigan

9. Payments or gifts

Burden payments have been proposed for the principal survey, the principal log, and the 

teacher survey. Payment will be made in the form of a check or gift card. During each round of 

data collection, principals completing the survey will receive a $30 payment. In addition, 

principals will be given $10 for each day a log is completed for a possible total of $50 per round 

of data collection and a total of $200 across all four rounds of data collection. While all aspects 

of this data collection are vital to the success of the study, the principal log is a critical outcome 

for which the control teachers will have no incentive to fill out. Adequately incentivizing 

principals to regularly log time spent performing the activities throughout the day in addition to 

entering the data into the online database, will increase the likelihood that accurate data is 

collected across the span of the data collection period.

Teacher will receive a payment of $30 for completing the teacher survey. The payments are

proposed because high response rates are needed to make the survey findings reliable, and we are 

aware that principals and teachers are the target of numerous requests to complete surveys on a 

wide variety of topics from state and district offices, independent researchers, and ED. The 

proposed amounts are within the incentive guidelines outlined in the March 22, 2005 memo, 

“Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

12

10. Assurances of confidentiality

The study team has established procedures to ensure the confidentiality and security of its

data. This approach will be in accordance with all relevant regulations and requirements, in 

particular the Education Sciences Institute Reform Act of 2002, Title I, Subsection (c) of Section 

183, which requires the director of IES to “develop and enforce standards designed to protect the 

confidentiality of persons in the collection, reporting, and publication of data.” The study will 

also adhere to requirements of Subsection (d) of Section 183 prohibiting disclosure of 

individually identifiable information as well as making the publishing or inappropriate 

communication of individually identifiable information by employees or staff a felony.

The study team will protect the full privacy and confidentiality of all individuals who 

provide data. The study will not have data associated with personally identifiable information 

(PII), as study staff will be assigning random ID numbers to all data records and then stripping 

any PII from the data records. In addition to the data safeguards described here, the study team 

will ensure that no respondent names, schools, or districts are identified in publicly available 

reports or findings, and if necessary, the study team will mask distinguishing characteristics. A 

statement to this effect will be included with all requests for data:

“Mathematica Policy Research and its subcontractors AIR and SPR follow the 

confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 

2002, Title I, Part E, Section 183). Responses to this data collection will be used only for 

research purposes. The reports prepared for the study will summarize findings across the 

sample and will not associate responses with a specific district, school, or individual. We will not 

provide information that identifies respondents to anyone outside the study team, except as 

required by law.”

Mathematica employs the following safeguards to ensure confidentiality:

 All Mathematica employees sign a pledge that emphasizes the importance of confidentiality

and describes their obligation (see Appendix F).

 Secure FTP services allow participating school districts to securely transfer encrypted data 

files to the study team. Internal networks are all protected from unauthorized access utilizing 

defense-in-depth best practices, which incorporate firewalls and intrusion detection and 

prevention systems. The networks are configured so that each user has a tailored set of 

rights, granted by the network administrator, to files approved for access and stored on the 

LAN. Access to hard-copy documents is strictly limited. Documents are stored in locked 

files and cabinets. Discarded materials are shredded.

 Computer data files are protected with passwords, and access is limited to specific users, 

who must change their passwords on a regular basis and conform to strong password 

policies.

 Especially sensitive data are maintained on removable storage devices that are kept 

physically secure when not in use.

After the study concludes, the study data may be transmitted to the National Center for 

Education Statistics (NCES) for safekeeping as a restricted-use file. All other versions of the data 
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

13

will be destroyed. Prior to transmittal, the data will undergo careful screening, and modification 

if necessary, to ensure that there is no unacceptably high level of disclosure risk for protected 

respondents. Researchers wishing to access the data for secondary analysis must apply for an 

NCEE license and agree to the applicable rules and procedures guiding the use of restricted-use 

files.

11. Additional justification for sensitive questions

No questions of a sensitive nature will be included in this study.

12. Estimates of hours burden

Table A.6 provides an estimate of time burden for the data collections, broken down by 

instrument and respondent. These estimates are based on our experience collecting 

administrative data from districts and administering surveys to school principals and teachers. 

The total of 2,235 hours covers all five years of the evaluation, and includes the following 

efforts: up to 12 hours, annually for 3 years, for each of the 10 districts to collect and assemble 

administrative records on teachers, principals, and students participating in the evaluation; 30 

minutes, annually for two years, for 85 principals (85 percent of the anticipated 100 principals in 

the sample) to complete the principal survey, plus 15 minutes, 20 times per year for 2 years, for 

85 principals to complete the principal log; and 30 minutes, annually for two years, for 1,020 

teachers (85 percent of the anticipated 1,200 teachers in the sample) to complete the teacher 

survey. Annual number of respondents is 372 and annual responses for three years of this 

collection are 1,880. The total annual burden for this collection is 745 hours.

13. Estimates of cost burden to respondents

There are no additional respondent costs associated with this data collection beyond the 

burden estimated in item A12.

14. Estimates of annual costs to the federal government

The estimated cost for this five-year study, including selection of the intervention provider, 

data collection instruments, justification package, district recruitment, provision of professional 

development, support for the provider, data collection, data analysis, and report preparation, is 

$12,198,695 or approximately $2,439,739 per year.

15. Reasons for program changes or adjustments

This is a new information collection request.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

14

Table A.6. Annual Reporting and recordkeeping hour burden

Respondent/data request

Total number 

of respondents 

(target/expected 

response rate)

Total annual 

responses

Total annual 

hours

Districts

Teacher and principal grade and school 

assignment request (3 times), performance 

request (2 times), and student records request 

(2 times) (Appendix D)

10 10 93

Principals

Principal survey (2 times) (Appendix A), and

Principal log (40 times) (Appendix B)

85 1190 312

Teachers

Teacher survey (2 times) (Appendix C) 1020 680 340

Total 1,115 1,880 745

Note: Reporting on an annualized data collection over 3 years: Annual number of respondents is 372 (1,115/3)

and the total annual burden hours for this collection is 745 hours.

16. Plan for tabulation and publication of results

a. Tabulation plans

Implementation analysis

The implementation analysis will contextualize findings from the impact analysis, detailing 

the fidelity of program implementation and challenges encountered, identifying mechanisms 

through which the professional development might improve outcomes, and documenting the 

extent of the contrast between the intervention and the counterfactual (that is, whether the 

experiences of principals in the treatment group differed significantly from those of principals in 

the control group).

Descriptive analysis of professional development. Drawing on information on the 

intended scope and sequence of the professional development gathered by the study’s technical 

assistance team, we will first describe the intervention as planned, including its objectives, and 

the timing, hours of professional development (session-based and coaching), delivery mode, and 

content of support.4 We will draw on principal surveys to describe the professional development 

received. We will describe the number of hours and focus of professional development received 

by principals and principals’ perceptions of the quality or usefulness of the professional 

development. Using information from the principal surveys, teacher surveys, and district records 

data, we will also describe the characteristics of the districts in which the intervention was 

delivered.

 

4 These data will be collected by the study’s technical assistance team from the professional development provider, 

and their collection will not impose burden on study participants.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

15

Comparison of the experiences of treatment and control group principals. Using data 

from the year 1 principal survey, we will describe the professional development received by the 

treatment group principals relative to that received by control group principals. The analysis will 

detail the content and frequency of professional development both groups received and test for 

statistically significant differences between the two groups. 

Impact analysis

The impact analysis will examine the effects of CEL on the principal, teacher, and student 

outcomes in Table A.7. Key outcomes of interest include (1) teachers’ receipt of professional 

development and perceived quality and frequency of feedback from the principal; (2) school 

climate outcomes, including staff collaboration and perceptions of support; (3) teachers’ 

perceptions of principals’ leadership practices, school climate, and educator behaviors;

(4) teacher effectiveness (as measured by the district’s central office) and teacher retention; and

(5) student test scores, attendance, and behavior. 

Table A.7. Principal, teacher, and student outcomes for the impact analysis

Level Outcomes

Principal  District central office performance ratings

Teacher  Perceptions of school climate

– Staff collaboration

– Perceived support

 Perceptions of principals’ leadership practices

 Teachers’ receipt of professional development and principal feedback (quality and usefulness)

 Retention

 Performance ratings

– All current teachers

– New teachers

 Assignment

– Difference between teacher performance in high-stakes, low-stakes grades

– Variance of average teacher performance across grades (to examine whether high- and lowperforming teachers grouped within grades)

Student  Achievement 

 Attendance 

 Behavior

We will estimate impacts using the following model:

(1) ,

where Yij is the outcome of interest for individual (principal, teacher, or student) i in school j; α is 

an intercept term; Tj is an indicator equal to one if the school is assigned to the treatment group 

and zero otherwise; Pij is a vector of baseline school- or student-level characteristics, and Zj is a 

vector of fixed effects corresponding to the study’s random assignment blocks; δ and γ are 

coefficient vectors; and εij is a random error term, clustered at the school level. The coefficient β

          Y T P Z ij j ij j ij
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

16

represents the average impact of the principal professional development.5 We will apply a 

weighting scheme that gives an equal weight to each of the schools in the sample (regardless of 

the number of principals, teachers, or students in the sample in each school).

For teacher assignment, we will investigate how assignment to the CEL program affects the 

way principals assign teachers to classes—for instance, whether it leads principals to pair higherperforming teachers with lower-performing teachers in the same grade (to facilitate mentoring), 

or whether it leads them to reassign low-performing teachers to grades not covered by highstakes tests. We will also examine principal and teacher performance ratings as measures of 

effectiveness, but we will interpret these data with caution. Some districts will rate principals at 

least partly based on school value added. For new principals, this rating is affected by the

teachers inherited from prior principals (Chiang et al. 2012). Similarly, teacher performance 

ratings may have two limitations: (1) our experience suggests that nearly all teachers are given 

‘satisfactory’ ratings, so there may be little variation in the data; and (2) teacher evaluations 

typically incorporate ratings by principals, so differences in ratings could reflect effects of the 

professional development program. 

Our main impact estimates will reflect the impact of the professional development on the 

schools whose principals were offered the professional development, whether or not they 

actually participated for the full two years. To examine the effects on schools that received the 

full intended “dosage”—that is, treatment schools whose principals received both years of the 

CEL professional development—we will estimate local average treatment effects (Imbens and 

Angrist 1994), using treatment status as an instrumental variable for the proportion of total hours 

of intended CEL activities the principal attended. Even schools with principal turnover during 

the study could receive the full intended dosage if both the original and replacement principals 

attend all offered CEL activities during their time at the school. 

Subgroup analyses. We will explore whether principal professional development is more 

effective in some contexts than others. These results may be useful to policymakers seeking to 

know whether principal professional development would be useful in particular kinds of districts 

or how best to target principal professional development within districts. Subgroups of interest 

will be (1) novice and experienced principals (because new principals may be more in need of 

and more receptive to coaching and formal group professional development sessions than more

experienced principals); and (2) districts with high versus low levels of autonomy (because 

principal professional development may be less effective in districts where principals have less 

influence on hiring, dismissal, and curricular decisions).6 Statistical power for subgroup analyses 

 

5 We will not adjust for principal and teacher characteristics in this model. Because the intervention could influence 

the composition of principals and teachers, though retention or mobility, observed characteristics of principals and 

teachers could reflect program effects, and including them could lead to an incorrect estimate of program impacts.

6 As discussed above, we will construct a district-level measure of principal autonomy by averaging reported levels 

of principal autonomy across treatment and control group principals in each district. However, if we find significant 

differences in reported autonomy between treatment and control group principals, suggesting that reported 

autonomy may have been influenced by the intervention, we will instead construct the district-level measure of 

autonomy based only on the reports of control group principals.
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

17

will be lower than for the full sample—subgroup findings will thus be less definitive than those 

for the full sample, but can provide suggestive evidence of impacts for these groups.

b. Publication plans

The study team will prepare two reports. The first will describe the implementation of the 

principal professional development program and present initial impacts of the program on 

intermediate outcomes, including school climate and educator behaviors, in year 1 (the 2015–

2016 school year). The second report will examine implementation and the effects of principal 

professional development on additional intermediate outcomes, including school climate and 

educator behaviors, in year 2 (the 2016–2017 school year), teacher effectiveness and retention, 

and student outcomes, including test scores, attendance, and behavior from years 1 and 2. Both 

reports will be released on the IES website. 

17. Approval not to display the OMB expiration date

All data collection instruments will include the OMB expiration date.

18. Explanation of exceptions

No exceptions are requested.

Table A.8. Timetable for project publications

Activity Date

Year 1 report

First draft January 2017

Revised draft March 2017

Final report December 2017

Year 2 report

First draft August 2018

Revised draft October 2018

Final report June 2019
CONTRACT NUMBER: ED-IES-14-R-0008 MATHEMATICA POLICY RESEARCH

18

REFERENCES

Camburn, E., J. Spillane, and J. Sebastian. “Assessing the Utility of a Daily Log for Measuring

Principal Leadership Practice.” Educational Administration Quarterly, vol. 46, 2010a, 

pp. 707–737.

Camburn, E., J. Huff, E. Goldring, and H. May. “Assessing the Validity of an Annual Survey for 

Measuring Principal Leadership Practice.” Elementary School Journal, vol. 111, no. 2, 

2010b, pp. 314–335.

Camburn, E. Goldring, J. Sebastian, H. May, and J. Huff. “Conducting Randomized Experiments 

with Principals: A Case Study.” Educational Administration Quarterly, forthcoming.

Goddard, R., A. Miller, M. Kim, Y. Goddard, P. Schroeder, and E.S. Kim. “Evaluation of 

Principals’ Professional Development Learning: Results from A Randomized Control Trial.” 

Paper presented at the Annual University Council for Educational Administration 

Convention, Pittsburgh, Pennsylvania, 2011;

Goff, P.J., E. Guthrie, E. Goldring, and L. Bickman. “Changing Principals’ Leadership Through 

Feedback and Coaching.” Journal of Education Administration, vol. 52, no. 5, 2014, 

pp. 682-704.

Kottkamp, R.B., and E.A. Rusch. “The Landscape of Scholarship on the Education of School 

Leaders, 1985-2006.” In Handbook of Research on the Education of School Leaders, edited 

by M.D. Young, G.M. Crow, J. Murphy, and R.T. Ogawa. New York: Routledge, 2009, 

pp. 23-85.

Nunnery, J.A., S.M. Ross, and C. Yen. “An Examination of the Effect of a Pilot of the National 

Institute for School Leadership’s Executive Development Program on School Performance 

Trends in Massachusetts.” Norfolk, VA: Old Dominion University, Center for Educational 

Partnerships, 2010a.

Nunnery, J., S.M. Ross, and C. Yen. “The Effect of a Pilot of the National Institute for School 

Leadership’s Executive Development Program on School Performance Trends in 

Pennsylvania.” Norfolk, VA: Old Dominion University, Center for Educational 

Partnerships, 2010b.

Nunnery, J., S.M. Ross, and C. Yen. “The Effect of a Pilot of the National Institute for School 

Leadership’s Executive Development Program on School Performance Trends in 

Pennsylvania: 2006-2010 Pilot Cohort Results.” Norfolk, VA: Old Dominion University, 

Center for Educational Partnerships, 2011.
www.mathematica-mpr.com

Improving public well-being by conducting high quality, 

objective research and data collection

PRINCETON, NJ ■ ANN ARBOR, MI ■ CAMBRIDGE, MA ■ CHICAGO, IL ■ OAKLAND, CA ■ WASHINGTON, DC

Mathematica®is a registered trademark 

of Mathematica Policy Research, Inc.