P A R T A



# Impact Evaluat ion of Support for Principals: OMB Data Collection Package

April 1 , 2015

Submitted to : Institute of Education Sciences 555 New Jersey Ave NW, Suite 502A Washington, DC 20208

Project Officer: Elizabeth Warner
Contract Number: ED

- IES

- 14

- R

- 0008

Submitted by: Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543

- 2393 Telephone: (609) 799

- 3535 Facsimile: (609) 799

- 0005
Project Director: Susanne James

- Burdumy

Reference Number: 40412 .492CONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

iii CONTENTS

PART A. SUPPORTING S TATEMENT FOR PAPERWO RK REDUCT ION ACT SUBMISSION

................. 1

A. Justification

................................ ................................ ................................ ................................ 1

1. Circumstances necessitating the collection of information

................................ ................. 1

2. Purposes and uses of the data

................................ ................................ ........................... 5

3. Use of technology to reduce burden

................................ ................................ ................... 8

7. Special circumstances

................................ ................................ ................................ ...... 10

8. Federal Register announcement and consultation

................................ ........................... 10

9. Payments or gifts

................................ ................................ ................................ .............. 11

10. Assurances of confidentiality

................................ ................................ ............................ 12

11. Additional justific ation for sensitive questions

................................ ................................ ... 13

12. Estimates of hours burden

................................ ................................ ................................ 13

13. Estimates of cost burden to respondents

................................ ................................ ......... 13

14. Estimates of annual costs to the federal government

................................ ....................... 13

15. Reasons for program changes or adjustments

................................ ................................ . 13

16. Plan for tabulation and publication of results

................................ ................................ .... 14

17. Approval not to display the OMB expiration date

................................ .............................. 17

18. Explanation of exceptions

................................ ................................ ................................ . 17

REFERENCES

................................ ................................ ................................ ................................ ............ 18

APPENDIX A : PRINCIPAL SURVEY

APPENDIX B : PRINCIPAL DAILY LOG

APPENDIX C : TEACHER SURVEY

APPENDIX D : DISTRICT

- LEVEL DATA REQUEST MEMO

APPENDIX E : ADVANCE LETTERS

APPENDIX F: CONFIDENTIALITY AGRE EMENTCONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

iv TABLES

A.1 Study data collection activities, by data source

................................ ................................ ............... 4

A.2 Time line for data collection activities

................................ ................................ .............................. 5

A.3 Research questions and data sources

................................ ................................ ............................. 5

A.4 Source, mode, and timing

................................ ................................ ................................ ................ 9

A.5 Technical Working Group members

................................ ................................ .............................. 11

A.6 Annual Reporting and recordkeeping hour burden

................................ ................................ ........ 14

A.7 Principal, teacher, and student outcomes for the impact analysis

................................ ................. 15

A.8 Timetable for project publications

................................ ................................ ................................ .. 17

EXHIBITS

A.1 Logic model for the CEL Program

................................ ................................ ................................ .... 3CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

1 PART A. SUPPORTING S TATEMENT FOR PAPERWO RK REDUCTION ACT
SUBMISSION

This Office of Management and Budget (OMB) package requests clearance for data collection activities to support the Impact Evaluation of Support for Principals. The evaluation will provide important information on the implementation and impacts of intensive professional development for principals. The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) has contracted with Mathematica Policy Research and its subcontractors, the American Institutes for Research (AIR), Social Policy Research Associ ates (SPR), and Pemberton Resea rch to conduct this evaluation.

The evaluation will include implementation and impact analyses. The implementation analysis will draw on information on the scope and sequence of the professional development and data from prin cipal surveys to describe principals’ professional development experiences. 1 The impact analysis will be based on a random assignment design in which participating schools in each district are randomly assigned to a treatment group whose principals are off ered intensive professional development or to a control group whose principals are not. The impact analysis will draw on data from principal and teacher surveys, principal daily logs, and district administrative records to estimate the impacts of the profe ssional development on principal performance, school climate, teacher retention and performance, and student achievement.

A. Justification

1. Circumstances necessitating the collection of information

a
. Statement of need for a rigorous evaluation of pri ncipal professional development
Principals play a n important role in the academic performance of students in their schools (Hallinger and Heck 1998; Harris et al. 2010; Knapp et al. 2006; Leithwood et al. 2004), and there is widespread interest in the pote ntial of intensive principal professional development programs to improve principals’ performance. However, little is known about the effectiveness of these programs and their ability to improve principals’ leadership skills and school quality (Huff et al. 2013)

. The data collection described in this request will provide essential information for describing implementation of a principal professional development and providing rigorous estimates of its effects on principal performance, school climate, teacher retention and performance, and student achievement.

Legislative authorization for this evaluation is found in Title II, Part A of the Elementary and Secondary Education Act (ESEA), Section 2121

- 2123, as amended by the No Child Left Behind Act (NCLB) (20 USC 6621

- 6623). Title II, Part A of ESEA provides funding to states to carry out professional development activities designed to improve the quality of principals

. Part F, Section 9601 of ESEA permits program funds to be used to evaluate activities authori zed under the act.

1 We are not requesting OMB approval for collection of data on the scope and sequence of the professional development because this information will be collected from the professional development provider and will not impose any burden on study participants.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

2 b. Study design and research questions
To learn about the effectiveness of intensive principal professional development, the study team will evaluate an intensive professional development program using a rigorous random assignment design . In fall 2014 we held a competition to select a promising principal professional development program, and, with input from a panel of experts, selected the Center for Educational Leadership (CEL) at the University of Washington. In the coming months we pl an to recruit 100 elementary schools from 10 school districts across the country and randomly assign them to a treatment group whose principals will be offered CEL’s principal professional development program or to a control group whose principals will not be offered this program. We will support and monitor the program for two school years to ensure high quality implementation, and will collect comprehensive data on program implementation and school, principal, teacher, and student outcomes. We will use th ese data to compare outcomes between treatment and control group schools to obtain rigorous evidence on the effectiveness of the
program.
The study will address the following research questions about the implementation and impacts of the CEL principal pro fessional development program:

 Implementation. What are the professional development experiences of principals in the study?

 Impacts. What are the impacts of intensive principal professional development on (a) school climate and educator behaviors

, (b) teacher retention and effectiveness , and (c) student achievement and behavior ?

c. The CEL principal professional development program
CEL’s professional development program is based on a theory of action that principals’ instructional

- focused leadership boost s teaching quality and raise s achievement

. According to the program’s logic model (Exhibit A.1) , if principals have a clear understanding of what quality instruction looks like, as well as a common language to describe and promote elements of quality instruction, and if they know how to lead instructional improvement, instructional practice will become more effective and student achievement will improve.

CEL seeks to develop principals’ knowledge of high

- quality instruction and to equip them to lead t heir teachers to deliver quality instruction, by honing their skills in three content areas:

1. Improving teacher instruction through observation, analysis of data (including information from district evaluation systems), and implementation of “inquiry cycles ” that generate useful feedback for teachers ( instructional leadership );

2. Guiding staffing plans and staff development ( human capital management ); and

3. Creating a school

- wide culture of learning to facilitate improved academic success for all students ( organ izational leadership ).CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

3 Exhibit A.1. Logic model for the CEL Program

CEL’s professional development sessions aim to sharpen principals’ skills in observing classrooms and identifying areas in which teachers need to improve. The professional development sessions are tailored to each district’s instructional framework an d teacher evaluation process , using crosswalks between the different frameworks . CEL also provides individual coaching and professional learning communities (PLCs) aligned with the group

se ssions

. While the length of CEL’s program varies across districts, for the purposes of th is study CEL will provide its program for 24 months, beginning with a summer institute in 2015. 2 During the 4

- day summer institute, CEL will introduce principals to co re program content; establish relationships between principals and coaches; analyze school data; and help principals plan specific leadership actions. After the summer institute, CEL will support principals in treatment schools throughout the 2015

– 2016 sch ool year, to help them apply lessons from the summer institute in their schools. T his support will include 8 one

- day professional development sessions in each district; 10 half

- day one

- on

- one coaching sessions (primarily in

- person but some may be virtual); and quarterly online PLCs

. In this way, p rincipals will interact at least twice a month with the prof

essional development provider.

The second year of the program will begin with a summer institute in 2016 for any principals who are new to one of the 50 t reatment schools in the study. Principal coaching will continue for all principals in treatment group schools throughout the 2016

– 2017 school year.

2 The institute will be offered on two dates, to help ensure all treatment group principals can attend.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

4 d. Data c ollection needs

T his study includes several data collection efforts that are summarized in Table A.1. The purposes and uses of these data are described in Section A.2 below.

Table A

. 1. Study data collection activities , by data source

Data

source Mode , timing , and respondent Key constructs and outcomes
Principal survey (treatment and control groups) Annual surveys 30

- minute web

- based survey, with telephone option

and in

- person follow

- up, administered spring 2016 and spring 2017 to treatment and control group principals Amount and usefulness of professional development received by principal during the current school year and the preceding summer Principals’ leadership practices during the school year (human capital management, instructional leadership, and organizational leadership) District and school context (resource availability, degree of principal autonomy, and district policies affecting principal practice) Principal background characteristics (demographic traits, educational degrees, certification/licensure, and professional experience) Principal log (treatment and control groups) Daily logs 1 5

- minute web

- based logs completed on 5

consecutive days by treatment and control group principals, during four weeklong periods of the 2015

– 2016 and 2016

– 2017 school years Time spent by principal on professional development activities during th e school year Time spent by principal on various leadership practices (human capital management, instructional leadership, organizational leadership, or other) during the school year Teacher survey (treatment and control groups) Annual surveys 30

- minute web

- based survey, with telephone and

hard

- copy options and in

- person follow

- up, administered to treatment and control group teachers in spring 2016 and spring 2017 Principals’ use of leadership practices and quality of principals’ instructional a nd organizational leadership during the school year Amount and usefulness of formal and informal professional development received by teachers during the school year

Amount and usefulness of instructional feedback teachers received from principal or other school leaders during the school year Teachers’ use of student achievement data and changes to instructional practice (teaching methods, student work, student assessment, and topics or materials) during the school year School climate and culture (professional climate, staff collaboration, student engagement and belonging, family engagement, and school safety) Teacher background characteristics (demographic traits, educational degrees, certification/licensure, and pro fessional experience)
District records (treatment and control groups)

Staff records Electronic grade and school assignment data for prior

and current school years, and background characteristics for prior school year, requested from all districts in fall 2015, fall 2016, and fall 2017 Principal and teacher mobility, teacher retention, teacher hiring, and principal and teacher background characteristics

Staff performance records Electronic performance data for prior school years,

requested from all distr icts in fall 2016 and fall 2017 b Teachers’ instructional effectiveness and principals’ leadership effectiveness

Student records Electronic student records data for prior school years,

requested from all districts in fall 2016 and fall 2017 b Student achievement (test scores), behavioral outcomes (attendance and disciplinary records), and background characteristics (demographic traits, grade level, English language learner status, and special education status)

b We will request teacher and principal pe rformance data and student records for the 2014

– 2015 and 2015

– 2016 school years in fall 2016 and for the 2016

– 2017 school year in fall 2017.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

5 e. Time line for data collection activities
The evaluation is expected to be completed in five years, with 2.5 years of data collection. Table A.2 shows the schedule of data collection activities.

2. Purposes and uses of the data

T o address the study ’s researc h questions, the evaluation will collect and analyze data from several sources. IES will use these data to better understand principals’ professional development experiences, and to estimate the impact of the professional development on school, teacher, and student outcomes. Table A.3 lists research questions and specific data sources that will be used to answer them

— in this table we have broken Research Question 2 into three parts, to correspond to three sets of outcomes we will examine. We describe how the study will use each data source below. Information will be collected by Mathem atica Policy Research and its partners AIR, SPR, and Pemberton Research, under contract with ED [contract number ED

- IES

14

- R

- 0008].

Table A.2. T ime line for data collection activities

Activity Date

Data for 2015

- 201 6 school year Collect principal daily logs 9 /2015 through 5/2016

Collect principal and teacher survey data 3/2016 through 5/2016

Collect district records a 8/2015 through 12/2015; 8/2016 through 12/2016
Data for 2016

- 201 7 school year Collect principal daily logs 9 /2016 through 5/2017

Collect principal and teacher survey data 3/2017 through 5/2018

Collect district records b 8/2017 through 12/2017

a In fall 2015, we will only collect data on teacher assignments for fall 2015. In fall 2016 we will collect district records data for the prior (2014

– 2015) school year, as well as teacher assignment data for the 2015

– 2016 school year.

b In this data collection, we will also collect district records data on teacher and principal assignments for the 2017

– 2018 school year.

Table A. 3 . Research qu estions and data sources

Research question Data source(s)

1. Implementation What are the professional development experiences of principals in the study ? Principal surveys

2. Impacts

a

. Impacts on intermediate outcomes

. What are the impacts of intensive principal professional development on school climate and educator behaviors? Principal surveys Principal daily logs Teacher surveys

b

. Impacts on teacher outcomes

. What are the impacts of intensive principal professional development on teacher retention and effectiveness? District records Staff records Staff performance records

c

. Impacts on student outcomes

. What are the impacts of intensive principal professional development on student achievement and behavior? District records Student recordsCONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

6 Principal S urveys. S urveys administered to treatment

- and control

- group principal s in years 1 and 2 will collect information on their leadership practices and perceptions of school climate. We will use this information to examine the effect of professional development on leadership practices and school climate (research question 2a). We will also ask principals to report the amount, cont ent, and usefulness of their professional development experiences. We will use this information to document treatment and control group principals’ perceptions of their professional development experiences (research question 1). In addition, p rincipals wil l provide data on their backgrounds, professional experience and educational attainment, and the district
context in which they operate ( including, for example , degree of perceived autonomy and barriers to school improvement ). We will use this information to document the context of principals’ professional development experiences (research question 1) and to measure whether impacts of intensive professional development vary across districts with high and low levels of principal autonomy (research question 2 ). 3

Principal Daily L ogs. D aily logs will collect information on how principals allocate their time across different categories of leadership practice. Logs will be administered to treatment and control groups on five consecutive days during four weeks of school years 1 and 2. We will use this information to examine the effect of professional development on daily time

- use in different domains of leadership (research question 2a). As compared to annual surveys, d aily logs have been shown to more accurately c apture frequently occurring behaviors, such as day

- to

day instructional leadership and managerial responsibilities, because they allow for measurement of daily fluctuations and are less susceptible to recall error (Camburn et al. 2010a, 2010b). P rincipal l ogs will complement annual principal surveys by ensur ing that the study team can examine impacts on how principals are actually spen ding their time on leadership behaviors on a
g iven day, rather than relying only on annual recall; and allow ing detection of changes in principals’ leadership behaviors within a year.

Teacher S urveys. Surveys administered to teachers in years 1 and 2 will collect information on teachers’ experiences during the school year, including the amount and perceived usefulne ss of professional development and instructional support they received , instructional improvement and data

- use practices, and perceptions of their principal ’s leadership practices and school improvement efforts. We will use this information to examine the effect of principal professional development on teacher practices and teacher perceptions of principal s’ leadership practices. Teachers will also be asked about their schools’ working conditions, staff collaboration and support, and academic culture during the school year. We will use this information to examine the effect of principal professional development on school climate (research question 2a). Finally, t eachers will provide data on their backgrounds , educational attainment, and professional experien ce

. We will use this information to provide context for understanding principal’s professional development experiences (research question 1) and to examine how principal

3 To obtain a district

- level measure of principal autonomy, we plan to average reported levels of principal autonomy across treatment and control group principals in each district. However, if we find significant differences in reported autonomy between tre atment and control group principals, suggesting that reported autonomy may have been influenced by the intervention, we will instead construct the district

- level measure of autonomy based only on the reports of control group principals.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

7 professional development affects the mix of teachers hired and retained in the study s chools (research question 2b).

District R ecords. W e will collect data on principals, teachers, and students from the district records listed below

. We will use this information to es timate effects of principal professional development on staff mobility, re tention, and performance (research question 2b) and on s tudent achievement and behavior (research question 2c), and we will also use district data to draw a sample of teachers to take the teacher survey in years 1 and 2.

 Staff records. We will collect data on principals’ school assignments and teachers’ school assignments and the grades and subjects taught. We will use this information to examine effects of principal professional development on principal turnover and teacher retention (research question 2b) . W e will also use it to draw a random sample of teachers to take the survey in years 1 and 2 of the study. We will collect available information on characteristics of the study sample, such as principal and teacher demographics (age, sex, race, and ethnic ity), educational attainment (certifications, degrees, and scores on licensure or certification exams), and years of teaching experience. We will use this information to describe the study context in the implementation analysis (research question 1), and t o examine whether impacts of principal professional development vary depending on baseline values of these characteristics, such as principal experience (research question 2).

 Staff performance records. We will collect principal and teacher performance ratings from district evaluation systems. These may include composite ratings, observation ratings, teacher value

- added scores, school value

- added scores, or scores on student learning objectives. We will use this information to examine the effect of principal professional development on teacher and principal effectiveness as assessed by their district systems (research question 2b). In conjunction with staff records described above, we will also use this i nformation to examine effects on hiring and retention of effective teachers, dismissal of ineffective teachers, and ways that principals assign teachers of differ ent performance levels to classes (research question 2b).

 Student records

. W e will collect inf ormation from student records for years 1 and 2, as well as the school year before the program started . We will obtain student standardized test scores in math and English/ l anguage a rts for all tested grades , which we will use t o examine the effect of prin cipal professional development on student achievement (research question 2c). We will also obtain data on student attendance and discipline, which we will use to examine effects on student behavior. Finally, we will collect data on student demographics (ag e, sex, race , and ethnicity ) and participation in other programs ( free or reduced

- price lunch eligibility , English Language Learner status, and individualized education plan status)

. We will use this information to describe students in the study sample and use baseline values of these measures and student test scores to improve precision of estimates in the impact analysis (research question 2).

Data collection in year 1. We will use information collected during year 1 to examine principals’ professional de velopment experiences (research question 1) and impacts of principal professional development during the initial year of program im plementation (research question 2). We will use information on year 1 educator behaviors and teacher and student
outcomes to inform research question 2, by examining whether principal professionalCONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

8 development resulted in differences between the treatment and control groups after the first year of the intervention’s implement ation

.

Data collection in year 2. We will use the infor mation collected during year 2 to examine the impacts of intensive principal professional development one year later , after two years of the intervention’s implementation (research question 2). More specifically, we will use the data collected in year 2 to examine whether principal professional development resulted in differences in school climate, teacher and principal practices

, and teacher , principal, and student outcomes after two years of professional development

.

3. Use of technology to reduce burd en The data collection plan is designed to obtain information in an efficient way that minimizes respondent burden. When feasible

, we will gather information from existing data sources , using the most efficient methods available

. Table A. 4 provides inform ation on the source, mode, and timing for each data collection activity

.

A web

- based survey will be the primary mode of data collection for principals and teachers in the study. The web

- based surveys and log will enable respondents to complete the data col lection instrument at a location and time of their choice, and its built

- in editing checks and programmed skips will reduce the level of response errors. We will also offer two additional modes (telephone and in

- person) of interviewing for educators who pr efer these methods to reduce burden on them.

District records we will collect will be computer files provided by districts. Whenever possible, districts will upload the files to a secure Mathematica FTP site. While we will specify the required data element s, we will accept any format in which the data are provided, to reduce burden for the district. Regardless of the form in which it is received, these data will be converted to a consistent format so that they can be combined with data submitted by other di stricts and will be suitable for analysis.

4. Efforts to avoid duplication of effort

Whenever possible, the study team will use existing data from districts, such as from their records for staff and students. This will reduce respondent burden and minimize duplication of data collection efforts.

5. Methods to minimize burden on small enti ties

No small businesses or entities will be involved as respondents.

6. Consequences of not collecting data

These data are needed to evaluate intensive principal professional development. Little is known about the ability of principal professional develop ment programs to improve principals’ leadership skills and school quality. Failing to collect these data would miss an opportunity to inform the field with rigorous evidence on the effectiveness of intensive principal professional development.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

9 Table A. 4

. Source, mode, and timing

Data source Mode and timing Respondent grou p Principal s urvey Annual Survey 30

- minute web

- based survey, with telephone option and in

- person follow

- up , administered in spring 2016 and spring 2017 Treatment and c ontrol

Principal l og Daily logs 1 5

- minute web

- based logs completed on 5 consecutive days, during four weeklong periods of the 2015

– 2016 and 2016

– 2017 school years Treatment and c ontrol

Teacher survey Annual Survey 30

- minute web

- based survey, with telephone and hard

- copy options and in

- person follow

- up, administered in spring 2016 and spring 2017 Treatment and control

District records
Staff records Electronic grade and school assignment data and background characteristics for prior and current school years, requested from districts in fall 2016 and fall 2017 Treatment and control

Staff performance records Electronic performance data for prior school years, requested from districts in fall 2016 and fall 2017 Treatment and control

Student records Electronic student records data for prior school years, requested from districts in fall 2016 and fall 2017 Treatment and control

Consequences of not collecting data from specific sources

. Each of the four data sources provides information needed for the evaluation.

Without information from principal surveys , we will be unable to examine the professi onal development experiences of principals in the study, nor to compare the experiences of control

group principals to those of treatment

- group principals. In addition, we will be unable to examine intermediate impacts of intensive principal professional development on principals’ leadership practices and school

- wide safety. Finally, without information on principal s’ demographic background, professional experience, and district context, we will be unable to describe the sample of principals participating in the study or capture principal and district characteristics that may influence the implementation or effectivenes s of the prin cipal professional development.

Without the information from principal daily logs , we will be unable to examine how professional development impacts principals’ daily time

- use in different domains of school leadership. Without collecting this information on five consecutive days per observation period, we will be unable to reliably measure principal time

- use. Without collecting this information during four weeklong observation periods per school year, we will be unable to assess whether princip al professional development induces changes in principals’ time

- use over the school year.

Without teacher surveys, we will be unable to examine how intensive principal professional development impacts teachers’ behaviors and their perceptions of profession al development opportunities, principals’ leadership practices, and school improvement efforts. Without the information on teacher perceptions of school climate, we will be unable to assess school

- wide changes in working conditions, staff collaboration and support, and academic culture. Finally,CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

10 without the information on teachers’ demographic backgrounds, educational attainment, and professional experience, we will be unable to capture teacher characteristics that may influence the implementation or effect iveness of the principal professional development.

Without district records , we will not be able to analyze the ultimate impact of the principal professional development on teacher or student outcomes. W ithout the following information,

w e will not be able to assess changes in principals’ ability to hire and retain effective teachers, determine effective teaching assignments, improve teachers’ performance, or improve student achievement and behavior :

 Without information from staff records , we will be unable to examine impacts of the principal professional development on staff mobility (principal turnover and teacher turnover), hiring and retention outcomes (hiring and retention of effective teachers and dismissal of ineffective teachers), a nd teacher assignments to grade levels or subjects. We will also be unable to draw a random sample of teachers for the teacher surveys, preventing us from capturing teacher perceptions about principal practices

.

 Without information from staff performance r ecords , we will be unable to examine impacts of the principal professional development on principal and teacher effectiveness or teacher

- effectiveness

- based staffing outcomes (e.g., hiring or retaining effective teachers).

 Without information from student records , we will be unable to examine impacts of the principal professional development on student outcomes (achievement, attendance, and disciplinary problems). Moreover, without information from prior

- year student records, we will be unable to assess the validity of random assignment (i.e., establish baseline equivalence between treatment and control groups).

Consequences of not collecting data at the end of years 1 and 2

. If we do not collect data at the end of year 1, we will be unable to examine progra m impacts

— such as changes in principal or teacher practices

— after the initial year of program implementation . In addition, we will not be able to examine differences across the treatment and control groups in principals’ professional development experience s

— such as the amount, focus, or quality of professional development received

— during the first year of program implementation

. If we do not collect this data at the end of year 2, we will be unable to examine effects of the principal professional developmen t after the second year of the program. It may take time for principals to implement leadership improvements learned through professional development. Consequently, school improvements and changes in teacher behaviors, teacher outcomes, or student outcomes may require time to become apparent. Without the year 2 information, we will be unable to detect any such effects that require time to develop.

7. Special circumst ances There are no special circumstances involved with this data collection

.

8. Federal Register announcement and consultation

a. Federal Register annou ncement

The 60

- day notice to solicit public co mments was published in [insert volume, number, and page] of the Federal Register on [insert date]

. [Insert text regarding comments]. The 30

- dayCONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

11 n otice to solicit public comments was published in [ insert volume, number , and page ] of the Federal Register on [insert date ].

b. Consultations outside the age ncy In formulating the evaluation design, the study team sought input from the technical working group ( TWG )

, which includes practitioners and experts in evaluation methods and data analysis, state assessment programs, and education reform. We will continue to consult with the TWG throughout the study on other issues that would benefit from their inpu t. Table A.5 lists the TWG members.

Table A. 5 . Technical Working Group m embers

Name Title and a ffiliation

Eric Camburn Associate Professor and Senior Researcher, Consortium for Policy Research in Education

(CPRE ), University of Wisconsin

- Madison Roger Goddard Professor and Fawcett Chair, Department of Educational Studies, the Ohio State University

Jason Grissom Assistant Professor and Faculty Affiliate, Center for the Study of Democratic Institutions,

Vanderbilt University

Jason Huff Educational Consu ltant, Seattle Public Schools Carolyn Kelley Professor, Educational Leadership and Policy Analysis , University of Wisconsin

- Madison Jim Kemple Executive Director of the Research All iance for New York City Schools; Research Professor at

the Steinhardt School of Culture, Educatio n, and Human Development, New York University

Susanna Loeb Barnett F amily Professor of Education, Stanford University John Nunnery Executive Director, Center for Educational Partnerships , Old Dominion University

Jeff Smith Pro fessor, Economics, University of Michigan

9. Payments or g ifts Burden payments have been proposed for the principal survey, the principal log, and the teacher survey. Payment will be made in the form of a check or gift card. During each round of data collection, principals completing the survey will receive a $ 3 0 payment . In addition, principals will be given $10 for each day a log is completed for a possible total of $50 per round of data collection and a total of $200 across all four rounds of data collection

. While all aspects of this data collection are vital to the success of the study, the principal log is a critical outcome for which the control teachers will have no incentive to fill out. Adequately incentivizing principals to regularly log time spent per forming the activities throughout the day in addition to entering the data into the online database, will increase the likelihood that accurate data is collected across the span of the data collection period.

Teacher will receive a payment of $30 for comp leti ng the teacher survey. Th e payment s are proposed because high response rates are needed to make the survey findings reliable, and we are aware that principals and teachers are the target of numerous requests to complete surveys on a wide variety of top ics from state and district offices, independent researchers, and ED

. The proposed amounts are within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB.CONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

12 10. Assurances of c onfidentiality The study team has established procedures to ensure the confidentiality and security of its data

. This approach will be in accordance with all relevant regulations and requirements , in particular the Education Sciences Institute Reform Act of 2002, Title I, Subsection (c) of Section 183, which requires the director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of S ubsection (d) of Section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.

The stud y team will protect the full privacy and confidentiality of all individuals who provide data . The study will not have data associated with personally identifiable information (PII), as study staff will be assigning random ID numbers to all data records and then stripping any PII from the data records . In addition to the data safeguards described here, the study team will ensure that no respondent names, schools, or districts are identified in publicly available reports or findings, and if necessary, the stu dy team will mask distinguishing characteristics . A statement to this effect will be included with all requests for data :

“ Mathematica Policy Research and its subcontractors AIR and SPR follow the confidentiality and data protection requirements of IES (Th e Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies respondents to anyone outside the study team, except as required by law. ”

Mathematica employs t he following safeguards to ensure confidentiality:

 All Mathemat ica employees sign a pledge that emphasiz es the importance of confidentiality and describ es their obligation (see Appendix F )

.

 Secure FTP services allow participating school districts to securely transfer encrypted data files to the study team

. Internal networks are all protected from unauthorized access utilizing defense

- in

- depth best practices, which incorporate firewalls and intrusion detection and prevention systems. The networks are configured so that each user has a tailored set of rights, granted by the network administrator, to files approved for access and stored on the LAN. Access to hard

- copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

 Computer data files are protected with passwords

, and access is limited to specific users

, who must change their passwords on a regular basis and conform to strong password policies

.

 Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use.

After the study concludes, the study data may be transmitted to the National Cente r for Education Statistics (NCE S ) for saf e keeping as a restricted

- use file. All other versions of the dataCONTRACT NUMBER: ED

- IE S

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

13 will be destroyed. Prior to transmittal, the data will unde rgo careful screening, and modification if necessary, to ensure that there is no unacceptably high level of disclosure risk for protected respondents. Researchers wishing to access the data for secondary analysis must apply for an NCE E license and agree to the applicable rules and procedures guiding the use of restricted

- use files.

11. Additional justification for sensitive questions No questions of a sensitive nature will be included in this study.

12. Estimates of hours burden Table A.6 provides an estimate of time burden for the data collections, broken down by instrument and respondent. These estimates are based on our experience collecting administrative data from districts and administering surveys to school principals and t eachers.

The total of 2,235 hours covers all five years of the evaluation, and includes the following efforts: up to 12 hours, annually for 3 years, for each of the 10 districts to collect and assemble administrative records on teachers, principals, and s tudents participating in the evaluation; 30 minutes, annually for two years, for 85 principals (85 percent of the anticipated 100 principals in the sample) to complete the principal survey, plus 15 minutes, 20 times per year for 2 years, for 85 principals to complete the principal log; and 30 minutes, annually for two years, for 1,020 teachers (85 percent of the anticipated 1,200 teachers in the sample) to complete the teacher survey. Annual number of respondents is 372 and annual responses for three years of this collection are 1,880

. T he total annual burden for this collection is 745 hours.

13. Estimates of cost burden to respondents There are no additional respondent costs associated with this data collection beyond the burden estimated in item A12.

14. Estimates of annual costs to the federal government

The estimated cost for this five

- year study, including selection of the intervention provider, data collection instruments, justification package, district recruitment, provision of professional developme nt, support for the provider, data collection, data analysis, and report preparation, is $12,198,695 or approximately $2,439,739 per year.

15. Reasons for program changes or adjustments This is a new information collection request.CONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

14 Table A. 6

. Annual Reporting and recordkeeping hour burden

Respondent/data r equest Total n umber
of respondents (target/expected

response r ate ) Total annual

r esponses Total annual
h ours

Districts Teacher and principal grade and school assignment request (3 times), performance request (2 times), and s tudent records request (2 times) (Appendix D ) 10 10 93

Principals Principal survey (2 times) (Appendix A) , and Principal log (40 times) (Appendix B) 85 1 190 312

Teachers Teacher survey (2 times) (Appendix C) 1020 680 340

T otal 1,115 1, 880 745

Note: Reporting on an annualized data collection over 3 years: Annual number of respondents is 372 (1,115/3)

and the total annual burden hours for this collection is 745 hours

.

16. Plan for tabulation and publication of results

a. Tabulation p lans
Implementation ana lysis The implementation analysis will contextualize findings from the impact analysis

, detailing the fidelity of program implementation and challenges encountered, identifying mechanisms through which the professional development might improve outcomes, and documenting the
extent of the contrast between the intervention and the counterfactual (that is, wh ether the experiences of principals in the treatment group differed significantly from those of principals in the control group).

Descriptive analysis of professional development. Drawing on information on the intended scope and sequence of the professiona l development gathered by the study’s technical assistance team, we will first describe the intervention as planned, including its objectives, and the timing, hours of professional development (session

- based and coaching), delivery mode, and content of sup port. 4 We will draw on principal surveys to describe the professional development received . We will describe the number of hours and focus of professional development received by principals and principal s’ perceptions of the quality or usefulness of the pr ofessional development. Using information from the principal surveys, teacher surveys, and district records data, w e will also describe the characteristics of the districts in which the intervention was delivered.

4 These data will b e collected by the study’s technical assistance team from the professional development provider, and their collection will not impose burden on study participants.CONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

15 Comparison of the experiences of treatment and control group principals. Using data from the year 1 principal survey, we will describe the professional development received by the treatment group principals relative to that received by control group principals. The analysis will detail the content and frequency of professional development both groups received and test for statistically significant differences between the two groups.

Impact analys is The impact analysis will examine the effects of CEL on the principal, teacher, and student
outcomes in Table A. 7

. Key outcomes of interest include (1) teachers’ receipt of professional development and perceived quality and frequency of feedback from the principal; (2) school climate outcomes, including staff collaboration and perceptions of support; (3) teachers’ perceptions of principals’ leadership practices

, school climate, and educator behaviors ; (4) teacher effectiveness (as measured by the district’s central office) and teacher retention ; and (5) student test scores, attendance, and behavior.

Table A. 7

. Principal , teacher, and student outcomes for the impact analysis

Level Outcomes
Principal  District central office performance ratings
Teacher  Perceptions of s chool climate

– Staff collaboration

– Perceived support

 Perceptions of principals’ leadership practices

 Teachers’ r eceipt of professional development and principal feedback (quality and usefulness)

 Retention

 Performance ratings

– All current teachers

– New teachers

 Assignment

– Difference between teacher performance in high

- stakes, low

- stakes grades

– Variance of average teacher performance across grades (to examine whether high

- and low

performing teachers grouped within grades)
Student  Achievement

 Attendance

 Behavior

We will estimate impacts using the following model:

(1)

,

where Y

ij is the outcome of interest for individual (principal, teacher, or student) i in school j ; α is an intercept term; T j is an indicator equal to one if the school is assigned to the treatment group and zero otherwise; P ij is a vector of baseline school

- or student

- level characteristics, and Z j is a vector of fixed effects corresponding to the study’s random assignment blocks; δ and γ are coefficient vectors; and ε ij is a random error term, clustered at the school level. The coefficient βCONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

16 represents the average impact of the principal professional development. 5 We will apply a weighting scheme that gives an equal weight to each of the schools in the sample (regardless of the number of principals, teachers, or students in the sample in each school).

For teacher assignment, we will investigate how assignment to the CEL program affects the way principals assign teachers to classes

— for instance, whether it leads principals to pair higher

performing teachers with lower

- performing teachers in the same grade (to facilitate mentoring)

, or whether it leads them to reassign low

- performing teachers to grades not covered by high

stakes tests. We will also examine p rincipal and teac her performance ratings as measures of effectiveness , but we will interpret these data with caution

. S ome districts will rate principals at least partly based on school value added . For new principals, this rating is affected by the teachers inherited from prior principals (Chiang et al. 2012). Similarly, teacher performance ratings may have two limitations: (1) our experience suggests that nearly all teachers are given

‘ satisfactory ’ ratings , so there may be little variation in the data; and (2) teacher ev aluations typically incorporate ratings by principals, so differences in ratings could reflect effects of the professional development program.
Our main impact estimates will reflect the impact of the professional development on the schools whose principa ls were offered the professional development, whether or not they actually participated for the full two years . To examine the effects on schools that received the full intended “dosage”

— that is, treatment schools whose principal s received both years of th e CEL professional development

— we will estimate local average treatment effects (Imbens and Angrist 1994), using treatment status as an instrumental variable for the proportion of total hours of intended CEL activities the principal attended. Even schools with principal turnover during the study c ould receive the full intended dosage if both the original and replacement principals attend all offered CEL activities during their time at the school

. Subgroup analyses. We will explore whether principal professional development is more effective in some contexts than others . These results may be useful to policymakers seeking to know whether principal professional development would be useful in particular kinds of districts or ho w best to target principal professional development within districts. S ubgroups of interest will be (1) novice and experienced principals (because new principals may be more in need of and more receptive to coaching and formal group professional developmen t sessions than more experienced principals); and (2 ) districts with high versus low levels of autonomy (because principal professional development may be less effective in districts where principals have less influence on hiring, dismissal, and curricular decisions). 6 S tatistical power for subgroup analyses

5 We will not adjust for principal and teacher characteristics in this model. Because the in tervention could influence the composition of principals and teachers, though retention or mobility, observed characteristics of principals and teachers could reflect program effects, and including them could lead to an incorrect estimate of program impact s.

6 As discussed above, we will construct a district

- level measure of principal autonomy by averaging reported levels of principal autonomy across treatment and control group principals in each district. However, if we find significant differences in repo rted autonomy between treatment and control group principals, suggesting that reported autonomy may have been influenced by the intervention, we will instead construct the district

- level measure of autonomy based only on the reports of control group princi pals.CONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

17 will be lower than for the full sample

— subgroup findings will thus be less definitive than those for the full sample, but can provide suggestive evidence of impacts for these groups.

b. Publication p lan s The study team will prepare two reports. The first will describe the implementation of the principal professional development program and present initial impacts of the program on intermediate outcomes, including school climate and educator behaviors, in year 1 (the 2015

– 2016 school year). The second report will examine implementation and the effects of principal professional development on additional intermediate outcomes, including school climate and educator behaviors, in year 2 (the 2016

– 2017 school y ear), teacher effectiveness and retention, and student outcomes, including test scores, attendance, and behavior from years 1 and 2. Both reports will be released on the IES website.

17. Approval not to display the OMB expiration date

All data collection instruments will include the OMB expiration date.

18. Explanation of exceptions No exceptions are requested.

Table A. 8

. Timetable for project publications

Activity Date

Year 1 repor t

First draft January 2017

Revised draft March 2017 Final report December 2017

Year 2 rep ort

First draft August 2018

Revised draft October 2018 Final report June 2019CONTRACT NUMBER: ED

- IES

- 14

- R

- 0008 MATHEMATICA POLICY R ESEARCH

18 REFERENCES

Camburn, E., J. Spillane, and J. Sebastian. “ Assessing the Utility of a Daily Log for Measuring Principal Leadership Practice.” Educational Administration Quarterly, vol. 46, 2010a,
pp. 707

– 737.

Camburn, E., J. Huff, E. Goldring, and H. May. “ Assessing the Validity of an Annual Survey f or Measuring Principal Leadersh ip Practice.” Elementary School Journal, vol. 111 , no. 2, 2010b, pp. 314

– 335.

Camburn, E. Goldring, J. Sebastian, H. May, and J. Huff. “Conducting Randomized Experiments with Principals: A Case Study.” Educational Administration Quarterly

, forthcoming.

Goddard, R., A. Miller, M. Kim, Y. Goddard, P. Schroeder, and E.S. Kim. “Evaluation of Principals’ Professional Development Learning: Results from A Randomized Control Trial.” Paper presented at the Annual University Council for Educational Administration Convention, Pittsburgh, Pennsylvania, 2011;

Goff, P.J., E. Guthrie, E. Goldring, and L. Bickman. “Changing Principals’ Leadership Through Feedback and Coaching.” Journal of Education Administration, vol. 52, no. 5, 2014,
pp. 682

- 704.

Kottkam p, R.B., and E.A. Rusch. “The Landscape of Scholarship on the Education of School Leaders, 1985

- 2006.” In Handbook of Research on the Education of School Leaders, edited by M.D. Young, G.M. Crow, J. Murphy, and R.T. Ogawa. New York: Routledge, 2009,
pp. 23

- 85.

Nunnery, J.A., S.M. Ross, and C. Yen. “An Examination of the Effect of a Pilot of the National Institute for School Leadership’s Executive Development Program on School Performance Trends in Massachusetts.” Norfolk, VA: Old Dominion University, Center for Educational Partnerships, 2010a.

Nunnery, J., S.M. Ross, and C. Yen. “The Effect of a Pilot of the National Institute for School Leadership’s Executive Development Program on School Performance Trends in Pennsylvania.” Norfolk, VA: Old Dominion Univer sity, Center for Educational Partnerships, 2010b.

Nunnery, J., S.M. Ross, and C. Yen. “The Effect of a Pilot of the National Institute for School Leadership’s Executive Development Program on School Performance Trends in Pennsylvania: 2006

- 2010 Pilot Cohor t Results.” Norfolk, VA: Old Dominion University, Center for Educational Partnerships, 2011.www.mathematica

- mpr.com Improving public well

- being by conducting high quality, objective research and data collection

PRINCETON, NJ ■ ANN ARBOR, MI ■ CAMBRIDGE, MA ■ CHICAGO, IL ■ OAKLAND, CA ■ WASHINGTON, DC

Mathematica ® is a registered trademark of Mathematica Policy Research, Inc.