1 

# I **MPACT** 

# E **VALUATION OF** 

# T **RAINING IN** 

# M **ULTI** 

# - **TIERED** 

# S **YSTEMS OF** 

# S **UPPORT FOR**



# B **EHAVIOR** 

# (**MTSS -**  **B )**  **OMB C**  **LEARANCE** 

# R**EQUEST** 

# : Supporting Statements A 

# D **ATA** 

# C **OLLECTION April**  **201 5**  **P REPARED FOR**  

## : Institute of Education Sciences United States Department of Education Contract No. ED - IES - 14 - C - 003 **P REPARED**  

## B **Y** 

## : MDRC 16 East 34 th 

## Street, 19 th 

## Floor New York, NY 10016 Fred Doolittle, Project Director Fred.Doolittle@mdrc.org (212) 340 - 86382 **TABLE OF CONTENTS** IN TRODUCTION 4 THEORY OF ACT ION AND RESEARCH QUESTI ONS 5 CHARACTERI STICS OF T HE TREATMENT TO BE TESTED 7 STUDY DESCRIPTION 10 SUPPORTING STATEMENT FOR PAPERWORK REDU CTI ON ACT SUBMI SSI ON 1 7 A. JUSTIFICATION 1 7 1. Circumstances Making Collecti on of Information Necessary 1 7 2. Purposes and Use of the Data 1 7 3. Use of Tec hnology to Reduce Burden 1 8 4. Efforts to Avoid Duplication 1 8 5. Methods to Minimize Burden on Small Entities 1 8 6. Consequences of Less Frequent Data Collection 1 8 7. Spe cial Circumstances Relating to Federal Guidelines 1 8 8. Federal Register Comments and Persons Consulted Outside the Agency 1 8 9. Pay

ment or Gifts to Respondents 1 9 10. Assurances of Confidentiali ty Provided to Respondents 2 1 11. Justification of Sensitive Questions 2 2 12. Estimates of Annuali zed Burden Hours and Cost 2 3 13. Estimate of Other Total Annual Cost Burden to Respondents and Record Keepers 2 5 14. Annualized Cost t o the Federal Government 2 5 15. Explanation for Program Changes or Adjustments 2 53

16. Time , Schedule , Publication and Anal ys is Plan 2 5 17. Approval to Not Di splay OMB Expiration Date 2 6 18. Exp lanation of Exceptions to the Paperwork Reduction Act 2 6 REFERENCES 2 7 APPENDI CES A. Interview Protocols: MTSS - B Coach B. Interview Protocols: Administrators C. Interview Protocols: Behavior Team Leader D. Interview P rotocol : Students E. Interview P rotocol: Staff F. Staff and Teacher Survey G. Student Survey H. Teaching Ratings of Student Behavior I. District Records Data Collection Request Letter J. Excerpt from IDEA, P.L. 108 - 4464 **INTRODUCTION** Creating and maintaining safe and orderl y s chool and classroom environments to support student learning is a top federal priority , as indicated by the U.S. Department of Education and the U.S. Department of Justice jointly issued guidance on school discipl ine. The implementation of Multi - Tiered Sy stems of Support for Behavior (MTSS - B) is seen as a promising approach to improve school and classroom climate, student behavior, and academic achievement, and potentially

reduce the inappropriate identification o f students for special education. MTSS - B is a multi - tiered, sy stematic framework for teaching and reinforcing positive behavior for all students, that includes universal , school wide supports (often called Tier I), targeted interventions to address the problems of students not responding to the universal components (often called Tier II), and intensive interventions (Tier III) for students not responding to Tier II services. The widespread use of MTSS - B nationally contributes to the relev ance of this evaluation. The U. S. Department of Education’s (ED’s) Office of Special Education Programs has supported technical assistance around MTSS - B since 1998. Recent data from the Office of Special Education Programs TA Center on Positive Behavior Interventions and Support indicate that as of 2014, MTSS - B is reportedly implemented in over 21,000 schools across the nation. 1 As the support for and implementation of MTSS - B have increased, researchers have been developing an evidence base for it and other related programs. Numerous descriptive and correlational studies have associated aspects of MTSS - B implementation, or MTSS - B implementation overall, with student outcomes. More recentl y, researchers have begun to conduct causal studies of MTSS - B, although some of thes e initial studies have been on a relativel y small scale (or conducted within a single state) or have been somewhat limited in duration and outcomes. 2 , 3 , 4 A recent meta - anal ys is of research on a broad range of 213 individual school - based, universal social and emotional learning (SEL) programs (including interventions than can be implemented within an MTSS - B program) found that compared to controls, SEL participants demonstrated significantl y improved social and emotional skills, attitudes, behavior, and acad emic performance. 5 Much of this research has focused on individual programs and not examined the impact of a large - scale implementation of both universal, school - wide programs and targeted programs within a MTSS - B framework. Therefore, this evaluation f ocuses on a comprehensive MTSS - B program that includes each of the components usually

associated with MTSS - B: school - wide and classroom - level strategies (universal supports, or Tier I) and individual or small group strategies (targeted interventions, or Ti er II), with appropriate infrastructure (staffing sand data s ys tem). This evaluation will not support intensive supports for students not responding to Tier II interventions (i.e., there will be no Tier III interventions). IE S has contracted with MDRC, AIR , Harvard University , and Decision Information Resources (DIR) to conduct the Impact Evaluation of MTSS - B.

1 Horner (2014). 2 Bradshaw , Mitchell, and Leaf (2010). 3 Horner et al. (2009). 4 OSEP Center on Positive Behavior Interventions and Supports (2007). 5 Durlak et al. (2011).5

The goal of the evaluation is to implement an MTSS - B program with fidelity and determine whether universal supports (Tier I) and a targeted intervention (Tier II) with appropriate infrastructure are effective when implemented in a large number of dis tricts and elementary schools. Using a school - level random assignment design, this evaluation will examine the impact of training elementary

school staff in the implementation of an MTSS - B framework including staff and data infrastructure with universal, school - wide components, as well as classroom - based activities (Tier I) in the first year of implementation as compared to outcomes for schools undert aking ty

pical behavior support practices (These schools are hereafter called business - as usual or BAU schools.) In addition, the study will also examine the impact of this Tier I training plus additional training in a targeted intervention (Tier II) in a s econd year of implementation. Schools that receive MTSS - B training and support are hereafter referred to as “ Program Schools .” The study

will also investigate the differences in behavioral support services between program and BAU schools. Finally , the evaluation team will assess whether the MTSS - B training and support activities are implemented as intended as well as the fidelity of school’s implementation of the intended MTSS - B Tier I and Tier II activities.

**THEORY OF ACTION & RESEARCH QUESTIONS** The impl ementation of MTSS - B is ultimately intended to improve student outcomes. The conduct of school - wide and classroom strategies (Tier I) and individual or small group strategies (Tier II), supported b y appropriate infrastructure, are designed to improve schoo l staff behavior practices as well as school and classroom climate. The improved practices and climate are, in turn, hy pothesized to benefit student behavior and academic outcomes. The individual and small group strategies may

affect the students who need additional support (Tier II) as well as all students if these strategies lessen the disruption of students needing additional support. Finall y, improved student behavior could lead to better student academic outcomes (see Exhibit A - 1).6 **Exhibit A**  **- 1: Conceptual Model of MTSS**  **- B and Intended Outcomes** 

The evaluation is anchored on this conceptual model and will focus on the following research questions:  What MTSS - B training and support activities were conducted?  Is the MTSS - B program implemented with fidelity? What are the key challenges to the implementation of MTSS - B with fidelity?  What is the difference in behavior support practices occurring in program and BAU schools?  What is the impact on school staff practices, school climate and student outcomes of providing training in the MTSS - B framework plus universal, school - wide and classroom Tier I positive behavior supports and a targeted Tier II intervention?  What are the i mpacts on school staff practices , school climate, and student outcomes for relevant subgroups (e.g ., teachers with less experience; at - risk students)?  Does the implementation of MTSS - B reduce any differences in student discipline outcomes across racial an d ethnic groups? Does it reduce the identification of students for special education services?7 **CHARACTERISTICS OF THE TREATMENT TO BE TESTED** In the spring of 2014, MDRC and partner organizations issued a Request for Proposals to providers of training in MTSS - B. In November 2014, we announced the winner of this competition, The Center for Social Behavior Support (CSBS), which is collaboration between The Illinois - Midwest PBIS Network at the School Association for Special Education in DuPage, Illinois (SASE D) and the PBI S Regional Training and Technical Assistance Center at Sheppard Pratt, in Maryland. The training and support activities proposed by CSBS are described below. Following that summary , the memo describes CSBS’ plan for how the MTSS - B Tier I and Tier II program will be implemented in schools. **CSBS Training and Support Activities** CSBS will mostly

follow a “train the trainer” model with some direct support to all school staff and on - site support to the school staff and C oaches throughout implementation. Below we describe the key

phases of their work with schools. **Readiness Period -**  **Training and Support by CSBS to Develop Infrastructure :**  The first step is the development of school - level infrastructures that will support implementation of Tier I a nd Tier II program components, which includes a staffing structure and data s ystem. Tier I infrastructure will be put in place in the late spring of 2015 for all Program Schools . In frastructure for Tier II will be developed in the spring of 2016. **Staffing structure:** CSBS trainers will work with each school to identify a representative School L eadership Team (SLT) and each district to identify MTSS - B Coaches. This will be accomplished during a two - day site visit to each district as well as webinars, email exchanges and phone calls. In addition to helping schools and districts identify the SLT and MTSS - B Coach es , CSBS will explain the MTSS - B program to the district coordinator, administrator, SLT and MTSS - B Coach es as well as the roles and functions o f each individual. During the spring of 2016 CSBS will work with schools to identify a Targeted Team (TT) to support Tier II implementation. Often, the TT is a subset of the SLT. **Behavioral monitoring data system:** During the spring of 2015 CSBS trainers will support the SLT in putting in place the data infrastructure that relies on a web - based behavior monitoring data sy stem called the School Wide Information Sy stem (SWI S). Minimally

three SLT members and MTSS - B Co aches will be trained to use the SWI S data sy stem to drive decision - making about issues such as which teachers may need additional supports with classroom management and which locations in the school at what time may

need more school staff to monitor student behavior. Appropriate staff will be identified and trained to enter data and generate reports. The data infrastructure will be augmented in the spring 2016 to include the data sy stem for Check In Check Out, the primary Tier II intervention. **Initial an d Booster Summer Trainings**  **:** During the summer of 2015, CSBS will host training within each district for the SLT, MTSS - B Coaches and administrators in all Program Schools .8

The goal of the initial CSBS Tier I training during the summer of 2015 is to introduc e the key school - level implementers (MTSS - B Coach es , SLT and administrators) to the core features of MTSS - B Tier I 6 as well as the tools to track implementation fidelity and engage in data - based problem solving. During the summer training each school team drafts a school - level implementation plan. Each school team should leave the summer training prepared to further develop and implement the school - level implementation plan. During the summer of 2016, the key Tier I implementers engage in a booster training in which they review core components, reflect on the previous ye ar and develop a ye ar 2 school - level implementation plan. During this booster training, new school staff are also introduced to the core components of MTSS - B Tier I. Initial summer training f

or Tier II also takes place in the summer of 2016 and includes training for the MTSS - B Coach es , Targeted Team and administrator in the core features of Check In Check Out 7 and how to use CI CO - SWI S data to monitor student progress. The goal of this initial training is for schools to develop and implement their CI CO implementation plan. **Ongoing Training and Support :**  During the 2015 - 2016 and 2016 - 2017 school years, CSBS will provide ongoing support to the MTSS - B Coaches , SLT and administrators and some direc t trainings to all school - staff as schools implement Tier I activities. CSBS trainers will visit all Program Schools four times over the course of the school year . During these visits, they will work with the SLT , administrator and MTSS - B Coaches to introd uce Tier I core components to staff and to train staff in their version of the Good Behavior Game (referred to as the Positive Behavior Game (PBG)), which helps teachers to teach and acknowledge classroom rules aligned with the school - wide expectations. Additionally , they will gather and use data regarding school’s implementation fidelity

to help the SLT to create and modify school’s implementation plans. 8 During site visits, monthly webinars and email/phone exchanges CSBS will support C oaches and the SLT t o train teachers in the 8 CSBS classroom components.

6 Core components of MTSS - B Tier I include teaching and reinforcing specific school - wide behavioral expectations, a classroom management system to support behavio ral expectations, and the use of fidelity data to monitor and im prove implementation. 7 Core components of MTSS - B Tier II (CICO) include use of data to identify and progress monitor students needing additional supports, additional instruction and time for students to develop behavioral skills, additional structures with increased opportunity for feedback from staff, and sharing information about student behavior between school and families. 8 CSBS will use fidelity measures to monitor schools’ progress, i dentify areas in need of improvement, and make adjustments in training when necessary. CSBS plans to use the Effective Behavior Support (EBS) survey and the Tiered Fidelity Inventory (TFI) im plementation tool for this purpose. CSBS will use the Behavior Education Program Fidelity of Implementation (BEP - FIM) to assess fidelity of implementation for the Tier II intervention.9

CSBS’ provision of ongoing support for the Tier II intervention during the 2016 - 2017 school ye ar will follow the same model as their provision of ongoing support for Tier I. However, CSBS will conduct an extra site visit during year 2 of the study so they have sufficient time to provide ongoing support for the implementation of Tier I and Tier II. **MTSS -**  **B Program Implementation** The key components of the Tier I and Tier II MTSS - B program evaluated in this study are described in the paragraphs below. **Tier I Practices:** The school - level implementation of Tier I practices will occur in all Program Sc hools during both the 2015 - 2016 and 2016 - 2017 school years. These practices include school wide activities as well as specific classroom components that are overseen by the SLT and MTSS - B Coach es . **Leadership provided by SLT and MTSS -**  **B Coach es**  **:** The SLT an d MTSS - B Coach es are the lead implementers of the program and are responsible for introducing the program model and implementation plan to other staff members with support from CSBS trainers. They meet regularl y to monitor implementation of the school acti on plan and, based on data collected on student behavior and implementation fidelity, provide support to other school staff to improve implementation. **Ongoing Support for Teachers by the SLT and MTSS -**  **B Coach es**  **:** The SLT and MTSS - B Coach es provide ongoing training and coaching to teachers. These supports include collection of data on teacher performance as well as the provision of targeted feedback to teachers and opportunities to practice the Positive Behavior Game. **School -** **wide implementation of program:** A key component of the Tier I practices of MTSS - B is that the program is implemented school - wide and not just by particular administrators, teachers, or staff members. All school staff will be responsible for implementing the school’s 3 - 5 behavioral expec tations, responding appropriately to positive and problem student behavior, and using student behavior data to develop solutions to specific problems.

**CSBS ’**  **Eight Classroom Components**  Arrang e orderly physical environment  Define, teach, and acknowledge rules aligned with school - wide expectations by modeling desired behaviors and using a gaming strategy;  Explain and teach routines;  Provide specific and contingent praise for appropriate behavior;  Use class - wide group contingencies;  Provide error correction through prompt, re - teach, and provision of choices;  Em ploy active supervisi on – mo ve, scan , interact;  P rovide multiple opportunities to respond.10

**Classroom components:** Within the classroom, teachers establish and maintain a classroom management system, which includes 8 specific components (see text box above). **Tier II Practices** : Tier II will begin in all Program Schools during the 2016 - 2017 school year . During this period, the SLT and MTSS - B Coach es will review behavioral monitoring data to identify students who are not responding to Tier I supports. They will also solicit referrals from school staff and families. These students will receive the Check In Check Out (CI CO ) intervention, which is the Tier II intervention in this study . CI CO is a small group intervention that reinforces and extends the universal expectations by sy stematically

providing a higher frequency of scheduled prompts, pre - correction, and acknowledg ement to participating students. The program consists of students daily “checking in” with an adult at the start of school day to retrieve a goal sheet and receive encouragement and help in setting specific behavioral goals (including skills to practice) f or the day . Throughout the day , teachers (or other school staff) provide feedback to the student and document student behavior on the goal sheet. Students “check out” at the end of the day with an adult, who summarizes the feedback and recognizes accomplis hments. The Targeted Team (TT), usually a subset of the SLT, monitors students’ progress on CI CO and establishes s ys tems of communication with families. **STUDY DE SCRIPTION**  **Study Design** School level random assignment is appropriate for this study because, by

definition, MTSS - B is a whole school intervention delivered at the school level. The anal ys is of MTSS - B impacts will rely on a random assignment design in which schools within each participating district (approximately

10) are randomly

assigned to the following two groups: (1) Program Schools : Training, infrastructure and support to implement Tier I components in ye ar 1 (SY2015 - 2016) and Tier I and Tier II components in year 2 (SY2016 - 20 17) (2) Business As Usual Schools: Continue with any existing practices related to student behavior, and will not receive any additional training and support in MTSS - B from the evaluation. First year impacts of Tier I will be calculated by

comparing first ye ar outcomes in all 60 Program Schools implementing Tier I to first ye ar outcomes in the 30 BAU schools. The second ye ar impacts of Tier I plus II will be calculated b y comparing second ye ar outcomes in the Program Schools to second year outcomes in the BAU schools. A baseline teacher assessment of student behavior (in fall 2015) allows for the identification of a high risk subgroup and the estimation the net impact for high risk students of Tier I (year 1) and Tier I plus II services (year 2) compared t o BAU schools.11

The study team will recruit approximately 90 elementary

schools across approximately ten districts that are willing and able to comply with the requirements of MTSS - B training providers and the research and data collection procedures involved in the evaluation. The impact study

will examine several outcomes of MTSS - B

—teacher practice, school climate, student behavior and academic achievement -- following the theory of action presented above. The study will also consider the difference in behavior support practices in program and BAU schools (service contrast). Finally , the evaluation will describe the CSBS’ training and support activities as well as the implementation of the MTSS - B in programs in schools.

**Data Collection Strategy & Anal ytic Approach**  Exhibit A - 2 provides an overview of data collection activities. Data sources that present a burden to students and/or staff are indicated with an asterisk. A longer description of and justification for the data sources that present a burden i s provided in the paragraphs below the table.12

Exhibit A - 2 . Overview of MTSS - B Data Sources **Timeline # of**  **Respondents per School**  **# of**

**Schools Spring** 

**2015 Summer** 

**2015 Fall 2015**  **Spring**

**2016 Summer** 

**2016 Fall 2016**  **Spring**

**2017 Observation of CSBS training and support**  **activities in Program Schools**  0 60 X X X X X X X **Site v**  **isits*** School tour to observe school rules 0 90 X X X Document reviews 0 90 X X X MTSS - B Coach interview **˅** • 20 minutes per wave in Program Schools . 33 60 X X X Administrator interview^ • 4 5 minutes per wave in BAU schools • 6 0 minutes per wave in Program Schools 1 90 X X X Team leader interview^ • 30 minutes per wave in BAU schools • 45 minutes per wave in Program Schools 1 90 X X X Student interview (randomly selected) • 2 minutes per wave 15 90 X X X Staff interview (randomly selected) • 5 minutes per wave 15 90 X X X13

**Review of w**  **eb -**  **based forms (Coach logs, meeting minute forms and TIPS school**  **forms)** 0 60 X X X X X X **Review of sample of CSBS records (CSBS trainer logs, attendance records, webinars,**  **fidelity data)** 0 60 X X X X X X X **Review of MTSS -**  **B behavior monitoring and fidelity data (SWIS data, SWIS**  **- CICO data &**  **PBIS Apps data)** 0 60 X X X X X X X **School staff survey (All grades k -**  **5 teachers and other relevant staff) ** * 35 minutes for non - teaching staff 40 minutes for teachers 30 90 X X **Student survey ( All students in grades 3**  **- 5)** * 50 minutes to administer survey per wave 240 90 X X **Classroom observations with the ASSIST and possibly CLASS [augmented in**  **Program Schools**  **to included assessment of implementation fidelity]**  0 90 X

**Teacher rating**  **s of student behavior (All**  **teachers in grades 1 -**  **5; all consenting students in class)** * 10 minutes per student per wave 20 90 X X X **District records data collection** (10 districts) 0 90 X X X Note: Activities that present a time burden to students and staff are indicated with an asterisk (*). ^ The structured interviews with the team leader and administrator listed under site visits are longer in Pr ogram Schools because the interview protocol includes questions related to these staff members’ perception of the quality and utility of CSBS training and support. **˅** The MTSS - B Coach role is designated as 0.2 FTE for each program school. On average, however, districts will need 2 different MTSS - B coaches, though one of them may be only part - time. This increases the number of respondents for data collection purposes, b ut does not change the actual staffing.14 **Description of Data Sources that Present a Burden to Staff and Students Site visits**  **:** We will conduct site visits in Program and BAU schools three times over the study period (Fall 2015, Spring 2016 and Spring 2017). The site visit protocols involve document review, observation and interview. The time s required for the interviews conducted as part of the site visit are indicated in Exhibits A - 2 and A - 3 . The data collected from these visits will serve the following purposes: (1) Assess the difference in behavior support practices in program and BAU schools (service contrast) (2) Assess implementation fidelity to the core feature of the MTSS - B Tier I and Tier II programs in Program Schools (3) Ass ess key imp lementers’ (MTSS - B C oach es , SLT team leader and administrator) perception of the quality and utility

of CSBS training and support in Program Schools . To develop the protocols for these site visits, the study team has pulled items from School wide Evaluatio n Tool (SET), which has been found be a valid and reliable measure of MTSS - B Tier I implementation fidelity ; 9 it has also been used in prio r randomized control trials of MTSS - B as a measure of service contrast between Program and BAU schools and found to a ccuratel y discriminate between schools implementing the core features of MTSS - B and those not. 10 The I - SSET assesses the fidelity of Tier II and Tier III implementation , 11 and it has been combined with SET in prior studies to provide a complete picture of th e three tiers of MTSS - B implementation. 12 Understanding staff’s perception of the quality and utility of CSBS training and support in the Program Schools is critical to fully understanding implementation fidelity and to identifying challenges associated w ith the implementation of MTSS - B in a large number of elementary school and districts. The SET/I SSET protocols do not ask school staff to reflect upon the quality and utility

of the MTSS - B training and support they received from CSBS. As a result, the study team will be augmenting the site visit interview protocols in Program Schools with questions for the MTSS - B C oach es , SLT team leader and administrator about their perceptions of the quality and utility

of CSBS training and support. The interview protocol s for the site visits can be found in Appendices A - E . **Staff Survey:** School staff (teachers, administrators and other staff) in all study schools will be asked to participate in a web - based survey about school climate, staff practices related to behavior, and training for behavior support practices and programs in the spring of 2016 and spring of 2017. The survey s also include a number of questions related to individual background characteristics and experiences. The survey for non - teaching staff will take approximately 35 minutes to complete and the survey for teaching staff will take approximately 40 minutes to complete.

9 Horner et al. (2004). 10 Bradshaw , Waasdorp, and Leaf (2012). 11 Anderson et al. (2008). 12 Debnam, Pas and Bradshaw (2012).15

To assess staff practices related to b ehavior, we are drawing items from the Effective Behavior Support S urvey

(EBS). 13 These items assess the extent to which schools are implementing core features of MTSS - B in classroom and non - classroom settings. We have also drawn upon questions from the multi - site evaluation from IES’ Social and Character Development study to identify differences between BAU and Program Schools in the extent to which they have received training for and/or are implementing school - wide and classroom based programs similar to the MTSS - B approach being implemented in Program schools. Additionally , to assess implementation fidelity

the study team has designed questions that ask about strategies specific to CSBS MTSS - B approach; the questions are general enough to appl y to Program and BAU schools but specific enough to the CSBS model that they will inform our assessment of implementation fidelity . We have also designed questions to assess staff’s exposure to training and coaching related to behavior support practices. Staff’s answer s to these questions will inform our understanding of service contrast; full y understanding service contrast is critical to the interpretation of the impact estimates. Improved school climate is theorized to be proximal outcome of the intervention and is therefore critical to measure to assess staff’s perception of school climate we are drawing from two well validated measures

—Organizational Health Inventory (OHI) and th e Malasch Burnout Inventory . 14 Prior studies of MTSS - B have found impacts on the OHI 15

. The staff survey can be found in Appendix F . **Student Survey (Grades 3 -**  **5):** Students in grades 3 - 5 in Program and BAU schools will be asked to complete a survey in the Spring of 2015 and Spring of 2016. The survey will take approximately

50 minutes for all fielding. This survey will provide critical information about students’ perception of their own behavior and classroom climate, which are both predicted b y the theory of action to be positively impacted by

MTSS - B. Many of the scales included were selected because they

have been widely

used in previous research on school - wide interventions; many have been found to have adequate ps ychometric properties. 16 We have also included a limited number of questions to assess students’ awareness of core features of MTSS - B implementation such as the establishment of school rules, a positive reinforcement s ys tem and a gaming strategy

to reinforce the school wide expectations. These que stions will be asked in the survey administered in BAU and Program schools so they can inform the assessment of implementation and service contract. The student survey can be found in Appendix G . **Teac her Ratings of Student Behavior:**  Teachers of students in grades 1 - 5 in Program and BAU schools will be asked to rate their students’ behavior in fall of 2015, 17 the spring of 2016 and the spring of 2017. Most districts will require that we obtain active parental consent from parents for these ratings and so te achers will only be asked to rate the behavior of students in their classes 13 Sugai, To dd, and Horner (2000). 14 Hoy and Tartar (1997); Maslach and Jackson (1981). 15 Bradshaw

et al., 2008. 16 Gresham et al. (2011); Haynes et al. (2001); Huesmann and Guerra (1997); Roberts, Hom, and Battistich (1995); Social and Character Development Research Consortium (2010). 17 Fielding this survey at baseline, will allow the study team to identify a high - ris k sub - group at baseline. Identification of this subgroup is necessary for answering one of the study ’ s research questions regarding the effects of MTSS - B on at - risk students.16

with consent. The rating form is expected to take approximately ten minutes per student to complete. The teacher ratings for

m will be used to assess the impact of the program on student behavior. The study team has adopted the Teacher Observation of Student Adaptation - Checklist (TOCA C) for this instrument. The TOCA - C has been shown to be a valid and reliable measure of students’ behavior (concentration problems, disruptive behavior and prosocial behavior). 18 Ithas been used in prior evaluations of MTSS - B where it has been shown to be sensitive to the intervention. 19 The study

team is augmenting this measure with questions that will provide information regarding students’ referral to and receipt of targeted supports for behavior and special education services. Asking this information on the teacher ratings forms is important because it cannot alway s be reliabl y ascertained from administrative datasets in Program and BAU schools. Fin all y, the rating form includes one scale that assesses teachers’ perception of students’ academic behaviors. 20 This scale has been added because the theory of action predicts that MTSS - B will improve students’ academic engagement, which is not assessed on the TOCA C. The teacher ratings form can be found in Appendix H . **District Records Data Collection:** The study team will request extant data from school district records regarding student background characteristics, teacher background characteristics and experiences, student behavior incidences, student academic achievement (grades 3 - 5) and special education. Data will be collected at the end of each school year: baseline (SY 2014 - 2015), year one (SY 2015 - 2016) and year 2 (SY 2016 - 2017). Requests will be mad e of district research offices. Appendix I provides a sample of the letter that will be sent to districts requesting these data.

18 Koth, Bradshaw

and Leaf (2009). 19 Bradshaw , Waasdorp, and Leaf (2012). 20 The study team will be adopting the “approaches to learning” scale from NCES’ ECLS - K survey to assess this construct. All necessary permissions will be acquired before the survey is fielded.17 **SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION**  **Part A: Justification** 1. Circumstances Making Collection ofInformation Necessary This evaluation arises from a growing interest in multi - tiered sy stems of support for behavior [MTSS - B]. The Office of Special Education Programs ’ (OSEP) National Technical Assistance Center on Positive Behavioral Intervention and Supports (positive behavior intervention supports is an example of an MTSS - B program) reports that more than 21,000 schools in the United States report implementation of MTSS - B. 21 Since the implementation of MTSS - B is so prevalent , information on the effectiv eness of this approach is necessary . Researchers have been developing an evidence base for these programs. Numerous descriptive and correlational studies have associated aspects of MTSS - B implementation, or MTSS - B implementation overall, with student outco mes. 22 , 23 , 24 More recentl y, researchers have begun to conduct causal studies of MTSS - B, although some of these initial studies have been on a relativel y s mall scale (or conducted within a single state) or have been somewhat limited in duration and outcomes. This study will provide the first national, random assignment study of the impacts of MTSS - B providing impact estimates for universal supports (Tier I) as well as universal and targeted support together (Tiers I and II combined). This impact evaluation of multi - tiered sy stems of support for behavior (MTSS - B) is authorized in Section 664 of the Individuals with Disabilities Education Act [IDEA] , as amended, which assigns to IES the responsibility to conduct studies and evaluations o f the implementation and impact of programs supported under IDEA , which includes MTSS - B . 25 This statute is provided in Appendix J . 2. Purpose and Use of the Data: How the data will be collected, by whom, and for what purpose. The information gathered thro ugh this data collection will be anal yz ed b y the IES evaluation contractor (MDRC) and its subcontractors to study the implementation, and impacts of MTSS - B in elementary

schools. This will involve new data collection where needed through staff and student surveys, teacher ratings of student behavior, brief interviews with random ly selected students and staff, and structured interviews with principals and key program implementers (MTSS - B Coach es and team leaders) . Exhibit A - 2 show s a timeline of these activi ties. The data collected for this study will be used b y ED to report to Congress on the assessment of activities using federal funds under the Individuals with Disabilities Education Act National Assessment. 26 Failure to collect these data may result in ED being unable to adequatel y report to

21 Horner (2014). 22 Bradshaw, Mitchell, and Leaf (2010). 23 Horner et al. (2009). 24 OSEP Center on Positive Behavior Interventions and Supports (2007). 25 IDEA, P.L. 108 - 446. 26 IDEA, P.L. 108 - 446.18

Congress on the assessment of these activities. Additionally, if this evaluation were not completed, ED and Congress would not have an accurate understanding of implementation and impact of MTSS - B, which is reportedl y being implemented in over 21,000 schools across the county and supported b y Federal funds . 3. Use of Technology to Reduce Burden All staff survey s and teacher ratings will be administered using web - based surve ys so they are easily accessible to respondents. Administration of web - based surveys enables reduced burden through complex skip patterns that are invisible to respondents, as well as prefilled information based on responses to previous items when appropri ate. Web - based survey s also leads to decreased costs associated with processing and increased data collection speed. Paper survey options will be offered to respondents as part of the follow - up effort with individuals who do not respond. 4. Efforts to Avoid Duplication As described in the introduction, the data c ollection effort planned for this project will produce data that are unique and targets specifically the research questions identified for this project. Valid and reliable measure of teacher practic e, school climate and student behavior are not available from extant data for the participating districts and schools . 5. Methods to Minimize Burden on Small Entities The data will be collected from district and school staff and no small businesses or entities will be involved in the data collection. 6. Consequences of Less Frequent Data C ollection If the proposed data collection is not done, it will not be possible fo r ED to report rigorous impact findings from a large scale study of MTSS - B to Congress, other policy makers, and practitioners seeking effective way s to support student learning. 7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 None appl y to this data collection. 8. Federal Register Comments and Persons Consulted Outside the Agency The following experts serve on an Expert Panel for the Selection of Training Providers and have been consulted on the design of the study.

**Expert Organization**  Leonard Bickman, Ph.D. Vanderbilt University, Peabody College of19 **Expert Organization**  Education Catherine Bradshaw, Ph.D. 27 University of Virginia, Curry School of Education Lori Newcomer, Ph.D. University of Missouri, Department of Educational, School, and Counseling Psychology The following individuals participate d in the Technical Working Group for the Evaluation that convene d on March 26 , 2015 .

**Expert Organization**  David Cordray, Ph.D Vanderbilt University, Department of Psychology and Human Development Brian Flay Oregon State University, School of Social and Behavioral Health Sciences Mary Louise Hemmeter Vanderbilt University, Department of Special Education Sara E. Rimm - Kaufman, Ph.D. University of Virginia, Curry School of Education an d Center for the Advanced Study of Teaching and Learning Lori Newcomer, Ph.D. University of Missouri, Missouri Prevention Center / Department of Educational School and Counseling Psychology 9. Payment or Gifts to Respondents We are aware that teachers are the targets of numerous requests to complete data collection instruments on a wide variety of topics from state and district offices, independent researchers, and ED and several decades of survey research support the benefits of offering incentives. Further, high response rates are needed to make the study measures reliable and offering honoraria for staff and teachers will help ensure high response rates. In fact, the importance of providing data collection incentives in federal studies has been described by other researchers, given the recognized burden and need for high response rates. 28 The use of incentives has been shown to be effective in lowering non - response rates and the level of effort required to obtain completions. 29 Studies have shown that when used appropriatel y, incentives are a cost - effective means of significantly increasing response rates. 30 Although much of the research on the value of using incentives has focused on low - income populations, the value of using incentives in educational s ettings has also been demonstrated. In the Reading First Impact Study

commissioned by

ED, monetary

incentives proved to have significant effects on response rates among teachers. A sub - study requested by

OMB on the effect of incentives on survey response rates for teachers showed significant increases when an incentive of $15 or $30

27 Dr. Bradshaw is also a member of the evaluation team but served in this role because of her special expertise on training providers and MTSS - B training. 28 Berry, Pevar, and Zander - Contugno (2008); Singer and Kulka (200 2 ). 29 James (1997). 30 For example, see Dillman (2007).20

was offered to teachers as opposed to no incentive. 31 In2005, the National Center for Education Evaluation submitted a memorandum to OMB outlining guidelines for incentives for NCEE Evaluation Studies and tyingrecommended incentive levels to the level of burden (represented by the length of the survey). The amount of incentives planned for the survey respondents for this study is consistent with that proposed in the NCEE memo Gu idelines for Incentives for NCEE Evaluation Studies. 32 **Payments to teachers and staff**: Teachers and school staff will be asked to complete surveys and rating forms at multiple times over the study period (fall 2015 - spring 2017). We expect that these activities will be completed outside of the normal workday covered in their labor agreements , given the level of demand already m ade upon their time. Teachers and other school staf f will be asked to complete a 35 - 40 minute survey at up to three points in time during the course of the study. If district rules allow, w e will offer a gift card value d at $25 for each completion. Teachers also will be asked to complete a 10 minute short behavior rating for each of their students ( from pool of consenting students if active consent is required by district) at three times during the st

udy . Each rating wil l take approximately 10 minutes per student and , if district rules allow, teachers will be provided a gift card valued at $5 for each student rating completed. 33 The incentive amounts are consistent with the 2005 NCEE memo recommendation for a “medium burden” data collection effort. In addition, if district rules allow, we will provide a $25 incentive to classroom teachers or the school data collection liaison to encourage high rates of return for the parental consent forms needed for data collection activities [1

. Our preferred method of distributing consent forms is to include the forms in a packet parents receive at the school’s open house during enrollment at the beginning of the school year. Alternativel y, homeroom teachers may distribute consent form s to their students or the school may wish to have the forms mailed directl y to the students’ home. We will work closely

with each school to ensure that parents who do not return the consent form are sent reminder notices with replacement forms. This proc ess will be repeated, as needed, to achieve the target active parental consent rate. We will provide an incentive ($25 gift card) for each classroom in which at least 90% of the student parental consent forms are returned, whether or not the parent allows the student to participate in the data collection activities. The incentive will be provided either to the classroom teacher or to the school data collection liaison based on who is responsible for monitoring the consent form returns to encourage the teacher, or the data collection liaison, to monitor consent returns carefully and follow up with students for whom a form has not been returned, encouraging students to return a signed form. Firms have used similar approaches successfully in previous studies. For example, in the cross - site evaluation of the Safe Schools/Healthy Students Initiative teachers and staff were told that each school would be provided an additional $25 for each classroom in which a minimum of 70% of active parental consent forms were completed by parents and returned to the school. In schools where this

31 Gamse et al. (2008). 32 National Center for Education Evaluation (March 22, 2005). 33 Teachers will be informed that they will only be awarded the $5 per response if they complete rating forms for a proportion of their student above a pre - specified threshold (from pool of consenting students if district rules require active parental consent for this activity). [1] MDRC’s IRB will likely require that we seek active parental consent for the teacher ratings of student behavi or and the student surveys. Some district IRBs will also require active parental consent for the sharing of extant student records.21

approach was used, the return rates of parental consent forms of 78% were achieved, on average. This incentive opportunity will be available for all classrooms in grades 1 - 5. 10. Assuran ces of Confidentiality Provided to Respondents All data collection activities will be conducted in full compliance with The Department of Education regulations to maintain the confidentiality of data obtained on private persons and to protect the rights and welfare of human research subjects as contained in the Department of Education regulations. These activities will also be conducted in compliance with other Federal regulations in particular with The Privacy Act of 1974, P.L . 93 - 579, 5 USC 552 a; the “ Buckley Amendment,” Family

Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of In formation Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1 - 1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. An explicit statement describing the project, the data collection, and confidentiality will be provided to study

participants. These participants will include parents of student s in all study schools as well as staff and administrators in all study schools. Most districts recruited for the study have rules requiring active informed consent for data collection concerning students . In these district s, parents will be asked to sign an informed consent form that meets federal human subjects requirements acknowledging their willingness to allow their student to participate and for the study team to access extant data. Indistricts that do not require active infor med consent for data collection about students, we will review whether to shift to passive consent for student data collection. We will review with our Institutional Review Board whether it believes passive consent would be appropriate for this data collec tion. **Confidentiality as surances during data collection:**   All MTSS - B data collection employees at MDRC , DIR and AIR and any data collection sub - contractors thereof

, sign confidentiality

agreements that emphasize the importance of confidentiality

and specify

employees’ obligations to maintain it.  Personally

identifiable information (PII) is maintained on separate forms and files, which are linked onl y b y sample identification numbers.  Access to a crosswalk file linking sample identification numbers to personally identifiable information and contact information is limited to a small number of individuals who have a need to know this information  Access to hard cop y documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.  Access to electronic files is protected b y secure usernames and passwords, which are onl y available to approved users. Access to identifying information for sample members is limited to those who have direct responsibility for prov iding and maintaining sample crosswalk and contact information. At the conclusion of the study , these data are destroyed.  The plan for maintaining confidentiality includes staff training regarding the meaning of confidentiality , particularly as it relates to handling requests for information and22

providing assurance to respondents about the protection of their responses. It also includes built - in safeguards concerning status monitoring and receipt control sy stems. **Confidentiality assurance during analysis :**  The data collected for this study will be used onl y for broadl y descriptive and statistical purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific district or individual. Inn o instances will the study team provide information that identifies districts, schools, principals, teachers, or students to anyone outside the study team, except as required b y law. More detail on MDRC’s procedures to ensure data security are described be low: **Data storage:** MDRC and subcontractors will store all data in compliance with its federall y approved data security

plan with state - of

- the - art data security

protections that have passed reviews b y the Social Security Administration and other federal a gencies. All quantitative data will be stored on a secure section of our network, accessible onl y to specific project staff identified by

the data manager. Data that is used for anal ys is is stored with research ID numbers rather than actual student or staf f identification information; Notes/digital recordings from interviews, and all survey data will only

be reviewed by

the research team and stored in a secure area accessible onl y to the research team. All information from these research activities will be reported in the aggregate, and no disaggregated information from our observations or interviews will be shared with teachers, school or district leadership, or the general public. Once the study is completed, original data collected for the project will b e destroyed at the end of the project and onl y the restricted access file without any actual identifiers for the district, school, or individual respondent will remain [to be discussed below]. **Method of data de struction**  **:** All data containing individually identifiable records will be destroyed by

an appropriate fail - safe method, including phy sical destruction of the media itself or deletion of the contents on our servers. After the study is completed, the study

team will create a restricted - access file of the data collected and submit that file to IES once the project ends. This file will have been stripped of all student, teacher/staff, school and district identifiers. 11. Justification of Sensitive Questions Questions in some components of the MTSS - B student survey are potentially

“sensitive.” Students in grade 3 - 5 are asked about personal topics including students’ perception of fairness and as their perceived level of aggression and feelings that they have in school. The questions we have included were se lected assess constructs that we need to measure in order to understand the impacts of the intervention. The specific scales selected for the survey s were chosen in part because they

have been widely

used in previous research and found to be valid and reliable measures of the constructs we intend to measure . 34 Moreover, trained assessors will explain questions to students before they are posed. Finally , students and their parents will be informed by research staff prior to the start of the survey that th eir answers are confidential, that they may refuse to answer any question, that results will only be reported in the aggregate, and that their responses will not have any effect on any

services or benefits they

or their family

members

34 Aber, Brown, and Jones (2003); Jones, Brown, and Aber (2011a); Jones, Brown, and Aber (2011b); Jones, Brown, Hoglund, and Aber (2010).23

receive. When require d by districts and if required by the MDRC IRB, the study team will be asking for active parental consent for this activity. Questions on the teacher ratings of student behavior are also potentially sensitive because teachers are providing an assessment o f individual student’s concentration problems and disruptive behavior. This rating scale is essential to the study because it is so closely

tied to the MTSS - B theory

of action and prior evaluations of MTSS - B have found effects on these rating scales . 35 Many of the questions on this measure have been used in other studies including the recent IES - funded evaluation of the Good Behavior Game. When required by districts and if required b y the MDRC IRB, the study team will be asking for active parental consent for this activity . 12. Estimates of Annualized Burden Hours and Costs Exhibit A - 3 summarizes reporting burden on respondent s for each d ata source. The respondent pool will include school staff and students. For each data collection activity , there is a corresponding number of respondents assumed per school, an assumed number of schools associated with the activity , and a total number of respondents estimated (assuming a 90% response rate). Exhibit A - 3 also provides the number of administrations each data collection activity will be conducted over the course of the two years of the study , average estimates for the amount of time required for each activity in minutes, as well as the total burden hours calculated for each activity

to be completed by

all of the required respondents. This proposed information collection does not impose a financial burden on any respondents and respondents will not incur any expenses. The introduction to supporting statement A and the responses to question B - 2 provide more de tail about the timeline of data collection activities listed in the exhibit and describe why these activities are essential to the study .

35 Bradshaw, Waasdorp, and Leaf (2012).24 Exhibit A - 3. Estimated Burden to Respondents **Data Collection Activity Assumed # of**  **Respondents per school**  **Assumed**

**# of schools**  **Estimated # of**

**Respondents per**  **Administration (90% consent**  **rate assumed) Number of** 

**Administrations (Over 2 Years)**  **Average**

**Burden Hours**  **per Response**  **(M inutes)**  **Total**

**Burden (Hours)**  **Total**

**Number of Responses**  **(Over 2 Years)**  **Total #of**

**Respondents (over**  **2 years; 90% consent**  **rate ) Teacher ratings of student behavior**  (assumed 20 students per teacher and 10 min. per student) 20 90 1,620 3 200 16,200 4,860 1,620 **Student survey** (grades 3 - 5) 240 90 19,440 2 50 32,400 38,880 25920 **Non -**  **teaching staff survey**  10 90 810 2 35 945 1,620 810 **Teacher survey** 20 90 1,620 2 40 2,160 3,240 1,620 **Site Visit:** Brief interviews with 15 randomly selected staff 15 90 1,215 3 5 304 3,645 3,645 **Site Visit:** Brief interviews with 15 randomly selected students 15 90 1,215 3 2 122 3,645 3,645 **Site Visit:** Interview with administrator in program schools 1 60 54 3 60 162 162 54 **Site Visit:** Interview with administrator in BAU schools for site visit 1 30 27 3 45 61 81 27 **Site Visit:** Interview with behavior team leader in program schools for site visits 1 60 54 3 45 122 162 54 **Site Visit:** Interview with behavior team leader in BAU schools 1 30 27 3 30 41 81 27 **Site Visit:** Interview with MTSS - B coach .33 60 18 3 20 18 54 18 **District Records Data Collection** from 10 districts 10 3 1080 540 30 10 **Total 53,073**  **56,460 37,450** 25 **Note.** The assumption of 80 students per grade is based on the average 3rd grade enrollment for elementary schools (defined as schoo ls with no grade higher than the 6th grade) across the U.S. for 2008 - 2009 school year, which was 77.9 using data reported by the Common Core of Data (CCD). The assumption of 20 students per class is based on the 2007 - 2008 Schools and Staffing Survey which shows an average elementary school class size of 20.3 students. The assumption of 20 grade 1 - 5 teachers is based on the assumption regarding class size and average number of students per grade. The assumption regarding the number of MTSS - B coaches per schools is based on the assumption that there will be six p rogram schools per district, which will require each of the ten districts to hire 2 part -time MTSS - B coaches (.33 *60 = 20). The assumption of 10 non -teaching staff in a school is an estimate based on the Schools and Staffing Survey. Assumptions regarding the time to fulfill district records requests are based on MDRC's experience requesting district records for other studies that request comparable data from districts. The estimated total number of respondents is different from the estimates number of resp ondents per administration because of a new cohort of third graders in year 2 and a random sample of students and staff being selected for site visit interviews at each wave. In calculating the number of responses per data source, we assume that each teacher's rating of all the students' in their class is one response and that each district records data collection request counts as one response for each wave of collection.26

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keeper s Not applicable. The information collection activities do not place any capital cost or cost of maintaining capital requirements on respondents. 14. Annualized Cost to the Federal Government The total cost for the study is $17,770,151 over 5 ye ars for an annualized cost of $3,5 44,030 .20 15. Explanation for Program Changes or Adjustments This submission to OMB is a new request for approval. 16. Time, Schedule, Publication and Analysis Plan **Schedule & Publication Plan** The project schedule is as follows:

 Final selection and randomization of sites by spring of 2015  MTSS - B training and support activities spring 2015 - spring 2017  Completion of data collection and creation of analysis files for final report by December 2017  Analy sis of findings for fi nal report in winter and spring of 2018, and  Preparation of a project final report and public - use data file for review by the Department and release in November 2018 We anticipate a report that includes an introductory chapter, a chapter on project design and data collection, a chapter discussing the nature and implementation of the MTSS - B training and support, a chapter discussing the nature and implementation of the MTSS - B program implemented in schools as well as services in the BAU schools, and a chap ter presenting impact findings. The report will follow guidance provided in the National Center for Education Statistics (NCES) Statistical Standards 36 and the IES Style Guide. 37 **Analysis Plan** To assess the implementation of MTSS - B training and support, the study team will use data from the direct observation of CSBS training and support activities, interviews with key program implementers (MTSS - B coach, administrator and behavior team leader), and review of program data (e.g. CSBS trainer logs and training attendance sheet s ) . S ys tematic review of these data will allow the team to describe the content and frequency of CSBS activities, key

implementers’

36 National Center for Education Statistics (2002). 37 U.S. Department of Education, Institute for Education Sciences (2005).27

perception of the quality

of CSBS activitie s and variation in the content of training delivered between and within districts. To assess implementation of the MTSS - B program in schools, the study team will use data obtained from site visits to program schools (interview, observation and document review) and through a review of program data (e.g. web - based program forms, MTSS - B coach logs). Analy sis of these data will allow the team to describe the observed implementation of Tier I and Tier II activities and to describe variation between and within districts in schools’ implementation of MTSS - B. To estimate the impacts of MTSS - B on school staff practice, school climate and student outcomes (e.g. student behavior, student achievement and receipt of special education services) the study team will rel y upon teacher and student survey s, teacher ratings of student behavior, classroom observation and district records. The study team will select one primary outcome per domain. For each outcome we will estimate program impacts after one and then two years o f implementation using a school level random assignment design, with random assignment occurring at the level of the districts. 38 Impacts will be estimated using fixed effects, multi - level hierarchical models. The primary student sub - group anal ys is will be for students identified as being at - risk for future behavior problems at baseline according to the teacher ratings of student behavior fielded in the Fall of 2015. Comparison of the behavior support practices in Program and BAU Schools through data obtaine d from site visits, staff and student survey s and classroom observations will help the team to interpret the findings from the impact analy sis. 17. Approval to Not Display O MB Expiration Date All data collection instruments will include the OMB expiration date. 18. Explanation of Exceptions to the Paperwork Reduction Act No exceptions are needed for this data collection.

38 I f there are subgroups of similar schools

– wemay have blocks of schools wi thin a district and randomize at that level . These decisions will be made upon revie w of district data.28 **REFERENCES USED IN PART A** Aber, J. L awrence, Joshua L . Brown, and Step hanie M. Jones. 2003. Developmental trajectories toward violence in middle childhood: Course, demographic differences, and response to school - based intervention. Developmental Psychology - Special Issue on Violent Children 39, 2: 324 - 348; Anderson, C. M., Teri Lewis - Palmer, Anne W. Todd, Robert H. Horner., George Sugai, N.K. Samson. 2008. "Individual student sy stems evaluation tool, version 2.6." Educational and Community

Supports, University

of Oregon. Berry , Sandra H., Jennifer Pevar, and Megan Zander- Co tugno. 2008. “The use of incentives in survey s supported b y federal grants.” Santa Monica, CA: RAND Corporation. Bradshaw, Catherine P., Christine W. Koth, Katherine B. Bevans, Nicholas Ialongo, and Philip J. Leaf. 2008a. “The Impact of School - Wide Positiv e Behavioral Interventions and Supports (PBI S) on the Organizational Health of Elementary Schools.” School Psychology Quarterly, 23, 4: 463 - 473. Bradshaw, Catherine P., Mary M. Mitchell, and Philip J. Leaf. 2010. “Examining the effects of School - Wide Positive Behavioral Interventions and Supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools.” Journal of Positive Behavior Interventions, 12, 133 - 148. Bradshaw, Catherine P., Tracy E. Waasdorp, and Philip J . L eaf. 2012. "Effects of School - Wide Positive Behavioral Interventions and Supports on Child Behavior Outcomes." PEDIATRICS 130, 5: e1136 - e1145. Debnam, Katrina J., Pas, Elise T., and Bradshaw, Catherine P. (2012). Secondary and tertiary support sy stems i n schools implementing school - wide positive behavioral interventions and supports: A preliminary descriptive anal ys is. Journal of Positive Behavior Interventions, 14 (3), 142

– 152. Dillman, Don A. 2007. Mail and Internet Surveys: The Tailored Design Method ( 2nd ed.). New Jersey : John Wiley

& Sons, Inc. Durlak, Joseph A., Roger P. Weissberg, Allison B. Dy mnicki, Rebecca D. Taylor, and Kriston B. Schellinger. 2011. "The impact of enhancing students’ social and emotional learning: A meta - anal ys is of school - based universal interventions." Child Development 82, 1: 405 - 432. Gamse, Beth C., Howard S. Bloom, H. S., James J. Kemple, and Robin Tepper Jacob. 2008. “Reading First Impact Study : Interim Report.” NCEE 2008 - 4016. National Center for Education Evaluation and Regional Assistance . Gresham, Frank M., Stephen N. Elliott, Michael J. Vance, and Clay ton R. Cook. 2011. "Comparability of the Social Skills Rating System to the Social Skills Improvement29

Sy stem: Content and Psychometric Comparisons Across Elementary and S econdary

Age Levels." School Psychology Quarterly 26, 1: 27 - 44. Hay nes, N. M., C. L. Emmons, M. Ben - Avie, and J. P. Comer. 2001. "The school development program student, staff, and parent school climate survey s." New Haven, CT: Yale Child Study Center. Hor ner, Robert H. 2014. “The Role of District Leadership Teams in PBIS Implementation”. Presentation. Available at http://www.pbis.org/Common/Cms/files/pbisresources/1 - Implementing%20PBIS.pptx. Horner, R. H., Todd, A. W., Lewis - Palmer, T., Irvin, L . K., Sugai , G., & Boland, J. B. 2004. The school - wide evaluation tool (SET): A research instrument for assessing school - wide positive behavior support. Journal of Positive Behavior Intervention s, 6, 3

– 12. Horner, Robert H., George Sugai, Keith Smolkowski, Lucille Eb er, Jean Nakasato, Anne W. Todd, and Jody

Esperanza. 2009. "A randomized, wait - list controlled effectiveness trial assessing school - wide positive behavior support in elementary schools." Journal of Positive Behavior Interventions 11, 3: 133 - 144. Hoy Way ne K. and Clemens John Tarter. 1997. The Road to Open and Healthy Schools: A Handbook for Change, Elementary Edition . Thousand Oaks, CA: Corwin Press. Huesmann, Rowell L., and Nancy G. Guerra. 1997. "Children's normative beliefs about aggression and aggressi ve behavior." Journal of Personality and Social Psychology 72, 2: 408 - 419. James, Tracy . 1997. “Results of the Wave Iincentive experiment in the 1996 survey of income and program participation.” In Pp 834 - 839 in Proceedings of the Survey Methods Section , American Statistical Association. Jones, Stephanie M., Joshua L .Brown and J. Lawrence Aber. 2011a. The longitudinal impact of a universal school - based social - emotional and literacy intervention: An experiment in translational developmental research. Child Development 82,2: 533 - 554; Jones, Stephanie M., Joshua L . Brown, and J. Lawrence Aber. 2011b “Two - Year Impacts of a Universal School - Based Social - Emotional and Literacy Intervention: An Experiment in Translational Developmental Research”. Psychology Facul ty Publications. Paper 131. (http://fordham, bepress.com/psych_facultypubs/131). Jones, Stephanie M., Joshua L . Brown, Wendy LG Hoglund, and J. L awrence Aber. 2010. "A school - randomized clinical trial of an integrated social

–emotional learning and literacy intervention: Impacts after 1 school year." Journal of Consulting and Clinical Psychology 78, 6: 829. Koth, Christine W., Catherine P. Bradshaw, and Philip J. Leaf. 2009. "Teacher Observation of Classroom Adaptation

—Checklist: Development and factor struc ture." Measurement and Evaluation in Counseling and Development 42, 1: 15 - 30.30

Maslach, Christina, and Susan E. Jackson. 1981. "The measurement of experienced burnout." Journal of Organizational Behavior 2, 2: 99 - 113. National Center for Education Evaluation. 2005. Guidelines for incentives for NCEE impact evaluations . (March 22). National Center for Education Evaluation: Washington, DC. OSEP Center on Positive Behavioral Interventions and Supports. 2007. “Is School - Wide Positive Behavior Support An Evidence - Based Practice? A Research Summary .” Website: www.pbismaryland.org. Roberts, William, Allen Hom, and Victor Battistich. 1995. “Assessing Students’ and Teachers’ sense of the School as a Caring Community .” Oakland, CA: Developmental Studies Center. Sugai, George, Todd, Anne W., and Horner, Robert H. (2000). Effective Behavior Support (EBS) Survey : Assessing and planning behavior supports in schools. Eugene, OR: University

of Oregon. Sugai, G., Lewis - Palmer, T., Todd, A., & Horner, R. H. (2005). School - wi de evaluation tool. Eugene: University of Oregon. Social and Character Development Research Consortium. 2010. "Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary

school children." Washington, DC: National Center for Education Research, Institute of Educational Sciences, U.S. Department of Education.