1 

# I **MPACT** 

# E **VALUATION OF** 

# T **RAINING IN** 

# M **ULTI** 

# - **TIERED** 

# S **YSTEMS OF** 

# S **UPPORT FOR**



# B **EHAVIOR** 

# (MTSS **- B)**  **OMB C**  **LEARANCE** 

# R**EQUEST** 

# : **D ATA**  

# C **OLLECTION** 

# S UPPORTING 

# S TATEMENTS 

# B

**April 2015 P**  **REPARED FOR** 

## : Institute of Education Sciences United States Department of Education Contract No. ED - IES - 14 - C - 003 **P REPARED**  

## B **Y** 

## : MDRC 16 East 34 th 

## Street, 19 th 

## Floor New York, NY 10016 Fred Doolittle, Project Director Fred.Doolittle@mdrc.org (212) 340 - 86382 **TABLE OF CONTENTS** IN TRODUCTION 3 THEORY OF ACTION AND RESEARCH QUESTI ONS 4 CHARACTERI STICS OF THE TREATMENT TO BE TESTED 5 STUDY DESCRIPTION 9 SUPPORTING STATEMENT FOR PAPERWORK REDUCTI ON ACT SUBMI SSI ON 1 6 B. DESCRIPTION OF STATI STICAL METHODS 15 1 . Respondent Universe and Sampling Methods 15 2. Procedures for Data Collection 16 3. Procedures to Maximize Response Rates 21 4. Testing of Data Collection Procedures 22 5. Individuals Consulted on Statistical and Methodological Aspect s of Data Collection 22 REFERENCES 24 APPENDI CES A. Interview Protocols: MTSS - B Coach B. Interview P rotocols : Administrators C. Interview Protocols: Behavior Team Leader D. Interview P rotocol : Students E. Interview P rotocol: Staff F. Staff and Teacher Survey G. Student Survey H. Teaching Ratings of Student Behavior I. District Records Data Collection Request Letter J. Excerpt from IDEA, P.L. 108 - 4463 **INTRODUCTION** Creating and maintaining safe and orderl y s chool and classroom environments to support stu dent learning is a top federal priority , as indicated by the U.S. Department of Education and the U.S. Department of Justice jointly issued guidance on school discipline. The implementation of Multi - Tiered Sy stems of Support for Behavior (MTSS - B) is seen as a promising approach to improve school and classroom climate, student behavior, and academic achievement, and potentially

reduce the inappropriate identification of students for special education. MTSS - B is a multi - tiered, sy stematic framework for teach ing and reinforcing positive behavior for all students, that includes universal , school wide supports (often called Tier I), targeted interventions to address the problems of students not responding to the universal components (often called Tier II), and i ntensive interventions (Tier III) for students not responding to Tier II services. The widespread use of MTSS - B nationally contributes to the relevance of this evaluation. The U.S. Department of Education’s (ED’s) Office of Special Education Programs ha s supported technical assistance around MTSS - B since 1998. Recent data from the Office of Special Education Programs TA Center on Positive Behavior Interventions and Support indicate that as of 2014, MTSS - B is reportedly implemented in over 21,000 schools across the nation. 1 As the support for and implementation of MTSS - B have increased, researchers have been developing an evidence base for it and other related programs. Numerous descriptive and correlational studies have associated aspects of MTSS - B imp lementation, or MTSS - B implementation overall, with student outcomes. More recentl y, researchers have begun to conduct causal studies of MTSS - B, although some of these initial studies have been on a relativel y small scale (or conducted within a single stat e) or have been somewhat limited in duration and outcomes. 2 , 3 , 4 A recent meta - anal ys is of research on a broad range of 213 individual school - based, universal social and emotional learning (SEL) programs (including interventions than can be implemented with in an MTSS - B program) found that compared to controls, SEL participants demonstrated significantl y improved social and emotional skills, attitudes, behavior, and academic performance. 5 Much of this research has focused on individual programs and not exami ned the impact of a large - scale implementation of both universal, school - wide programs and targeted programs within a MTSS - B framework. Therefore, this evaluation focuses on a comprehensive MTSS - B program that includes each of the components usually

asso ciated with MTSS - B: school - wide and classroom - level strategies (universal supports, or Tier I) and individual or small group strategies (targeted interventions, or Tier II), with appropriate infrastructure (staffing sand data s ys tem). This evaluation will not support intensive supports for students not responding to Tier II interventions (i.e., there will be

1 Horner (2014). 2 Bradshaw , Mitchell, and Leaf (2010). 3 Horner et al. (2009). 4 OSEP Center on Positive Behavior Interventions an d Supports (2007). 5 Durlak et al. (2011).4

no Tier III interventions). IE S has contracted with MDRC, AIR, Harvard University , and Decision Information Resources (DIR) to conduct the Impact Evalu ation of MTSS - B. The goal of the evaluation is to implement an MTSS - B program with fidelity and determine whether universal supports (Tier I) and a targeted intervention (Tier II) with appropriate infrastructure are effective when implemented in a large n umber of districts and elementary schools. Using a school - level random assignment design, this evaluation will examine the impact of training elementary

school staff in the implementation of an MTSS - B framework including staff and data infrastructure with universal, school - wide components, as well as classroom - based activities (Tier I) in the first year of implementation as compared to outcomes for schools undertaking ty

pical behavior support practices (These schools are hereafter called business - as usual or BAU schools.) In addition, the study will also examine the impact of this Tier I training plus additional training in a targeted intervention (Tier II) in a second year of implementation. Schools that receive MTSS - B training and support are hereafter r eferred to as “Program Schools.” The study

will also investigate the differences in behavioral support services between program and BAU schools. Finally , the evaluation team will assess whether the MTSS - B training and support activities are implemented as intended as well as the fidelity of school’s implementation of the intended MTSS - B Tier I and Tier II activities.

**THEORY OF ACTION & RESEARCH QUESTIONS** The implementation of MTSS - B is ultimately intended to improve student outcomes. The conduct of school - wide and classroom strategies (Tier I) and individual or small group strategies (Tier II), supported b y appropriate infrastructure, are designed to improve school staff behavior practices as well as school and classroom climate. The improved practices and climate are, in turn, hy pothesized to benefit student behavior and academic outcomes. The individual and small group strategies may

affect the students who need additional support (Tier II) as well as all students if these strategies lessen the disruption of students needing additional support. Finall y, improved student behavior could lead to better student academic outcomes (see Exhibit B - 1).5 **Exhibit B**  **- 1: Conceptual Model of MTSS**  **- B and Intended Outcomes** 

The evaluation is anchored on this conc eptual model and will focus on the following research questions:  What MTSS - B training and support activities were conducted?  Is the MTSS - B program implemented with fidelity? What are the key challenges to the implementation of MTSS - B with fidelity?  What i s the difference in behavior support practices occurring in program and BAU schools?  What is the impact on school staff practices, school climate and student outcomes of providing training in the MTSS - B framework plus universal, school - wide and classroom T ier I positive behavior supports and a targeted Tier II intervention?  What are the impacts on school staff practices , school climate, and student outcomes for relevant subgroups (e.g ., teachers with less experience; at - risk students)?  Does the implementation of MTSS - B reduce an y differences in student discipline outcomes across racial and ethnic groups? Does it reduce the identification of students for special education services?6 **CHARACTERISTICS OF THE TREATMENT TO BE TESTED** In the spring of 2014, MDR C and partner organizations issued a Request for Proposals to providers of training in MTSS - B. In November 2014, we announced the winner of this competition, The Center for Social Behavior Support (CSBS), which is collaboration between The Illinois - Midwest PBIS Network at the School Association for Special Education in DuPage, Illinois (SASED) and the PBI S Regional Training and Technical Assistance Center at Sheppard Pratt, in Maryland. The training and support activities proposed by CSBS are described belo w. Following that summary , the memo describes CSBS’ plan for how the MTSS - B Tier I and Tier II program will be implemented in schools. **CSBS Training and Support Activities** CSBS will mostly

follow a “train the trainer” model with some direct support to all school staff and on - site support to the school staff and Coaches throughout implementation. Below we describe the key

phases of their work with schools. **Readiness Period -**  **Training and Support by CSBS to Develop Infrastructure:** The first step is the d evelopment of school - level infrastructures that will support implementation of Tier I and Tier II program components, which includes a staffing structure and data s ystem. Tier I infrastructure will be put in place in the late spring of 2015 for all Program Schools. In frastructure for Tier II will be developed in the spring of 2016. **Staffing structure:** CSBS trainers will work with each school to identify a representative School L eadership Team (SL T) and each district to identify MTSS - B Coaches. This will be accomplished during a two - day site visit to each district as well as webinars, email exchanges and phone calls. In addition to helping schools and districts identify the SLT and MTSS - B Coaches, CSBS will explain the MTSS - B program to the district coordi nator, administrator, SL T and MTSS - B Coaches as well as the roles and functions of each individual. During the spring of 2016 CSBS will work with schools to identify a Targeted Team (TT) to support Tier II implementation. Often, the TT is a subset of the S LT. **Behavioral monitoring data system:** During the spring of 2015 CSBS trainers will support the SL T in putting in place the data infrastructure that relies on a web - based behavior monitoring data sy stem called the School Wide Information Sy stem (SWI S). M inimally

three SLT members and MTSS - B Coaches will be trained to use the SWI S data sy stem to drive decision - making about issues such as which teachers may need additional supports with classroom management and which locations in the school at what time may need more school staff to monitor student behavior. Appropriate staff will be identified and trained to enter data and generate reports. The data infrastructure will be augmented in the spring 2016 to include the data sy stem for Check In Check Out, the primary Tier II intervention. **Initial and Booster Summer Trainings:** During the summer of 2015, CSBS will host training within each district for the SL T, MTSS - B Coaches and administrators in all Program Schools.7

The goal of the initial CSBS Tier I training during the summer of 2015 is to introduce the key school - level implementers (MTSS - B Coaches, SLT and administrators) to the core features of MTSS - B Tier I 6 as well as the tools to track implementation fidelity and engage in data - based problem solving. Duri ng the summer training each school team drafts a school - level implementation plan. Each school team should leave the summer training prepared to further develop and implement the school - level implementation plan. During the summer of 2016, the key Tier I i mplementers engage in a booster training in which they review core components, reflect on the previous ye ar and develop a ye ar 2 school - level implementation plan. During this booster training, new school staff are also introduced to the core components of MTSS - B Tier I. Initial summer training for Tier II also takes place in the summer of 2016 and includes training for the MTSS - B Coaches, Targeted Team and administrator in the core features of Check In Check Out 7 and how to use CI CO - SWI S data to monitor stu dent progress. The goal of this initial training is for schools to develop and implement their CI CO implementation plan. **Ongoing Training and Support:** During the 2015 - 2016 and 2016 - 2017 school years, CSBS will provide ongoing support to the MTSS - B Coache s, SLT and administrators and some direct trainings to all school - staff as schools implement Tier I activities. CSBS trainers will visit all Program Schools four times over the course of the school year. During these visits, they will work with the SL T, administrator and MTSS - B Coaches to introduce Tier I core components to staff and to train staff in their version of the Good Behavior Game (referred to as the Positive Behavior Game (PBG)), which helps teachers to teach and acknowledge classroom rules align ed with the school - wide expectations. Additionally , they will gather and use data regarding school’s implementation fidelity

to help the SL T to create and modify school’s implementation plans. 8 During site visits, monthly webinars and email/phone exchanges CSBS will support Coaches and the SL T to train teachers in the 8 CSBS classroom components.

6 Core components of MTSS - B Tier I include teaching and reinforcing specific school - wide behavioral expectations, a classroom management system to support behavioral expectations, and the use of fidelity data to mo nitor and im prove implementation. 7 Core components of MTSS - B Tier II (CICO) include use of data to identify and progress monitor students needing additional supports, additional instruction and time for students to develop behavioral skills, additional structures with increased opportunity for feedback from staff, and sharing information about student behavior between school and families. 8 CSBS will use fidelity measures to monitor schools’ progress, identify areas in need of improvement, and make adj ustments in training when necessary. CSBS plans to use the Effective Behavior Support (EBS) survey and the Tiered Fidelity Inventory (TFI) im plementation tool for this purpose. CSBS will use the Behavior Education Program Fidelity of Implementation (BEP - FI M) to assess fidelity of implementation for the Tier II intervention.8

CSBS’ provision of ongoing support for the Tier II intervention during the 2016 - 2017 school ye ar will follow the same model as their provision of o ngoing support for Tier I. However, CSBS will conduct an extra site visit during year 2 of the study so they have sufficient time to provide ongoing support for the implementation of Tier I and Tier II. **MTSS -**  **B Program Implementation** The key components of the Tier I and Tier II MTSS - B program evaluated in this study are described in the paragraphs below. **Tier I Practices:** The school - level implementation of Tier I practices will occur in all Program Schools during both the 2015 - 2016 and 2016 - 2017 schoo l years. These practices include school wide activities as well as specific classroom components that are overseen by the SLT and MTSS - B Coaches. **Leadership provided by SLT and MTSS -**  **B Coaches:** The SLT and MTSS - B Coaches are the lead implementers of the p rogram and are responsible for introducing the program model and implementation plan to other staff members with support from CSBS trainers. They meet regularl y to monitor implementation of the school action plan and, based on data collected on student beh avior and implementation fidelity , provide support to other school staff to improve implementation. **Ongoing Support for Teachers by the SLT and MTSS -**  **B Coaches:** The SL T and MTSS - B Coaches provide ongoing training and coaching to teachers. These supports i nclude collection of data on teacher performance as well as the provision of targeted feedback to teachers and opportunities to practice the Positive Behavior Game. **School -**  **wide implementation of program:** A key component of the Tier I practices of MTSS - B is that the program is implemented school - wide and not just by particular administrators, teachers, or staff members. All school staff will be responsible for implementing the school’s 3 - 5 behavioral expectations, responding appropriately to positive and p roblem student behavior, and using student behavior data to develop solutions to specific problems.

**CSBS ’**  **Eight Classroom Components**

 Arrang e orderly physical environment  Define, teach, and acknowledge rules aligned with school - wide expectations by modeling desired be haviors and using a gaming strategy;  Explain and teach routines;  Provide specific and contingent praise for appropriate behavior;  Use class - wide group contingencies;  Provide error correction through prompt, re - teach, and provision of choices;  Em ploy ac tive supervisi on – mo ve, scan, interact;  P rovide multiple opportunities to respond.9

**Classroom components:** Within the classroom, teachers establish and maintain a classroom management system, which includes 8 specific components (see text box above). **Tier II Practices** : Tier II will begin in all Program Schools during the 2016 - 2017 school year. During this period, the SLT and MTSS - B Coaches will review behavioral monitoring data to identify students who are not responding to Tier I supports . They will also solicit referrals from school staff and families. These students will receive the Check In Check Out (CI CO) intervention, which is the Tier II intervention in this study . CI CO is a small group intervention that reinforces and extends the universal expectations by sy stematically

providing a higher frequency of scheduled prompts, pre - correction, and acknowledgement to participating students. The program consists of students daily “checking in” with an adult at the start of school day to retrieve a goal sheet and receive encouragement and help in setting specific behavioral goals (including skills to practice) for the day . Throughout the day , teachers (or other school staff) provide feedback to the student and document student behavior on the goal sheet. Students “check out” at the end of the day with an adult, who summarizes the feedback and recognizes accomplishments. The Targeted Team (TT), usually a subset of the SL T, monitors students’ progress on CI CO and establishes s ys tems of communication with families. **STUDY DESCRIPTION Study Design**  School level random assignment is appropriate for this study because, by

definition, MTSS - B is a whole school intervention delivered at the school level. The anal ys is of MTSS - B impacts will rely on a random assignment design in which schools within each participating district (approximately

10) are randomly

assigned to the following two groups: (1) Program Schools: Training, infrastructure and support to implement Tier I components in ye ar 1 (SY2015 - 2016) and Tier I and Tier II components in year 2 (SY2016 - 2017) (2) Business As Usual Schools: Continue with any existing practices related to student behavior, and will not receive an y additional training and support in MTSS - B from the evaluation. First year impacts of Tier Iwill be calculated by comparing first year outcomes in all 60 Program Schools implementing Tier I to first ye ar outcomes in the 30 BAU schools. The second ye ar impacts of Tier I plus II will be calculated b y comparing second ye ar outcomes in the Program Schools to second year outcomes in the BAU schools. A baseline teacher assessment of student behavior (in fall 2015) allows for the identification of a high risk subgroup and the estimation the net impact for high risk students of Tier I (year 1) and Tier I plus II services (year 2) compared to BAU schools.10

The study team will recruit approximately 90 elementary

schools across approximately ten districts that are willing and able to comply with the requirements of MTSS - B training providers and the research and data collection procedures involved in the evaluation. The impact study

will examine several outcomes of MTSS - B

—teacher practice, school climate, student behavior and academic achievement -- following the theory of action presented abo ve. The study will also consider the difference in behavior support practices in program and BAU schools (service contrast). Finally , the evaluation will describe the CSBS’ training and support activities as well as the implementation of the MTSS - B in programs in schools.

**Data Collection Strategy & Analytic Approach** Exhibit B - 2 provides an overview of data collection activities. Data sources that present a burden to students and/or staff are indicated with an asterisk. A longer description of and justificat ion for the data sources that present a burden is provided in the paragraphs below the table.11

Exhibit B - 2. Overview of MTSS - B Data Sources **Timeline # of**  **Respondents per School**  **# of**

**Schools Spring** 

**2015 Summer** 

**2015 Fall 2015**  **Spring**

**2016 Summer** 

**2 016**  **Fall 2015 Spring** 

**2016 Observation of CSBS training and support**  **activities in Program Schools** 0 60 X X X X X X X **Site visits*** School tour to observe school rules 0 90 X X X Document reviews 0 90 X X X MTSS - B Coach interview **˅** • 20 minutes per wave in Program Schools .33 60 X X X Administrator interview^ • 4 5 minutes per wave in BAU schools • 6 0 minutes per wave in Program Schools 1 90 X X X Team leader interview^ • 30 minutes per wave in BAU schools • 45 minutes per wave in Program Schools 1 90 X X X Student interview (randomly selected) • 2 minutes per wave 15 90 X X X Staff interview (randomly selected) • 5 minutes per wave 15 90 X X X **Review of web -**  **based forms (Coach logs, meeting minute forms and TIPS**  **school** 0 60 X X X X X X12

**forms) Review of sample of CSBS records (CSBS**  **trainer logs, attendance records, webinars, fidelity data)**  0 60 X X X X X X X **Review of MTSS -**  **B behavior monitoring and fidelity data (SWIS data, SWIS**  **- CICO data &**  **PBIS Apps data)** 0 60 X X X X X X X **School staff survey (All grades k -**  **5 teachers and other relevant staff) ** * 35 minutes for non - teaching staff 40 minutes for teachers 30 90 X X **Student survey (All students in grades 3 -**  **5)*** 50 minutes to administer survey per wave 240 90 X X **Classroom observations with the ASSIST and possibly CLASS [augmented in Program**  **Schools to included assessment of implementation fidelity]**  0 90 X **Teacher ratings of student behavior (All teachers in grades 1**  **- 5; all consenting**  **students in class)** * 10 minutes per student per wave 20 90 X X X **District records data collection** (10 districts) 0 90 X X X Note: Activities that present a time burden to students and staff are indicated with an asterisk(*). ^ The structured interviews wi th the team leader and administrator listed under site visits are longer in Program Schools because the interview protocol includes questions related to these staff members’ perception of the quality and utility of CSBS training and support. **˅** The MTSS - B Coach role is designated as 0.2 FTE for each program school. On average, however, districts will need 2 different MTSS - B coaches, though one of them may be only part - time. This increases the number of respondents for data collection purposes, but does not change the actual staffing.13 **Description of Data Sources that Present a Burden to Staff and Students Site visits:**  We will conduct site visits in Program and BAU schools three times over the study period (Fall 2015, Spring 2016 and Spring 2 017). The site visit protocols involve document review, observation and interview. The times required for the interviews conducted as part of the site visit are indicated in Exhibits B - 2 and B - 3 . The data collected from these visits will serve the followin g purposes: (1) Assess the difference in behavior support practices in program and BAU schools (service contrast) (2) Assess implementation fidelity to the core feature of the MTSS - B Tier I and Tier II programs in Program Schools (3) Assess key implementer s’ (MTSS - B Coaches, SLT team leader and administrator) perception of the quality and utility

of CSBS training and support in Program Schools. To develop the protocols for these site visits, the study team has pulled items from School wide Evaluation Tool ( SET), which has been found be a valid and reliable measure of MTSS - B Tier I implementation fidelity ; 9 it has also been used in prior randomized control trials of MTSS - B as a measure of service contrast between Program and BAU schools and found to accuratel y discriminate between schools implementing the core features of MTSS - B and those not. 10 The I - SSET assesses the fidelity of Tier II and Tier III implementation, 11 and it has been combined with SET in prior studies to provide a complete picture of the three tiers of MTSS - B implementation. 12 Understanding staff’s perception of the quality and utility of CSBS training and support in the Program Schools is critical to fully understanding implementation fidelity and to identifying challenges associated with the implementation of MTSS - B in a large number of elementary school and districts. The SET/I SSET protocols do not ask school staff to reflect upon the quality and utility

of the MTSS - B training and support they received from CSBS. As a result, the study team w ill be augmenting the site visit interview protocols in Program Schools with questions for the MTSS - B Coaches, SLT team leader and administrator about their perceptions of the quality and utility

of CSBS training and support. The interview protocols for th e site visits can be found in Appendi ces A - E . **Staff Survey:** School staff (teachers, administrators and other staff) in all study schools will be asked to participate in a web - based survey about school climate, staff practices related to behavior, and trai ning for behavior support practices and programs in the spring of 2016 and spring of 2017. The survey s also include a number of questions related to individual background characteristics and experiences. The survey for non - teaching staff will take approxim ately 35 minutes to complete and the survey for teaching staff will take approximately 40 minutes to complete. To assess staff practices related to behavior, we are drawing items from the Effective Behavior Support Survey

(EBS). 13 These items assess the extent to which schools are implementing core 9 Horner et al. (2004). 10 Bradshaw , Waasdorp, and Leaf (2012). 11 Anderson et al. (2008). 12 Debnam, Pas and Bradshaw (2012). 13 Sugai, Todd, and Horner (2000).14

features of MTSS - B in classroom and non - classroom settings. We have also drawn upon questions from the multi - site evaluation from IES’ Social and Character Development study to identify differences between BAU and Program Schools in the extent to which they have received training for and/or are implementing school - wide and classroom based programs similar to the MTSS - B approach being implemented in Program schools. Additionally , to assess implementation fidelity the study team has designed questions that ask about strategies specific to CSBS MTSS - B approach; the questions are general enough to appl y to Program and BAU schools but specific enough to the CSBS model that they will inform our assessment of implementation fidelity . We have also designed questions to assess staff’s exposure to training and coaching related to behavior support practices. Staff’s answers to these questions will inform our understanding of service contrast; full y understanding service contrast is critical to the interpretation of the impact estimates. Improved school climate is theorized to be proximal outcome of the intervention and is therefore critical to measure to assess staff’s perception of school climate we are drawing from two we ll validated measures

— Organizational Health Inventory (OHI) and the Malasch Burnout Inventory . 14 Prior studies of MTSS - B have found impacts on the OHI 15 . The staff survey can be found in Appendix F . **Student Survey (Grades 3 -**  **5):** Students in grades 3 - 5 in P rogram and BAU schools will be asked to complete a survey in the Spring of 2015 and Spring of 2016. The survey will take approximately

50 minutes for all fielding. This survey will provide critical information about students’ perception of their own behavi or and classroom climate, which are both predicted by the theory

of action to be positively impacted by

MTSS - B. Man y of the scales included were selected because they

have been widely

used in previous research on school - wide interventions; many have been f

ound to have adequate ps ychometric properties. 16 We have also included a limited number of questions to assess students’ awareness of core features of MTSS - B implementation such as the establishment of school rules, a positive reinforcement s ys tem and a gam ing strategy

to reinforce the school wide expectations. These questions will be asked in the survey administered in BAU and Program schools so they can inform the assessment of implementation and service contract. The student survey can be found in Appendi x G . **Teacher Ratings of Student Behavior:** Teachers of students in grades 1 - 5 in Program and BAU schools will be asked to rate their students’ behavior in fall of 2015, 17 the spring of 2016 and the spring of 2017. Most districts will require that we obtain active parental consent from parents for these ratings and so teachers will only be asked to rate the behavior of students in their classes with consent. The rating form is expected to take approximately ten minutes per student to complete. 14 Hoy and Tartar (1997); M aslach and Jackson (1981). 15 Bradshaw

et al., 2008. 16 Gresham et al. (2011); Haynes et al. (2001); Huesmann and Guerra (1997); Roberts, Hom, and Battistich (1995); Social and Character Development Research Consortium (2010). 17 Fielding this survey at base line, will allow the study team to identify a high - risk sub - group at baseline. Identification of this subgroup is necessary for answering one of the study ’ s research questions regarding the effects of MTSS - B on at - risk students.15

The teacher r atings form will be used to assess the impact of the program on student behavior. The study team has adopted the Teacher Observation of Student Adaptation - Checklist (TOCA - C) for this instrument. The TOCA - C has been shown to be a valid and reliable measure of students’ behavior (concentration problems, disruptive behavior and prosocial behavior). 18 It has been used in prior evaluations of MTSS - B where it has been shown to be sensitive to the intervention. 19 The study

team is augmenting this measure with questi ons that will provide information regarding students’ referral to and receipt of targeted supports for behavior and special education services. Asking this information on the teacher ratings forms is important because it cannot alway s be reliabl y ascertain ed from administrative datasets in Program and BAU schools. Finall y, the rating form includes one scale that assesses teachers’ perception of students’ academic behaviors. 20 This scale has been added because the theory of action predicts that MTSS - B will improve students’ academic engagement, which is not assessed on the TOCA C. The teacher ratings form can be found in Appendix H . **District Records Data Collection:** The study team will request extant data from school district records regarding student background characteristics, teacher background characteristics and experiences, student behavior incidences, student academic achievement (grades 3 - 5) and special education. Data will be collected at the end of each school year: baseline (SY 2014 - 2015), year one (SY 2015 - 2016) and year 2 (SY 2016 - 2017). Requests will be made of district research offices. Appendix I provides a sample of the letter that will be sent to districts requesting these data.

18 Koth, Bradshaw

and Leaf (2009). 19 Bradshaw , Waasdorp, and Leaf (2012). 20 The study team will be adopting the “approaches to learning” scale from NCES’ ECLS - K survey to assess this construct. All necessary permissions will be acquired before the survey is fielded.16 **SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION**  **P art B: Description of Statistical Methods**  1. Respondent Universe and Sampling Methods The sample for this school - level random assignment study will include approximately 90 schools across approximately

ten school districts. To arrive at this sample from the universe of schools districts and elementary

schools within districts, study team and the Department developed a set of district and school eligibility guidelines, which are summarized in Exhibit B - 4 . **Exhibit B -**  **4 : Screening guidelines used for recruitme**  **nt** District Screening Guidelines: A district must have 10 public schools serving students in Kindergarten Grade 5 or Grade 6 that meet the following criteria:  **School Size:** Have at least 50 students per grade, on average.  **Family Income:** 25% of students q ualify for Free and Reduced Priced Lunch or are Title 1 Eligible [According to 2011 - 2012 CCD or district data].  **Current Behavior Support Strategies:** Are not currently implementing MTSS - B Tier I or Tier II programs.  **Interest:** The district must be interested in implementing the MTSS - B program in nominated schools. The district must fully understand the random assignment process and the burden of data collection. School Screening Guidelines: Within an eligible district, a school will be considered for nomination if they meet the following criteria:  **School Size:** Do not have a single grade serving fewer than 40 students [according to 2011 - 2012 CCD or district data]. Across all nominated schools, there will be an average of at least 60 students per grade.  **Fam ily Income:**  Across all nominated schools within a district, the percentage of FRL students must be greater than 25% with preference given to schools that have at least 15% of students qualify for Free and Reduced Priced Lunch [according to 2011 - 2012 CCD or district data].  **Current Behavior Support Practices:** Are not currently implementing MTSS - B Tier I and Tier II programs.  **Interest:** The schools must be interested in implementing the MTSS - B program if they are selected as a program school and be willing to continue with business - as -usual if they are selected as a BAU school. Schools must fully understand the random assignment process and the burden of data collection. **Site Recruitment Process:** To recruit districts and schools and to determine eligibility , the study team first identified 480 districts across the country that met the final screening criteria (see17

Exhibit B - 4 ). The evaluation team then contacted potential sites and provided them with information on the project and the research design. The di stricts were discussed their current behavioral s ys tems in place, as well as the district's ability and willingness to implement research procedures related to random assignment and data collection. The team conducted site visits at all of the promising di stricts to discuss the study more in depth, provide detailed data collection plans, discuss the roles, responsibilities and expectations for all participants, and review which schools might be eligible for participation. Eligible and interested districts then negotiated and signed a tentative agreement to participate with the project director for the research team. The pool of eligible districts that signed this agreement was then sent to the Department of Education for final selection. Final selection of districts and schools took into account the advantage for the policy relevance of results of having sites that include a variety of regions, urban/suburban/rural settings, and race/ethnicity of the student body . The research team worked closely with the pro ject officer in the Department of Education to make final choices. Selected districts and schools then received formal notice of inclusion in the study . As of April 2015, five districts with 47 schools have been accepted for the study. 2. Procedures for Data Collection MDRC will work with partner organization Decision Information Resources (DIR) to conduct the data collection activities. The paragraphs below describe the data collection strategy for the data sources that impose a burden on students or schoo l staff: site visits to Program and BAU schools; staff survey s; student survey s; and teacher ratings of student behavior. The following sections present our approach for each data collection effort, including information on sample and timing, instruments , and training. More information about data sources can be found in the Description of Data Sources beginning on page 14. **Site Visits:** The site visits include document review, observations, and interviews with key implementers (MTSS - B Coaches, behavior te am leaders, and administrators). The data collected from site visits serves three purposes: to assess the difference in behavior support practices between program and BAU schools; to assess implementation fidelity of the core features of the MTSS - B Tier I and Tier II programs in program school; and to assess the key implementers’ perception of the quality

and utility

of CSBS training and support for the MTSS - B Tier I and Tier II program. As described in the introduction to Supporting Statement A, the site v isit protocol draws from the Schoolwide Evaluation Tool (SET) and the Individual Student Systems Evaluation Tool (I SSET), which assess of MTSS - B implementation fidelity and service contrast. 21 These protocols have been augmented with structured interview pr otocols in Program Schools to assess the quality and utility

of CSBS training and support. Site visits to all schools will occur three times over the course of the study, during the Fall and Spring of the first year (2015 - 2016), and during Spring of the second ye ar (2016 - 2017). During the visits to Program Schools, the study team will conduct interviews with the MTSS - B Coaches (20 minutes), an administrator (60 minutes), and the behavior team leader (45 minutes). Additionally , there will be interviews with 15 randomly selected staff members (5 minutes each) 21 Anderson et al., 2008; Horner et al., 2004; Debman, Pas and Bradshaw , 2012.18

and 15 randoml y selected students (2 minutes each). Interviews in BAU schools will include a 45 minute administrator interview, 30 minute interview with the behavior team leader, 5 minute interviews with 15 randomly selected staff and 2 minute interviews with 15 randomly selected students. Interviews will be conducted b y DI R and MDRC staff with training b y Catherine Bradshaw’s team at the University of Virginia and Johns Hopkins University22 and the MDRC staff. The data collection team understands the challenges involved with conducting site visits at schools during the school day . Therefore, our approach will be flexible to allow a range of options that will accommodate known or unknown challenges. The team will work closely with school liaisons to arrange access to each school, identify the availability

of space, determine best times for the individual interviews, and identify any “restricted” times or day s (for example, field trips, school closings, student testing, and so forth). Site visitors will be required to have experience conducting site visits in school settings and have at least a bachelor’s degree. Preference will be given to individuals with an education, child development, or similar educ ational background. All site visitors will attend a 3 - 4 day training in which they

will go over details about the study and site visit procedures. A portion of the training will include visits to non - study schools to assess the site visitors' reliability with the protocols The training agenda will include:

 An overview of the overall study  General information about conducting site visits  Proper administration of the measurement tool  Quality

assurance  Scoring each tool  Practice sessions immediately foll owing the training  Administrative procedures, responsibilities, and compensation  Review of the certification process and criteria to meet established standard Attendees will also review the instrument, professional conduct, policies and guidelines for reporting child maltreatment, and administrative policies and guidelines. A manual including information from the training will be provided to attendees. Site visitors who meet the certification requirements will be selected to conduct the site visits. The specific site visit activities will be carried out as follows:

 The team will work with the school liaison to schedule the visit, set up the required interviews, and notify

them of the required documents/procedures that will be reviewed.

22 Bradshaw

and her colleagues have used the site visit protocols being used in this study in multiple other studies of MTSS - B; for example, see Debman, Pas, and Bradshaw (2012).19

The agreed upon schedule will be provided to the school electronically and a reminder will be sent 1 week and 1 day prior to the scheduled visit.  On the day

of the site visit, the site visitor will arrive 30 - 60 minutes prior to scheduled start time, and upon arriving, the y will check - in with the school liaison and review the schedule for the day

and ensure that all necessary documents are available for review. In addition, the site visitor will confirm that the planned interviews will be carried out with the specified peop le.  After the schedule is set, the site visitor will begin carrying out the various required steps including selection of a random sample of students and staff for brief interview (students interview is 1 - 2 minute per student and the staff interview is 3 - 5 minutes long). The procedures of this random sampling will be worked out with the district and the school liaison ahead of time so that every effort is made to not disrupt instruction.  Upon completion of all pieces, the site visitor will alert the school liaison that the visit is finished, or if there were missing elements, that will be reviewed at that time.  Site visitors will be required to conduct the scoring of both protocols on the same day the visit was conducted. Scored protocols will be sent back to DIR and MDRC, reviewed for quality control, and then entered into a database. **Staff Survey:** The staff survey is designed to collect data on the school climate, staff practices related to behavior, and training for behavior support practices. This sur vey will be provided to teachers and non - teaching staff at all study schools in a web based format in the Spring of 2016 and the Spring of 2017. DIR is managing the fielding of this survey . The survey

will be administered to all teachers in grades 1 - 5 as w ell as all non - teaching staff and administrators (n=30/per school). School staff will be given the option to complete all survey s by

using the web version or paper version or a combination, which will maximize the attainment of the desired 90 - percent (or h igher) response rate. School staff emails will be obtained through the school data collection liaison. For staff that request to complete the questionnaires using the web version, the survey will launch with a web - only period, which will last for three ful l weeks. During the web - onl y period, reminder emails will be sent to non responders at intervals determined by web activity

but no more than once per week. DIR will format the questionnaire for self - administration via the web or hardcopy . Prior to completi ng the survey , teachers and other school staff will be asked to provide consent, and they may also withdraw their consent at any time. Training for the staff survey administration will be nominal. From experience, most school staff will opt to complete the report using the web - based version. For those opting to use hardcopies, DIR staff will disseminate hardcopies of the instrument to school staff by mailing them to school liaisons. School staff will be asked to put all completed hardcopies in a DIR - provi ded envelope, seal the envelope, and return them to the school liaison. They will be shipped to the Houston office and logged as completed. Field personnel will be trained in the areas of confidentiality, data security

(including collecting, securing, and transporting data), and tracking or logging receipt of complete and incomplete instruments.20

DIR will send hardcopy

questionnaires (through school liaisons) to staff that request to complete the questionnaires using the paper version hardcopies. At the sta rt of week 4, hardcop y questionnaires will be sent to school liaisons who will deliver them to the non - responders who requested to complete using the web version. A reminder postcard will be designed (to be delivered in person or via regular mail) and sent to all non - responders, and additional email reminders will be sent to teachers with email addresses during the last 2 weeks of the fielding period. DIR’s previous experience with attempting to reach teachers and school staff b y telephone has not been very successful. Ty picall y, messages are left without any certaint y of their delivery . Thus, DIR will ask the school liaisons to hand deliver reminder postcards and give the teachers and school staff a verbal reminder at specific times during the fielding peri od. Each of these steps will help to ensure that the desired response rate is reached by the end of the fielding period. In summary , DIR will complete the following steps in fielding the staff survey :  Two weeks prior to launching the web survey , DIR wil l send staff a personalized letter (via email and regular mail) with a description of the importance of the questionnaire, stressing the confidentiality of their responses, explaining how the results will be used, describing the honoraria, and indicating when they will receive their individualized link to complete the web version. DIR will also thank them in advance for their participation.  On the day

of the survey

launch, DIR will send staff an email announcing the launch of the survey . This email will co ntain a unique identification number and a link to the survey . For staff without an email address, DIR will send the same information via regular mail.  At the 3 - day , 10 - day , and 14 - day mark, DIR will send reminder emails and/or postcards to thank staff for their participation or encourage them to respond if they have not already done so. At the 14 - day

mark, the option to complete a hardcopy survey will be included in the reminder. Upon request, hardcop y survey s will be sent directl y to the teacher via FedEx overnight delivery, along with a pre - paid return envelope. DIR will ask school liaisons to send a reminder to staff to express district/school support for the study and encourage participation. DIR will provide the specific text for them to send, in an effort to minimize the burden on the liaisons. **Student Survey (Grades 3 -**  **5) :**  This survey will provide critical information about students’ perception of their own behavior and classroom climate, which are both predicted by the theory of action to be positi vely

impacted by

MTSS - B. DIR will field the survey to all students in grades 3 - 5 in approximately 90 participating elementary

schools across 10 districts (9 schools per district) during the spring of 2016 and 2017.The survey

will take approximately

50 mi nutes for all fielding (the actual survey will only take 40 - 45 minutes to complete). Data collection in each school will occur over a 2 - 3 day period. The study team anticipates needing to collect active parental consent for this activity since man y of the likely

districts will require it. See question A - 9 and B - 3 for more detail on our proposed strategy for maximizing response rates on the consent form.21

DIR will recruit and hire a team of 8

– 10 proctors per district. Each team will be led by an assessor with extensive classroom experience. Candidates will be required to have experience working with children between the ages of 8 - 11 and have some college education. Preference will be given to individuals with a child development or similar educational back ground. All proctors will attend a training in which they will go over details about the study and data collection procedures. The training agenda will include:

 An overview of the evaluation and student survey  General assessment techniques for working w ith children ages 8

— 11  Proper administration of the measurement tool  Quality

assurance  Live practice sessions immediately following the training  Administrative procedures, responsibilities, and compensation  Review of the certification process and criteria to meet established standard Attendees will also review the instrument, proctors’ responsibilities, professional conduct, policies and guidelines for reporting child maltreatment, and administrative policies and guidelines. A manual including informati on from the training will be provided to all proctors. The following approaches will be used to collect the student survey data:

 DIR will work closel y with school liaisons to arrange access to each school, identify the availability

of space, determine b est times for assessments, and identify any “restricted” times or day s (for example, field trips, school closings, and so forth).  DIR staff will call approximately 3 business day s before the scheduled visit to confirm the date, the school contact (if dif ferent from the liaison), will verify the location for the administration, and gather any additional necessary information.  DIR will assign 2 proctors to all third, fourth and fifth grade classrooms. For each day of data collection, DIR staff will arrive 1 hour before the actual administration is set to begin. One proctor will distribute the materials and verify all materials match the classroom roster and are distributed accuratel y. The second proctor will confirm that all students have their assigned mate rials, distribute pencils, walk the classroom during the survey

administration, and collect all materials at the end of the survey administration assigned.  Survey

instructions and all questions will be read aloud to students.  Completed survey s will be lo gged and returned to the DIR offices for scoring. **Teacher Ratings of Student Behavior:** The teacher ratings will be used to assess the impact of the program on student behavior and the students’ referral to and receipt of targeted supports for behavior and special education services. Teachers of students in grades 1 - 5 in all study schools (n=90) will be asked to rate their students’ behavior in the fall of 2015, the spring 2016 and the spring of 2017. DIR field staff will contact all teachers of students in grades 1 - 5 in 90 participating schools across 10 districts. The rating22

form is expected to take approximately ten minutes per student to complete and each teacher will rate all of their students. The study team anticipates needing to collect active pare ntal consent for this activity

since many

of the likely

districts will require it. See questions A - 9 and B - 3 for more detail on our proposed strategy for maximizing response rates on the consent form. The rating checklist will be administered electronic ally with DIR using paper - based survey s to follow - up with non - responders. Fielding procedures for this instrument are consistent with the web - based staff survey

described above. In summary , the teacher rating checklist data collection activities will be carried out as follows: Two weeks prior to launching the web link, DIR will send teachers a personalized letter (via email and regular mail) with a description of the importance of the rating checklists, stressing the confidentiality of their responses, exp laining how the results will be used, and indicating when they will receive their individualized link to complete the web version. DIR will also thank them in advance for their participation. On the day

of the web launch, DIR will send teachers an email announcing the launch of the web version. This email will contain a unique identification number and a link to complete the ratings checklist. For staff without an email address, DIR will send the same information via regular mail. At the 3 - day , 10 - day , and 14 - day mark, DIR will send reminder emails and/or postcards to thank teachers for their participation or encourage them to respond if they have not already done so. At the 14 - day mark, the option to complete a hard - cop y version of the ratings checklists will be included in the reminder. Upon request, hardcop y versions will be sent directl y to the teacher via FedEx overnight delivery , along with a pre - paid return envelope. DIR will ask school liaisons to send a reminder to teachers to express district/ school support for the study and encourage participation. DIR will provide the specific text for them to send, in an effort to minimize the burden on the liaisons. 3. Procedures to Maximize Response Rates The goal will be to achieve a 90 percent response rate or higher in each site for each data collection effort. Procedures for obtaining the maximum degree of cooperation include:

 Sending respondents cover materials that address the importance and value of the study .  Collecting all parental consent forms f or student data collection activities that require parental consent at the start of the school ye ar  Working closel y with school principals and/or liaisons to develop a schedule that will facilitate administration to all students.  Training proctors to mai ntain direct contact with respondents and provide assurances to reluctant responders.23

 Providing proctors packets of student surveys with blank labels in the event that new students are added to the class.  Allowing school staff and teachers multiple format s (web and paper - pencil) for completing the instruments.  Offering appropriate payments to school staff and teachers.  For several districts, active consent will be required, thus maximizing response rates for obtaining parent consent will be critical for the success of the student survey , student assessments, and teacher ratings. Maximizing the response rate will require a multilingual and multistep approach. All materials will be prepared in both English and Spanish and distributed in packets by

classroom to teachers with a request to distribute the forms to all children in their classrooms. Forms will be printed on 2 - ply carbonless paper so that parents could keep a signed cop y. The consent packet will include a written explanation of the study

and a reque st that the parent or guardian allow their child’s participation in the study by signing the forms. As an incentive, teachers or schools data collection liaisons (whomever is responsible for distributing the consent forms) will be offered a $25 gift card i f they

achieve an 90% response rate or better return of parental consent forms . 4. Testing of Data Collection Procedures Most of the questions proposed for this survey are either identical to questions used in prior MDRC or AIR evaluations 23 or are similar to questions used in previous national survey s or major evaluations. 24 Consequently , man y of the items have been thoroughl y tested on larger samples. The MTSS - B survey s and interview protocols have already undergone a number of revisions, following critiqu es by

internal staff, by

project consultants, and by staff at IES. MDRC will also work closel y with DIR and AIR senior staff to conduct formal pre - tests of these survey s and interview protocols, using fewer than nine appropriate respondents. 5. Individuals Consulted on Statistical and Methodological Aspects of Data Collection This project is being conducted under contract to the Department of Education by MDRC, AIR, Harvard Universit y, and DIR. The data collection strategy and instruments were developed b y Fred Doolittle, Barbara Condliffe of MDRC; Anja Kurki, L ouis Danielson, Muna Shami and Gail Chan of AIR, Stephanie Jones of Harvard Graduate School of Education, and Catherine Bradshaw of the University of Virginia. Pei Zhu of MDRC is leading the statisti cal design of the study and the estimation of the impacts of the MTSS - B training and support. The organizations responsible for data collection activities are as follows: 23 Some exam

ples of such evaluations include MDRC's current Evaluation of Response to Intervention for Elementary School Reading (report due in 2015; http://ies.ed.gov/ncee/projects/evaluation/disabilities_rti.asp ) and AIR's Good Behavior Game Professional Development Study (http://www.air.org/project/good - behavior - game - professional - development - study). 24 For example, Anderson et al. (2008); Bradshaw et al. (2008a); Bradshaw et al. (2008b); Bradshaw, Mitchell, and Leaf (2010); Bradshaw, Waasdorp, and Leaf (2012); Jones, Brown, Hoglund, and Aber (2010); Jones, Brown, and Aber (2011 a); Jones, Brown, and Aber (2011b); Koth, Bradshaw, and Leaf (2009).24

**Organization Primary Contact**  **Phone Number** MDRC Fred Doolittle 212 - 340 - 8638 American Institutes for Research Anja Kurki 202 - 403 - 6022 Decisions Information Resources (DIR) Russell Jackson 713 - 650 - 1425 Individuals outside these organizations who were consulted on the design of data collection strategy are as follows: **Name Organization**  **Phone number** David Cordray Vanderbilt University, Department of Psychology and Human Development 615 - 343 - 2699 Brian Flay Oregon State University, School of Social and Behavioral Health Sciences 541 - 737 - 3837 Mary Louise Hemmeter Vanderbi lt University, Department of Special Education 615 - 322 - 5648 Sara E. Rimm - Kaufman, Ph.D. University of Virginia, Curry School of Education and Center for the Advanced Study of Teaching and Learning 434 - 982 - 2863 Lori Newcomer, Ph.D. University of Missouri, Missouri Prevention Center / Department of Educational School and Counseling Psychology 573 - 882 - 7731 314 - 605 - 0235 Catherine Bradshaw, Ph.D. University of Virginia Jo hns Hopkins University 434 - 924 - 8121 Stephanie Jones, Ph.D. Harvard University 617 496 - 2223 The contracting offer for this study is Dr. Lauren Angelo of the National Center for Education Eva luation (IES). Her phone number is: 202 - 219 - 2180.25 **REFERENCES USED IN PART B**  Anderson, C. M., Teri Lewis - Palmer, Anne W. Todd, Robert H. Horner., George Sugai, N.K. Samson. 2008. "Individual student sy stems evaluation tool, version 2.6." Educational and C ommunity

Supports, University

of Oregon. Bradshaw, Catherine P., Christine W. Koth, Katherine B. Bevans, Nicholas Ialongo, and Philip J. Leaf. 2008a. “The Impact of School - Wide Positive Behavioral Interventions and Supports (PBI S) on the Organizational Hea lth of Elementary Schools.” School Psychology Quarterly, 23, 4: 463 - 473. Bradshaw, Catherine P., Mary M. Mitchell, and Philip J. Leaf. 2010. “Examining the effects of School - Wide Positive Behavioral Interventions and Supports on student outcomes: Results f rom a randomized controlled effectiveness trial in elementary schools.” Journal of Positive Behavior Interventions, 12, 133 - 148. Bradshaw, Catherine P., Wendy M. Reinke, Louis D. Brown, Katherine B. Bevans, and Philip J. Leaf. 2008b. “Implementation of sch ool - wide Positive Behavioral Interventions and Supports (PBI S) in elementary schools: Observations from a randomized trial.” Education and Treatment of Children, 31, 1

–26. Bradshaw, Catherine P., Tracy E. Waasdorp, and Philip J. L eaf. 2012. "Effects of Sch ool - Wide Positive Behavioral Interventions and Supports on Child Behavior Outcomes." PEDIATRICS 130, 5: e1136 - e1145. Debnam, Katrina J., Pas, Elise T., and Bradshaw, Catherine P. (2012). Secondary and tertiary support sy stems in schools implementing school - wide positive behavioral interventions and supports: A preliminary descriptive anal ys is. Journal of Positive Behavior Interventions, 14 (3), 142

– 152. Durlak, Joseph A., Roger P. Weissberg, Allison B. Dy mnicki, Rebecca D. Taylor, and Kriston B. Schellinger. 2011. "The impact of enhancing students’ social and emotional learning: A meta - anal ys is of school - based universal interventions." Child Development 82, 1: 405 - 432. Gresham, Frank M., Stephen N. Elliott, Michael J. Vance, and Clay ton R. Cook. 2011. "Compar abilit y of the Social Skills Rating System to the Social Skills Improvement Sy stem: Content and Psychometric Comparisons Across Elementary and Secondary

Age Levels." School Psychology Quarterly 26, 1: 27 - 44. Hay nes, N. M., C. L. Emmons, M. Ben - Avie, and J. P. Comer. 2001. "The school development program student, staff, and parent school climate survey s." New Haven, CT: Yale Child Study Center. Horner, Robert H. 2014. “The Role of District Leadership Teams in PBIS Implementation”. Presentation. Available at http://www.pbis.org/Common/Cms/files/pbisresources/1 - Implementing%20PBIS.pptx.26

Horner, R. H., Todd, A. W., Lewis - Palmer, T., Irvin, L . K., Sugai, G., & Boland, J. B. 2004. The school - wide evaluation tool (SET): A research instrument for assessing school - wi de positive behavior support. Journal of Positive Behavior Intervention s, 6, 3

– 12. Horner, Robert H., George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne W. Todd, and Jody

Esperanza. 2009. "A randomized, wait - list controlled effectiveness tri al assessing school - wide positive behavior support in elementary schools." Journal of Positive Behavior Interventions 11, 3: 133 - 144. Hoy Way ne K. and Clemens John Tarter. 1997. The Road to Open and Healthy Schools: A Handbook for Change, Elementary Editio n . Thousand Oaks, CA: Corwin Press. Huesmann, Rowell L., and Nancy G. Guerra. 1997. "Children's normative beliefs about aggression and aggressive behavior." Journal of Personality and Social Psychology 72, 2: 408 - 419. Jones, Stephanie M., Joshua L .Brown and J. Lawrence Aber. 2011a. The longitudinal impact of a universal school - based social - emotional and literacy intervention: An experiment in translational developmental research. Child Development 82,2: 533 - 554; Jones, Stephanie M., Joshua L . Brown, and J . Lawrence Aber. 2011b “Two - Year Impacts of a Universal School - Based Social - Emotional and L iteracy Intervention: An Experiment in Translational Developmental Research”. Psychology Facult y Publications. Paper 131. (http://fordham, bepress.com/psych_facult yp ubs/131). Jones, Stephanie M., Joshua L . Brown, Wendy LG Hoglund, and J. L awrence Aber. 2010. "A school - randomized clinical trial of an integrated social

– emotional learning and literacy intervention: Impacts after 1 school year." Journal of Consulting and Clinical Psychology 78, 6: 829. Koth, Christine W., Catherine P. Bradshaw, and Philip J. Leaf. 2009. "Teacher Observation of Classroom Adaptation

— Checklist: Development and factor structure." Measurement and Evaluation in Counseling and Development 42, 1: 15 - 30. Maslach, Christina, and Susan E. Jackson. 1981. "The measurement of experienced burnout." Journal of Organizational Behavior 2, 2: 99 - 113. OSEP Center on Positive Behavioral Interventions and Supports. 2007. “Is School - Wide Positive Behavior Support An Evidence - Based Practice? A Research Summary .” Website: www.pbismaryland.org. Roberts, William, Allen Hom, and Victor Battistich. 1995. “Assessing Students’ and Teachers’ sense of the School as a Caring Community .” Oakland, CA: Developmental Studies Ce nter.27

Sugai, George, Todd, Anne W., and Horner, Robert H. (2000). Effective Behavior Support (EBS) Survey : Assessing and planning behavior supports in schools. Eugene, OR: University

of Oregon. Sugai, G., Lewis - Palmer, T., Todd, A., & Horner, R. H. (2005). School - wide evaluation tool. Eugene: University of Oregon. Social and Character Development Research Consortium. 2010. "Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary

school children." Washington, DC: National Center for Education Research, Institute of Educational Sciences , U.S. Department of Education.