1

IMPACT EVALUATION OF TRAINING IN 

MULTI-TIERED SYSTEMS OF SUPPORT FOR 

BEHAVIOR (MTSS-B)

OMB CLEARANCE REQUEST:

DATA COLLECTION

SUPPORTING STATEMENTS B

April 2015

PREPARED FOR:

Institute of Education Sciences

United States Department of Education

Contract No. ED-IES-14-C-003

PREPARED BY:

MDRC

16 East 34th Street, 19th Floor

New York, NY 10016

Fred Doolittle, Project Director

Fred.Doolittle@mdrc.org

(212) 340-8638
2

TABLE OF CONTENTS

INTRODUCTION 3

THEORY OF ACTION AND RESEARCH QUESTIONS 4

CHARACTERISTICS OF THE TREATMENT TO BE TESTED 5

STUDY DESCRIPTION 9

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 

SUBMISSION 16

B. DESCRIPTION OF STATISTICAL METHODS 15

1. Respondent Universe and Sampling Methods 15

2. Procedures for Data Collection 16

3. Procedures to Maximize Response Rates 21

4. Testing of Data Collection Procedures 22

5. Individuals Consulted on Statistical and Methodological Aspects 

of Data Collection 22

REFERENCES 24

APPENDICES

A. Interview Protocols: MTSS-B Coach

B. Interview Protocols: Administrators

C. Interview Protocols: Behavior Team Leader

D. Interview Protocol: Students

E. Interview Protocol: Staff

F. Staff and Teacher Survey

G. Student Survey

H. Teaching Ratings of Student Behavior

I. District Records Data Collection Request Letter

J. Excerpt from IDEA, P.L. 108-446
3

INTRODUCTION

Creating and maintaining safe and orderly school and classroom environments to support student 

learning is a top federal priority, as indicated by the U.S. Department of Education and the U.S. 

Department of Justice jointly issued guidance on school discipline. The implementation of 

Multi-Tiered Systems of Support for Behavior (MTSS-B) is seen as a promising approach to 

improve school and classroom climate, student behavior, and academic achievement, and 

potentially reduce the inappropriate identification of students for special education. MTSS-B is a 

multi-tiered, systematic framework for teaching and reinforcing positive behavior for all 

students, that includes universal, school wide supports (often called Tier I), targeted

interventions to address the problems of students not responding to the universal components 

(often called Tier II), and intensive interventions (Tier III) for students not responding to Tier II 

services. 

The widespread use of MTSS-B nationally contributes to the relevance of this evaluation. The 

U.S. Department of Education’s (ED’s) Office of Special Education Programs has supported 

technical assistance around MTSS-B since 1998. Recent data from the Office of Special 

Education Programs TA Center on Positive Behavior Interventions and Support indicate that as 

of 2014, MTSS-B is reportedly implemented in over 21,000 schools across the nation.1 

As the support for and implementation of MTSS-B have increased, researchers have been 

developing an evidence base for it and other related programs. Numerous descriptive and 

correlational studies have associated aspects of MTSS-B implementation, or MTSS-B 

implementation overall, with student outcomes. More recently, researchers have begun to 

conduct causal studies of MTSS-B, although some of these initial studies have been on a 

relatively small scale (or conducted within a single state) or have been somewhat limited in 

duration and outcomes.2,3,4 A recent meta-analysis of research on a broad range of 213 individual 

school-based, universal social and emotional learning (SEL) programs (including interventions 

than can be implemented within an MTSS-B program) found that compared to controls, SEL

participants demonstrated significantly improved social and emotional skills, attitudes, behavior, 

and academic performance. 5 Much of this research has focused on individual programs and not 

examined the impact of a large-scale implementation of both universal, school-wide programs 

and targeted programs within a MTSS-B framework. 

Therefore, this evaluation focuses on a comprehensive MTSS-B program that includes each of 

the components usually associated with MTSS-B: school-wide and classroom-level strategies 

(universal supports, or Tier I) and individual or small group strategies (targeted interventions, or 

Tier II), with appropriate infrastructure (staffing sand data system). This evaluation will not 

support intensive supports for students not responding to Tier II interventions (i.e., there will be 

 

1 Horner (2014).

2 Bradshaw, Mitchell, and Leaf (2010).

3 Horner et al. (2009). 

4 OSEP Center on Positive Behavior Interventions and Supports (2007). 

5

Durlak et al. (2011).
4

no Tier III interventions). IES has contracted with MDRC, AIR, Harvard University, and 

Decision Information Resources (DIR) to conduct the Impact Evaluation of MTSS-B.

The goal of the evaluation is to implement an MTSS-B program with fidelity and determine 

whether universal supports (Tier I) and a targeted intervention (Tier II) with appropriate 

infrastructure are effective when implemented in a large number of districts and elementary

schools. 

Using a school-level random assignment design, this evaluation will examine the impact of 

training elementary school staff in the implementation of an MTSS-B framework including staff 

and data infrastructure with universal, school-wide components, as well as classroom-based 

activities (Tier I) in the first year of implementation as compared to outcomes for schools 

undertaking typical behavior support practices (These schools are hereafter called business-asusual or BAU schools.) In addition, the study will also examine the impact of this Tier I training 

plus additional training in a targeted intervention (Tier II) in a second year of implementation. 

Schools that receive MTSS-B training and support are hereafter referred to as “Program 

Schools.” The study will also investigate the differences in behavioral support services between 

program and BAU schools. Finally, the evaluation team will assess whether the MTSS-B 

training and support activities are implemented as intended as well as the fidelity of school’s 

implementation of the intended MTSS-B Tier I and Tier II activities. 

THEORY OF ACTION & RESEARCH QUESTIONS

The implementation of MTSS-B is ultimately intended to improve student outcomes. The 

conduct of school-wide and classroom strategies (Tier I) and individual or small group strategies 

(Tier II), supported by appropriate infrastructure, are designed to improve school staff behavior 

practices as well as school and classroom climate. The improved practices and climate are, in 

turn, hypothesized to benefit student behavior and academic outcomes. The individual and small 

group strategies may affect the students who need additional support (Tier II) as well as all 

students if these strategies lessen the disruption of students needing additional support. Finally, 

improved student behavior could lead to better student academic outcomes (see Exhibit B-1). 
5

Exhibit B-1: Conceptual Model of MTSS-B and Intended Outcomes

The evaluation is anchored on this conceptual model and will focus on the following research 

questions: 

 What MTSS-B training and support activities were conducted?

 Is the MTSS-B program implemented with fidelity? What are the key challenges to the 

implementation of MTSS-B with fidelity?

 What is the difference in behavior support practices occurring in program and BAU 

schools?

 What is the impact on school staff practices, school climate and student outcomes of 

providing training in the MTSS-B framework plus universal, school-wide and classroom 

Tier I positive behavior supports and a targeted Tier II intervention? 

 What are the impacts on school staff practices, school climate, and student outcomes for 

relevant subgroups (e.g., teachers with less experience; at-risk students)? 

 Does the implementation of MTSS-B reduce any differences in student discipline 

outcomes across racial and ethnic groups? Does it reduce the identification of students for 

special education services?
6

CHARACTERISTICS OF THE TREATMENT TO BE TESTED

In the spring of 2014, MDRC and partner organizations issued a Request for Proposals to 

providers of training in MTSS-B. In November 2014, we announced the winner of this 

competition, The Center for Social Behavior Support (CSBS), which is collaboration between 

The Illinois-Midwest PBIS Network at the School Association for Special Education in DuPage, 

Illinois (SASED) and the PBIS Regional Training and Technical Assistance Center at Sheppard 

Pratt, in Maryland. The training and support activities proposed by CSBS are described below. 

Following that summary, the memo describes CSBS’ plan for how the MTSS-B Tier I and Tier 

II program will be implemented in schools. 

CSBS Training and Support Activities

CSBS will mostly follow a “train the trainer” model with some direct support to all school staff 

and on-site support to the school staff and Coaches throughout implementation. Below we 

describe the key phases of their work with schools. 

Readiness Period - Training and Support by CSBS to Develop Infrastructure: The first step 

is the development of school-level infrastructures that will support implementation of Tier I and 

Tier II program components, which includes a staffing structure and data system. Tier I 

infrastructure will be put in place in the late spring of 2015 for all Program Schools. 

Infrastructure for Tier II will be developed in the spring of 2016. 

Staffing structure: CSBS trainers will work with each school to identify a representative 

School Leadership Team (SLT) and each district to identify MTSS-B Coaches. This will 

be accomplished during a two-day site visit to each district as well as webinars, email 

exchanges and phone calls. In addition to helping schools and districts identify the SLT 

and MTSS-B Coaches, CSBS will explain the MTSS-B program to the district 

coordinator, administrator, SLT and MTSS-B Coaches as well as the roles and functions 

of each individual. During the spring of 2016 CSBS will work with schools to identify a 

Targeted Team (TT) to support Tier II implementation. Often, the TT is a subset of the 

SLT. 

Behavioral monitoring data system: During the spring of 2015 CSBS trainers will 

support the SLT in putting in place the data infrastructure that relies on a web-based 

behavior monitoring data system called the School Wide Information System (SWIS). 

Minimally three SLT members and MTSS-B Coaches will be trained to use the SWIS 

data system to drive decision-making about issues such as which teachers may need 

additional supports with classroom management and which locations in the school at 

what time may need more school staff to monitor student behavior. Appropriate staff will 

be identified and trained to enter data and generate reports. The data infrastructure will be 

augmented in the spring 2016 to include the data system for Check In Check Out, the 

primary Tier II intervention. 

Initial and Booster Summer Trainings: During the summer of 2015, CSBS will host training 

within each district for the SLT, MTSS-B Coaches and administrators in all Program Schools. 
7

The goal of the initial CSBS Tier I training during the summer of 2015 is to introduce the key

school-level implementers (MTSS-B Coaches, SLT and administrators) to the core features of 

MTSS-B Tier I6as well as the tools to track implementation fidelity and engage in data-based 

problem solving. During the summer training each school team drafts a school-level 

implementation plan. Each school team should leave the summer training prepared to further 

develop and implement the school-level implementation plan. During the summer of 2016, the 

key Tier I implementers engage in a booster training in which they review core components, 

reflect on the previous year and develop a year 2 school-level implementation plan. During this 

booster training, new school staff are also introduced to the core components of MTSS-B Tier I. 

Initial summer training for Tier II also takes place in the summer of 2016 and includes training 

for the MTSS-B Coaches, Targeted Team and administrator in the core features of Check In 

Check Out7and how to use CICO-SWIS data to monitor student progress. The goal of this initial 

training is for schools to develop and implement their CICO implementation plan. 

Ongoing Training and Support: During the 2015-2016 and 2016-2017 school years, CSBS 

will provide ongoing support to the MTSS-B Coaches, SLT and administrators and some direct 

trainings to all school-staff as schools implement Tier I activities. CSBS trainers will visit all 

Program Schools four times over the course of the school year. During these visits, they will 

work with the SLT, administrator and MTSS-B Coaches to introduce Tier I core components to 

staff and to train staff in their version of the Good Behavior Game (referred to as the Positive 

Behavior Game (PBG)), which helps teachers to teach and acknowledge classroom rules aligned 

with the school-wide expectations. Additionally, they will gather and use data regarding school’s 

implementation fidelity to help the SLT to create and modify school’s implementation plans.8

During site visits, monthly webinars and email/phone exchanges CSBS will support Coaches and 

the SLT to train teachers in the 8 CSBS classroom components. 

 

6 Core components of MTSS-B Tier I include teaching and reinforcing specific school-wide behavioral expectations, 

a classroom management system to support behavioral expectations, and the use of fidelity data to monitor and 

improve implementation. 

7 Core components of MTSS-B Tier II (CICO) include use of data to identify and progress monitor students needing 

additional supports, additional instruction and time for students to develop behavioral skills, additional structures 

with increased opportunity for feedback from staff, and sharing information about student behavior between school 

and families. 

8 CSBS will use fidelity measures to monitor schools’ progress, identify areas in need of improvement, and make 

adjustments in training when necessary. CSBS plans to use the Effective Behavior Support (EBS) survey and the 

Tiered Fidelity Inventory (TFI) implementation tool for this purpose. CSBS will use the Behavior Education 

Program Fidelity of Implementation (BEP-FIM) to assess fidelity of implementation for the Tier II intervention.
8

CSBS’ provision of ongoing support for the Tier II intervention during the 2016-2017 school 

year will follow the same model as their provision of ongoing support for Tier I. However, CSBS 

will conduct an extra site visit during year 2 of the study so they have sufficient time to provide 

ongoing support for the implementation of Tier I and Tier II. 

MTSS-B Program Implementation 

The key components of the Tier I and Tier II MTSS-B program evaluated in this study are 

described in the paragraphs below. 

Tier I Practices: The school-level implementation of Tier I practices will occur in all Program 

Schools during both the 2015-2016 and 2016-2017 school years. These practices include schoolwide activities as well as specific classroom components that are overseen by the SLT and 

MTSS-B Coaches. 

Leadership provided by SLT and MTSS-B Coaches: The SLT and MTSS-B Coaches 

are the lead implementers of the program and are responsible for introducing the program 

model and implementation plan to other staff members with support from CSBS trainers. 

They meet regularly to monitor implementation of the school action plan and, based on 

data collected on student behavior and implementation fidelity, provide support to other 

school staff to improve implementation. 

Ongoing Support for Teachers by the SLT and MTSS-B Coaches: The SLT and 

MTSS-B Coaches provide ongoing training and coaching to teachers. These supports 

include collection of data on teacher performance as well as the provision of targeted 

feedback to teachers and opportunities to practice the Positive Behavior Game. 

School-wide implementation of program: A key component of the Tier I practices of 

MTSS-B is that the program is implemented school-wide and not just by particular 

administrators, teachers, or staff members. All school staff will be responsible for 

implementing the school’s 3-5 behavioral expectations, responding appropriately to 

positive and problem student behavior, and using student behavior data to develop 

solutions to specific problems. 

CSBS’ Eight Classroom Components

 Arrange orderly physical environment

 Define, teach, and acknowledge rules aligned with school-wide expectations by modeling 

desired behaviors and using a gaming strategy;

 Explain and teach routines; 

 Provide specific and contingent praise for appropriate behavior; 

 Use class-wide group contingencies; 

 Provide error correction through prompt, re-teach, and provision of choices;

 Employ active supervision – move, scan, interact; 

 Provide multiple opportunities to respond.
9

Classroom components: Within the classroom, teachers establish and maintain a 

classroom management system, which includes 8 specific components (see text box 

above).

Tier II Practices: Tier II will begin in all Program Schools during the 2016-2017 school year. 

During this period, the SLT and MTSS-B Coaches will review behavioral monitoring data to 

identify students who are not responding to Tier I supports. They will also solicit referrals from 

school staff and families. These students will receive the Check In Check Out (CICO) 

intervention, which is the Tier II intervention in this study. 

CICO is a small group intervention that reinforces and extends the universal expectations by

systematically providing a higher frequency of scheduled prompts, pre-correction, and 

acknowledgement to participating students. The program consists of students daily “checking in” 

with an adult at the start of school day to retrieve a goal sheet and receive encouragement and 

help in setting specific behavioral goals (including skills to practice) for the day. Throughout the 

day, teachers (or other school staff) provide feedback to the student and document student 

behavior on the goal sheet. Students “check out” at the end of the day with an adult, who 

summarizes the feedback and recognizes accomplishments. The Targeted Team (TT), usually a 

subset of the SLT, monitors students’ progress on CICO and establishes systems of 

communication with families. 

STUDY DESCRIPTION

Study Design

School level random assignment is appropriate for this study because, by definition, MTSS-B is 

a whole school intervention delivered at the school level. The analysis of MTSS-B impacts will 

rely on a random assignment design in which schools within each participating district 

(approximately 10) are randomly assigned to the following two groups: 

(1) Program Schools: Training, infrastructure and support to implement Tier I components in 

year 1 (SY2015-2016) and Tier I and Tier II components in year 2 (SY2016-2017) 

(2) Business As Usual Schools: Continue with any existing practices related to student 

behavior, and will not receive any additional training and support in MTSS-B from the 

evaluation.

First year impacts of Tier I will be calculated by comparing first year outcomes in all 60 Program 

Schools implementing Tier I to first year outcomes in the 30 BAU schools. The second year 

impacts of Tier I plus II will be calculated by comparing second year outcomes in the Program 

Schools to second year outcomes in the BAU schools. A baseline teacher assessment of student 

behavior (in fall 2015) allows for the identification of a high risk subgroup and the estimation the 

net impact for high risk students of Tier I (year 1) and Tier I plus II services (year 2) compared to 

BAU schools. 
10

The study team will recruit approximately 90 elementary schools across approximately ten 

districts that are willing and able to comply with the requirements of MTSS-B training providers 

and the research and data collection procedures involved in the evaluation.

The impact study will examine several outcomes of MTSS-B—teacher practice, school climate, 

student behavior and academic achievement-- following the theory of action presented above. 

The study will also consider the difference in behavior support practices in program and BAU 

schools (service contrast). Finally, the evaluation will describe the CSBS’ training and support 

activities as well as the implementation of the MTSS-B in programs in schools.

Data Collection Strategy & Analytic Approach

Exhibit B-2 provides an overview of data collection activities. Data sources that present a burden 

to students and/or staff are indicated with an asterisk. A longer description of and justification for 

the data sources that present a burden is provided in the paragraphs below the table. 
11

Exhibit B-2. Overview of MTSS-B Data Sources 

Timeline

# of 

Respondents 

per School 

# of 

Schools Spring 

2015

Summer 

2015 Fall 2015 Spring 2016

Summer 

2016 Fall 2015 Spring 2016

Observation of CSBS training and support 

activities in Program Schools 0 60 X X X X X X X

Site visits*

 School tour to observe school rules 0 90 X X X

 Document reviews 0 90 X X X

MTSS-B Coach interview˅

• 20 minutes per wave in Program 

Schools

.33 60 X X X

Administrator interview^ 

• 45 minutes per wave in BAU schools

• 60 minutes per wave in Program 

Schools

1 90 X X X

Team leader interview^

• 30 minutes per wave in BAU schools

• 45 minutes per wave in Program 

Schools

1 90 X X X

 Student interview (randomly selected)

• 2 minutes per wave 

15 90 X X X

 Staff interview (randomly selected)

• 5 minutes per wave

15 90 X X X

Review of web-based forms (Coach logs, 

meeting minute forms and TIPS school 0 60 X X X X X X
12

forms)

Review of sample of CSBS records (CSBS 

trainer logs, attendance records, webinars, 

fidelity data)

0 60 X X X X X X X

Review of MTSS-B behavior monitoring and 

fidelity data (SWIS data, SWIS-CICO data & 

PBIS Apps data)

0 60 X X X X X X X

School staff survey (All grades k-5 teachers 

and other relevant staff) *

35 minutes for non-teaching staff

40 minutes for teachers

30 90 X X

Student survey (All students in grades 3-5)*

50 minutes to administer survey per wave 240 90 X X

Classroom observations with the ASSIST and 

possibly CLASS [augmented in Program 

Schools to included assessment of 

implementation fidelity] 0 90 X

Teacher ratings of student behavior (All 

teachers in grades 1-5; all consenting 

students in class)*

10 minutes per student per wave

20 90 X X X

District records data collection (10 districts) 0 90 X X X

Note: Activities that present a time burden to students and staff are indicated with an asterisk(*).

^ The structured interviews with the team leader and administrator listed under site visits are longer in Program Schools because the interview protocol 

includes questions related to these staff members’ perception of the quality and utility of CSBS training and support. 

˅

The MTSS-B Coach role is designated as 0.2 FTE for each program school. On average, however, districts will need 2 different MTSS-B coaches, though one 

of them may be only part-time. This increases the number of respondents for data collection purposes, but does not change the actual staffing. 
13

Description of Data Sources that Present a Burden to Staff and Students 

Site visits: We will conduct site visits in Program and BAU schools three times over the study 

period (Fall 2015, Spring 2016 and Spring 2017). The site visit protocols involve document 

review, observation and interview. The times required for the interviews conducted as part of the 

site visit are indicated in Exhibits B-2 and B-3. The data collected from these visits will serve the 

following purposes: (1) Assess the difference in behavior support practices in program and BAU 

schools (service contrast) (2) Assess implementation fidelity to the core feature of the MTSS-B 

Tier I and Tier II programs in Program Schools (3) Assess key implementers’ (MTSS-B 

Coaches, SLT team leader and administrator) perception of the quality and utility of CSBS 

training and support in Program Schools. To develop the protocols for these site visits, the study

team has pulled items from School wide Evaluation Tool (SET), which has been found be a valid 

and reliable measure of MTSS-B Tier I implementation fidelity;

9

it has also been used in prior 

randomized control trials of MTSS-B as a measure of service contrast between Program and 

BAU schools and found to accurately discriminate between schools implementing the core 

features of MTSS-B and those not.10 The I-SSET assesses the fidelity of Tier II and Tier III 

implementation,11 and it has been combined with SET in prior studies to provide a complete 

picture of the three tiers of MTSS-B implementation.12

Understanding staff’s perception of the quality and utility of CSBS training and support in the 

Program Schools is critical to fully understanding implementation fidelity and to identifying 

challenges associated with the implementation of MTSS-B in a large number of elementary

school and districts. The SET/ISSET protocols do not ask school staff to reflect upon the quality

and utility of the MTSS-B training and support they received from CSBS. As a result, the study

team will be augmenting the site visit interview protocols in Program Schools with questions for 

the MTSS-B Coaches, SLT team leader and administrator about their perceptions of the quality

and utility of CSBS training and support. The interview protocols for the site visits can be found 

in Appendices A-E.

Staff Survey: School staff (teachers, administrators and other staff) in all study schools will be 

asked to participate in a web-based survey about school climate, staff practices related to 

behavior, and training for behavior support practices and programs in the spring of 2016 and 

spring of 2017. The surveys also include a number of questions related to individual background 

characteristics and experiences. The survey for non-teaching staff will take approximately 35 

minutes to complete and the survey for teaching staff will take approximately 40 minutes to 

complete. 

To assess staff practices related to behavior, we are drawing items from the Effective Behavior 

Support Survey (EBS).13 These items assess the extent to which schools are implementing core 

 

9

Horner et al. (2004).

10 Bradshaw, Waasdorp, and Leaf (2012).

11 Anderson et al. (2008).

12 Debnam, Pas and Bradshaw (2012). 

13 Sugai, Todd, and Horner (2000). 
14

features of MTSS-B in classroom and non-classroom settings. We have also drawn upon 

questions from the multi-site evaluation from IES’ Social and Character Development study to 

identify differences between BAU and Program Schools in the extent to which they have 

received training for and/or are implementing school-wide and classroom based programs similar 

to the MTSS-B approach being implemented in Program schools. Additionally, to assess 

implementation fidelity the study team has designed questions that ask about strategies specific 

to CSBS MTSS-B approach; the questions are general enough to apply to Program and BAU 

schools but specific enough to the CSBS model that they will inform our assessment of 

implementation fidelity. We have also designed questions to assess staff’s exposure to training 

and coaching related to behavior support practices. Staff’s answers to these questions will inform 

our understanding of service contrast; fully understanding service contrast is critical to the 

interpretation of the impact estimates. 

Improved school climate is theorized to be proximal outcome of the intervention and is therefore 

critical to measure to assess staff’s perception of school climate we are drawing from two wellvalidated measures—Organizational Health Inventory (OHI) and the Malasch Burnout 

Inventory.

14 Prior studies of MTSS-B have found impacts on the OHI15. The staff survey can be 

found in Appendix F. 

Student Survey (Grades 3-5): Students in grades 3-5 in Program and BAU schools will be 

asked to complete a survey in the Spring of 2015 and Spring of 2016. The survey will take 

approximately 50 minutes for all fielding. This survey will provide critical information about 

students’ perception of their own behavior and classroom climate, which are both predicted by

the theory of action to be positively impacted by MTSS-B. Many of the scales included were 

selected because they have been widely used in previous research on school-wide interventions; 

many have been found to have adequate psychometric properties.16 We have also included a 

limited number of questions to assess students’ awareness of core features of MTSS-B 

implementation such as the establishment of school rules, a positive reinforcement system and a 

gaming strategy to reinforce the school wide expectations. These questions will be asked in the 

survey administered in BAU and Program schools so they can inform the assessment of 

implementation and service contract. The student survey can be found in Appendix G.

Teacher Ratings of Student Behavior: Teachers of students in grades 1-5 in Program and BAU 

schools will be asked to rate their students’ behavior in fall of 2015,17 the spring of 2016 and the 

spring of 2017. Most districts will require that we obtain active parental consent from parents for 

these ratings and so teachers will only be asked to rate the behavior of students in their classes 

with consent. The rating form is expected to take approximately ten minutes per student to 

complete. 

 

14 Hoy and Tartar (1997); Maslach and Jackson (1981).

15 Bradshaw et al., 2008. 

16 Gresham et al. (2011); Haynes et al. (2001); Huesmann and Guerra (1997); Roberts, Hom, and Battistich (1995); 

Social and Character Development Research Consortium (2010).

17 Fielding this survey at baseline, will allow the study team to identify a high-risk sub-group at baseline. 

Identification of this subgroup is necessary for answering one of the study’s research questions regarding the effects 

of MTSS-B on at-risk students. 
15

The teacher ratings form will be used to assess the impact of the program on student 

behavior. The study team has adopted the Teacher Observation of Student Adaptation-Checklist 

(TOCA-C) for this instrument. The TOCA-C has been shown to be a valid and reliable measure 

of students’ behavior (concentration problems, disruptive behavior and prosocial behavior).18 It 

has been used in prior evaluations of MTSS-B where it has been shown to be sensitive to the 

intervention.19 The study team is augmenting this measure with questions that will provide 

information regarding students’ referral to and receipt of targeted supports for behavior and 

special education services. Asking this information on the teacher ratings forms is important 

because it cannot always be reliably ascertained from administrative datasets in Program and 

BAU schools. Finally, the rating form includes one scale that assesses teachers’ perception of 

students’ academic behaviors.20 This scale has been added because the theory of action predicts 

that MTSS-B will improve students’ academic engagement, which is not assessed on the TOCAC. The teacher ratings form can be found in Appendix H. 

District Records Data Collection: The study team will request extant data from school district 

records regarding student background characteristics, teacher background characteristics and 

experiences, student behavior incidences, student academic achievement (grades 3-5) and special 

education. Data will be collected at the end of each school year: baseline (SY 2014-2015), year 

one (SY 2015-2016) and year 2 (SY 2016-2017). Requests will be made of district research 

offices. Appendix I provides a sample of the letter that will be sent to districts requesting these 

data. 

 

18 Koth, Bradshaw and Leaf (2009). 

19 Bradshaw, Waasdorp, and Leaf (2012).

20 The study team will be adopting the “approaches to learning” scale from NCES’ ECLS-K survey to assess this 

construct. All necessary permissions will be acquired before the survey is fielded. 
16

SUPPORTING STATEMENT FOR PAPERWORK

REDUCTION ACT SUBMISSION

Part B: Description of Statistical Methods

1. Respondent Universe and Sampling Methods

The sample for this school-level random assignment study will include approximately 90 schools 

across approximately ten school districts. To arrive at this sample from the universe of schools 

districts and elementary schools within districts, study team and the Department developed a set 

of district and school eligibility guidelines, which are summarized in Exhibit B-4. 

Exhibit B-4: Screening guidelines used for recruitment

District Screening Guidelines: A district must have 10 public schools serving students in KindergartenGrade 5 or Grade 6 that meet the following criteria:

 School Size: Have at least 50 students per grade, on average.

 Family Income: 25% of students qualify for Free and Reduced 

Priced Lunch or are Title 1 Eligible [According to 2011-2012 

CCD or district data].

 Current Behavior Support Strategies: Are not currently 

implementing MTSS-B Tier I or Tier II programs.

 Interest: The district must be interested in implementing the 

MTSS-B program in nominated schools. The district must fully 

understand the random assignment process and the burden of 

data collection. 

School Screening Guidelines: Within an eligible district, a school will be considered for nomination if 

they meet the following criteria:

 School Size: Do not have a single grade serving fewer than 40 

students [according to 2011-2012 CCD or district data]. Across 

all nominated schools, there will be an average of at least 60 

students per grade.

 Family Income: Across all nominated schools within a district, 

the percentage of FRL students must be greater than 25% with 

preference given to schools that have at least 15% of students 

qualify for Free and Reduced Priced Lunch [according to 2011-

2012 CCD or district data]. 

 Current Behavior Support Practices: Are not currently 

implementing MTSS-B Tier I and Tier II programs.

 Interest: The schools must be interested in implementing the 

MTSS-B program if they are selected as a program school and 

be willing to continue with business-as-usual if they are 

selected as a BAU school. Schools must fully understand the 

random assignment process and the burden of data collection. 

Site Recruitment Process: To recruit districts and schools and to determine eligibility, the study

team first identified 480 districts across the country that met the final screening criteria (see 
17

Exhibit B-4). The evaluation team then contacted potential sites and provided them with 

information on the project and the research design. The districts were discussed their current 

behavioral systems in place, as well as the district's ability and willingness to implement research 

procedures related to random assignment and data collection. The team conducted site visits at 

all of the promising districts to discuss the study more in depth, provide detailed data collection 

plans, discuss the roles, responsibilities and expectations for all participants, and review which 

schools might be eligible for participation. Eligible and interested districts then negotiated and 

signed a tentative agreement to participate with the project director for the research team. The 

pool of eligible districts that signed this agreement was then sent to the Department of Education 

for final selection. Final selection of districts and schools took into account the advantage for the 

policy relevance of results of having sites that include a variety of regions, urban/suburban/rural 

settings, and race/ethnicity of the student body. The research team worked closely with the 

project officer in the Department of Education to make final choices. Selected districts and 

schools then received formal notice of inclusion in the study. As of April 2015, five districts with 

47 schools have been accepted for the study. 

2. Procedures for Data Collection

MDRC will work with partner organization Decision Information Resources (DIR) to conduct 

the data collection activities. The paragraphs below describe the data collection strategy for the 

data sources that impose a burden on students or school staff: site visits to Program and BAU 

schools; staff surveys; student surveys; and teacher ratings of student behavior. 

The following sections present our approach for each data collection effort, including 

information on sample and timing, instruments, and training. More information about data 

sources can be found in the Description of Data Sources beginning on page 14.

Site Visits: The site visits include document review, observations, and interviews with key

implementers (MTSS-B Coaches, behavior team leaders, and administrators). The data collected 

from site visits serves three purposes: to assess the difference in behavior support practices 

between program and BAU schools; to assess implementation fidelity of the core features of the 

MTSS-B Tier I and Tier II programs in program school; and to assess the key implementers’ 

perception of the quality and utility of CSBS training and support for the MTSS-B Tier I and 

Tier II program. As described in the introduction to Supporting Statement A, the site visit 

protocol draws from the Schoolwide Evaluation Tool (SET) and the Individual Student Systems 

Evaluation Tool (ISSET), which assess of MTSS-B implementation fidelity and service 

contrast.21 These protocols have been augmented with structured interview protocols in Program 

Schools to assess the quality and utility of CSBS training and support. 

Site visits to all schools will occur three times over the course of the study, during the Fall and 

Spring of the first year (2015-2016), and during Spring of the second year (2016-2017). During 

the visits to Program Schools, the study team will conduct interviews with the MTSS-B Coaches 

(20 minutes), an administrator (60 minutes), and the behavior team leader (45 minutes). 

Additionally, there will be interviews with 15 randomly selected staff members (5 minutes each) 

 

21 Anderson et al., 2008; Horner et al., 2004; Debman, Pas and Bradshaw, 2012. 
18

and 15 randomly selected students (2 minutes each). Interviews in BAU schools will include a 45 

minute administrator interview, 30 minute interview with the behavior team leader, 5 minute 

interviews with 15 randomly selected staff and 2 minute interviews with 15 randomly selected 

students. Interviews will be conducted by DIR and MDRC staff with training by Catherine 

Bradshaw’s team at the University of Virginia and Johns Hopkins University22 and the MDRC 

staff. 

The data collection team understands the challenges involved with conducting site visits at 

schools during the school day. Therefore, our approach will be flexible to allow a range of 

options that will accommodate known or unknown challenges. The team will work closely with 

school liaisons to arrange access to each school, identify the availability of space, determine best 

times for the individual interviews, and identify any “restricted” times or days (for example, field 

trips, school closings, student testing, and so forth). 

Site visitors will be required to have experience conducting site visits in school settings and have 

at least a bachelor’s degree. Preference will be given to individuals with an education, child 

development, or similar educational background. 

All site visitors will attend a 3-4 day training in which they will go over details about the study

and site visit procedures. A portion of the training will include visits to non-study schools to 

assess the site visitors' reliability with the protocols 

The training agenda will include:

 An overview of the overall study

 General information about conducting site visits

 Proper administration of the measurement tool 

 Quality assurance

 Scoring each tool

 Practice sessions immediately following the training 

 Administrative procedures, responsibilities, and compensation

 Review of the certification process and criteria to meet established standard

Attendees will also review the instrument, professional conduct, policies and guidelines for 

reporting child maltreatment, and administrative policies and guidelines. A manual including 

information from the training will be provided to attendees. Site visitors who meet the 

certification requirements will be selected to conduct the site visits. 

The specific site visit activities will be carried out as follows:

 The team will work with the school liaison to schedule the visit, set up the required 

interviews, and notify them of the required documents/procedures that will be reviewed. 

 

22 Bradshaw and her colleagues have used the site visit protocols being used in this study in multiple other studies of 

MTSS-B; for example, see Debman, Pas, and Bradshaw (2012). 
19

The agreed upon schedule will be provided to the school electronically and a reminder 

will be sent 1 week and 1 day prior to the scheduled visit.

 On the day of the site visit, the site visitor will arrive 30-60 minutes prior to scheduled 

start time, and upon arriving, they will check-in with the school liaison and review the 

schedule for the day and ensure that all necessary documents are available for review. In 

addition, the site visitor will confirm that the planned interviews will be carried out with 

the specified people.

 After the schedule is set, the site visitor will begin carrying out the various required steps 

including selection of a random sample of students and staff for brief interview (students 

interview is 1-2 minute per student and the staff interview is 3-5 minutes long). The 

procedures of this random sampling will be worked out with the district and the school 

liaison ahead of time so that every effort is made to not disrupt instruction.

 Upon completion of all pieces, the site visitor will alert the school liaison that the visit is 

finished, or if there were missing elements, that will be reviewed at that time. 

 Site visitors will be required to conduct the scoring of both protocols on the same day the 

visit was conducted.

Scored protocols will be sent back to DIR and MDRC, reviewed for quality control, and then 

entered into a database.

Staff Survey: The staff survey is designed to collect data on the school climate, staff practices 

related to behavior, and training for behavior support practices. 

This survey will be provided to teachers and non-teaching staff at all study schools in a webbased format in the Spring of 2016 and the Spring of 2017. DIR is managing the fielding of this 

survey. The survey will be administered to all teachers in grades 1-5 as well as all non-teaching 

staff and administrators (n=30/per school). School staff will be given the option to complete all 

surveys by using the web version or paper version or a combination, which will maximize the 

attainment of the desired 90-percent (or higher) response rate. School staff emails will be 

obtained through the school data collection liaison. For staff that request to complete the 

questionnaires using the web version, the survey will launch with a web-only period, which will 

last for three full weeks. During the web-only period, reminder emails will be sent to nonresponders at intervals determined by web activity but no more than once per week. DIR will 

format the questionnaire for self-administration via the web or hardcopy. Prior to completing the 

survey, teachers and other school staff will be asked to provide consent, and they may also 

withdraw their consent at any time. 

Training for the staff survey administration will be nominal. From experience, most school staff 

will opt to complete the report using the web-based version. For those opting to use hardcopies, 

DIR staff will disseminate hardcopies of the instrument to school staff by mailing them to school 

liaisons. School staff will be asked to put all completed hardcopies in a DIR-provided envelope, 

seal the envelope, and return them to the school liaison. They will be shipped to the Houston 

office and logged as completed. Field personnel will be trained in the areas of confidentiality, 

data security (including collecting, securing, and transporting data), and tracking or logging 

receipt of complete and incomplete instruments.
20

DIR will send hardcopy questionnaires (through school liaisons) to staff that request to complete 

the questionnaires using the paper version hardcopies. At the start of week 4, hardcopy 

questionnaires will be sent to school liaisons who will deliver them to the non-responders who 

requested to complete using the web version. A reminder postcard will be designed (to be 

delivered in person or via regular mail) and sent to all non-responders, and additional email 

reminders will be sent to teachers with email addresses during the last 2 weeks of the fielding 

period. DIR’s previous experience with attempting to reach teachers and school staff by 

telephone has not been very successful. Typically, messages are left without any certainty of 

their delivery. Thus, DIR will ask the school liaisons to hand deliver reminder postcards and give 

the teachers and school staff a verbal reminder at specific times during the fielding period. Each 

of these steps will help to ensure that the desired response rate is reached by the end of the 

fielding period. 

In summary, DIR will complete the following steps in fielding the staff survey:

 Two weeks prior to launching the web survey, DIR will send staff a personalized letter 

(via email and regular mail) with a description of the importance of the questionnaire, 

stressing the confidentiality of their responses, explaining how the results will be used, 

describing the honoraria, and indicating when they will receive their individualized link 

to complete the web version. DIR will also thank them in advance for their participation. 

 On the day of the survey launch, DIR will send staff an email announcing the launch of 

the survey. This email will contain a unique identification number and a link to the 

survey. For staff without an email address, DIR will send the same information via 

regular mail.

 At the 3-day, 10-day, and 14-day mark, DIR will send reminder emails and/or postcards 

to thank staff for their participation or encourage them to respond if they have not already 

done so. At the 14-day mark, the option to complete a hardcopy survey will be included 

in the reminder. Upon request, hardcopy surveys will be sent directly to the teacher via 

FedEx overnight delivery, along with a pre-paid return envelope. 

DIR will ask school liaisons to send a reminder to staff to express district/school support for the 

study and encourage participation. DIR will provide the specific text for them to send, in an 

effort to minimize the burden on the liaisons.

Student Survey (Grades 3-5): This survey will provide critical information about students’ 

perception of their own behavior and classroom climate, which are both predicted by the theory

of action to be positively impacted by MTSS-B. 

DIR will field the survey to all students in grades 3-5 in approximately 90 participating 

elementary schools across 10 districts (9 schools per district) during the spring of 2016 and 

2017.The survey will take approximately 50 minutes for all fielding (the actual survey will only

take 40-45 minutes to complete). Data collection in each school will occur over a 2-3 day period. 

The study team anticipates needing to collect active parental consent for this activity since many 

of the likely districts will require it. See question A-9 and B-3 for more detail on our proposed 

strategy for maximizing response rates on the consent form. 
21

DIR will recruit and hire a team of 8 – 10 proctors per district. Each team will be led by an 

assessor with extensive classroom experience. Candidates will be required to have experience 

working with children between the ages of 8-11 and have some college education. Preference 

will be given to individuals with a child development or similar educational background. 

All proctors will attend a training in which they will go over details about the study and data 

collection procedures. The training agenda will include:

 An overview of the evaluation and student survey

 General assessment techniques for working with children ages 8—11 

 Proper administration of the measurement tool 

 Quality assurance

 Live practice sessions immediately following the training 

 Administrative procedures, responsibilities, and compensation

 Review of the certification process and criteria to meet established standard

Attendees will also review the instrument, proctors’ responsibilities, professional conduct, 

policies and guidelines for reporting child maltreatment, and administrative policies and 

guidelines. A manual including information from the training will be provided to all proctors. 

The following approaches will be used to collect the student survey data:

 DIR will work closely with school liaisons to arrange access to each school, identify

the availability of space, determine best times for assessments, and identify any

“restricted” times or days (for example, field trips, school closings, and so forth). 

 DIR staff will call approximately 3 business days before the scheduled visit to 

confirm the date, the school contact (if different from the liaison), will verify the 

location for the administration, and gather any additional necessary information.

 DIR will assign 2 proctors to all third, fourth and fifth grade classrooms. For each day

of data collection, DIR staff will arrive 1 hour before the actual administration is set 

to begin. One proctor will distribute the materials and verify all materials match the 

classroom roster and are distributed accurately. The second proctor will confirm that 

all students have their assigned materials, distribute pencils, walk the classroom 

during the survey administration, and collect all materials at the end of the survey

administration assigned. 

 Survey instructions and all questions will be read aloud to students. 

 Completed surveys will be logged and returned to the DIR offices for scoring.

Teacher Ratings of Student Behavior: The teacher ratings will be used to assess the impact of 

the program on student behavior and the students’ referral to and receipt of targeted supports for 

behavior and special education services. 

Teachers of students in grades 1-5 in all study schools (n=90) will be asked to rate their students’ 

behavior in the fall of 2015, the spring 2016 and the spring of 2017. DIR field staff will contact 

all teachers of students in grades 1-5 in 90 participating schools across 10 districts. The rating 
22

form is expected to take approximately ten minutes per student to complete and each teacher will 

rate all of their students. The study team anticipates needing to collect active parental consent for 

this activity since many of the likely districts will require it. See questions A-9 and B-3 for more 

detail on our proposed strategy for maximizing response rates on the consent form. 

The rating checklist will be administered electronically with DIR using paper-based surveys to 

follow-up with non-responders. Fielding procedures for this instrument are consistent with the 

web-based staff survey described above. In summary, the teacher rating checklist data collection 

activities will be carried out as follows:

Two weeks prior to launching the web link, DIR will send teachers a personalized letter (via 

email and regular mail) with a description of the importance of the rating checklists, 

stressing the confidentiality of their responses, explaining how the results will be used, 

and indicating when they will receive their individualized link to complete the web 

version. DIR will also thank them in advance for their participation. 

On the day of the web launch, DIR will send teachers an email announcing the launch of the 

web version. This email will contain a unique identification number and a link to 

complete the ratings checklist. For staff without an email address, DIR will send the same 

information via regular mail.

At the 3-day, 10-day, and 14-day mark, DIR will send reminder emails and/or postcards to 

thank teachers for their participation or encourage them to respond if they have not 

already done so. At the 14-day mark, the option to complete a hard-copy version of the 

ratings checklists will be included in the reminder. Upon request, hardcopy versions will 

be sent directly to the teacher via FedEx overnight delivery, along with a pre-paid return 

envelope. 

DIR will ask school liaisons to send a reminder to teachers to express district/school support 

for the study and encourage participation. DIR will provide the specific text for them to 

send, in an effort to minimize the burden on the liaisons.

3. Procedures to Maximize Response Rates

The goal will be to achieve a 90 percent response rate or higher in each site for each data 

collection effort. Procedures for obtaining the maximum degree of cooperation include:

 Sending respondents cover materials that address the importance and value of the study. 

 Collecting all parental consent forms for student data collection activities that require 

parental consent at the start of the school year 

 Working closely with school principals and/or liaisons to develop a schedule that will 

facilitate administration to all students. 

 Training proctors to maintain direct contact with respondents and provide assurances to 

reluctant responders. 
23

 Providing proctors packets of student surveys with blank labels in the event that new 

students are added to the class.

 Allowing school staff and teachers multiple formats (web and paper-pencil) for 

completing the instruments.

 Offering appropriate payments to school staff and teachers. 

 For several districts, active consent will be required, thus maximizing response rates for 

obtaining parent consent will be critical for the success of the student survey, student 

assessments, and teacher ratings. Maximizing the response rate will require a multilingual 

and multistep approach. All materials will be prepared in both English and Spanish and 

distributed in packets by classroom to teachers with a request to distribute the forms to all 

children in their classrooms. Forms will be printed on 2-ply carbonless paper so that 

parents could keep a signed copy. The consent packet will include a written explanation 

of the study and a request that the parent or guardian allow their child’s participation in 

the study by signing the forms. As an incentive, teachers or schools data collection 

liaisons (whomever is responsible for distributing the consent forms) will be offered a 

$25 gift card if they achieve an 90% response rate or better return of parental consent 

forms.

4. Testing of Data Collection Procedures

Most of the questions proposed for this survey are either identical to questions used in prior 

MDRC or AIR evaluations23 or are similar to questions used in previous national surveys or 

major evaluations.24 Consequently, many of the items have been thoroughly tested on larger 

samples.

The MTSS-B surveys and interview protocols have already undergone a number of revisions, 

following critiques by internal staff, by project consultants, and by staff at IES. MDRC will also 

work closely with DIR and AIR senior staff to conduct formal pre-tests of these surveys and 

interview protocols, using fewer than nine appropriate respondents. 

5. Individuals Consulted on Statistical and Methodological Aspects of Data Collection

This project is being conducted under contract to the Department of Education by MDRC, AIR, 

Harvard University, and DIR. The data collection strategy and instruments were developed by 

Fred Doolittle, Barbara Condliffe of MDRC; Anja Kurki, Louis Danielson, Muna Shami and 

Gail Chan of AIR, Stephanie Jones of Harvard Graduate School of Education, and Catherine 

Bradshaw of the University of Virginia. Pei Zhu of MDRC is leading the statistical design of the 

study and the estimation of the impacts of the MTSS-B training and support. 

The organizations responsible for data collection activities are as follows:

 

23 Some examples of such evaluations include MDRC's current Evaluation of Response to Intervention for Elementary School 

Reading (report due in 2015; http://ies.ed.gov/ncee/projects/evaluation/disabilities_rti.asp) and AIR's Good Behavior Game 

Professional Development Study (http://www.air.org/project/good-behavior-game-professional-development-study).

24 For example, Anderson et al. (2008); Bradshaw et al. (2008a); Bradshaw et al. (2008b); Bradshaw, Mitchell, and Leaf (2010); 

Bradshaw, Waasdorp, and Leaf (2012); Jones, Brown, Hoglund, and Aber (2010); Jones, Brown, and Aber (2011a); Jones, 

Brown, and Aber (2011b); Koth, Bradshaw, and Leaf (2009).
24

Organization Primary Contact Phone Number 

MDRC Fred Doolittle 212-340-8638

American Institutes for Research Anja Kurki 202-403-6022

Decisions Information Resources 

(DIR) 

Russell Jackson 713-650-1425

Individuals outside these organizations who were consulted on the design of data collection 

strategy are as follows:

Name Organization Phone number

David Cordray Vanderbilt University, Department 

of Psychology and Human 

Development

615-343-2699

Brian Flay Oregon State University, School of 

Social and Behavioral Health 

Sciences

541-737-3837

Mary Louise Hemmeter Vanderbilt University, Department 

of Special Education 615-322-5648

Sara E. Rimm-Kaufman, Ph.D. University of Virginia, Curry 

School of Education and Center for 

the Advanced Study of Teaching 

and Learning

434-982-2863

Lori Newcomer, Ph.D. University of Missouri, Missouri 

Prevention Center / Department of 

Educational School and Counseling 

Psychology

573-882-7731 

314-605-0235

Catherine Bradshaw, Ph.D. University of Virginia

Johns Hopkins University 434-924-8121

Stephanie Jones, Ph.D. Harvard University 617 496-2223

The contracting offer for this study is Dr. Lauren Angelo of the National Center for Education 

Evaluation (IES). Her phone number is: 202-219-2180.
25

REFERENCES USED IN PART B

Anderson, C. M., Teri Lewis-Palmer, Anne W. Todd, Robert H. Horner., George Sugai, N.K. 

Samson. 2008. "Individual student systems evaluation tool, version 2.6." Educational and 

Community Supports, University of Oregon.

Bradshaw, Catherine P., Christine W. Koth, Katherine B. Bevans, Nicholas Ialongo, and Philip J. 

Leaf. 2008a. “The Impact of School-Wide Positive Behavioral Interventions and 

Supports (PBIS) on the Organizational Health of Elementary Schools.” School 

Psychology Quarterly, 23, 4: 463-473.

Bradshaw, Catherine P., Mary M. Mitchell, and Philip J. Leaf. 2010. “Examining the effects of 

School-Wide Positive Behavioral Interventions and Supports on student outcomes: 

Results from a randomized controlled effectiveness trial in elementary schools.” Journal 

of Positive Behavior Interventions, 12, 133-148.

Bradshaw, Catherine P., Wendy M. Reinke, Louis D. Brown, Katherine B. Bevans, and Philip J. 

Leaf. 2008b. “Implementation of school-wide Positive Behavioral Interventions and 

Supports (PBIS) in elementary schools: Observations from a randomized trial.” 

Education and Treatment of Children, 31, 1–26.

Bradshaw, Catherine P., Tracy E. Waasdorp, and Philip J. Leaf. 2012. "Effects of School-Wide 

Positive Behavioral Interventions and Supports on Child Behavior Outcomes." 

PEDIATRICS 130, 5: e1136-e1145.

Debnam, Katrina J., Pas, Elise T., and Bradshaw, Catherine P. (2012). Secondary and tertiary

support systems in schools implementing school-wide positive behavioral interventions 

and supports: A preliminary descriptive analysis. Journal of Positive Behavior 

Interventions, 14(3), 142–152.

Durlak, Joseph A., Roger P. Weissberg, Allison B. Dymnicki, Rebecca D. Taylor, and Kriston B. 

Schellinger. 2011. "The impact of enhancing students’ social and emotional learning: A 

meta-analysis of school-based universal interventions." Child Development 82, 1: 405-

432.

Gresham, Frank M., Stephen N. Elliott, Michael J. Vance, and Clayton R. Cook. 2011. 

"Comparability of the Social Skills Rating System to the Social Skills Improvement 

System: Content and Psychometric Comparisons Across Elementary and Secondary Age 

Levels." School Psychology Quarterly 26, 1: 27-44.

Haynes, N. M., C. L. Emmons, M. Ben-Avie, and J. P. Comer. 2001. "The school development 

program student, staff, and parent school climate surveys." New Haven, CT: Yale Child 

Study Center.

Horner, Robert H. 2014. “The Role of District Leadership Teams in PBIS Implementation”. 

Presentation. Available at http://www.pbis.org/Common/Cms/files/pbisresources/1-

Implementing%20PBIS.pptx.
26

Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. 2004. The 

school-wide evaluation tool (SET): A research instrument for assessing school-wide 

positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.

Horner, Robert H., George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne W. 

Todd, and Jody Esperanza. 2009. "A randomized, wait-list controlled effectiveness trial 

assessing school-wide positive behavior support in elementary schools." Journal of 

Positive Behavior Interventions 11, 3: 133-144.

Hoy Wayne K. and Clemens John Tarter. 1997. The Road to Open and Healthy Schools: A 

Handbook for Change, Elementary Edition. Thousand Oaks, CA: Corwin Press. 

Huesmann, Rowell L., and Nancy G. Guerra. 1997. "Children's normative beliefs about 

aggression and aggressive behavior." Journal of Personality and Social Psychology 72, 2: 

408-419.

Jones, Stephanie M., Joshua L.Brown and J. Lawrence Aber. 2011a. The longitudinal impact of a 

universal school-based social-emotional and literacy intervention: An experiment in 

translational developmental research. Child Development 82,2: 533-554; 

Jones, Stephanie M., Joshua L. Brown, and J. Lawrence Aber. 2011b “Two-Year Impacts of a 

Universal School-Based Social-Emotional and Literacy Intervention: An Experiment in 

Translational Developmental Research”. Psychology Faculty Publications. Paper 131. 

(http://fordham, bepress.com/psych_facultypubs/131).

Jones, Stephanie M., Joshua L. Brown, Wendy LG Hoglund, and J. Lawrence Aber. 2010. "A 

school-randomized clinical trial of an integrated social–emotional learning and literacy

intervention: Impacts after 1 school year." Journal of Consulting and Clinical Psychology

78, 6: 829.

Koth, Christine W., Catherine P. Bradshaw, and Philip J. Leaf. 2009. "Teacher Observation of 

Classroom Adaptation—Checklist: Development and factor structure." Measurement and 

Evaluation in Counseling and Development 42, 1: 15-30.

Maslach, Christina, and Susan E. Jackson. 1981. "The measurement of experienced burnout." 

Journal of Organizational Behavior 2, 2: 99-113. 

OSEP Center on Positive Behavioral Interventions and Supports. 2007. “Is School-Wide Positive 

Behavior Support An Evidence-Based Practice? A Research Summary.” Website: 

www.pbismaryland.org.

Roberts, William, Allen Hom, and Victor Battistich. 1995. “Assessing Students’ and Teachers’ 

sense of the School as a Caring Community.” Oakland, CA: Developmental Studies 

Center.
27

Sugai, George, Todd, Anne W., and Horner, Robert H. (2000). Effective Behavior Support 

(EBS) Survey: Assessing and planning behavior supports in schools. Eugene, OR: 

University of Oregon.

Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. H. (2005). School-wide evaluation tool. 

Eugene: University of Oregon.

Social and Character Development Research Consortium. 2010. "Efficacy of schoolwide 

programs to promote social and character development and reduce problem behavior in 

elementary school children." Washington, DC: National Center for Education Research, 

Institute of Educational Sciences, U.S. Department of Education.