1

IMPACT EVALUATION OF TRAINING IN 

MULTI-TIERED SYSTEMS OF SUPPORT FOR 

BEHAVIOR (MTSS-B)

OMB CLEARANCE REQUEST:

Supporting Statements A

DATA COLLECTION

April 2015

PREPARED FOR:

Institute of Education Sciences

United States Department of Education

Contract No. ED-IES-14-C-003

PREPARED BY:

MDRC

16 East 34th Street, 19th Floor

New York, NY 10016

Fred Doolittle, Project Director

Fred.Doolittle@mdrc.org

(212) 340-8638
2

TABLE OF CONTENTS

INTRODUCTION 4

THEORY OF ACTION AND RESEARCH QUESTIONS 5

CHARACTERISTICS OF THE TREATMENT TO BE TESTED 7

STUDY DESCRIPTION 10

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 

SUBMISSION 17

A. JUSTIFICATION 17

1. Circumstances Making Collection of Information Necessary 17

2. Purposes and Use of the Data 17

3. Use of Technology to Reduce Burden 18

4. Efforts to Avoid Duplication 18

5. Methods to Minimize Burden on Small Entities 18

6. Consequences of Less Frequent Data Collection 18

7. Special Circumstances Relating to Federal Guidelines 18

8. Federal Register Comments and Persons Consulted Outside the Agency 18

9. Payment or Gifts to Respondents 19

10. Assurances of Confidentiality Provided to Respondents 21

11. Justification of Sensitive Questions 22

12. Estimates of Annualized Burden Hours and Cost 23

13. Estimate of Other Total Annual Cost Burden to Respondents and Record

Keepers 25

14. Annualized Cost to the Federal Government 25

15. Explanation for Program Changes or Adjustments 25
3

16. Time, Schedule, Publication and Analysis Plan 25

17. Approval to Not Display OMB Expiration Date 26

18. Explanation of Exceptions to the Paperwork Reduction Act 26

REFERENCES 27

APPENDICES

A. Interview Protocols: MTSS-B Coach

B. Interview Protocols: Administrators

C. Interview Protocols: Behavior Team Leader

D. Interview Protocol: Students

E. Interview Protocol: Staff

F. Staff and Teacher Survey

G. Student Survey

H. Teaching Ratings of Student Behavior

I. District Records Data Collection Request Letter

J. Excerpt from IDEA, P.L. 108-446
4

INTRODUCTION

Creating and maintaining safe and orderly school and classroom environments to support student 

learning is a top federal priority, as indicated by the U.S. Department of Education and the U.S. 

Department of Justice jointly issued guidance on school discipline. The implementation of 

Multi-Tiered Systems of Support for Behavior (MTSS-B) is seen as a promising approach to 

improve school and classroom climate, student behavior, and academic achievement, and 

potentially reduce the inappropriate identification of students for special education. MTSS-B is a 

multi-tiered, systematic framework for teaching and reinforcing positive behavior for all 

students, that includes universal, school wide supports (often called Tier I), targeted

interventions to address the problems of students not responding to the universal components 

(often called Tier II), and intensive interventions (Tier III) for students not responding to Tier II 

services. 

The widespread use of MTSS-B nationally contributes to the relevance of this evaluation. The 

U.S. Department of Education’s (ED’s) Office of Special Education Programs has supported 

technical assistance around MTSS-B since 1998. Recent data from the Office of Special 

Education Programs TA Center on Positive Behavior Interventions and Support indicate that as 

of 2014, MTSS-B is reportedly implemented in over 21,000 schools across the nation.1 

As the support for and implementation of MTSS-B have increased, researchers have been 

developing an evidence base for it and other related programs. Numerous descriptive and 

correlational studies have associated aspects of MTSS-B implementation, or MTSS-B 

implementation overall, with student outcomes. More recently, researchers have begun to 

conduct causal studies of MTSS-B, although some of these initial studies have been on a 

relatively small scale (or conducted within a single state) or have been somewhat limited in 

duration and outcomes.2,3,4 A recent meta-analysis of research on a broad range of 213 individual 

school-based, universal social and emotional learning (SEL) programs (including interventions 

than can be implemented within an MTSS-B program) found that compared to controls, SEL

participants demonstrated significantly improved social and emotional skills, attitudes, behavior, 

and academic performance. 5 Much of this research has focused on individual programs and not 

examined the impact of a large-scale implementation of both universal, school-wide programs 

and targeted programs within a MTSS-B framework. 

Therefore, this evaluation focuses on a comprehensive MTSS-B program that includes each of 

the components usually associated with MTSS-B: school-wide and classroom-level strategies 

(universal supports, or Tier I) and individual or small group strategies (targeted interventions, or 

Tier II), with appropriate infrastructure (staffing sand data system). This evaluation will not 

support intensive supports for students not responding to Tier II interventions (i.e., there will be 

no Tier III interventions). IES has contracted with MDRC, AIR, Harvard University, and 

Decision Information Resources (DIR) to conduct the Impact Evaluation of MTSS-B.

 

1 Horner (2014).

2 Bradshaw, Mitchell, and Leaf (2010).

3 Horner et al. (2009). 

4 OSEP Center on Positive Behavior Interventions and Supports (2007). 

5 Durlak et al. (2011).
5

The goal of the evaluation is to implement an MTSS-B program with fidelity and determine 

whether universal supports (Tier I) and a targeted intervention (Tier II) with appropriate 

infrastructure are effective when implemented in a large number of districts and elementary

schools. 

Using a school-level random assignment design, this evaluation will examine the impact of 

training elementary school staff in the implementation of an MTSS-B framework including staff 

and data infrastructure with universal, school-wide components, as well as classroom-based 

activities (Tier I) in the first year of implementation as compared to outcomes for schools 

undertaking typical behavior support practices (These schools are hereafter called business-asusual or BAU schools.) In addition, the study will also examine the impact of this Tier I training 

plus additional training in a targeted intervention (Tier II) in a second year of implementation. 

Schools that receive MTSS-B training and support are hereafter referred to as “Program 

Schools.” The study will also investigate the differences in behavioral support services between 

program and BAU schools. Finally, the evaluation team will assess whether the MTSS-B 

training and support activities are implemented as intended as well as the fidelity of school’s 

implementation of the intended MTSS-B Tier I and Tier II activities. 

THEORY OF ACTION & RESEARCH QUESTIONS

The implementation of MTSS-B is ultimately intended to improve student outcomes. The 

conduct of school-wide and classroom strategies (Tier I) and individual or small group strategies 

(Tier II), supported by appropriate infrastructure, are designed to improve school staff behavior 

practices as well as school and classroom climate. The improved practices and climate are, in 

turn, hypothesized to benefit student behavior and academic outcomes. The individual and small 

group strategies may affect the students who need additional support (Tier II) as well as all 

students if these strategies lessen the disruption of students needing additional support. Finally, 

improved student behavior could lead to better student academic outcomes (see Exhibit A-1). 
6

Exhibit A-1: Conceptual Model of MTSS-B and Intended Outcomes

The evaluation is anchored on this conceptual model and will focus on the following research 

questions: 

 What MTSS-B training and support activities were conducted?

 Is the MTSS-B program implemented with fidelity? What are the key challenges to the 

implementation of MTSS-B with fidelity?

 What is the difference in behavior support practices occurring in program and BAU 

schools?

 What is the impact on school staff practices, school climate and student outcomes of 

providing training in the MTSS-B framework plus universal, school-wide and classroom 

Tier I positive behavior supports and a targeted Tier II intervention? 

 What are the impacts on school staff practices, school climate, and student outcomes for 

relevant subgroups (e.g., teachers with less experience; at-risk students)? 

 Does the implementation of MTSS-B reduce any differences in student discipline 

outcomes across racial and ethnic groups? Does it reduce the identification of students for 

special education services?
7

CHARACTERISTICS OF THE TREATMENT TO BE TESTED

In the spring of 2014, MDRC and partner organizations issued a Request for Proposals to 

providers of training in MTSS-B. In November 2014, we announced the winner of this 

competition, The Center for Social Behavior Support (CSBS), which is collaboration between 

The Illinois-Midwest PBIS Network at the School Association for Special Education in DuPage, 

Illinois (SASED) and the PBIS Regional Training and Technical Assistance Center at Sheppard 

Pratt, in Maryland. The training and support activities proposed by CSBS are described below. 

Following that summary, the memo describes CSBS’ plan for how the MTSS-B Tier I and Tier 

II program will be implemented in schools. 

CSBS Training and Support Activities

CSBS will mostly follow a “train the trainer” model with some direct support to all school staff 

and on-site support to the school staff and Coaches throughout implementation. Below we 

describe the key phases of their work with schools. 

Readiness Period - Training and Support by CSBS to Develop Infrastructure: The first step 

is the development of school-level infrastructures that will support implementation of Tier I and 

Tier II program components, which includes a staffing structure and data system. Tier I 

infrastructure will be put in place in the late spring of 2015 for all Program Schools. 

Infrastructure for Tier II will be developed in the spring of 2016. 

Staffing structure: CSBS trainers will work with each school to identify a representative 

School Leadership Team (SLT) and each district to identify MTSS-B Coaches. This will 

be accomplished during a two-day site visit to each district as well as webinars, email 

exchanges and phone calls. In addition to helping schools and districts identify the SLT 

and MTSS-B Coaches, CSBS will explain the MTSS-B program to the district 

coordinator, administrator, SLT and MTSS-B Coaches as well as the roles and functions 

of each individual. During the spring of 2016 CSBS will work with schools to identify a 

Targeted Team (TT) to support Tier II implementation. Often, the TT is a subset of the 

SLT. 

Behavioral monitoring data system: During the spring of 2015 CSBS trainers will 

support the SLT in putting in place the data infrastructure that relies on a web-based 

behavior monitoring data system called the School Wide Information System (SWIS). 

Minimally three SLT members and MTSS-B Coaches will be trained to use the SWIS 

data system to drive decision-making about issues such as which teachers may need 

additional supports with classroom management and which locations in the school at 

what time may need more school staff to monitor student behavior. Appropriate staff will 

be identified and trained to enter data and generate reports. The data infrastructure will be 

augmented in the spring 2016 to include the data system for Check In Check Out, the 

primary Tier II intervention. 

Initial and Booster Summer Trainings: During the summer of 2015, CSBS will host training 

within each district for the SLT, MTSS-B Coaches and administrators in all Program Schools. 
8

The goal of the initial CSBS Tier I training during the summer of 2015 is to introduce the key

school-level implementers (MTSS-B Coaches, SLT and administrators) to the core features of 

MTSS-B Tier I6as well as the tools to track implementation fidelity and engage in data-based 

problem solving. During the summer training each school team drafts a school-level 

implementation plan. Each school team should leave the summer training prepared to further 

develop and implement the school-level implementation plan. During the summer of 2016, the 

key Tier I implementers engage in a booster training in which they review core components, 

reflect on the previous year and develop a year 2 school-level implementation plan. During this 

booster training, new school staff are also introduced to the core components of MTSS-B Tier I. 

Initial summer training for Tier II also takes place in the summer of 2016 and includes training 

for the MTSS-B Coaches, Targeted Team and administrator in the core features of Check In 

Check Out7and how to use CICO-SWIS data to monitor student progress. The goal of this initial 

training is for schools to develop and implement their CICO implementation plan. 

Ongoing Training and Support: During the 2015-2016 and 2016-2017 school years, CSBS 

will provide ongoing support to the MTSS-B Coaches, SLT and administrators and some direct 

trainings to all school-staff as schools implement Tier I activities. CSBS trainers will visit all 

Program Schools four times over the course of the school year. During these visits, they will 

work with the SLT, administrator and MTSS-B Coaches to introduce Tier I core components to 

staff and to train staff in their version of the Good Behavior Game (referred to as the Positive 

Behavior Game (PBG)), which helps teachers to teach and acknowledge classroom rules aligned 

with the school-wide expectations. Additionally, they will gather and use data regarding school’s 

implementation fidelity to help the SLT to create and modify school’s implementation plans.8

During site visits, monthly webinars and email/phone exchanges CSBS will support Coaches and 

the SLT to train teachers in the 8 CSBS classroom components. 

 

6 Core components of MTSS-B Tier I include teaching and reinforcing specific school-wide behavioral expectations, 

a classroom management system to support behavioral expectations, and the use of fidelity data to monitor and 

improve implementation. 

7 Core components of MTSS-B Tier II (CICO) include use of data to identify and progress monitor students needing 

additional supports, additional instruction and time for students to develop behavioral skills, additional structures 

with increased opportunity for feedback from staff, and sharing information about student behavior between school 

and families. 

8 CSBS will use fidelity measures to monitor schools’ progress, identify areas in need of improvement, and make 

adjustments in training when necessary. CSBS plans to use the Effective Behavior Support (EBS) survey and the 

Tiered Fidelity Inventory (TFI) implementation tool for this purpose. CSBS will use the Behavior Education 

Program Fidelity of Implementation (BEP-FIM) to assess fidelity of implementation for the Tier II intervention.
9

CSBS’ provision of ongoing support for the Tier II intervention during the 2016-2017 school 

year will follow the same model as their provision of ongoing support for Tier I. However, CSBS 

will conduct an extra site visit during year 2 of the study so they have sufficient time to provide 

ongoing support for the implementation of Tier I and Tier II. 

MTSS-B Program Implementation 

The key components of the Tier I and Tier II MTSS-B program evaluated in this study are 

described in the paragraphs below. 

Tier I Practices: The school-level implementation of Tier I practices will occur in all Program 

Schools during both the 2015-2016 and 2016-2017 school years. These practices include schoolwide activities as well as specific classroom components that are overseen by the SLT and 

MTSS-B Coaches. 

Leadership provided by SLT and MTSS-B Coaches: The SLT and MTSS-B Coaches

are the lead implementers of the program and are responsible for introducing the program 

model and implementation plan to other staff members with support from CSBS trainers. 

They meet regularly to monitor implementation of the school action plan and, based on 

data collected on student behavior and implementation fidelity, provide support to other 

school staff to improve implementation. 

Ongoing Support for Teachers by the SLT and MTSS-B Coaches: The SLT and 

MTSS-B Coaches provide ongoing training and coaching to teachers. These supports 

include collection of data on teacher performance as well as the provision of targeted 

feedback to teachers and opportunities to practice the Positive Behavior Game. 

School-wide implementation of program: A key component of the Tier I practices of 

MTSS-B is that the program is implemented school-wide and not just by particular 

administrators, teachers, or staff members. All school staff will be responsible for 

implementing the school’s 3-5 behavioral expectations, responding appropriately to 

positive and problem student behavior, and using student behavior data to develop 

solutions to specific problems. 

CSBS’ Eight Classroom Components

 Arrange orderly physical environment

 Define, teach, and acknowledge rules aligned with school-wide expectations by modeling 

desired behaviors and using a gaming strategy;

 Explain and teach routines; 

 Provide specific and contingent praise for appropriate behavior; 

 Use class-wide group contingencies; 

 Provide error correction through prompt, re-teach, and provision of choices;

 Employ active supervision – move, scan, interact; 

 Provide multiple opportunities to respond.
10

Classroom components: Within the classroom, teachers establish and maintain a 

classroom management system, which includes 8 specific components (see text box 

above).

Tier II Practices: Tier II will begin in all Program Schools during the 2016-2017 school year. 

During this period, the SLT and MTSS-B Coaches will review behavioral monitoring data to 

identify students who are not responding to Tier I supports. They will also solicit referrals from 

school staff and families. These students will receive the Check In Check Out (CICO) 

intervention, which is the Tier II intervention in this study. 

CICO is a small group intervention that reinforces and extends the universal expectations by

systematically providing a higher frequency of scheduled prompts, pre-correction, and 

acknowledgement to participating students. The program consists of students daily “checking in” 

with an adult at the start of school day to retrieve a goal sheet and receive encouragement and 

help in setting specific behavioral goals (including skills to practice) for the day. Throughout the 

day, teachers (or other school staff) provide feedback to the student and document student 

behavior on the goal sheet. Students “check out” at the end of the day with an adult, who 

summarizes the feedback and recognizes accomplishments. The Targeted Team (TT), usually a 

subset of the SLT, monitors students’ progress on CICO and establishes systems of 

communication with families. 

STUDY DESCRIPTION

Study Design

School level random assignment is appropriate for this study because, by definition, MTSS-B is 

a whole school intervention delivered at the school level. The analysis of MTSS-B impacts will 

rely on a random assignment design in which schools within each participating district

(approximately 10) are randomly assigned to the following two groups: 

(1) Program Schools: Training, infrastructure and support to implement Tier I components in 

year 1 (SY2015-2016) and Tier I and Tier II components in year 2 (SY2016-2017) 

(2) Business As Usual Schools: Continue with any existing practices related to student 

behavior, and will not receive any additional training and support in MTSS-B from the 

evaluation.

First year impacts of Tier I will be calculated by comparing first year outcomes in all 60 Program 

Schools implementing Tier I to first year outcomes in the 30 BAU schools. The second year 

impacts of Tier I plus II will be calculated by comparing second year outcomes in the Program 

Schools to second year outcomes in the BAU schools. A baseline teacher assessment of student 

behavior (in fall 2015) allows for the identification of a high risk subgroup and the estimation the 

net impact for high risk students of Tier I (year 1) and Tier I plus II services (year 2) compared to 

BAU schools. 
11

The study team will recruit approximately 90 elementary schools across approximately ten 

districts that are willing and able to comply with the requirements of MTSS-B training providers 

and the research and data collection procedures involved in the evaluation.

The impact study will examine several outcomes of MTSS-B—teacher practice, school climate, 

student behavior and academic achievement-- following the theory of action presented above. 

The study will also consider the difference in behavior support practices in program and BAU 

schools (service contrast). Finally, the evaluation will describe the CSBS’ training and support 

activities as well as the implementation of the MTSS-B in programs in schools.

Data Collection Strategy & Analytic Approach

Exhibit A-2 provides an overview of data collection activities. Data sources that present a burden 

to students and/or staff are indicated with an asterisk. A longer description of and justification for 

the data sources that present a burden is provided in the paragraphs below the table. 
12

Exhibit A-2. Overview of MTSS-B Data Sources 

Timeline

# of 

Respondents 

per School 

# of 

Schools Spring 

2015

Summer 

2015 Fall 2015 Spring 2016

Summer 

2016 Fall 2016 Spring 2017

Observation of CSBS training and support 

activities in Program Schools 0 60 X X X X X X X

Site visits*

 School tour to observe school rules 0 90 X X X

 Document reviews 0 90 X X X

MTSS-B Coach interview˅

• 20 minutes per wave in Program 

Schools

.33 60 X X X

Administrator interview^ 

• 45 minutes per wave in BAU schools

• 60 minutes per wave in Program 

Schools

1 90 X X X

Team leader interview^

• 30 minutes per wave in BAU schools

• 45 minutes per wave in Program 

Schools

1 90 X X X

 Student interview (randomly selected)

• 2 minutes per wave 15 90 X X X

 Staff interview (randomly selected)

• 5 minutes per wave 15 90 X X X
13

Review of web-based forms (Coach logs, 

meeting minute forms and TIPS school 

forms)

0 60 X X X X X X

Review of sample of CSBS records (CSBS 

trainer logs, attendance records, webinars, 

fidelity data)

0 60 X X X X X X X

Review of MTSS-B behavior monitoring and 

fidelity data (SWIS data, SWIS-CICO data & 

PBIS Apps data)

0 60 X X X X X X X

School staff survey (All grades k-5 teachers 

and other relevant staff) *

35 minutes for non-teaching staff

40 minutes for teachers

30 90 X X

Student survey (All students in grades 3-5)*

50 minutes to administer survey per wave 240 90 X X

Classroom observations with the ASSIST and 

possibly CLASS [augmented in Program 

Schools to included assessment of 

implementation fidelity] 0 90 X

Teacher ratings of student behavior (All 

teachers in grades 1-5; all consenting 

students in class)*

10 minutes per student per wave

20 90 X X X

District records data collection (10 districts) 0 90 X X X

Note: Activities that present a time burden to students and staff are indicated with an asterisk(*).

^ The structured interviews with the team leader and administrator listed under site visits are longer in Program Schools because the interview protocol 

includes questions related to these staff members’ perception of the quality and utility of CSBS training and support. 

˅

The MTSS-B Coach role is designated as 0.2 FTE for each program school. On average, however, districts will need 2 different MTSS-B coaches, though one 

of them may be only part-time. This increases the number of respondents for data collection purposes, but does not change the actual staffing. 
14

Description of Data Sources that Present a Burden to Staff and Students 

Site visits: We will conduct site visits in Program and BAU schools three times over the study 

period (Fall 2015, Spring 2016 and Spring 2017). The site visit protocols involve document 

review, observation and interview. The times required for the interviews conducted as part of the 

site visit are indicated in Exhibits A-2 and A-3. The data collected from these visits will serve the 

following purposes: (1) Assess the difference in behavior support practices in program and BAU 

schools (service contrast) (2) Assess implementation fidelity to the core feature of the MTSS-B 

Tier I and Tier II programs in Program Schools (3) Assess key implementers’ (MTSS-B 

Coaches, SLT team leader and administrator) perception of the quality and utility of CSBS 

training and support in Program Schools. To develop the protocols for these site visits, the study

team has pulled items from School wide Evaluation Tool (SET), which has been found be a valid 

and reliable measure of MTSS-B Tier I implementation fidelity;

9

it has also been used in prior 

randomized control trials of MTSS-B as a measure of service contrast between Program and 

BAU schools and found to accurately discriminate between schools implementing the core 

features of MTSS-B and those not.10 The I-SSET assesses the fidelity of Tier II and Tier III 

implementation,

11 and it has been combined with SET in prior studies to provide a complete 

picture of the three tiers of MTSS-B implementation.12

Understanding staff’s perception of the quality and utility of CSBS training and support in the 

Program Schools is critical to fully understanding implementation fidelity and to identifying 

challenges associated with the implementation of MTSS-B in a large number of elementary

school and districts. The SET/ISSET protocols do not ask school staff to reflect upon the quality

and utility of the MTSS-B training and support they received from CSBS. As a result, the study 

team will be augmenting the site visit interview protocols in Program Schools with questions for 

the MTSS-B Coaches, SLT team leader and administrator about their perceptions of the quality

and utility of CSBS training and support. The interview protocols for the site visits can be found 

in Appendices A-E.

Staff Survey: School staff (teachers, administrators and other staff) in all study schools will be 

asked to participate in a web-based survey about school climate, staff practices related to 

behavior, and training for behavior support practices and programs in the spring of 2016 and 

spring of 2017. The surveys also include a number of questions related to individual background 

characteristics and experiences. The survey for non-teaching staff will take approximately 35 

minutes to complete and the survey for teaching staff will take approximately 40 minutes to 

complete. 

 

9

Horner et al. (2004).

10 Bradshaw, Waasdorp, and Leaf (2012).

11 Anderson et al. (2008).

12 Debnam, Pas and Bradshaw (2012). 
15

To assess staff practices related to behavior, we are drawing items from the Effective Behavior 

Support Survey (EBS).13 These items assess the extent to which schools are implementing core 

features of MTSS-B in classroom and non-classroom settings. We have also drawn upon 

questions from the multi-site evaluation from IES’ Social and Character Development study to 

identify differences between BAU and Program Schools in the extent to which they have 

received training for and/or are implementing school-wide and classroom based programs similar 

to the MTSS-B approach being implemented in Program schools. Additionally, to assess 

implementation fidelity the study team has designed questions that ask about strategies specific 

to CSBS MTSS-B approach; the questions are general enough to apply to Program and BAU 

schools but specific enough to the CSBS model that they will inform our assessment of 

implementation fidelity. We have also designed questions to assess staff’s exposure to training 

and coaching related to behavior support practices. Staff’s answers to these questions will inform 

our understanding of service contrast; fully understanding service contrast is critical to the 

interpretation of the impact estimates. 

Improved school climate is theorized to be proximal outcome of the intervention and is therefore 

critical to measure to assess staff’s perception of school climate we are drawing from two wellvalidated measures—Organizational Health Inventory (OHI) and the Malasch Burnout 

Inventory.

14 Prior studies of MTSS-B have found impacts on the OHI15

. The staff survey can be 

found in Appendix F. 

Student Survey (Grades 3-5): Students in grades 3-5 in Program and BAU schools will be 

asked to complete a survey in the Spring of 2015 and Spring of 2016. The survey will take 

approximately 50 minutes for all fielding. This survey will provide critical information about 

students’ perception of their own behavior and classroom climate, which are both predicted by 

the theory of action to be positively impacted by MTSS-B. Many of the scales included were 

selected because they have been widely used in previous research on school-wide interventions; 

many have been found to have adequate psychometric properties.16 We have also included a 

limited number of questions to assess students’ awareness of core features of MTSS-B 

implementation such as the establishment of school rules, a positive reinforcement system and a 

gaming strategy to reinforce the school wide expectations. These questions will be asked in the 

survey administered in BAU and Program schools so they can inform the assessment of 

implementation and service contract. The student survey can be found in Appendix G.

Teacher Ratings of Student Behavior: Teachers of students in grades 1-5 in Program and BAU 

schools will be asked to rate their students’ behavior in fall of 2015,17 the spring of 2016 and the 

spring of 2017. Most districts will require that we obtain active parental consent from parents for 

these ratings and so teachers will only be asked to rate the behavior of students in their classes 

 

13 Sugai, Todd, and Horner (2000). 

14 Hoy and Tartar (1997); Maslach and Jackson (1981).

15 Bradshaw et al., 2008. 

16 Gresham et al. (2011); Haynes et al. (2001); Huesmann and Guerra (1997); Roberts, Hom, and Battistich (1995); 

Social and Character Development Research Consortium (2010).

17 Fielding this survey at baseline, will allow the study team to identify a high-risk sub-group at baseline. 

Identification of this subgroup is necessary for answering one of the study’s research questions regarding the effects 

of MTSS-B on at-risk students. 
16

with consent. The rating form is expected to take approximately ten minutes per student to 

complete. 

The teacher ratings form will be used to assess the impact of the program on student behavior. 

The study team has adopted the Teacher Observation of Student Adaptation-Checklist (TOCAC) for this instrument. The TOCA-C has been shown to be a valid and reliable measure of 

students’ behavior (concentration problems, disruptive behavior and prosocial behavior).18 It has 

been used in prior evaluations of MTSS-B where it has been shown to be sensitive to the 

intervention.19 The study team is augmenting this measure with questions that will provide 

information regarding students’ referral to and receipt of targeted supports for behavior and 

special education services. Asking this information on the teacher ratings forms is important 

because it cannot always be reliably ascertained from administrative datasets in Program and 

BAU schools. Finally, the rating form includes one scale that assesses teachers’ perception of 

students’ academic behaviors.20 This scale has been added because the theory of action predicts 

that MTSS-B will improve students’ academic engagement, which is not assessed on the TOCAC. The teacher ratings form can be found in Appendix H. 

District Records Data Collection: The study team will request extant data from school district 

records regarding student background characteristics, teacher background characteristics and 

experiences, student behavior incidences, student academic achievement (grades 3-5) and special 

education. Data will be collected at the end of each school year: baseline (SY 2014-2015), year 

one (SY 2015-2016) and year 2 (SY 2016-2017). Requests will be made of district research 

offices. Appendix I provides a sample of the letter that will be sent to districts requesting these 

data. 

 

18 Koth, Bradshaw and Leaf (2009). 

19 Bradshaw, Waasdorp, and Leaf (2012).

20 The study team will be adopting the “approaches to learning” scale from NCES’ ECLS-K survey to assess this 

construct. All necessary permissions will be acquired before the survey is fielded. 
17

SUPPORTING STATEMENT FOR PAPERWORK

REDUCTION ACT SUBMISSION

Part A: Justification 

1. Circumstances Making Collection of Information Necessary

This evaluation arises from a growing interest in multi-tiered systems of support for behavior 

[MTSS-B]. The Office of Special Education Programs’ (OSEP) National Technical Assistance 

Center on Positive Behavioral Intervention and Supports (positive behavior intervention supports 

is an example of an MTSS-B program) reports that more than 21,000 schools in the United 

States report implementation of MTSS-B.21 Since the implementation of MTSS-B is so 

prevalent, information on the effectiveness of this approach is necessary. Researchers have been 

developing an evidence base for these programs. Numerous descriptive and correlational studies 

have associated aspects of MTSS-B implementation, or MTSS-B implementation overall, with 

student outcomes. 22,23,24 More recently, researchers have begun to conduct causal studies of 

MTSS-B, although some of these initial studies have been on a relatively small scale (or 

conducted within a single state) or have been somewhat limited in duration and outcomes. This 

study will provide the first national, random assignment study of the impacts of MTSS-B 

providing impact estimates for universal supports (Tier I) as well as universal and targeted 

support together (Tiers I and II combined).

This impact evaluation of multi-tiered systems of support for behavior (MTSS-B) is authorized 

in Section 664 of the Individuals with Disabilities Education Act [IDEA], as amended, which 

assigns to IES the responsibility to conduct studies and evaluations of the implementation and 

impact of programs supported under IDEA, which includes MTSS-B.

25 This statute is provided 

in Appendix J. 

2. Purpose and Use of the Data: How the data will be collected, by whom, and for what 

purpose.

The information gathered through this data collection will be analyzed by the IES evaluation 

contractor (MDRC) and its subcontractors to study the implementation, and impacts of MTSS-B 

in elementary schools. This will involve new data collection where needed through staff and 

student surveys, teacher ratings of student behavior, brief interviews with randomly selected 

students and staff, and structured interviews with principals and key program implementers 

(MTSS-B Coaches and team leaders). Exhibit A-2 shows a timeline of these activities. 

The data collected for this study will be used by ED to report to Congress on the assessment of 

activities using federal funds under the Individuals with Disabilities Education Act National 

Assessment.26 Failure to collect these data may result in ED being unable to adequately report to 

 

21 Horner (2014). 

22 Bradshaw, Mitchell, and Leaf (2010). 

23 Horner et al. (2009). 

24 OSEP Center on Positive Behavior Interventions and Supports (2007).

25 IDEA, P.L. 108-446.

26 IDEA, P.L. 108-446.
18

Congress on the assessment of these activities. Additionally, if this evaluation were not 

completed, ED and Congress would not have an accurate understanding of implementation and 

impact of MTSS-B, which is reportedly being implemented in over 21,000 schools across the 

county and supported by Federal funds. 

3. Use of Technology to Reduce Burden

All staff surveys and teacher ratings will be administered using web-based surveys so they are 

easily accessible to respondents. Administration of web-based surveys enables reduced burden 

through complex skip patterns that are invisible to respondents, as well as prefilled information 

based on responses to previous items when appropriate. Web-based surveys also leads to 

decreased costs associated with processing and increased data collection speed. Paper survey

options will be offered to respondents as part of the follow-up effort with individuals who do not 

respond. 

4. Efforts to Avoid Duplication

As described in the introduction, the data collection effort planned for this project will produce 

data that are unique and targets specifically the research questions identified for this project. 

Valid and reliable measure of teacher practice, school climate and student behavior are not 

available from extant data for the participating districts and schools. 

5. Methods to Minimize Burden on Small Entities

The data will be collected from district and school staff and no small businesses or entities will 

be involved in the data collection. 

6. Consequences of Less Frequent Data Collection

If the proposed data collection is not done, it will not be possible for ED to report rigorous 

impact findings from a large scale study of MTSS-B to Congress, other policymakers, and 

practitioners seeking effective ways to support student learning. 

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

None apply to this data collection. 

8. Federal Register Comments and Persons Consulted Outside the Agency

The following experts serve on an Expert Panel for the Selection of Training Providers and have 

been consulted on the design of the study. 

Expert Organization

Leonard Bickman, Ph.D. Vanderbilt University, Peabody College of 
19

Expert Organization

Education

Catherine Bradshaw, Ph.D.27 University of Virginia, Curry School of Education

Lori Newcomer, Ph.D. University of Missouri, Department of Educational, 

School, and Counseling Psychology

The following individuals participated in the Technical Working Group for the Evaluation that 

convened on March 26, 2015.

Expert Organization

David Cordray, Ph.D Vanderbilt University, Department of Psychology 

and Human Development

Brian Flay Oregon State University, School of Social and 

Behavioral Health Sciences

Mary Louise Hemmeter Vanderbilt University, Department of Special 

Education

Sara E. Rimm-Kaufman, Ph.D. University of Virginia, Curry School of Education 

and Center for the Advanced Study of Teaching 

and Learning

Lori Newcomer, Ph.D. University of Missouri, Missouri Prevention Center 

/ Department of Educational School and 

Counseling Psychology

9. Payment or Gifts to Respondents

We are aware that teachers are the targets of numerous requests to complete data collection 

instruments on a wide variety of topics from state and district offices, independent researchers, 

and ED and several decades of survey research support the benefits of offering incentives. 

Further, high response rates are needed to make the study measures reliable and offering 

honoraria for staff and teachers will help ensure high response rates. In fact, the importance of 

providing data collection incentives in federal studies has been described by other researchers, 

given the recognized burden and need for high response rates.28

The use of incentives has been shown to be effective in lowering non-response rates and the level 

of effort required to obtain completions.29 Studies have shown that when used appropriately, 

incentives are a cost-effective means of significantly increasing response rates.30 Although much 

of the research on the value of using incentives has focused on low-income populations, the 

value of using incentives in educational settings has also been demonstrated. In the Reading First 

Impact Study commissioned by ED, monetary incentives proved to have significant effects on 

response rates among teachers. A sub-study requested by OMB on the effect of incentives on 

survey response rates for teachers showed significant increases when an incentive of $15 or $30 

 

27 Dr. Bradshaw is also a member of the evaluation team but served in this role because of her special expertise on training 

providers and MTSS-B training. 

28 Berry, Pevar, and Zander-Contugno (2008); Singer and Kulka (2002).

29 James (1997).

30 For example, see Dillman (2007).
20

was offered to teachers as opposed to no incentive.31 In 2005, the National Center for Education 

Evaluation submitted a memorandum to OMB outlining guidelines for incentives for NCEE 

Evaluation Studies and tying recommended incentive levels to the level of burden (represented 

by the length of the survey). The amount of incentives planned for the survey respondents for 

this study is consistent with that proposed in the NCEE memo Guidelines for Incentives for 

NCEE Evaluation Studies.32

Payments to teachers and staff: Teachers and school staff will be asked to complete surveys 

and rating forms at multiple times over the study period (fall 2015-spring 2017). We expect that 

these activities will be completed outside of the normal workday covered in their labor 

agreements, given the level of demand already made upon their time. Teachers and other school 

staff will be asked to complete a 35-40 minute survey at up to three points in time during the 

course of the study. If district rules allow, we will offer a gift card valued at $25 for each 

completion. Teachers also will be asked to complete a 10 minute short behavior rating for each 

of their students (from pool of consenting students if active consent is required by district) at 

three times during the study. Each rating will take approximately 10 minutes per student and, if 

district rules allow, teachers will be provided a gift card valued at $5 for each student rating 

completed.33 The incentive amounts are consistent with the 2005 NCEE memo recommendation 

for a “medium burden” data collection effort.

In addition, if district rules allow, we will provide a $25 incentive to classroom teachers or the 

school data collection liaison to encourage high rates of return for the parental consent forms 

needed for data collection activities[1. Our preferred method of distributing consent forms is to 

include the forms in a packet parents receive at the school’s open house during enrollment at the 

beginning of the school year. Alternatively, homeroom teachers may distribute consent forms to 

their students or the school may wish to have the forms mailed directly to the students’ home. 

We will work closely with each school to ensure that parents who do not return the consent form 

are sent reminder notices with replacement forms. This process will be repeated, as needed, to 

achieve the target active parental consent rate. We will provide an incentive ($25 gift card) for 

each classroom in which at least 90% of the student parental consent forms are returned, whether 

or not the parent allows the student to participate in the data collection activities. The incentive 

will be provided either to the classroom teacher or to the school data collection liaison based on 

who is responsible for monitoring the consent form returns to encourage the teacher, or the data 

collection liaison, to monitor consent returns carefully and follow up with students for whom a 

form has not been returned, encouraging students to return a signed form. Firms have used 

similar approaches successfully in previous studies. For example, in the cross-site evaluation of 

the Safe Schools/Healthy Students Initiative teachers and staff were told that each school would 

be provided an additional $25 for each classroom in which a minimum of 70% of active parental 

consent forms were completed by parents and returned to the school. In schools where this 

 

31 Gamse et al. (2008). 

32 National Center for Education Evaluation (March 22, 2005).

33 Teachers will be informed that they will only be awarded the $5 per response if they complete rating forms for a 

proportion of their student above a pre-specified threshold (from pool of consenting students if district rules require 

active parental consent for this activity). 

[1] MDRC’s IRB will likely require that we seek active parental consent for the teacher ratings of student behavior 

and the student surveys. Some district IRBs will also require active parental consent for the sharing of extant student 

records. 
21

approach was used, the return rates of parental consent forms of 78% were achieved, on average. 

This incentive opportunity will be available for all classrooms in grades 1-5.

10. Assurances of Confidentiality Provided to Respondents 

All data collection activities will be conducted in full compliance with The Department of 

Education regulations to maintain the confidentiality of data obtained on private persons and to 

protect the rights and welfare of human research subjects as contained in the Department of 

Education regulations. These activities will also be conducted in compliance with other Federal 

regulations in particular with The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley

Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of 

Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 

1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations 

on the protection of human research participants. 

An explicit statement describing the project, the data collection, and confidentiality will be 

provided to study participants. These participants will include parents of students in all study

schools as well as staff and administrators in all study schools. Most districts recruited for the 

study have rules requiring active informed consent for data collection concerning students. In 

these districts, parents will be asked to sign an informed consent form that meets federal human 

subjects requirements acknowledging their willingness to allow their student to participate and 

for the study team to access extant data. In districts that do not require active informed consent 

for data collection about students, we will review whether to shift to passive consent for student 

data collection. We will review with our Institutional Review Board whether it believes passive 

consent would be appropriate for this data collection. 

Confidentiality assurances during data collection: 

 All MTSS-B data collection employees at MDRC, DIR and AIR and any data collection 

sub-contractors thereof, sign confidentiality agreements that emphasize the importance of 

confidentiality and specify employees’ obligations to maintain it.

 Personally identifiable information (PII) is maintained on separate forms and files, which 

are linked only by sample identification numbers.

 Access to a crosswalk file linking sample identification numbers to personally

identifiable information and contact information is limited to a small number of 

individuals who have a need to know this information

 Access to hard copy documents is strictly limited. Documents are stored in locked files 

and cabinets. Discarded materials are shredded.

 Access to electronic files is protected by secure usernames and passwords, which are only 

available to approved users. Access to identifying information for sample members is 

limited to those who have direct responsibility for providing and maintaining sample

crosswalk and contact information. At the conclusion of the study, these data are 

destroyed.

 The plan for maintaining confidentiality includes staff training regarding the meaning of 

confidentiality, particularly as it relates to handling requests for information and 
22

providing assurance to respondents about the protection of their responses. It also 

includes built-in safeguards concerning status monitoring and receipt control systems.

Confidentiality assurance during analysis: The data collected for this study will be used only 

for broadly descriptive and statistical purposes. The reports prepared for the study will 

summarize findings across the sample and will not associate responses with a specific district or 

individual. In no instances will the study team provide information that identifies districts, 

schools, principals, teachers, or students to anyone outside the study team, except as required by 

law. More detail on MDRC’s procedures to ensure data security are described below: 

Data storage: MDRC and subcontractors will store all data in compliance with its federally 

approved data security plan with state-of-the-art data security protections that have passed 

reviews by the Social Security Administration and other federal agencies. All quantitative data 

will be stored on a secure section of our network, accessible only to specific project staff 

identified by the data manager. Data that is used for analysis is stored with research ID numbers 

rather than actual student or staff identification information; Notes/digital recordings from 

interviews, and all survey data will only be reviewed by the research team and stored in a secure 

area accessible only to the research team. All information from these research activities will be 

reported in the aggregate, and no disaggregated information from our observations or interviews 

will be shared with teachers, school or district leadership, or the general public.

Once the study is completed, original data collected for the project will be destroyed at the end of 

the project and only the restricted access file without any actual identifiers for the district, 

school, or individual respondent will remain [to be discussed below]. 

Method of data destruction: All data containing individually identifiable records will be 

destroyed by an appropriate fail-safe method, including physical destruction of the media itself 

or deletion of the contents on our servers. After the study is completed, the study team will create 

a restricted- access file of the data collected and submit that file to IES once the project ends.

This file will have been stripped of all student, teacher/staff, school and district identifiers. 

11. Justification of Sensitive Questions

Questions in some components of the MTSS-B student survey are potentially “sensitive.”

Students in grade 3-5 are asked about personal topics including students’ perception of fairness 

and as their perceived level of aggression and feelings that they have in school. The questions we 

have included were selected assess constructs that we need to measure in order to understand the 

impacts of the intervention. The specific scales selected for the surveys were chosen in part 

because they have been widely used in previous research and found to be valid and reliable 

measures of the constructs we intend to measure.

34 Moreover, trained assessors will explain 

questions to students before they are posed. Finally, students and their parents will be informed 

by research staff prior to the start of the survey that their answers are confidential, that they may

refuse to answer any question, that results will only be reported in the aggregate, and that their 

responses will not have any effect on any services or benefits they or their family members 

 

34 Aber, Brown, and Jones (2003); Jones, Brown, and Aber (2011a); Jones, Brown, and Aber (2011b); Jones, Brown, Hoglund, 

and Aber (2010).
23

receive. When required by districts and if required by the MDRC IRB, the study team will be 

asking for active parental consent for this activity.

Questions on the teacher ratings of student behavior are also potentially sensitive because 

teachers are providing an assessment of individual student’s concentration problems and 

disruptive behavior. This rating scale is essential to the study because it is so closely tied to the 

MTSS-B theory of action and prior evaluations of MTSS-B have found effects on these rating 

scales.

35 Many of the questions on this measure have been used in other studies including the 

recent IES-funded evaluation of the Good Behavior Game. When required by districts and if 

required by the MDRC IRB, the study team will be asking for active parental consent for this 

activity. 

12. Estimates of Annualized Burden Hours and Costs

Exhibit A-3 summarizes reporting burden on respondents for each data source. The respondent 

pool will include school staff and students. For each data collection activity, there is a 

corresponding number of respondents assumed per school, an assumed number of schools 

associated with the activity, and a total number of respondents estimated (assuming a 90% 

response rate). Exhibit A-3 also provides the number of administrations each data collection 

activity will be conducted over the course of the two years of the study, average estimates for the 

amount of time required for each activity in minutes, as well as the total burden hours calculated 

for each activity to be completed by all of the required respondents. This proposed information 

collection does not impose a financial burden on any respondents and respondents will not incur 

any expenses. 

The introduction to supporting statement A and the responses to question B-2 provide more 

detail about the timeline of data collection activities listed in the exhibit and describe why these 

activities are essential to the study. 

 

35 Bradshaw, Waasdorp, and Leaf (2012). 
24

Exhibit A-3. Estimated Burden to Respondents

Data Collection Activity

Assumed # of 

Respondents 

per school

Assumed 

# of 

schools

Estimated # of 

Respondents 

per 

Administration 

(90% consent 

rate assumed)

Number of 

Administrations 

(Over 2 Years)

Average 

Burden 

Hours 

per 

Response 

(Minutes)

Total 

Burden 

(Hours)

Total 

Number of 

Responses 

(Over 2 

Years)

Total #of 

Respondents 

(over 2 years; 

90% consent 

rate )

Teacher ratings of student behavior

(assumed 20 students per teacher and 

10 min. per student)

20 90 1,620 3 200 16,200 4,860 1,620

Student survey (grades 3-5) 240 90 19,440 2 50 32,400 38,880 25920

Non-teaching staff survey 10 90 810 2 35 945 1,620 810

Teacher survey 20 90 1,620 2 40 2,160 3,240 1,620

Site Visit: Brief interviews with 15 

randomly selected staff 15 90 1,215 3 5 304 3,645 3,645

Site Visit: Brief interviews with 15 

randomly selected students 15 90 1,215 3 2 122 3,645 3,645

Site Visit: Interview with 

administrator in program schools 1 60 54 3 60 162 162 54

Site Visit: Interview with 

administrator in BAU schools for site 

visit

1 30 27 3 45 61 81 27

Site Visit: Interview with behavior 

team leader in program schools for 

site visits

1 60 54 3 45 122 162 54

Site Visit: Interview with behavior 

team leader in BAU schools 1 30 27 3 30 41 81 27

Site Visit: Interview with MTSS-B 

coach .33 60 18 3 20 18 54 18

District Records Data Collection 

from 10 districts 10 3 1080 540 30 10

Total 53,073 56,460 37,450
25

Note. The assumption of 80 students per grade is based on the average 3rd grade enrollment for elementary schools (defined as schools with no grade higher than 

the 6th grade) across the U.S. for 2008-2009 school year, which was 77.9 using data reported by the Common Core of Data (CCD). The assumption of 20 students 

per class is based on the 2007-2008 Schools and Staffing Survey which shows an average elementary school class size of 20.3 students. The assumption of 20 grade 

1-5 teachers is based on the assumption regarding class size and average number of students per grade. The assumption regarding the number of MTSS-B coaches 

per schools is based on the assumption that there will be six program schools per district, which will require each of the ten districts to hire 2 part-time MTSS-B 

coaches (.33 *60 = 20). The assumption of 10 non-teaching staff in a school is an estimate based on the Schools and Staffing Survey. Assumptions regarding the 

time to fulfill district records requests are based on MDRC's experience requesting district records for other studies that request comparable data from districts. The 

estimated total number of respondents is different from the estimates number of respondents per administration because of a new cohort of third graders in year 2 

and a random sample of students and staff being selected for site visit interviews at each wave. In calculating the number of responses per data source, we assume 

that each teacher's rating of all the students' in their class is one response and that each district records data collection request counts as one response for each wave 

of collection.
26

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

Not applicable. The information collection activities do not place any capital cost or cost of 

maintaining capital requirements on respondents.

14. Annualized Cost to the Federal Government

The total cost for the study is $17,770,151 over 5 years for an annualized cost of $3,544,030.20 

15. Explanation for Program Changes or Adjustments

This submission to OMB is a new request for approval.

16. Time, Schedule, Publication and Analysis Plan 

Schedule & Publication Plan 

The project schedule is as follows:

 Final selection and randomization of sites by spring of 2015

 MTSS-B training and support activities spring 2015-spring 2017

 Completion of data collection and creation of analysis files for final report by December 

2017

 Analysis of findings for final report in winter and spring of 2018, and 

 Preparation of a project final report and public-use data file for review by the Department 

and release in November 2018

We anticipate a report that includes an introductory chapter, a chapter on project design and data 

collection, a chapter discussing the nature and implementation of the MTSS-B training and 

support, a chapter discussing the nature and implementation of the MTSS-B program 

implemented in schools as well as services in the BAU schools, and a chapter presenting impact 

findings. The report will follow guidance provided in the National Center for Education Statistics 

(NCES) Statistical Standards36 and the IES Style Guide.37

Analysis Plan

To assess the implementation of MTSS-B training and support, the study team will use data from 

the direct observation of CSBS training and support activities, interviews with key program 

implementers (MTSS-B coach, administrator and behavior team leader), and review of program 

data (e.g. CSBS trainer logs and training attendance sheets). Systematic review of these data will 

allow the team to describe the content and frequency of CSBS activities, key implementers’ 

 

36 National Center for Education Statistics (2002).

37 U.S. Department of Education, Institute for Education Sciences (2005).
27

perception of the quality of CSBS activities and variation in the content of training delivered 

between and within districts. 

To assess implementation of the MTSS-B program in schools, the study team will use data 

obtained from site visits to program schools (interview, observation and document review) and 

through a review of program data (e.g. web-based program forms, MTSS-B coach logs). 

Analysis of these data will allow the team to describe the observed implementation of Tier I and 

Tier II activities and to describe variation between and within districts in schools’ 

implementation of MTSS-B. 

To estimate the impacts of MTSS-B on school staff practice, school climate and student 

outcomes (e.g. student behavior, student achievement and receipt of special education services) 

the study team will rely upon teacher and student surveys, teacher ratings of student behavior, 

classroom observation and district records. The study team will select one primary outcome per 

domain. For each outcome we will estimate program impacts after one and then two years of 

implementation using a school level random assignment design, with random assignment 

occurring at the level of the districts.38 Impacts will be estimated using fixed effects, multi-level 

hierarchical models. The primary student sub-group analysis will be for students identified as 

being at-risk for future behavior problems at baseline according to the teacher ratings of student 

behavior fielded in the Fall of 2015. Comparison of the behavior support practices in Program 

and BAU Schools through data obtained from site visits, staff and student surveys and classroom 

observations will help the team to interpret the findings from the impact analysis. 

17. Approval to Not Display OMB Expiration Date 

All data collection instruments will include the OMB expiration date. 

18. Explanation of Exceptions to the Paperwork Reduction Act

No exceptions are needed for this data collection.

 

38 If there are subgroups of similar schools – we may have blocks of schools within a district and randomize at that 

level. These decisions will be made upon review of district data. 
28

REFERENCES USED IN PART A

Aber, J. Lawrence, Joshua L. Brown, and Stephanie M. Jones. 2003. Developmental trajectories 

toward violence in middle childhood: Course, demographic differences, and response to 

school-based intervention. Developmental Psychology - Special Issue on Violent Children

39, 2: 324-348; 

Anderson, C. M., Teri Lewis-Palmer, Anne W. Todd, Robert H. Horner., George Sugai, N.K. 

Samson. 2008. "Individual student systems evaluation tool, version 2.6." Educational and 

Community Supports, University of Oregon.

Berry, Sandra H., Jennifer Pevar, and Megan Zander-Cotugno. 2008. “The use of incentives in 

surveys supported by federal grants.” Santa Monica, CA: RAND Corporation.

Bradshaw, Catherine P., Christine W. Koth, Katherine B. Bevans, Nicholas Ialongo, and Philip J. 

Leaf. 2008a. “The Impact of School-Wide Positive Behavioral Interventions and 

Supports (PBIS) on the Organizational Health of Elementary Schools.” School 

Psychology Quarterly, 23, 4: 463-473.

Bradshaw, Catherine P., Mary M. Mitchell, and Philip J. Leaf. 2010. “Examining the effects of 

School-Wide Positive Behavioral Interventions and Supports on student outcomes: 

Results from a randomized controlled effectiveness trial in elementary schools.” Journal 

of Positive Behavior Interventions, 12, 133-148.

Bradshaw, Catherine P., Tracy E. Waasdorp, and Philip J. Leaf. 2012. "Effects of School-Wide 

Positive Behavioral Interventions and Supports on Child Behavior Outcomes." 

PEDIATRICS 130, 5: e1136-e1145.

Debnam, Katrina J., Pas, Elise T., and Bradshaw, Catherine P. (2012). Secondary and tertiary

support systems in schools implementing school-wide positive behavioral interventions 

and supports: A preliminary descriptive analysis. Journal of Positive Behavior 

Interventions, 14(3), 142–152.

Dillman, Don A. 2007. Mail and Internet Surveys: The Tailored Design Method (2nd ed.). New 

Jersey: John Wiley & Sons, Inc.

Durlak, Joseph A., Roger P. Weissberg, Allison B. Dymnicki, Rebecca D. Taylor, and Kriston B. 

Schellinger. 2011. "The impact of enhancing students’ social and emotional learning: A 

meta-analysis of school-based universal interventions." Child Development 82, 1: 405-

432.

Gamse, Beth C., Howard S. Bloom, H. S., James J. Kemple, and Robin Tepper Jacob. 2008. 

“Reading First Impact Study: Interim Report.” NCEE 2008-4016. National Center for 

Education Evaluation and Regional Assistance.

Gresham, Frank M., Stephen N. Elliott, Michael J. Vance, and Clayton R. Cook. 2011. 

"Comparability of the Social Skills Rating System to the Social Skills Improvement 
29

System: Content and Psychometric Comparisons Across Elementary and Secondary Age 

Levels." School Psychology Quarterly 26, 1: 27-44.

Haynes, N. M., C. L. Emmons, M. Ben-Avie, and J. P. Comer. 2001. "The school development 

program student, staff, and parent school climate surveys." New Haven, CT: Yale Child 

Study Center.

Horner, Robert H. 2014. “The Role of District Leadership Teams in PBIS Implementation”. 

Presentation. Available at http://www.pbis.org/Common/Cms/files/pbisresources/1-

Implementing%20PBIS.pptx.

Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. 2004. The 

school-wide evaluation tool (SET): A research instrument for assessing school-wide 

positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.

Horner, Robert H., George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne W. 

Todd, and Jody Esperanza. 2009. "A randomized, wait-list controlled effectiveness trial 

assessing school-wide positive behavior support in elementary schools." Journal of 

Positive Behavior Interventions 11, 3: 133-144.

Hoy Wayne K. and Clemens John Tarter. 1997. The Road to Open and Healthy Schools: A 

Handbook for Change, Elementary Edition. Thousand Oaks, CA: Corwin Press. 

Huesmann, Rowell L., and Nancy G. Guerra. 1997. "Children's normative beliefs about 

aggression and aggressive behavior." Journal of Personality and Social Psychology 72, 2: 

408-419.

James, Tracy. 1997. “Results of the Wave I incentive experiment in the 1996 survey of income 

and program participation.” In Pp 834-839 in Proceedings of the Survey Methods Section, 

American Statistical Association.

Jones, Stephanie M., Joshua L.Brown and J. Lawrence Aber. 2011a. The longitudinal impact of a 

universal school-based social-emotional and literacy intervention: An experiment in 

translational developmental research. Child Development 82,2: 533-554; 

Jones, Stephanie M., Joshua L. Brown, and J. Lawrence Aber. 2011b “Two-Year Impacts of a 

Universal School-Based Social-Emotional and Literacy Intervention: An Experiment in 

Translational Developmental Research”. Psychology Faculty Publications. Paper 131. 

(http://fordham, bepress.com/psych_facultypubs/131).

Jones, Stephanie M., Joshua L. Brown, Wendy LG Hoglund, and J. Lawrence Aber. 2010. "A 

school-randomized clinical trial of an integrated social–emotional learning and literacy

intervention: Impacts after 1 school year." Journal of Consulting and Clinical Psychology

78, 6: 829.

Koth, Christine W., Catherine P. Bradshaw, and Philip J. Leaf. 2009. "Teacher Observation of 

Classroom Adaptation—Checklist: Development and factor structure." Measurement and 

Evaluation in Counseling and Development 42, 1: 15-30.
30

Maslach, Christina, and Susan E. Jackson. 1981. "The measurement of experienced burnout." 

Journal of Organizational Behavior 2, 2: 99-113. 

National Center for Education Evaluation. 2005. Guidelines for incentives for NCEE impact 

evaluations. (March 22). National Center for Education Evaluation: Washington, DC. 

OSEP Center on Positive Behavioral Interventions and Supports. 2007. “Is School-Wide Positive 

Behavior Support An Evidence-Based Practice? A Research Summary.” Website: 

www.pbismaryland.org.

Roberts, William, Allen Hom, and Victor Battistich. 1995. “Assessing Students’ and Teachers’ 

sense of the School as a Caring Community.” Oakland, CA: Developmental Studies 

Center.

Sugai, George, Todd, Anne W., and Horner, Robert H. (2000). Effective Behavior Support 

(EBS) Survey: Assessing and planning behavior supports in schools. Eugene, OR: 

University of Oregon.

Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. H. (2005). School-wide evaluation tool. 

Eugene: University of Oregon.

Social and Character Development Research Consortium. 2010. "Efficacy of schoolwide 

programs to promote social and character development and reduce problem behavior in 

elementary school children." Washington, DC: National Center for Education Research, 

Institute of Educational Sciences, U.S. Department of Education.