DOI: 10.3886/icpsr38863.v1
title: Impact Evaluation of Complementarities Between Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice, Maryland, 2018-2021
title: Version 1

description: Across the United States (U.S.), school districts have grappled with how to create safe community- and achievement-oriented schools and how to ensure the necessary discipline is applied transparently, fairly, and without bias. Two programs that many schools have turned to in order to achieve these goals are Schoolwide Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice (RJ). PBIS is an evidence-driven schoolwide behavioral management approach that aims to outline clear expectations for students and to cultivate shared norms and practices across classrooms and school spaces. PBIS has become a popular approach in schools and districts: as of 2020; over 19,000 schools in the U.S. have implemented PBIS.
A second program, Restorative Justice (RJ), has grown in popularity in recent years. RJ typically focuses on restorative relationship building between affected parties, peaceful reconciliation, and non-punitive approaches to rectifying harm, using a structured circle discussion format. RJ schools use both community circles, designed to build a safe space for students and staff to share and listen to each other, and restorative circles, designed to share perspectives on and redress a behavioral issue.
Working with a large school district in a mid-Atlantic state, researchers set out to test whether these two programs substitute for or complement each other. In partnership with the school district researchers conducted two separate school-level randomized controlled trials (RCTs). The first RCT (RCT 1) sought to uncover the marginal impact of RJ by adding the program to a set of randomly selected schools that were already implementing PBIS. The second RCT (RCT 2) was designed to discover the impact of introducing both programs together into schools that had neither program at baseline. Researchers conducted student and staff surveys to collect measures of school climate, teacher logs to record program implementation, and researchers also received administrative data from the district on student test scores, teacher and student absences, student disciplinary infractions, and school costs. 
There is growing evidence of the effectiveness of each of these programs in isolation. A recent meta-analysis of 32 experimental and quasi-experimental impact studies of PBIS found that PBIS reduced disciplinary exclusions and problem behavior and increased academic achievement. The findings were statistically significant and showed small to medium effect sizes. Individual studies have found that PBIS reduces the use of office disciplinary referrals and other exclusionary disciplinary measures (including the use of in-school and out-of-school suspensions), while improving student behavior and attitudes across school levels. Individual studies show variable--some statistically significant and some null--impacts on academic outcomes.
The empirical evidence on the effect of RJ in U.S. schools is more limited, with little rigorous casual evidence published to date. Based on patterns across rigorous and non-rigorous research, restorative justice is associated with decreases in suspension rates and disciplinary disparities, improved student behavior, and improved school climate and relationships.
description: The goal of the study was to hypothesize that the Positive Behavioral Interventions and Supports (PBIS) and Restorative Justice (RJ) programs would be complementary, with the two being more effective than either program alone at improving school safety and discipline, and that the prosocial approach shared by both programs would reduce racial disparities in disciplinary responses. The formal research questions were as follows:

1. Do RJ and PBIS+RJ affect behavioral outcomes (i.e., disciplinary referrals, bullying, harassment, feeling isolated, social skills)?
2. Do RJ and PBIS+RJ reduce use of exclusionary practices (such as suspensions and expulsions)?
3. Do RJ and PBIS+RJ affect the disparities between African American, Hispanic/Latino, and White and Asian students on indicators of problem behavior, such as suspension rates?
4. Do RJ and PBIS+RJ affect students' attendance or academic outcomes?
5. Is PBIS+RJ more effective on the outcome measures than PBIS alone or neither program?

For both randomized controlled trials 1 (RCT 1) and randomized controlled trials 2 (RCT 2), researchers believed that the intervention (adding RJ to PBIS for RCT 1 and introducing RJ+PBIS for RCT 2) would positively affect behavioral outcomes and would reduce exclusionary practices and disparities in exclusionary practices. Researchers did not believe that there would be negative on academic outcomes and they believed that student and teacher attendance would increase. Although the RCTs were not set up to directly evaluate adding both programs to adding only one program (for example, as if all treatment conditions had been in a single RCT), researchers believed that the effect sizes in RCT 2 would be larger than the effect sizes in RCT 1 (i.e., that going from zero programs to two programs would have greater impact than going from one program to two programs).
description: The school district informed schools of the study in school year (SY) 2018-2019 and invited schools to volunteer to take part. Schools were told that they would be randomly assigned to treatment or control groups and that district personnel would introduce Restorative Justice (RJ) or RJ+ Positive Behavioral Interventions and Supports (PBIS) into treatment schools over the following two academic years and control schools at the end of the two-year intervention. The latter was intended to keep the control schools engaged in study data collection during the period in which they did not receive interventions. Because schools were allowed to select into the experiment, it is likely that the schools that volunteered were more interested in positive behavioral interventions. Results should be interpreted as applying to such interested schools and may not be characteristic of all schools in the district.
Interested schools were then divided into two groups: those that were already implementing PBIS and those that were not (no elementary or middle schools in the district were implementing RJ at the time). The first group consisted of 23 schools (18 elementary schools and 5 middle schools), of which 12 were randomized into treatment (9 elementary schools and 3 middle schools) and the rest into the control condition. The second group consisted of 21 schools (15 elementary schools and 6 middle schools), of which 11 were randomized into treatment (8 elementary schools and 3 middle schools) and the rest into the control condition. The randomization was conducted in February 2019 to allow training for the treatment schools to start in the summer before the 2019 - 2020 academic year (SY 2019-2020). One treatment elementary school in randomized controlled trial 2 (RCT 2) dropped out of the experiment at the beginning of the 2019 - 2020 school year. 
The final sample of 43 excludes one treatment elementary school in RCT 2 that dropped out of the experiment at the beginning of the 2019-2020 school year.
description: The study includes five datasets (3 administrative datasets and 2 survey datasets) and a total of 115 variables with unique IDs: <strong>School ID</strong>, <strong>Student ID</strong>, and <strong>Staff ID</strong>. The administrative data files had asked about the average teacher salary, average per pupil expenditure - Federal, if there was any suspension or expulsion, and Measures of Academic Progress (MAP) Reading score.
The survey data files asked the respondents to describe if students get in fights, if students spread mean rumors or lies about others on social media or the internet, if they participated in meetings designed to address a problem or disagreement, the frequency of physical conflicts among students and if parents treat staff with respect. 
The datasets also included additional demographic variables such as; male, minority, teachers, elementary school among others.
description: Response Rates: The study originally randomized 44 schools into treatment and control conditions. One treatment elementary school in randomized controlled trial 2 (RCT 2) dropped out of the experiment at the beginning of the 2019-2020 school year, leaving 43 schools in the final sample.


 <strong>Administrative data</strong>: Administrative data was obtained for all students, staff, and schools for school years 2018-2019, 2019-2020, and 2020-2021.

 <strong>Student survey</strong>: The student survey was administered to students in grades 5 and above. Active parental consent was required for student participation. The survey administration took place during classroom time: students with parental consent were given the choice to assent and to take the survey in English or Spanish. The logistics of seeking active parental consent and administering the survey in each school limited the response rates, which were 22.2 percent in Wave 1, 18.7 percent in Wave 2, and 17.5 percent in Wave 3.

 <strong>Staff survey</strong>: Invitations to the staff survey were sent out by e-mail to all employed staff. Response rates were 49.1 percent for Wave 1, 32.9 percent for Wave 2, and 35.7 percent for Wave 3.

 <strong>Site visits</strong>: Site visits were conducted at three times over the timespan of the study in nine study schools: five schools implementing both Restorative Justice (RJ) and Positive Behavioral Interventions and Supports (PBIS), two schools implementing PBIS only, and two schools implementing neither. Four were middle schools and five were elementary schools.

 <strong>Teacher logs</strong>: A total of 2 teachers monthly per study school were randomly selected to fill out each monthly log. The total was increased to 4 per month starting in December 2020. The average response rate was 27.4 percent.
description: Presence of Common Scales: Various Likert-type scales were used.
description: Datasets: 
DS0: Study-Level Files
DS1: School Administrative Data
DS2: Student Administrative Data
DS3: Staff Administrative Data
DS4: Student Survey Data
DS5: Staff Survey Data
description: Student and staff in 43 elementary and middle schools in a United States school district.
description: web-based survey