DOI: 10.3886/icpsr37954.v1
title: A Randomized Impact Evaluation of Capturing Kids' Hearts, South Carolina, 2016-2018
title: Version 1

description: Funded by the National Institute of Justice, and in collaboration with the Flippen Group, the South Carolina Department of Education, and the districts of Charleston, Georgetown, Greenville, and Richland County Schools, WestEd conducted a randomized controlled trial of violence prevention and a school climate improvement program known as Capturing Kids' Hearts Campus by Design (CKH). CKH is a skill intensive, systematic process, whole-school intervention, designed to strengthen students' connectedness to school through enhancing protective factors (strong bonds with teachers, clear rules of conduct that are consistently enforced) and targeting modifiable risk factors (inappropriate behavior, poor social coping skills). The CKH training program aims to address the mechanisms and processes of social skills instruction that lead to improved student behavior by promoting skills acquisition (i.e., modeling, coaching, and behavioral rehearsal), enhancing skills performance, removing competing behavior, and facilitating maintenance of social skills. Components of CKH have been widely used throughout the United States. As of 2013, CKH training has been offered to over 200,000 staff in more than 7,000 schools. Although widely used, the whole package of CKH training and service has not been sufficiently subjected to a rigorous evaluation to assess the effectiveness of the program. The purpose of this study is to test and evaluate the Capturing Kids' Hearts package of programs and processes offered in the school-wide model, <em>CKH Campus by Design</em>, which includes <em>Capturing Kids' Hearts, Process Champions-Plus, Campus TrAction Pacs, Teen Leadership, CKH Recharged,</em> and <em>Leadership Blueprint</em>.
description: The purpose of this study is to conduct an evaluation of the Capturing Kids' Hearts (CKH) programs that will add to the literature on promising programs that improve school safety and climate, reduce bullying, and improve student social competencies. This randomized controlled trial provides the first rigorous test of the CKH package of programs to determine if its promise holds up in a rigorous evaluation. <strong>The study is designed to address the following research question:</strong> <br> What are the impacts of CKH on violence perpetration and victimization, relationship bonds between and among students and teachers, and social competencies? <br>Specifically, Does Capturing Kids' Hearts enhance relationship bonds between and among students and teachers, increase students' social competencies, and reduce aggression and violence? Does Capturing Kids' Hearts reduce absenteeism, truancy, suspensions, and disciplinary referrals, and increase language arts and math scores? Research question 1 is examined using data collected from surveys administered to students and teachers, while research question 2 relies on student survey and archival data.
description: To examine the impact of CKH, this study used a true, cluster-randomized, experimental design with repeated measures involving 27 middle schools (grades 6-8) served by four school districts in South Carolina - Charleston County, Georgetown County, Greenville County, and Richland County. Schools were randomly assigned to one of two treatment groups - a treatment group and a wait-listed control group. 15 schools were assigned to the treatment group, while 12 schools were assigned to the control group. The treatment group received the CKH package, with training provided to all teachers and administrators - including the foundational CKH, Process Champions-Plus, Campus TrAction Pacs, CKH Recharged, and the Leadership Blueprint - and implemented the one-semester Teen Leadership course for grade 6 students. Schools in the control condition operated as "business as usual" during this time and received training following the two-year intervention and data collection period in July 2018. Data collection started with the administration of an online survey to all school staff in May 2016 to assess baseline measures of school climate. CKH foundational training was delivered to all staff in treatment schools at the start of the school year (August 2016). Select teachers were trained to teach the one-semester <em>Teen Leadership</em> course for grade 6 students in fall 2016. <em>Process Champion</em> training and <em>Campus TrAction</em> workshops and consultations - which aim to enhance the capacity of schools to implement CKH with fidelity - was provided to select staff during the first implementation year. In addition, refresher training for intervention schools (<em>Capturing Kids' Hearts Recharged</em>) was provided at the start of year 2. All students in grades 6-8 were surveyed in each school, with baseline data collected in September 2016 and at the end of each implementation year (May 2017 and 2018). The same survey instrument was used at baseline and subsequent administrations. <strong>Data Collection Systems for Outcomes</strong><br> Two data collection systems were implemented to facilitate outcome data collection: an <em>incident tracking system</em> for disciplinary referrals and an <em>online survey data system</em>. Data from these systems were merged using unique ID/access codes assigned to each student. <em>Incident Tracking</em><br> WestEd tracked discipline referrals in both treatment and control schools utilizing the districts' current systems for recording infractions. Site coordinators were responsible for "tagging" incident reports with the student ID codes for students involved in the incident so that the two data collection systems could be linked. Data on disciplinary referrals in the prior academic year (2015-16) were obtained to examine whether and how referrals changed once CKH was implemented. However, because the intervention will likely have an impact on disciplinary referrals, the incident data was not used to estimate program impacts. <em>Online Survey System</em><br> Students (and teachers) were assigned unique access codes that allowed WestEd to link baseline and outcome survey data, and link survey data to student involvement in referred incidents. The student rosters of the relevant grade 6-8 classrooms in participating schools were obtained from the participating school districts at the start of each academic year, and unique access coded were assigned to each student. Only this access code will be entered into the online system to access the survey and provide linkage to follow-up surveys and to involvement in referred incidents.
description: <strong>Staff Survey Data</strong> The surveys asked staff to report their perceptions of students' connectedness to school, bonds between and among students and school staff, and students' personal and social competencies. <strong>Student Survey Data</strong> The surveys asked students to report information regarding demographics and various experiences at school, such as relationships with adults, connectedness to school, as well as aggressive or violent behaviors. <strong>Archival Record Data</strong> The archival data comprised of student-level demographics, achievement, discipline, and attendance data. Variables obtained from archival data and used as outcomes include the number of total absences, excused absences, unexcused absences, tardies, total suspensions, out-of-school suspensions, in-school suspensions, discipline referrals, as well as test scores for ELA and math assessments. Two grouping variables were used to identify unique schools. The variable <em>school ID 2</em> is a unique school identifier that represents the school a student attended in Year 1. The variable <em>school ID 3</em> is a school identifier that represents the school identified by the <em>school ID 2</em> variable but represents the school in Year 3 if the school is missing for the <em>school ID 2</em> variable, but not missing in Year 3.
description: Not applicable.
description: Response Rates: Student survey response rates:72% in 201676% in 201786% in 2018 Staff survey response rates:90% in 201687% in 201779% in 2018
description: Presence of Common Scales: Several Likert-type scales were used. Some of the measures and/or subscales included the California School Climate, Healthy, and Learning Survey (Cal-SCHLS) and the Psychological Sense of School Membership (PSSM) scale (Goodenow, 1993).
description: <strong>Staff Survey Data</strong><br> All school staff were surveyed. School staff surveys were collected using the same survey instruments at three time periods: spring of 2016, 2017, and 2018. The data collected at the first time point, in May 2016, serves as the baseline staff survey data. <strong>Student Survey Data</strong><br> In all participating schools, all students with parental consent in grades 6-8 were surveyed over a three-year period. The same survey instrument was administered at all three time points with baseline data collected in spring 2016 and subsequent administration in May 2017 and 2018. The sample consisted of 22,591 students which had data for at least one of the years. <strong>Archival Record Data</strong><br> Archival (i.e., administrative) data on students were provided by the districts of Charleston, Georgetown, Greenville, and Richland for the years 2015-16, 2016-17, and 2017-18. The archival data were received separately by district for each year, and the separate data were merged to create a single data file. <br>To prepare the archival record data for analysis, the archival record data were merged with the student survey data resulting in 25,836 total unique observations. Of those students, 21,242 had both archival data and CKH survey data in at least one year, while 3,245 students had archival data in at least one year but no survey data, and 1,349 students had CKH survey data in at least on year, but not archival data.
description: Datasets: DS0: Study-Level Files DS1: Merged Wide Archival Student Survey Data DS2: Capturing Kids' Hearts (CKH) Staff Survey Data DS3: Capturing Kids' Hearts (CKH) Student Survey Data
description: All students (grades 6-8), administrators, and staff on campus who interact with students at 27 middle schools served by four school districts in South Carolina - Charleston County, Georgetown County, Greenville County, and Richland County. Smallest Geographic Unit: County
description: web-based survey