Development and Pilot Testing of a Remote Training Strategy for the Implementation of Mental Health Evidence-Based Practices in Rural Schools
Phase: NA
Conditions: Behavior Disorders
Sponsor: Children's Hospital of Philadelphia

 
 
 
 
 
 
 
 
Evaluation of a Remote Training Strategy for School Personnel 
NCT05034198 
November 27, 2023 
 
 

 
 
 
 
 
 
 
 
ii
Evaluation of a Remote Training Strategy for School Personnel 
Title: 
Evaluation of a Remote Training Strategy for School Personnel 
Short Title 
Evaluation of a Remote Training Strategy  
eIRB Number 
IRB 21-018863 
Protocol Date:  
8/16/2021 
Amendment 1 Date: September 15, 2022  
Amendment 4 Date: June 27, 2023 
Amendment 2 Date: November 16, 2022 
Amendment 5 Date: November 27, 2023 
Amendment 3 Date: February 1, 2023 
 
 
Study Principal Investigator:   
 
Ricardo B. Eiraldi, PhD  
Children’s Hospital of Philadelphia 
The Roberts Center for Pediatric Research 
2716 South Street, Room 8293 
Philadelphia, PA 19146 
Phone 215-590-7759 
email: eiraldi@ chop.edu 
 
 
 

 
 
 
 
 
 
 
 
iii
TABLE OF CONTENTS 
Table of Contents .................................................................................................................. iii 
Abbreviations and Definitions of Terms.............................................................................. v 
Abstract .................................................................................................................................. 1 
1 
BACKGROUND INFORMATION AND RATIONALE ....................................................................... 2 
1.1 
INTRODUCTION ..................................................................................................................................... 2 
1.2 
NAME AND DESCRIPTION OF INTERVENTION ........................................................................................ 2 
1.2.1 
Remote Video Training Strategy Development ............................................................................... 2 
1.2.2 
Asynchronous and Synchronous Trainings for Implementation of EBPs ........................................ 3 
1.2.3 
Benefits of a Remote Training Strategy ........................................................................................... 3 
1.2.4 
Tier 2 Interventions ......................................................................................................................... 3 
1.3 
FINDINGS FROM NON-CLINICAL AND CLINICAL STUDIES ..................................................................... 4 
1.4 
RELEVANT LITERATURE AND DATA ..................................................................................................... 5 
1.4.1 
Mental Health Services in Rural Schools ........................................................................................ 5 
1.5 
COMPLIANCE STATEMENT .................................................................................................................... 5 
2 
STUDY OBJECTIVES .............................................................................................................................. 6 
2.1 
PRIMARY OBJECTIVE (OR AIM) ............................................................................................................ 6 
2.2 
SECONDARY OBJECTIVES (OR AIM) ...................................................................................................... 6 
3 
INVESTIGATIONAL PLAN .................................................................................................................... 6 
3.1 
GENERAL SCHEMA OF STUDY DESIGN ................................................................................................. 6 
3.1.1 
Pre-Trial Activities and Screening Phase ....................................................................................... 6 
3.1.2 
Phase 1: Training of Behavioral Health Staff ................................................................................. 7 
3.2 
STUDY DURATION, ENROLLMENT AND NUMBER OF SITES ................................................................... 8 
3.2.1 
Total Number of Study Sites/Total Number of Subjects Projected .................................................. 8 
3.3 
STUDY POPULATION ............................................................................................................................. 9 
3.3.1 
Inclusion Criteria ............................................................................................................................ 9 
3.3.2 
Exclusion Criteria ........................................................................................................................... 9 
4 
STUDY PROCEDURES .......................................................................................................................... 10 
4.1 
SCREENING AND CONSENT PROCEDURES ........................................................................................... 10 
4.2 
PHASE 1: TRAINING OF BEHAVIORAL HEALTH STAFF ........................................................................ 10 
4.2.1 
Initial Synchronous Trainings ....................................................................................................... 10 
4.2.2 
Guided Video for Remote Video and Remote Video plus Coaching .............................................. 10 
4.3 
PHASE 2: IMPLEMENTATION OF EBPS ................................................................................................ 10 
4.3.1 
Remote Consultation for Remote Video plus Coaching ................................................................ 11 
4.4 
SUBJECT COMPLETION/WITHDRAWAL ............................................................................................... 12 
5 
STUDY EVALUATIONS AND MEASUREMENTS ............................................................................ 12 
5.1 
SCREENING MEASURES ...................................................................................................................... 12 
5.2 
DEMOGRAPHIC INFORMATION ............................................................................................................ 12 
5.3 
IMPLEMENTATION FIDELITY ............................................................................................................... 12 
5.4 
STUDENT OUTCOME MEASURES......................................................................................................... 14 
5.5 
REMOTE TRAINING STRATEGY FEEDBACK ......................................................................................... 14 
6 
STATISTICAL CONSIDERATIONS .................................................................................................... 15 
6.1 
PRIMARY AND SECONDARYENDPOINTS.............................................................................................. 15 
6.2 
STATISTICAL ANALYSIS ..................................................................................................................... 15 
6.3 
STATISTICAL METHODS...................................................................................................................... 15 
6.4 
SAMPLE SIZE AND POWER .................................................................................................................. 16 
7 
SAFETY MANAGEMENT ..................................................................................................................... 17 

  
 
 
 
 
 
 
 
iv
7.1 
CLINICAL ADVERSE EVENTS .............................................................................................................. 17 
7.2 
ADVERSE EVENT REPORTING ............................................................................................................. 17 
8 
STUDY ADMINISTRATION ................................................................................................................. 18 
8.1 
TREATMENT ASSIGNMENT METHODS................................................................................................. 18 
8.1.1 
Randomization ............................................................................................................................ 18 
8.2 
DATA COLLECTION AND MANAGEMENT ............................................................................................ 18 
8.3 
CONFIDENTIALITY .............................................................................................................................. 19 
8.4 
REGULATORY AND ETHICAL CONSIDERATIONS.................................................................................. 20 
8.4.1 
Data and Safety Monitoring Plan ................................................................................................. 20 
8.4.2 
Risk Assessment ............................................................................................................................. 20 
8.4.3 
Potential Benefits of Trial Participation ....................................................................................... 21 
8.5 
RECRUITEMENT STRATEGY ................................................................................................................ 21 
8.6 
INFORMED CONSENT .......................................................................................................................... 22 
8.6.1 
Behavioral Health Staff ................................................................................................................. 22 
8.6.2 
School Administrators ................................................................................................................... 22 
8.6.3 
Parents and Students ..................................................................................................................... 23 
8.6.4 
Individuals with Limited English Proficiency ............................................................................... 23 
8.6.5 
Teachers providing data about eligible and enrolled students ..................................................... 24 
8.7 
PAYMENT TO SUBJECTS/FAMILIES ..................................................................................................... 24 
9 
PUBLICATION ........................................................................................................................................ 24 
10 
REFERENCES ......................................................................................................................................... 25 
  

  
 
 
 
 
 
 
 
v
ABBREVIATIONS AND DEFINITIONS OF TERMS 
 
 
 
BASC-3 
 
Behavior Assessment System for Children – 3rd Edition 
BHS 
 
Behavioral Health Staff 
CATS 
 
CBT for Anxiety Treatment in Schools  
CBPR 
 
Community-based participatory research  
CC 
 
Control Condition  
CICO 
 
Check-in/Check-out 
CoP SBBH 
 
Community of Practice on School-based Behavioral Health 
CPP 
 
Coping Power Program 
EBPs 
EvsD 
 
Evidence-based practices 
Engagement Versus Disaffection with Learning: Teacher 
Report 
FRIENDS 
 
Friends for Life 
ISF 
 
Interactive System Framework for Dissemination & 
Implementation 
NCES 
 
The National Center for Education Statistics 
PaPBS 
 
Pennsylvania Positive Behavior Support 
PBIS 
RV 
RV+ 
 
Positive Behavioral Interventions and Supports  
Remote Video 
Remote Video plus Coaching  
SAEBRS 
SCARED 
 
Social, Academic, & Emotional Behavior Screener 
Screen for Child Anxiety Related Disorders   
SDQ 
 
Strengths and Difficulties Questionnaire  
 
 
 
 
 
 

 
  
 
 
 
1
ABSTRACT 
Context:  
An increasing number of schools in rural settings are employing the multi-tier positive 
behavioral interventions and supports  (PBIS) framework to improve school-climate. PBIS 
can also be used as a framework for the adoption and integration of evidence-based practices 
(EBPs) for children’s mental health concerns. A key challenge is that school personnel need 
technical assistance (training plus ongoing consultation) in order to implement EBPs with 
fidelity. In urban and suburban schools, this support can be provided to school staff on site. 
However, providing ongoing on-site support is not feasible or sustainable in the majority of 
rural schools, due to their remote physical location. For this reason, video technology has 
been recommended for the training of behavioral health staff (BHS) in under-served rural 
communities. 
Objectives:  
The purpose of this study is to test the effectiveness of a remote training platform with BHS 
in schools serving rural communities.  
Study Design:  
This study is a pilot randomized controlled trial to compare implementation and student 
outcomes of three training strategies: Control Condition (CC), Remote Video (RV), and 
Remote Video plus Coaching (RV+).  
Setting/Participants: 
Participants will be school personnel with and without prior mental health training and 
students in grades 4-8 in schools serving rural communities who are deemed at risk for 
externalizing and internalizing mental health disorders. The pilot trial will be conducted in 
24 schools (8 in CC, 8 in RV, and 8 in RV+). It is estimated that a total of 72 behavioral 
health staff (3 staff per school- 24 per arm) and 312 students (13 students per school - 104 
per arm) will participate in this study. Additonaly, we estimate about 24 school 
administrators such as principals or assistant principals will approve school’s participation.  
Study Interventions and Measures:  
This study is designed to test the effectiveness of initial training compared to remote video 
training and remote video training plus coaching to train BHS to implement EBPs. Each 
school will be randomly assigned to receive one of three training supports: (a) Initial 
training (CC) (b) Initial training, plus video (RV) or (c) Initial training plus video, plus 
remote coaching (RV+). BHS in the RV+ condition will receive consultation from a CHOP 
consultant regarding implementation of EBPs, whereas BHS in the RV condition will be 
given access to the asynchronous training video modules. Participants in all three conditions 
will receive treatment manuals for the interventions they have chosen. The EBPs to be 
implemented by the BHS include: Coping Power Program (CPP), CBT for Anxiety 
Treatment in Schools (CATS), and Check-in/Check-out (CI/CO). All BHS will conduct one 
CPP or CATS group with 5 students of similar developmental level (e.g., a group of 4th & 

 
  
 
 
 
2
5th graders), and that each school staff in CI/CO will implement the intervention with 4 
separate individual students of any eligible school grade. 
1 
BACKGROUND INFORMATION AND RATIONALE 
1.1 Introduction 
Eighty-four percent of Mental Health Professional Shortage Areas are located in rural and 
frontier areas.1 Children and adolescents in rural settings are less likely to receive services 
compared to their urban and suburban counterparts and even fewer are likely to receive 
evidence-based care.2,3 Schools have become more involved in the delivery of mental health 
services and hold great potential for increasing access for children and adolescents. 
Innovations in training and service delivery are needed to improve mental health care quality 
and availability in rural schools.4,5 Evidence-based practices (EBPs) can be incorporated into 
school-wide multi-tiered systems that are currently being used to improve school climate 
and safety. School-wide Positive Behavioral Interventions and Supports (PBIS), a service-
delivery strategy based on the public health model is one example. 6-8 A growing number of 
schools in rural areas are employing PBIS. 9-12 Given the large service disparities for 
children in rural areas, offering EBPs through PBIS can improve access and lead to better 
long-term outcomes. 13 
PBIS is a multi-tiered framework for defining and organizing services, including mental 
health services. Emphasis is placed on teaching key behavioral expectations and routines to 
all students. Two key features of PBIS are its scalability and its compatibility with an array 
of mental health EBPs. Tier 1 strategies focus on preventing new cases of problem behaviors 
by using universal strategies such as effective instructional practices, classroom behavior 
management, and school-wide discipline. Tier 2 strategies focus on supporting children 
with, or at risk for, mental health disorders, and Tier 3 strategies offer individualized support 
for more severe cases. In urban schools, school-wide PBIS has been used to incorporate 
EBPs at Tier 2 and Tier 3.14-16 Previous research has demonstrated that school personnel, 
with or without prior mental health training, can implement Tier 2 interventions with fidelity 
and clinical effectiveness if given adequate consultation support.15,16,17,18 In urban and 
suburban schools, this support (training plus ongoing consultation) can be provided to 
school staff on site. However, providing ongoing on-site support is not feasible or 
sustainable in the majority of rural schools, due to their remote physical location. For this 
reason, video technology has been recommended for the training of behavioral health staff 
(BHS) in under-served rural communities. 19,20  
1.2 Name and Description of Intervention 
1.2.1 Remote Video Training Strategy Development  
We will use a remote training strategy to train BHS to implement EBPs to students in 
schools serving rural areas. We have obtained approval from the IRB for Phase 1 of this 
study (protocol ID: 20-017895). The remote video training strategy was developed during 
Phase 1 (years 1-2) of this 5-year grant using a community-based participatory research 
(CBPR) 21,22 approach to assess barriers and facilitators to participation in a remote training. 
We  used an iterative development framework to build the training strategy. The framework, 
rapid prototyping, originally used for software development, 23,24 is based on a cyclic 

 
  
 
 
 
3
process of analyzing data from users in order to improve successive prototypes. Applied to 
the development of psychosocial interventions, prototyping involves the creation of a “low 
fidelity” version of the intervention that contains key functions of interest in order to test a 
concept and facilitate rapid evaluation and feedback. 25 Following the evaluation of the early 
prototypes, a fully functional “high fidelity” prototype is created that is more similar to the 
final product and that offers fully interactive content.26   
1.2.2 Asynchronous and Synchronous Trainings for Implementation of EBPs 
Our online strategy, which will utilize WebEx video technology, will offer protected access 
to asynchronous training materials such as training “modules” that include didactic content, 
audio and visual examples, as well as treatment materials and resources, all of which can be 
viewed at the trainee’s individual pace and convenience. The remote training strategy will 
also allow synchronous training and consultation for the implementation of EBPs by school 
staff supported by expert consultants, and communication between consultants and school 
personnel to coordinate services in a child-centered approach for student participants who 
are already receiving other services at school. 
 
1.2.3 Benefits of a Remote Training Strategy 
Advantages of web-based remote training include flexibility, accessibility, cost-efficiency, 
potential for both didactic and interactive learning, and consistency in quality. 20, 27 Remote 
online training allows for synchronous (i.e., interactive) supervision and feedback from a 
supervisor anywhere in the country (or the world). This allows for the trainee to be able to 
receive ongoing consultation or supervision on site without the time and cost of travel. 27-29  
Though findings are preliminary, remote technology-enhanced programs have been found to 
be acceptable and feasible in community settings. 30,31 Studies suggest that technology-based 
training methods, particularly when developed using the latest multimedia and interactive 
design formats, may be more effective than manuals alone and as effective as face-to-face 
training workshops in disseminating EBPs to community mental health professionals. 20,31-33  
1.2.4 Tier 2 Interventions  
CBT for Anxiety Treatment in Schools (CATS): CATS is an adaptation of Friends for Life 
(FRIENDS). 34 The adapted protocol retains the core elements of evidence-based CBT for 
anxiety and the FRIENDS group format. We implemented planned adaptations to the 
protocol based on our collective experience with the protocol, two previous implementation 
studies, 17, 35 and focus groups and qualitative interviews with stakeholders. We followed 
procedures developed by Lee and colleagues 36 and Bernal et al. 37 regarding the 
appropriateness of FRIENDS for the target population. We conducted qualitative interviews 
with five consultants and six focus groups with school counselors who used FRIENDS. We 
made changes to the language (idioms, metaphors, words), cultural fit (cultural values), 
methods (session length [40 minutes], number of sessions [8 sessions]), and activities (in-
session practices), which resulted in additions and substitutions in these areas, while 
maintaining the 5 essential components of the treatment. 38,39 This resulted in a briefer (8-
session) and more feasible, engaging and culturally appropriate protocol for urban under-
resourced schools than the original FRIENDS.   

 
  
 
 
 
4
Coping Power Program (CPP): Coping Power Program is an evidence-based intervention 
designed for students with externalizing behavior disorders. CPP is comprised of twelve 45-
minute sessions. This EBP has been shown to be effective at reducing aggressive behavior 
and substance abuse among aggressive boys, with gains maintained at one-year follow-up. 40 
Growth curve analyses showed that CPP had linear effects for three years after intervention 
on reductions in aggressive behavior and academic behavior problems. 41 
Check-in/Check-out (CI/CO):  CI/CO is a targeted, Tier 2 intervention for students at risk of 
developing externalizing and internalizing mental health disorders. 42,43 CI/CO is designed to 
provide immediate feedback (i.e., at the end of each class period) to students, based on the 
use of a daily report card. This feedback is developmentally sensitive. 42,43 CI/CO 
implementers meet individually with students for a brief ‘check-in’ in the morning and a 
brief ‘check-out’ in the afternoon. Research on the use of CI/CO has shown it to be effective 
in reducing externalizing and internalizing problems with elementary school students.43,44,45  
CI/CO will be offered to individual students for a variable period of time, depending on 
need.  
1.3 Findings from Non-Clinical and Clinical Studies  
We conducted a needs assessment and resource-mapping study in Pennsylvania schools that 
currently implement PBIS. According to the Pennsylvania Positive Behavior Support 
(PaPBS) Network, 171 elementary and middle schools in Pennsylvania are currently 
implementing PBIS with fidelity. We asked a member of each school leadership team at 85 
schools, randomly selected from the pool of 171 schools, to complete an electronic survey 
about their school’s PBIS program (e.g., How do the PBIS and mental health teams’ 
function?). Participants completed the Survey on School Readiness for Interconnecting 
Positive Behavior Interventions and Supports and School Mental Health.46 We collected 48 
responses (56% response rate) from PBIS Leadership Team members. Schools were 
classified as being located in a suburb (40%), rural area (35%), town (15%), or city (10%).47 
Compared to non-rural schools (N = 31), rural schools (N = 17) showed a significantly 
higher Total Readiness 48 for interconnecting PBIS with mental health (t = 2.34, p < .05). 
Scores for all rural schools were within the “ready” range for interconnecting with mental 
health supports.22 Also, rural schools reported having significantly more staff support for 
PBIS and school mental health programs, on average, compared to non-rural schools (t = 
2.49, p < .05). Of the rural schools, all reported offering Tier 2 services, including 77% 
Social Skills Training, and 47% Check and Connect. None of the rural schools reported the 
use of manualized group-based mental health interventions or interventions for internalizing 
disorders. Approximately 65% of participants in rural schools reported receiving training, 
supervision/consultation for Tier 2; however, these implementers received initial training 
only (e.g., conferences, in-service professional development) with little ongoing or follow-
up consultation or supervision. The study shows that rural schools have the capacity and 
readiness to implement Tier 2 mental health EBPs because all of them are already offering 
Tier 2 services. However, there is a clear need for training and consultation on manualized 
group and individual interventions for externalizing and internalizing mental health. 

 
  
 
 
 
5
1.4 Relevant Literature and Data 
1.4.1 Mental Health Services in Rural Schools 
There is a pressing need to increase capacity to provide quality mental health services in 
rural schools. Rural areas have fewer and less well trained health care providers than non-
rural areas. 2 Problems specific to mental health services in rural schools include staff 
turnover 23 and low levels of funding for mental health services. 50 Even though most rural 
schools have some school personnel with mental health training, the approach most schools 
take with regard to dealing with mental health problems is to assess and refer to an outside 
provider. Given the shortages of mental health providers in most rural communities, this 
option simply leads to children not being served. The second option taken by BHS in rural 
schools is to assess and treat students individually. However, given the high need for 
services, BHS in rural schools are not always able to see individual students for more than a 
brief period of time and many children in need of services never receive help. 47 . There is 
mounting evidence that school-based mental health programs can effectively address 
emotional and behavioral disorders. 51 A number of comprehensive school-wide prevention 
approaches have been found to have a positive effect on an array of behavioral, emotional 
and academic outcomes. One such approach is school-wide PBIS 52, a multi-tiered 
framework for defining and organizing practices and interventions. 53 One potential 
innovation to increase access to services is to incorporate EBPs into PBIS. 6-8 PBIS 
programs can also incorporate targeted group-based mental health EBPs for children at risk 
(Tier 2) and individualized EBPs for more severe cases (Tier 3). A growing number of 
schools in rural areas are employing PBIS at Tier 1. 9-12 However, rural schools are not able 
to provide Tier 2 support unless BHS are given adequate training and consultation on EBP 
implementation.    
The problems of shortage of providers with appropriate training in EBP implementation and 
the common practice of assess-and-refer can be addressed via remote training of school 
personnel and by integrating mental health services into existing service delivery strategies. 
Remote video technology offers the potential to provide training and consultation for 
behavioral health staff in rural schools. Based on our reading of the relevant literature and 
our collective experience developing programs in the school setting, we propose that the 
development of a training approach for BHS in rural school settings ought to (a) include a 
training system for BHS to enhance knowledge and skill needed for implementation, (b) use 
remote video technologies, and (c) incorporate implementer and school context factors to 
increase the strategy’s perceived feasibility, appropriateness and acceptability by 
stakeholders.   
 
1.5 Compliance Statement 
This study will be conducted in full accordance with all applicable Children’s Hospital of 
Philadelphia Research Policies and Procedures and all applicable Federal and state laws and 
regulations including 45 CFR 46. 21 CFR Parts 50, 54, 56, 312, 314 and 812.  
The investigators will perform the study in accordance with this protocol, will obtain 
consent and assent (when appropriate), and will report unanticipated problems involving 
risks to subjects or others in accordance with The Children’s Hospital of Philadelphia IRB 

 
  
 
 
 
6
Policies and Procedures and all federal requirements. Collection, recording, and reporting of 
data will be accurate and will ensure the privacy, health, and welfare of research subjects 
during and after the study.  
2 
STUDY OBJECTIVES 
2.1 Primary Objective (or Aim) 
The purpose of this study is to conduct a pilot randomiozed controlled trial comparing three 
training strategies: Control Condition (CC), Remote Video (RV), and Remote Video plus 
Coaching (RV+).   
 
2.2 Secondary Objectives (or Aim) 
The secondary objectives are the evaluation of implementation and student outcomes. 
• To explore mediators and moderators of consultation support on behavioral health 
staff fidelity. 
• To qualitatively examine school staff’s perceived feasibility, appropriateness and 
acceptability of the support they received.  
3 
INVESTIGATIONAL PLAN 
3.1 General Schema of Study Design 
This study is a pilot randomized controlled trial. The three arms are: (a) Initial remote 
training (b) Initial remote training, plus video and (C) Initial remote training, plus video, 
plus remote coaching. Each school will be instructed to select CICO and one of the two 
group EBP protocols for use. 
 
3.1.1 Pre-Trial Activities and Screening Phase 
The Remote Training Strategy was developed during Phase 1 of this 5-year project using an 
iterative process (Rapid Prototyping) to develop the appropriateness, feasibility, acceptability 
and preliminary efficacy of the online training platform.     
School Selection: The National Center for Education Statistics (NCES) subdivides rural and 
town school areas by their proximity to an urbanized area into fringe, distant, or remote 
locations.47 Participant schools will be located in any of these three rural locations or town, 
remote area and/or the school will serve rural communities. We will invite 24 schools 
representing the 2 geographic locations (urban/rural) to participate in the pilot efficacy study. 
After receiving school approval to participate schools will be stratified based on the 
geographic location and a computer-generated block randomization list will be prepared to 
randomize the 24 schools in a 1:1:1 ratio to either RV, RV+ or CC (8 schools/arm) stratified 
by geographic locations (urban/rural). The block randomization with varying will ensure 
ensure equal number of schools in each arm and eleminate the predictability of 
schools’assignment.   

 
  
 
 
 
7
Behavioral Health Staff Screening: Recruitment will begin with assistance from the school 
principal who may nominate school behavioral health staff to participate in the study. 
Additonally, the research staff will reach out to school behavioral health staff concurrently 
with the school principal. The research staff will speak with school staff by phone or via 
WebEx to clearly explain study procedures, as well as potential risks and benefits. BHS will 
be clearly informed of the voluntary nature of their participation and will receive ample 
opportunities to ask questions. A copy of the consent form will be provided to the BHS, and 
consent will be obtained electronically via REDCap. 
Student Screening: Recruitment of child participants will begin with research staff assisting 
school behavioral health staff who will examine teacher nomination data for students in 
specific grades and classes in order to identify students for Tier 2 interventions. We will 
train BHS remotely on providing a brief in-service training to teachers tasked with 
nominating students on recognizing warning signs of excessive anxiety, depression, and 
disruptive behavior problems in children, and completing the screening measures. Following 
the remote training, BHS will ask teachers from specific grades or classes to nominate 
students. First, teachers will be asked to rank order students and then select the top three 
students with externalizing problems and the top three students with internalizing problems. 
Second, parents and teachers will complete the SDQ (paper or electronically) on these 
identified students. Third, parents and teachers will give the SDQ forms to the BHS 
member (when applicable). Fourth, BHS will scan the SDQ to CHOP staff to score. Fifth, 
students who score above the specified cut-off on the externalizing or internalizing 
dimension will be selected for the next step, which will require BHS to contact the parents 
and request that they complete the SCARED or to ask a teacher of the student to complete 
the SAEBRS. Sixth, children who score at least 25 or there is an elevated subscale for the 
SCARED or if the student score on the Social Behavioral subscale of the SAEBRS is 12 or 
less, they will be considered for participation in one of the groups or in CI/CO. The 
research aspect of the project will begin at this point. Children who meet inclusion 
criteria will be invited to participate in the study. Research staff will ask parents to provide 
consent for their child to participate in the intervention. The children wil also be asked to 
provide assent to participate in the intervention. Informed consent (parent) and assent (child) 
will be obtained by research staff prior to enrollment.  Participating BHS will facilitate 
consent meetings with eligible children and their parents. A research staff member will join 
these meetings by phone or via a videoconference.  
 
3.1.2 Phase 1: Training of Behavioral Health Staff  
All initial trainings will be lead by research staff and will be conducted remotely via Webex. 
These initial trainings wil be recorded and made available to all participants to reference for 
the duration of their participation in the study.  
Initial Red Flags and Screening Training: Research staff will train school staff from all 
conditions remotely on identifying “red flags” that a student is struggling with anxiety, 
anger, or other mood disorders and how this information can help Tier 2 Team members and 
teachers make informed referrals for the interventions. Study participants should use this 
information to hold a brief professional development with school staff to help guide their 
referrals. Additionally, participants will be taught about how the referral and screening 

 
  
 
 
 
8
process operates, including who is responsible for each aspect of the process, how the 
screening measures are scored and what those scores mean. We will train BHS using 
strategies that have been proven effective, including the teaching of the theoretical approach 
used in the intervention, active demonstration and practice of skills, and coaching. 56,57  
Initial Evidence-Based Programs Training: Research staff will train school staff from all 
conditions remotely on the three EBPs. The three EBPs that schools staff will be trained on 
are Coping Power Program (CPP) 58 for externalizing behavior disorders, CBT for Anxiety 
Treatment in Schools (CATS) 59 for anxiety disorders, and Check-in/Check-out (CI/CO) 60 
for externalizing and internalizing 61 disorders. Three separate trainings will be held to 
introduce participants to the interventions and provide an overview and understanding of 
each EBP.  
Video Training Modules: We will offer protected access to asynchronous training materials 
to BHS in the Remote Video and Remote video plus coaching conditions. These trainings 
will include “modules” that are comprised of didactic content, audio and visual examples, as 
well as treatment materials and resources, all of which can be viewed at the trainee’s 
individual pace and convenience. Participants will have access to the training modules via 
CHOP OPEN and they will be able to login using a unique username and password.  
3.2 Study Duration, Enrollment and Number of Sites 
Behavioral Health Staff: BHS will be asked to commit to participate in the study for one 
school year. The school the BHS works at will be randomly selected as a participating 
school for either years 1,2, or 3 of the study. BHS will be free to implement the EBPs 
outside the research project after their participation in the study has concluded. 
Students: Students who are eligible, based on the SDQ, SCARED, and SAEBRS, and for 
whom consent/assent is obtained, will participate in the study for the time it takes them to 
receive one of the interventions. Participation includes receiving interventions consisting of 
CI/CO, CPP, or CATS. The interventions will consist of one 40-minute/class period 
intervention session conducted at school or two brief daily check-ins also conducted during 
the school day. Post-treatment data will be collected by a parent/caregiver and teacher using 
the BASC-3 and Engagement versus Disaffection with Learing Teacher Report (EvsD-
Teacher). Students will be asked to complete the Behavior and Feelings Survey. 
Administrators: School principals and/or administrators will be asked to complete a 
qualitative interview about their experience with the project.. The interview will occur at the 
end of the academic year and their participation will last the time it takes to complete the 
interview 
3.2.1 Total Number of Study Sites/Total Number of Subjects Projected 
We will conduct the pilot trial in 24 schools (8 in Control Condition, 8 in  Remote Video, 
and 8 in Remote Video plus Coaching.  We anticipate enrolling 24 administrators (1 per 
school), 72 behavioral health staff (3 from each school; 24 in CC, 24 in RV, 24 in RV+) and 
312 students (104 in CC, 104 in RV, 104 in RV+). We expect that each school will have one 
staff member with prior mental health training (e.g., school counselor). This staff member 
will be expected to implement one of the group-EBPs. The two other staff will be tasked 

 
  
 
 
 
9
with implementing CICO, as this intervention can be implemented by school staff without 
prior mental health training. 62,63 We estimate that each school staff will conduct one CPP or 
CATS group with 5-6 students of similar developmental level (e.g., a group of 4th & 5th 
graders; 144 students total), and that each school staff in CICO will implement the 
intervention with 6 individual students of any school grade (144 students total).  
3.3 Study Population 
3.3.1 Inclusion Criteria 
Administrator: Any school principal or assistant principal from participating schools 
implementing PBIS.  
Behavioral Health Staff: Any counselor, social worker, or teacher from participating schools 
implementing PBIS who work with students in grades 4-8.  
Students: Inclusion of students to receive a Tier 2 intervention is as follows:  
• 
attending one of the participating schools 
• 
being in grades 4-8 
• 
identified by the Tier 2 team as not responding to Tier 1 intervention, thus 
needing Tier 2 support 
• 
scoring ≥ 1 SD above the mean on the Emotional Symptoms or Conduct 
Problems scales of the Strength and Difficulties Questionnnaire (SDQ) 54 
completed by a parent or a teacher 
The cut-off score level for the SDQ is appropriate for identifying students at risk for a 
behavioral/mental health disorder54,55. We limit participation to students in grades 4-8 
because the group EBPs are appropriate for this age group. 
3.3.2 Exclusion Criteria 
Administrator: School staff who are not principals or assistant principals.  
Behavioral Health Staff: School staff who are not part of the PBIS team and who do not 
work with students in grades 4-8.  
Students: Students who do not meet screening or group participation criteria will not be 
included in the study. Students with a history of intellectual disability or serious 
developmental delays according to school records will not be included because they would 
be unlikely to benefit from the interventions used in the study. Students with a history of 
psychotic or autistic spectrum disorders as reported by parents will not be included.  
Subjects that do not meet all the enrollment criteria may not be enrolled. Any violations of 
these criteria must be reported in accordance with IRB Policies and Procedures.  

 
  
 
 
 
10 
4 
STUDY PROCEDURES 
4.1 Screening and Consent Procedures 
Please refer to Section 3.1.1 for detailed description of Screening and Consent Procedures.  
Teachers will assist with screening procedures by completing screening measures for 
potentially eiligible students in their class. Teachers are not considered study participants.  
4.2 Phase 1: Training of Behavioral Health Staff 
4.2.1 Initial Synchronous Trainings 
Research consultants will conduct a synchronous (live) training workshop at the beginning 
of each year via a video-conferencing platform (Webex) for behavioral health staff. BHS 
will be instructed on the use of data to identify and assign students at risk for behavioral and 
emotional disorders into Tier 2. The BHS will also be taught a competency framework for 
mental health and PBIS77,78 strategies for enhancing school personnel knowledge of mental 
health “warning signs” among students, and how to access the online materials. The training 
related to “warning signs” will be conducted in order to help BHS identify students who 
could benefit from the interventions. 
BHS will be instructed to use training manuals and will be trained on the use of a mental 
health screening instrument (SDQ)54 and a multi-axial parent rating scale (Behavioral 
Assessment System for Children, Third Edition (BASC-3)68 and other instruments used in 
the study. BHS will be introduced to a competency model for CBT79 . They will also learn 
about how to deal with implementation barriers (e.g., scheduling sessions, conducting 
exposure tasks)80. Training content and procedures will be based on adult learning 
characteristics (e.g., propensity to learn from experience, capacity to reflect on performance 
and apply knowledge, and selfmotivation)81,82. 
4.2.2 Guided Video for Remote Video and Remote Video plus Coaching  
Following the initial training workshop and after BHS have identified students for 
interventions, BHS in the RV and RV+ conditions will be given access to the training 
videos. The videos will be made available through CHOP OPEN that has the ability to 
measure how many times each video has been accessed by the BHS and for how long. Each 
participant will be given access to the videos that correspond to the interventions that they 
plan to use. All Tier 2 implementers in RV and RV+ will be given access to videos that 
address general topics (e.g., how to use data to identify students for participation; dealing 
with implementation barriers).  
4.3 Phase 2: Implementation of EBPs 
We expect that each school will have one staff member with prior mental health training 
(e.g., school counselor). This person will be expected to implement one of the group-EBPs. 
The two other staff members will be tasked with implementing CI/CO, as this intervention 
can be implemented by school staff without prior mental health training.63, 68 We estimate 
that the staff member with mental health training will conduct one CPP or CATS group with 
5-6 students of similar developmental level (e.g., a group of 4th & 5th graders; 144 students 

 
  
 
 
 
11 
total), and the other implementers will lead CI/CO and will implement the intervention with 
6 individual students of any school grade (144 students total). Following the initial live 
training, BHS will be instructed to conduct targeted screenings with the grade group they 
would like to select for participation in the study.  
Research consultants will provide technical assistance to support behavioral health staff. The 
three EBPs that schools will use for Tier 2 are the Coping Power Program (CPP) 58 for 
externalizing behavior disorders, CBT for Anxiety Treatment in Schools (CATS) 59 for 
anxiety disorders, and Check-in/Check-out (CI/CO) 60 for externalizing 61 disorders. CPP 
and CATS will be implemented in group format with students of similar developmental 
level (e.g., 4th & 5th or 7th & 8th grade together) during the lunch or free period. BHS 
could also use CI/CO for individual students who do not meet inclusion criteria for CPP or 
CATS. Each school will be instructed to select CI/CO and one of the two CBT protocols for 
use. 
4.3.1 Remote Consultation for Remote Video plus Coaching  
BHS will receive a study-purchased audio recorder to use to record the student groups.They 
will be instructed to upload the recording to OneDrive. Research team consultants will 
provide synchronous consultation to BHS in the RV+ condition via Webex. The consultation 
will have two main components: (a) session preparation (CPP/CATS) or review and 
planning (CICO) and (b) coaching. 
Session Preparation: Session preparation for CPP and CATS will consist of (a) discussing 
referrals to the groups; (b) conducting a step-by-step walkthrough of the session objectives, 
(c) reviewing the CBT principles behind the treatment components for that session; (d) 
encouraging adherence and the use of active learning strategies; (e) problem-solving barriers 
to implementation and helping implementers reflect on past challenges (e.g., attendance 
problems, organizational barriers, materials/resources) in order to successfully implement 
the next session with appropriate adaptations as needed; and (f) enhancing BHS’s use of 
empathy and positive reinforcement through modeling. Research consultants for CICO will 
(a) review the school’s CICO implementation manual and (b) provide feedback and 
suggestions for program implementation.  
Coaching: Coaching for CPP and CATS will consist of (a) goal setting, 65 (b) self-reflection, 
66 and (c) performance feedback. 67 BHS will be told that they are expected to reach high 
fidelity when implementing the intervention. Then, implementers will be asked to reflect on 
the previous session (e.g., “How do you think you did during the last CPP/CATS session? 
What do you think went right? What do you think did not go well?”). Finally, the consultant 
will use audio clips from the previous session that contain examples of well-executed 
implementation objectives and brief audio clips that contain ineffective or less than optimal 
execution of objectives. The consultant will also discuss how the implementer handled 
student behavior in session, including overall level of participation and enthusiasm, and 
disruptive or withdrawn behavior. The audio clips will be contained in a OneDrive server 
uploaded by implementers. All consultation procedures will be detailed in a consultation 
manual. Coaching for CICO will consist of (a) providing performance feedback to the CICO 
coordinator and data analyst about their program (e.g., use of data to refer students to CICO, 
student progress monitoring) and (b) problem solving implementation barriers. 

 
  
 
 
 
12 
 
4.4 Subject Completion/Withdrawal 
Subjects may withdraw from the study at any time without prejudice to their care, education 
or employment.  They may also be discontinued from the study at the discretion of the 
Investigator for lack of adherence to the study and/or development of exclusionary criteria.  
Student participants will be removed from the study if they develop exclusionary criteria 
during the trial or if it is determined that the intervention group is not age/developmentally 
appropriate based on the grade level of the majority of enrolled students.  It will be 
documented whether or not each subject completes the clinical study. If the Investigator 
becomes aware of any serious, related adverse events after the subject completes or 
withdraws from the study, they will be recorded in the study files. 
5 
STUDY EVALUATIONS AND MEASUREMENTS 
5.1 Screening Measures  
Screening for Tier 2: The Strengths and Difficulties Questionnaire plus Impact Supplement 
(SDQ).54 The SDQ is a 25-item, 3-point scale questionnaire used to assess the psychological 
adjustment of children and youth, ages 4-17. 54 The SDQ has excellent psychometric 
properties as a screening instrument for behavioral and emotional problems. 54 We will use 
the scores on the Conduct Problems and Emotional Problems 62 scales plus impact 
supplement scores (impairment) for screening students for Tier 2.  
The Screener for Child Anxiety Related Disorders (SCARED): is a 41-item questionnaire 
used to screen for children with anxiety disorders, using five sub-scales and a Total Score. It 
has excellent psychometric properties and has been used in community settings as a 
screening instrument for anxiety disorders. 
 
The Social, Academic, and Emotional of Behavioral Risk Screener (SAEBRS): is a 19-item, 
psychometrically sound screening instrument for assessing emotional and behavioral risk. 
The SAEBRS is completed byschool staff using a 4-point Likert scale.  We will use the 
“Social Behavior” subscale of the SAEBRS to determine eligibility for CPP and CICO. 
5.2 Demographic Information 
Demographic information for students will be collected at baseline from a parent. 
Demographic information for BHS will be collected via a REDCap survey following 
consent to participate.  
Demographic information for administrators will be collected via a REDCap survey 
following consent to participate.  
5.3 Implementation Fidelity  
Content Fidelity to Tier 2 Group CBT (CPP & CATS). The Content Fidelity Checklist 
(CFC) reflects each activity component of the session agenda of the treatment protocols. 
Raters use a yes/no response scale to indicate whether or not the implementer covered a 
particular component. All GCBT sessions will be audio-recorded. 100% of sessions will be 

 
  
 
 
 
13 
coded by the first coder and approximately 20% of sessions will be randomly selected and 
coded by the second coder. Interrater reliability will be evaluated using Kappa coefficients69   
Process Fidelity to Tier 2 Group CBT (CPP & CATS). Process fidelity will be assessed 
via an adapted version of the rating system developed by Lochman and colleagues. 70 Ten 
items are rated on a scale of 0 to 4, with 0 being "Not At All" and 4 being "Very Often". 
Ratings are given on the extent to which school staff members delivered the intervention in 
an orderly fashion, using active learning strategies. etc. Exploratory factor analysis (EFA) 
yielded two factors on the 10-item scale. We will use the average Total and factor scores for 
analysis. 100% of sessions will be coded by the first coder and approximately 20% of 
sessions will be coded by the second coder. Interrater reliability will be evaluated using ICC.   
Content Fidelity for CI/CO. Content fidelity for CI/CO will be assessed via the CI/CO 
Implementation Guide Fidelity Checklist (IGFC). The CI/CO IGFC is completed by the 
consultant (research staff) after school teams submit their completed CI/CO Implementation 
Guide.  The IGFC is completed once and measures both the content and quality of the 
customized manual components for each school. The CI/CO IGFC/Content Measure is a 26-
item measure of CI/CO customization of the following CI/CO components: (a) staff roles 
and responsibilities, (b) CI/CO process logistics, (c) daily progress report, (d) motivation 
system, (e) fading and self-monitoring procedures, (f) staff training procedures, and (g) data 
collection and progress monitoring procedures. Each item is scored according to completion 
status as “yes,” “no,” or “not applicable.” The CI/CO IGFC/Content Measure will be scored 
by recording the frequency of “yes” scores over the frequency of “yes,” and “no” scores. 
The CI/CO IGFC/Quality Measure, or part 2 of the IGFC/Content Measure, is also 
completed by the consultant (research staff) once teams submit their completed CI/CO 
Implementation Guide. The IGFC/Quality Measure is a 14-item measure of the quality of, or 
degree to which, team responses in customizing their respective CI/CO Implementation 
Guides are comprehensive, clear, feasible. Measurement entails examining team responses 
to items related to the (a) implementation roles and responsibilities, (b) implementation 
process, (c) motivation system, (d) fading and self-monitoring, (e) training/orientation of 
staff, mentors, students and families, and (f) data entry and progress monitoring. The 
IGFC/Quality Measure will be scored by recording a score of 0-3 for each item depending 
on whether or not the team response meets the criteria for being comprehensive, clear, and 
feasible (with 1 point awarded for each). 
Process Fidelity to CI/CO. Process fidelity for CI/CO will be assessed with the Check-
In/Check-Out Fidelity Checklist, a measure of the daily implementation of CI/CO. The 
CI/CO Fidelity Checklist is completed by the consultant (research staff) on a monthly basis 
after collecting the completed Daily Progress Report (DPR) student forms for each student 
involved in CICO. The Check-in/Check-Out Fidelity Checklist is an 11-item form that 
indicates whether or not students received daily CICO feedback based on the occurrence of 
the following: (a) student identification, (b) feedback date, (c) recorded goal, (d) behavioral 
expectations rated by the teacher for each period, (e) the number of daily points earned out 
of the number of possible points, (f) an indication of whether or not the goal was met, and 
(g) the signature of the teacher/mentor. The 11 items will be scored as “occurred” or “did 
not occur.” The Check-In/Check-Out Fidelity Checklist will be scored by recording 
occurrence over occurrence-plus-non-occurrence and multiplying the decimal by 100. 

 
  
 
 
 
14 
Adoption Inventory (AI). The AI will track the number of times each intervention is used 
per school, per condition. This information will be collected one time at the end of the 
intervention via a REDCap survey.  
Dosage Inventory (DI). The DI is an Excel tracking sheet exported from the website 
hosting the video modules listing the number of times and length of time each video module 
is accessed by BHS. 
Penetration Inventory (PI). The PI will list EBP penetration at the student level (students 
receiving EBPs at Tier 2). This information will be collected one time at the end of the 
intervention via a REDCap survey.  
5.4 Student Outcome Measures  
Parent, teacher, and student measures will be collected at pre-treatment and post-treatment 
of the pilot trial to assess changes in mental health symptoms and academic engagement. 
Mental health symptoms. The BASC-3 55 is a 138-item, 4-point, Likert-type rating scale 
for assessing parental report of child mental health functioning, standardized for ages 2.5 to 
18 years. The BASC-3 has excellent psychometric properties. 55 The BASC-3 (Aggression, 
Conduct Problems, and Anxiety scales) will be administered at pre- and post-treatment for 
children who participate in CPP, CATS or CICO. Parents will complete the paper and pencil 
version of the BASC-355 over the phone with a research assistant or they may complete 
electronically. When this measure is administered to Spanish proficient parents, the research 
assistant will call language services to connect to an interpreter and the measure will be 
completed by phone.  
Student academic engagement.  The Engagement versus Disaffection with Learning: 
Teacher Report (EvsD) 71 will be completed by teachers for all children receiving Tier 2 
interventions. This is a 20-item, four-point instrument with four sub-scales: (a) Behavioral 
Engagement, (b) Emotional Engagement, (c) Behavioral Disaffection, and (d) Emotional 
Disaffection. The survey will be completed by hand or electronically via REDCap survey. 
We will use the average score for each of the four scales at pre- and post-participation in 
CPP, CATS or CICO. 
Externalizing and Internalizing Problems: Children will complete The Behavior and 
Feeling Survey-Your Report (YFS-SR). The YFS-SR is a brief (10 items) measure of 
internalizing and externalizing problems rated on a 5-point scale. The YFS-SR has excellent 
psychometric properties and it is sensitive to change.50 The YFS-SR provides three scores 
(Internalizing, Externalizing, Total). The measure will be completed at pre- and post-
treatment for children who participate in  group CBT or CICO. 
 
5.5 Remote Training Strategy Feedback  
Perception of Training Support. Qualitative Interview. We will conduct semi-structured 
qualitative interviews with BHS and administrators in each condition. These interviews will 
be audio-recorded and conducted over the phone or via Webex and should last 
approximately 30 minutes in length. Teachers will not be interviewed. We will seek to 
interview one administrator at each school. We will utilize purposive sampling with BHS, 

 
  
 
 
 
15 
focused on variation in interventions implemented and levels of experience. In interviews 
with administrators, we will utilize an interview guide designed to understand existing Tier 
2 programs at their school as well as their perceptions of the training support’s feasibility, 
acceptability, appropriateness, and usability. In interviews with BHS, we will utilize an 
interview guide that seeks to elicit views and perspectives about the perceived feasibility, 
acceptability, appropriateness, and usability of the training support they received. The audio 
recorded interviews will be transcribed by ADA transcription services. We will conduct a 
thematic analysis of these data and compare across conditions. 
6 
STATISTICAL CONSIDERATIONS 
The statistical analysis plan (SAP) will be updated and finalized before the data base lock. 
The SAP will provide comprehensive descriptive information of the statistical analysis plan, 
including approaches for summarizing primary and secondary endpoints at baseline and post 
treatment. All statistical analyses will be performed using SAS®, 73version 9.4 or higher 
6.1 Primary and SecondaryEndpoints 
The primary endpoints related to school staff implementing the interventions are measures 
of content and process fidelity. Primary endpoints related to student outcomes are pre-to-
post changes in student mental health symptoms (measured by Behavior Assessment System 
for Children- 3rd Edition (BASC-3),55  which include Aggression, Conduct Problems, 
Anxiety, and Academic engagement measured by Behavioral Engagement, Emotional 
Engagement, Behavioral Disaffection and Emotional Disaffection. 
Secondary endpoints are: the mediators and moderators effects of consultation support on 
behavioral health staff fidelity and the qualitative examination of school staff’s perceived 
feasibility, appropriateness and acceptability of the support they received. 
6.2 Statistical Analysis 
Prior to statistical comparison between groups (RV, RV+, and CC), all pertinent variables 
collected for the pilot study will be reported by groups and presented as mean, standard 
deviation, median, minimum, maximum and the 95% confidence intervals for continuous 
variables, while frequencies and proportions will be used for categorical variables. 
Continuous data measures will also be checked for normality and outliers. Non-normally 
distributed continuous data variables will be examined and transformed “normalized” if 
needed using appropriate transformation functions. Outliers not found to be in error will be 
kept in the analysis. Since the school is the randomization unit, BHS and student 
characteristics (Demographic and other potential confounders will be compared between the 
two groups using the two independent samples t-test or the non-parametric Wilcoxon signed 
rank test) to identify pre-treatment differences between the two groups. If the two groups are 
found to be statistically different in a pre-measured outcome, the pre-measurement(s) will be 
included in the subsequent analyses as a covariate using analysis of covariance (ANCOVA). 
6.3 Statistical Methods  
Data will be analyzed using an intent-to-treat approach, wherein each participant (BHS or 
student) will be kept in the arm to which the school was randomized, regardless of treatment 

 
  
 
 
 
16 
received. In addition to creating pre/post change scores and analyzing the data using t-tests 
(or the Wilcoxon signed rank test), a mixed-effects model will be explored for analyzing the 
pre/post repeated-measures endpoints related to student academic engagement and mental 
health symptoms. A student-level random effect will be included and modeled using the 
unstructured covariance matrix to account for within-subject dependence due to repeated 
measurements. The models will include study arm (RTS or CC) and time of measurement 
(pre/post) as fixed effect. Arm x Time interaction effects will be included in the model as 
well. This modeling approach will allow us to compare pre-to-post changes and the extent to 
which these changes differ across study arms. For the purpose of generating statistical 
estimates for an anticipated phase 3 larger scale study, the nested nature of students within 
the school as well as BHS within the school will be explored using a hierarchal mixed effect 
multilevel model. Furthermore, we will explore marginal models using generalized 
estimating equations (GEE; 74,75), which will produce robust estimates that adjust for 
clustering at the school level. Although SAS PROC MIXED will not remove participants 
with post-missing data from the analysis, missing data will be kept to a minimum. As a 
sensitivity analysis, we will impute the potential missing data using the maximum likelihood 
method for imputation. 76  
For the mixed-methods analyses (survey and semi-structured interview data about perceived 
appropriateness, feasibility, and acceptability of the training and consultation procedures, 
and BHS’s opinions about the support they received), we will use the same data analytical 
approach described in IRB Protocol # 20-017895.  
The statistical analysis related to the secondary aims to examine the mediators and 
moderators effects of consultation support on behavioral health staff fidelity will utilize 
PROCESS85 a modeling tool freely-available for SPSS and SAS. The many features of 
PROCESS integrate many existing functions and published statistical tools for analyzing 
mediation and moderation. The SAS software 83; and the SPSS software 84 will be utilized 
for data analyses.  The qualitative examination of  school staff’s perceived feasibility, 
appropriateness and acceptability of the support they received will be in NVivo and 
analyzed thematically. 
6.4 Sample Size and Power 
The proposed pilot study aims to collect data and estimate effect sizes measuring the effect 
of RV and RV+ when compared to the CC. Although, this pilot study is not powered to test 
for specific hypotheses regarding effect sizes, the sample size estimates were based on data 
obtained from our recently completely NIH funded study, 15,77 we estimated that a total of 72 
BHS (24 per arm) and 312 students will participate in this study. With 24 BHS per arm, a 
two-sided 95% confidence interval in mean differences in content fidelity (CF) and process 
fidelity (PF between) RV or RV+ vs. CC will be produced. The minimum distance from the 
mean difference to the limits (margins of error) will equal to 9.3 (CF) and 0.35 (PF) with the 
estimated pooled standard deviations equal to 16 and 0.6 respectively. It is anticipated that 
104 students per arm will be recruited (total = 312).Assuming  a uniform attrition rate of 
15%,  a total of 268 evaluable students will be included in the final statistical analysis. With 
89 students in each of the three study arms, a two-sided 95% confidence interval for mean 
differences in pre/post changes in student mental health symptoms and academic 
engagement between the two arms ((RV or RV+) vs. CC) will equal to 2.4 with estimated 

 
  
 
 
 
17 
pooled standard deviation equal to 8. Sample size justification was reported using PASS 13 
software. 78 
7 
SAFETY MANAGEMENT 
7.1 Clinical Adverse Events 
Clinical adverse events (AEs) will be monitored throughout the study.  
7.2 Adverse Event Reporting 
Since the study procedures are not greater than minimal risk, SAEs are not expected. If any 
unanticipated problems related to the research involving risks to subjects or others happen 
during the course of this study (including SAEs) they will be reported to the IRB in 
accordance with CHOP IRB SOP 408: Unanticipated Problems Involving Risks to Subjects. 
AEs that are not serious but that are notable and could involve risks to subjects will be 
summarized in narrative or other format and submitted to the IRB at the time of continuing 
review.  
If a participant becomes more symptomatic (based on parent/child report or clinical 
judgment), without resolution for at least four weeks, the study treatment provider will 
review the participant’s clinical status with the project team and PI.  Concerns about 
deterioration will trigger an evaluation conducted by school’s behavioral health staff and 
then discussion with the PI regarding whether discontinuation from the study is in the best 
interests of the child.  Such participants will be removed from the study and treated openly 
with appropriate clinical care.    
In cases where concern about danger to self or others is identified, the behavioral health staff 
– with consultation with CHOP consultants, as needed – will conduct further assessment of 
the child’s mental status and decide on a course of action. If the assessment leads to the 
conclusion that the child is at risk for suicide, the behavioral health staff will work with the 
family to take the child to the emergency room for evaluation and treatment. If the child 
returns to school after visiting the emergency room and it is determined that an outpatient, 
school-based level of care is appropriate, he or she may continue to participate in study 
interventions. If it is determined that a higher level of care is appropriate, the child will be 
withdrawn from the study and will receive the higher level of care.   
If previously undocumented sexual or physical abuse is discovered during eligibility 
assessment, behavioral health staff at the schools will implement standard procedures for 
notifying the State social welfare authorities (per state law and standard procedure in the 
respective school district), and defer further evaluation if needed until the child has been 
released from agency review.  If state involvement is identified or mandated at any point 
during the study, the child may or may not be continued in treatment depending on what is 
in the best interests of the child. This decision will be made by the PI and treating counselor 
at the site in coordination with the child's legal guardians and state/local authorities. If the 
child returns to school after receiving emergency services, he or she will continue to 
participate in interventions in the school setting. 

 
  
 
 
 
18 
8 
STUDY ADMINISTRATION 
8.1 Treatment Assignment Methods 
8.1.1 Randomization 
The 24 invited schools will be recruited from 2 geographic locations (urban/rural). Schools 
will be stratified based on the geographic location and a computer-generated block 
randomization list will be prepared to randomize the 24 schools in a 1:1:1 ratio to either RV, 
RV+ or CC (8 schools/arm) stratified by geographic locations (urban/rural). The block 
randomization with varying will ensure ensure equal number of schools in each arm and 
eleminate the predictability of schools’assignment.  It is estimated that a total of 72 
behavioral health staff (3 staff per school- 24 per arm) and 312 students (13 students per 
school- 104 per arm) will participate in this randomized parallel groups pilot study. 
8.2 Data Collection and Management 
The data collection and management plan is consistent with CHOP Policy A-3-6: 
Acceptable Use of Technology Resources that defines the requirements for encryption and 
security of computer systems.  
• Identifiable data will be collected as part of this study.  This data includes full 
names/initials, date of birth, address, telephone number, e-mail addresses.  However, 
participants will be identified by alphanumeric code only.  This precautionary step 
allows for the electronic transfer of data without using data encryption techniques. At 
each stage of data collection and maintenance, measures are taken to ensure that all 
identifying information is taken out of data archives, and any hard copies of data that 
could identify participants are stored in locked file cabinets with restricted access, 
and that data files are password protected.  Participant identification numbers are 
used that do not reveal the identity of participants (e.g., no use of birth dates, initials, 
social security numbers, etc.). Only members of the research team will have access 
to the data.  If the results of this study are presented at scientific meetings or 
published in professional journals, they will not contain information that could be 
used to identify BHS, or parents/students.  
• Hard data will be kept in locked file cabinets at the Roberts Center for Pediatric 
Research.  Audio recordings (containing participant identifiers) will also be stored in 
locked file cabinets at CHOP.  The files will be transferred and stored to a CHOP 
computer network drive and analyzed by study team members for the coding of 
integrity/fidelity scoring.  After all analyses are complete, the files will be 
maintained for 6 years in accordance with CHOP policy A-3-9, then be destroyed 
and personal identifiers will not be retained with the data.  All computerized study 
databases for questionnaire data will be kept on a secure Windows NT server located 
at one of the hospital’s research buildings. This server is also protected by a firewall 
to reduce the risk of unauthorized access to study information. 
• Remote data collection will only take place via secure video chat platforms and/or 
the telephone.  Study team members will record the data onto paper CRFs in the 

 
  
 
 
 
19 
same manner as in-person data collection, and no digital video or audio recordings 
will be created. 
• Data collected as part of this study will be entered (data entry from CRFs or upload 
from Excel files) and stored using REDCap (Research Electronic Data Capture) 
database, a secure web-based software database supporting clinical and translational 
research databases. The database will be password-protected, stored, and backed up 
on a daily basis by CHOP’s Research Institute.  REDCap provides data management 
functionality; including automated export procedures for seamless data downloads to 
Excel and commonly used statistical packages (SPSS, SAS, Stata, R). The database 
will incorporate range checks and between-variables consistency checks to ensure 
quality control. The system will signal the presence of questionable or potentially 
incorrect items.  After data cleaning and quality assurance procedures are completed, 
pertinent sets of data will be converted into SAS format for statistical analysis. The 
Biostatistics and Data Management Core (BDMC) of the Research Institute will 
provide staff expertise in REDCap database design and implementation, 
randomization, and statistical analysis under the direction of the study’s 
biostatistician. 
8.3 Confidentiality 
All data and records generated during this study will be kept confidential in accordance with 
Institutional policies on subject privacy and that the Investigator and other site personnel 
will not use such data and records for any purpose other than conducting the study.  
The following steps will be taken to maintain confidentiality:  (1) the interview protocols and 
case record forms will be kept in a locked file cabinet in the secure lab space; (2) rating scales 
will be collected using REDCap (3) subject identity will be de-identified using alphanumeric 
codes keyed to a master list; (4) de-identified data (i.e., no names or other possible identifiers) 
will be entered directly into a database that will be password protected; (5) all computerized 
study databases for questionnaire data will be kept on a secure CHOP server located at one of 
the hospital’s research buildings which is password and firewall protected to reduce the risk 
of unauthorized access to study information; (6) all project staff will be required to complete 
training in research protocol and Human Subjects safety and privacy before starting, and will 
promise in writing to protect subject confidentiality; (7) only trained research staff will have 
access to the data; and (8) if the results of the study are published in scientific journals or 
presented at scientific meetings, data which might reveal the identity of any particular subject 
will be disguised. Subjects and their families will be informed about the limits of 
confidentiality (e.g., in cases of danger to self or others or when previously undocumented 
physical or sexual abuse is discovered).     
 
No identifiable data will be used for future study without first obtaining IRB approval. The 
investigator will obtain a data use agreement between the provider (the PI) of the data and 
any recipient researchers (including others at CHOP) before sharing a limited dataset (PHI 
limited to dates and zip codes).  

 
  
 
 
 
20 
8.4 Regulatory and Ethical Considerations 
8.4.1 Data and Safety Monitoring Plan 
The PI (Dr. Eiraldi) will assume overall responsibility for monitoring the integrity of data 
collection and management, and the safety of the interventions. The PI will inform each 
member of the research team of his or her responsibility in this area and will also meet 
weekly with the team to monitor all efforts.  This includes the external consultants who have 
the responsibility of training the behavioral health staff in each school, and the research team 
who has the responsibility for informed consent procedures and data collection and 
management activities. All adverse events will be reported immediately to the IRBs at 
CHOP. 
8.4.2 Risk Assessment 
Risks are not greater than minimal.  There are no known physical or legal risks to 
participation in the study. The experimental study protocol is designed to identify and 
remediate depression, anxiety, and disruptive behavior; however, it is possible that 
participation in the study may exacerbate some children’s emotional and/or behavioral 
difficulties. Children’s emotional and behavioral difficulties normally fluctuate over the 
course of a school year. We do not expect that this study will pose any additional risk to 
study participants compared to non-participants. In the event that a parent, child, or teacher 
becomes distressed as a result of study participation or present potential safety concerns 
(e.g., suicidal thoughts or behaviors), Dr. Eiraldi licensed psychologist, will work to 
stabilize the situation and address any outstanding concerns or needs.   
Potential Risks Associated with Assessment. Child assessments will be administered by 
school behavioral health staff who have had training in depression, anxiety, and disruptive 
behaviors. Participants could develop mild to moderate emotional discomfort or frustration 
associated with completing questionnaires.  Participants will be allowed any breaks during 
data collection that are needed. Parents and children may experience a loss of privacy as a 
result of providing information about the child’s mental health history during the 
assessment. Parents and children will be reminded about confidentiality, the secure storage 
of study data, and authorized data sharing via a signed release of information. Teachers, 
some of whom may feel uncomfortable providing personal information about the child, will 
be provided a copy of the parent and child’s authorization for the release of confidential 
data.  
Given the time required for recruitment and assessment of study participants, it is possible 
that eligible participants may experience a waiting period of up to two months between the 
point of assessment and the start of study interventions. Research staff will work with 
behavioral health staff and parents to identify appropriate care, especially in cases where the 
presenting symptoms require immediate attention. Their school counselor or other 
behavioral health providers in the school may serve some children. Other children may 
require a referral to a community provider, which the behavioral health staff will facilitate in 
consultation with research staff.  
Potential Risks Associated with Group Interventions. One risk associated with the 
selected interventions, as with all therapeutic interventions, is that they might not be 

 
  
 
 
 
21 
effective for every participant that enters treatment.  If following the course of any of these 
treatments, no improvement in symptoms (based on post-intervention measures) is noted, if 
deterioration is noted during treatment (based on self-report, parent- or teacher-report, or 
clinical judgment), or a family member is interested in further treatment for the child, we 
will work with families and behavioral health staff to identify appropriate referrals in their 
community. Participants may experience at least some subjective distress during treatment.  
Their reactions will be closely monitored and addressed therapeutically. In the unlikely 
event that a child manifests unusually high levels of distress, suggesting prohibitive side 
effects or intolerance of treatment, the PI will withdraw him or her from the study and 
clinical recommendations in the best interest of the child will be made on a case by case 
basis. Should clinical need arise (e.g., need to address clinical worsening, address parent 
concerns, prevent premature termination), the research team will support the behavioral 
health staff to schedule in-person parent meetings. Should there be concerns regarding the 
child’s symptoms that cannot be addressed in the context of these treatments, we will work 
together with the family to identify appropriate next steps including the option to seek 
services from other providers. 
Please refer to Section 7.2 for procedures related to suicidal ideation. 
Dr. Eiraldi, a licensed psychologist, will provide the team with on-call support to address 
any potential safety concerns, including suicidal ideation or behavior and concerns about 
abuse or neglect.  
 
8.4.3 Potential Benefits of Trial Participation 
We believe that this study will result in the development of a feasible and effective strategy 
for the training of behavioral health staff in schools serving rural communities. Linking 
PBIS with state-of-the-art mental health supports and having well trained behavioral health 
staff in underserved schools would likely lead to improvement in the behavioral health and 
academic productivity of students who would otherwise not have access to evidence-based 
care. Both of these would be important contributions to public health in rural school settings 
8.5 Recruitement Strategy  
Recruitment of schools will be selected from the existing pool of schools that participated in 
phase one of the grant (IRB number 20-017895). If necessary additional schools will be 
selected that are eligible if the school is implementing PBIS and serves rural communities.  
Recruitment of BHS may begin with assistance from the school principal or an additional 
school staff member. The principal and school staff member may nominate school staff to 
participate in the study. Additonally, the research staff will reach out to school behavioral 
health staff concurrently with the school principal. 
Recruitment of child participants will begin with research staff assisting school behavioral 
health staff who will examine teacher nomination data for students in specific grades and 
classes in order to identify students for Tier 2 interventions. The study team will approach 
the family of referred students via phone call, email and/or text message. After five 
unsuccessful attempts at contact, a recruitment letter may be distributed to families by the 
school. We will train BHS remotely on providing a brief in-service training to teachers 

 
  
 
 
 
22 
tasked with nominating students on recognizing warning signs of excessive anxiety, 
depression, and disruptive behavior problems in children, and completing the Strengths and 
Difficulties Questionnaire (SDQ). Following the remote training, BHS will ask teachers 
from specific grades or classes to nominate students. First, teachers will be asked to rank 
order students and then select the top three students with externalizing problems and the top 
three students with internalizing problems. Second, parents and teachers will complete the 
SDQ on these identified students. Third, parents and teachers will give the SDQ forms to 
the BHS member. Fourth, BHS will scan the SDQ to CHOP staff to score. Fifth, students 
who score above the specified cut-off on the externalizing or internalizing dimension will be 
selected for the next step, which will require BHS to contact the parents and request that 
they complete the SCARED or to ask a teacher of the student to complete the SAEBRS. 
Sixth, children who score at least 25 or there is an elevated subscale for the SCARED or if 
the student scored on the Social Behavioral subscale of the SAEBRS is 12 or less, they will 
be considered for participation in one of the groups or in CI/CO. The research aspect of the 
project will begin at this point. Children who meet inclusion criteria will be invited to 
participate in the study. Research staff will ask parents provide consent for their child to 
participate and the child will provide assent to participate in the intervention. Informed 
consent (parent) and assent (child) will be obtained by research staff prior to enrollment.  
Participating BHS will facilitate consent meetings with eligible children and their parents. A 
research staff member will join these meetings by phone or via a videoconference. 
8.6 Informed Consent  
8.6.1 Behavioral Health Staff  
Behavioral health staff will be provided with a public survey link from REDCap to review 
the IRB approved verbal consent form. After reviewing the consent form they will be 
prompted to enter their email address to receive a copy of the consent form and the 
following statement will be included in REDCap, “By clicking “next” you agree to 
participate in Project CARE”. Consent will be documented electronically via survey 
completion. If consent is not documented within 5 business days after distribution, the study 
team will contact BHS via email or phone to discuss any questions or concerns they have. 
The consent form will clearly explain study procedures, potential risks and benefits, and the 
voluntary nature of their participation. A copy of the consent form will be provided to the 
BHS, and consent will be obtained electronically via REDCap.  
8.6.2 School Administrators  
School administrators will consent for the school’s participation in the study, the 
demographics survey, and participation in the qualitative interview. The consent procedures 
for school administrarors will follow the same method as consent for BHS. Administrators 
will be provided a public survey link from REDCap to review the IRB approved verbal 
consent form. After reviewing the consent form they will be prompted to enter their email 
address to receive a copy of the consent form and the following statement will be included 
in REDCap, “By clicking “next” you agree to participate in Project CARE”. Consent will be 
documented electronically via survey completion. If consent is not documented within 5 
business days after distribution, the study team will contact administrators via email or 
phone to discuss any questions or concerns they have. The consent form will clearly explain 

 
  
 
 
 
23 
study procedures, potential risks and benefits, and the voluntary nature of their participation. 
A copy of the consent form will be provided to the administrators, and consent will be 
obtained electronically via REDCap. 
8.6.3 Parents and Students  
Once a family has been identified by BHS, they will be approached by the study team to 
verbally consent for screening procedures. Study staff will speak with parents by phone for 
the verbal screening consent and email them a copy of the screening consent form for their 
records. Students will be contacted by phone or WebEx with a BHS member to verbally 
assent to screening procedures. Verbal screening consent and assent will be documented by 
the study team using the verbal consent/assent signature blocks.  After the child has been 
screened and determined to be eligible, the family will meet with research staff again to 
verbally consent for the main study. 
Informed consent (parent) and assent (child) will be obtained by research staff prior to 
enrollment.  Participating BHS will facilitate consent meetings with eligible children and 
their parents by either scheduling an appointment to go over the consent process or by 
giving research staff the parents’ phone number with permission from the parents. The 
research staff will explain the study procedures, as well as potential risks, benefits, and 
treatment alternatives to both the parent, and child in a language understandable to each. No 
child will be asked to participate if demonstrably unwilling, even if his or her parent wishes 
it. Children and parents will be clearly informed of the voluntary and confidential nature of 
their participation, that treatment services will be provided in the school setting, and of their 
right to terminate their participation at any time without penalty.  Ample opportunity will be 
provided for them to ask any questions they may have of the professional staff.  If the family 
agrees to participate, parents will provide verbal consent and children will provide verbal 
assent. Informed consent and assent will be documented by research staff at the conclusion 
of the consent meeting. Families will be sent a copy of the consent and assent forms via 
email or by mail. Consent and assent will be documented using the verbal consent/assent 
signature blocks. Children who need treatment but do not qualify for this study or whose 
family decides not to participate will be provided appropriate treatment referrals. Treatment 
referrals will occur without disclosure of PHI.  
8.6.4 Individuals with Limited English Proficiency  
Once a family has been identified by BHS, they will be approached by the study team to 
verbally consent for screening procedures. Study staff will call language services to speak 
with an interpreter and once the connection with the interpreter is established, study staff 
will proceed with reviewing the screening consent form with the Spanish proficient parent. 
Study staff will speak with parents by phone for the verbal screening consent and email 
them a copy of the screening consent form for their records.Verbal screening consent will be 
documented by the study team using the verbal consent signature block. 
Informed consent will be obtained by study staff prior to enrollment. Study staff will 
facilitate consent meetings with Spanish proficient parents via phone. Study staff will call 
language services to speak with an interpreter and once the connection with the interpreter is 
established, study staff will proceed with reviewing the intervention consent form with the 
Spanish proficient parent. Informed consent will be documented by research staff at the 

 
  
 
 
 
24 
conclusion of the consent meeting. Families will be sent a copy of the consent form via 
email or by mail. Consent will be documented using the verbal consent signature blocks.  
8.6.5 Teachers providing data about eligible and enrolled students  
Teachers will be provided with a public survey link from REDCap to review the IRB 
approved information sheet. After reviewing the information sheet they will be prompted to 
enter their email address to receive a copy of the consent form and the following statement 
will be included in REDCap, “By clicking “next” you agree to participate in Project CARE”. 
Participation will be documented electronically via survey completion. If teachers want to 
discuss or have questions about the information sheet, study staff will be available to speak 
via phone or WebEx. A copy of the information sheet will be provided to the teachers. 
8.7 Payment to Subjects/Families 
Study interventions will be provided at no cost to families. Parents will receive $20 for 
completing screening and baseline measures and $20 for completing outcome measures. 
Teachers will receive $20 for completing screening and baseline measures and $20 for 
completing outcome measures at post. Children will receive small gifts (e.g., pencils, 
notebooks, small toys) as reinforcement for participating and reaching certain goals in-group 
sessions. Behavioral health staff will receive between $200 to $250 depending on condition  
for conducting targeted screenings with students, for participating in remote trainings, 
implementing EBPs, and participating in a qualitative interview. School administrators will 
receive $30 for the completion of the qualitative interview.  
9 
PUBLICATION 
The data generated in this grant project will be presented at national professional 
conferences and published in peer review publications. All final peer-reviewed manuscripts 
originating from this project will be submitted to the digital archive PubMed Central. No 
individually identifiable protected health information will be published. 
 
 
 

 
  
 
 
 
25 
10 REFERENCES 
1. 
Smalley KB, Yancey CT, Warren JC, Naufel K, Ryan R, Pugh JL. Rural mental 
health and psychological treatment: A review for practitioners. Journal of Clinical 
Psychology. 2010;66(5):479-489. 
2. 
Wagenfeld MO. A snapshot of rural and frontier America. In: Stamm BH, ed. Rural 
behavioral health care: An interdisciplinary guide. Washington, DC: American 
Psychological Association; 2003:33-40. 
3. 
Anderson NJ, Neuwirth SJ, Lenardson JD, Hartley D. Patterns of Care for Rural and 
Urban Children with Mental Health Problems. Portland, ME: University of Southern 
Maine;2013, June. 
4. 
Evans SW, Weist MD. Implementing empirically supported treatments in the 
schools: what are we asking? Clin Child Fam Psychol Rev. 2004;7(4):263-267. 
5. 
Owens JS, Watabe Y, Michael KD. Culturally responsive school mental health in 
rural communities. . In: Clauss-Ehlers CS, Serpell Z, Weist MD, eds. Handbook of 
culturally responsive school mental health: Advancing research, training, practice, 
and policy. New York, NY: Springer; 2013:31–42. 
6. 
Kelleher KJ, Gardner W. Out of Sight, Out of Mind - Behavioral and Developmental 
Care for Rural Children. N Engl J Med. 2017;376(14):1301-1303. 
7. 
Sugai G, Horner R. The evolution of discipline practices: School-wide positive 
behavior supports. Child and Family Behavior Therapy. 2002;24:23-50. 
8. 
Horner RH, Sugai G, Smolkowski K, et al. A randomized, wait-list controlled 
effectiveness trial assessing school-wide positive behavior support in elementary 
schools. Journal of Positive Behavior Interventions. 2009;11(3):133-144. 
9. 
Cavanaugh B, Swan M. Building SWPBIS capacity in rural schools through 
building-based coaching: Early findings from a district-based model. Rural Special 
Education Quarterly. 2015;34(4):29-39. 
10. 
Fitzgerald CB, Geraci LM, Swanson M. Scaling up in rural schools using positive 
behavioral interventions and supports. Rural Special Education Quarterly. 
2014;33(1). 
11. 
McCrary D, Lechtenberger D, Wang E. The effect of schoolwide positive behavioral 
supports on children in impoverished rural community schools. Preventing School 
Failure. 2012;56(1):1-7. 
12. 
Steed EA, Pomerleau T, Muscott H, Rohde L. Program-wide positive behavioral 
interventions and supports in rural preschools. Rural Special Education Quarterly. 
2013;32(1):38-46. 
13. 
Elias MJ, Zins JE, Graczyk PA, Weissberg RP. Implementation, Sustainability, and 
Scaling Up of Social-Emotional and Academic Innovations in Public Schools. 
School Psychology Review. 2003;32(3):303-319. 
14. 
Garbacz AS, Watkins ND, Diaz Y, Barnabas ERJ, Schwartz B, Eiraldi R. Using 
Conjoint Behavioral Consultation to Implement Evidence-based Practices for 
Students in Low-income Urban Schools. Journal of Educational and Psychological 
Consultation. 2016;61(3):198-210. 
15. 
Eiraldi R, Mautone JA, Khanna MS, et al. Group CBT for Externalizing Disorders in 
Urban schools: Effect of Training Strategy on Treatment Fidelity and Patient 
Outcomes. Behavior Therapy. 2018. 

 
  
 
 
 
26 
16. 
Eiraldi R, Power TJ, Schwartz BS, et al. Examining Effectiveness of Group 
Cognitive-Behavioral Therapy for Externalizing and Internalizing Disorders in 
Urban Schools. Behav Modif. 2016;40(4):611-639. 
17. 
Eiraldi R, McCurdy B, Khanna M, et al. A cluster randomized trial to evaluate 
external support for the implementation of positive behavioral interventions and 
supports by school personnel. Implementation Science. 2014;9(12). 
18. 
Eiraldi R, McCurdy B, Schwartz B, et al. Pilot Study for the Fidelity, Acceptability 
and Effectiveness of a PBIS Program plus Mental Health Supports in Under-
resourced Urban Schools. Psychology in the Schools. 2019:1-16. 
19. 
Riding-Malon R, Werth J, J. L. Psychological Practice in Rural Settings: At the 
Cutting Edge. Professional Psychology: Research and Practice. 2014;45(2):85–91. 
20. 
Khanna MS, Kendall PC. Bringing Technology to Training: Web-Based Therapist 
Training to Promote the Development of Competent Cognitive-Behavioral 
Therapists. Cognitive and Behavioral Practice. 2015;22(3):291–301. 
21. 
Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: 
assessing partnership approaches to improve public health. Annu Rev Public Health. 
1998;19:173-202. DOI:10.1146/annurev.publhealth.19.1.173 
22. 
Israel BA, Eng E, Schultz AJ, Parker EA. Methods for community-based 
participatory research for health. Jossey-Bass; 2013. 
23. 
Larman C, Basili VR. Iterative and Incremental Development: A Brief History. 
Computer. 2003;36:47-56. doi:10.1109/MC.2003.1204375 
24. 
Wilson J, Rosenberg D. Rapid prototyping for user interface design. In: Helander M, 
ed. Handbook of human-computer interaction. North-Holland; 1988: 859–875. 
25. 
Lyon AR, Koerner K. User-Centered Design for Psychosocial Intervention 
Development and Implementation. Clin Psychol (New York). Jun 2016;23(2):180-
200. doi:10.1111/cpsp.12154 
26. 
Maguire M. Methods to support human-centred design. International Journal of 
Human-Computer Studies. 2001;55:587–634. doi:10.1006/ijhc.2001.0503 
27. 
Barnett JE. Utilizing technological innovations to enhance psychotherapy 
supervision, training, and outcomes. Psychotherapy. 2011;48:103-108. 
28. 
Abbass A, Arthey S, Elliott J, et al. Web-conference supervision for advanced 
psychotherapy training: A practical guide. . Psychotherapy. 2011;48(2):109-118. 
29. 
Weisz JR. Agenda for child and adolescent psychotherapy research: On the need to 
put science into practice. Archives of General Psychiatry. 2000;57(9):837-8 
30. 
Cohen J, Mannarino AP. Disseminating and Implementing Trauma-focused CBT in 
community settings. Trauma Violence Abuse. 2008;9(4):214-226. 
31. 
Cully JA, Curry AD, Ryan SR, Malik A, Zeno D, Willcockson IU. Development of a 
computer-aided training program for brief cognitive-behavioral therapy in primary 
care. Acad Psychiatry. 2013;37(2):120-124. 
32. 
Kobak KA, Craske MG, Rose RD, Wolitsky-Taylor K. Web-based therapist training 
on cognitive behavior therapy for anxiety disorders: A pilot study. . Psychotherapy. 
2013;50:235-247. 
33. 
Rakovshik SG, McManus F, Westbrook D, Kholmogorova AB, al. e. Randomized 
trial comparing internet-based training in cognitive behavioural therapy theory, 
assessment and formulation to delayed-training control. Behav Res Ther. 
2013;51:231-239. 

 
  
 
 
 
27 
34. 
Barrett P. Friends for Life: Group leaders' manual for children. Pathways Health and 
Research Centre, Australia; 2008. 
35. 
Eiraldi R, Power TJ, Schwartz BS, et al. Examining Effectiveness of Group 
Cognitive-Behavioral Therapy for Externalizing and Internalizing Disorders in 
Urban Schools. Behavior modification. Feb 11 2016. 
doi:10.1177/0145445516631093 
36. 
Lee SJ, Altschul I, Mowbray CT. Using planned adaptation to implement evidence-
based programs with new populations. Research Support, N.I.H., Extramural. 
American journal of community psychology. Jun 2008;41(3-4):290-303. 
doi:10.1007/s10464-008-9160-5 
37. 
Bernal G, Bonilla J, Bellido C. Ecological validity and cultural sensitivity for 
outcome research: Issues for the cultural adaptation and development of 
psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology. 
1995;23(1):67-82. doi:10.1007/BF01447045 
38. 
Ollendick TH, King NJ. Empirically supported treatments for children with phobic 
and anxiety disorders: Current status. Journal of Clinical Child Psychology. 
1998;27:156-167. doi: 10.1207/s15374424jccp2702_3 
39. 
Kendall PC. Child and adolescent therapy: Cognitive-behavioral procedures. 2nd 
edition ed. the Guilford Press; 2000. 
40. 
Lochman JE, Wells KC. The coping power program for preadolescent aggressive 
boys and their parents: outcome effects at the 1-year follow-up. J Consult Clin 
Psychol. 2004;72(4):571-578. 
41. 
Ellis ML, Lindsey MA, Barker ED, Boxmeyer CL, Lochman JE. Predictors of 
engagement in a school-based family preventive intervention for youth experiencing 
behavioral difficulties. Prev Sci. 2013;14(5):457-467. 
42. 
Crone DA, Hawken LS, Horner RH. Responding to problem behavior in schools: 
The behavior education program (2nd ed.). New York: Guilford; 2010. 
43. 
Hunter KK, Chenier JS, Gresham FM. Evaluation of check-in/check out for students 
with internalizing behavior problems. Journal of Emotional and Behavioral 
Disorders. 2013;22:135-148. 
44. 
Hawken LS, MacLeod KS, Rawlings L. Effects of the Behavior Education Program 
(BEP) on office discipline referrals of elementary school students. Journal of 
Positive Behavior Interventions. 2007;9(2):94-101. 
45. 
McCurdy BL, Kunsch C, Reibstein S. Secondary Prevention in the Urban School: 
Implementing the Behavior Education Program. Preventing School Failure. 
2007;51(3):12-19. 
46. 
Anello V, Weist M. Survey on School Readiness for Interconneting Positive 
Behavior Interventions and Supports and School Mental Health. In: Barrett S, Eber 
L, Weist M, eds. Advancing Education Effectiveness: Interconnecting School Mental 
Health and School-Wide Positive Behavior Support.2013:128-133. 
47. 
Provasnik S, KewalRamani A, Coleman MM, Gilbertson L, Herring W, Xie Q. 
Status of Education in Rural America (NCES 2007-040). Washington, DC: National 
Center for Education Statistics, Institute of Education Sciences, U.S. Department of 
Education;2007. 

 
  
 
 
 
28 
48. 
Anello V, Weist M, Eber L, et al. Readiness for Positive Behavioral Interventions 
and Supports and School Mental Health Interconnection: Preliminary Development 
of a Stakeholder Survey. Journal of Emotional and Behavioral Disorders. 2016:1-14. 
49. 
Lee  SW, Lohmeier JH, Niileksela C, Oeth J. Rural schools' mental health needs: 
Educators' perceptions of mental health needs and services in rural schools. Journal 
of Rural Mental Health. 2009;33(1):26-31. 
50. 
Slade EP. The relationship between school characteristics and the availability of 
mental health and related health services in middle and high schools in the United 
States. J Behav Health Serv Res. 2003;30(4):382-392. 
51. 
Rones M, Hoagwood K. School-based mental health services: a research review. 
Clin Child Fam Psychol Rev. 2000;3(4):223-241. 
52. 
Sugai G, Horner R. Defining and describing school wide positive behavior support. 
In: Sailor W, Dunlap G, Sugai G, Horner R, eds. Handbook of positive behavior 
support. New York: Springer; 2009:307-326. 
53. 
Putnam R, McCart A, Griggs P, Hoon Choi J. Implementation of school-wide 
positive behavior support in urban settings. In: Sailor W, Dunlap G, Sugai G, Horner 
R, eds. Handbook of positive behavior support. New York: Springer; 2009:443-463 
54. 
Goodman R, Ford T, Simmons H, Gatward R, Meltzer H. Using the Strengths and 
Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a 
community sample. Br J Psychiatry. 2000;177:534–539. DOI:10.1192/bjp.177.6.534 
55. 
Reynolds CR, Kamphaus RW. Manual for the Behavior Assessment System for 
Children - Third Edition (BASC-3). Pearson, Psych Corp.; 2015. 
56. 
Showers B, Joyce B, Bennett B. Synthesis of research on staff development: a 
framework for future study and a state-of-the-art analysis. Educational Leadership. 
1987;45:77-87. 
57. 
Warner CM, Fox JK. Advances and Challenges in School-Based Intervention for 
Anxious and Depressed Youth: Identifying and Addressing Issues of Sustainability. 
School Mental Health. 2012;4:193–196. doi:10.1007/s12310-012-9087-8 
58. 
Lochman JE, Wells KC, Lenhart L. Coping Power Child Group Program: Facilitator 
Guide. Oxford University Press; 2008. 
59. 
Khanna MS, Eiraldi R, Schwartz B, Kendall PC. CBT for Anxiety Treatment in 
Schools. Unpublished; 2016. 
60. 
Crone DA, Hawken LS, Horner RH. Responding to problem behavior in schools: 
The behavior education program (2nd ed.). Guilford; 2010. 
61. 
Hunter KK, Chenier JS, Gresham FM. Evaluation of check-in/check out for students 
with internalizing behavior problems. Journal of Emotional and Behavioral 
Disorders. 2013;22:135-148. https://doi.org/10.1177/1063426613476091 
62. 
Stone LL, Otten R, Engels RC, Vermulst AA, Janssens JM. Psychometric properties 
of the parent and teacher versions of the strengths and difficulties questionnaire for 
4- to 12-year-olds: a review. Clin Child Fam Psychol Rev. 2010;13(3):254-274. 
63. 
Todd AW, Campbell AL, Meyer GB, Horner RH. The effects of a targeted 
intervention to reduce problem behaviors. Journal of Positive Behavior Intervention. 
2008;10(1):46-55. 
64. 
Barrett S, Eber L, Weist M. Advancing education effectiveness: Interconnecting 
school mental health and school-wide positive behavior support. 2013. 
http://www.pbis.org/common/cms/files/pbisresources/Final-Monograph.pdf 

 
  
 
 
 
29 
65. 
Locke EA, Latham GP. Building a practically useful theory of goal setting and task 
motivation. A 35-year odyssey. Am Psychol. Sep 2002;57(9):705-17. 
https://doi.org/10.1037/0003-066X.57.9.705 
66. 
Denton CA, Hasbrouck J. A Description of Instructional Coaching and its 
Relationship to Consultation. Journal of Educational and Psychological Consultation. 
2009;19:150-175. doi:10.1080/10474410802463296 
67. 
Kluger AN, DeNisi A. The Effects of Feedback Interventions on Performance: A 
Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention 
Theory. Psychological Bulletin. 1996;119(2):254-284. 
http://dx.doi.org/10.1037/0033-2909.119.2.254 
68. 
Filter KJ, McKenna MK, Benedict EA, Horner RH. Check in/ Check out: A Post-
Hoc Evaluation of an Efficient, Secondary-Level Targeted Intervention for Reducing 
Problem Behaviors in Schools. Education and Treatment of Children. 2007;30(1):69-
84. doi:10.1353/etc.2007.0000 
69. 
Elliot D, Mihalic S. Issues in disseminating and replicating effective prevention 
programs. Prevention Science. 2004;5:47-53. 
70. 
Lochman JE, Powell NP, Boxmeyer CL, Qu L, Wells KC, Windle M. 
Implementation of a school-based prevention program: Effects of counselor and 
school characteristics. Professional Psychology: Research and Practice. 
2009;40(5):476-482 
71. 
Skinner EA, Kindermann TA, Furrer CJ. A Motivational Perspective on Engagement 
and Disaffection: Conceptualization and Assessment of Children’s Behavioral and 
Emotional Participation in Academic Activities in the Classroom. Educational and 
Psychological Measurement. 2009;69(3):493-525. 
72. 
Weiner  BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly 
developed implementation outcome measures. Implementation Science. 
2017;12:108. 
73. 
SAS/STAT® 9.2 User’s Guide [computer program]. Cary, NC: SAS Institute Inc.; 
2002-2008. 
74. 
Zeger SL, Liang KY. Longitudinal data analysis for dicrete and continuous 
outcomes. Biometrics. 1986;42(1):121-130. 
75. 
Liang KW, Zeger SL. Longitudinal data analyses using generalized linear models. 
Biometrika. 1986;73:13-22. 
76. 
Allison PD. Handling Missing Data by Maximum Likelihood. Haverford, PA, USA: 
Statistical Horizons. 
77. 
American Psychological Association (APA), Guidelines for clinical supervision in 
health service psychology. American Psychologist. 2015;70(1): 33-46. 
78. 
Lewis, T.J., et al., Blueprint for school-wide positive behavior support training and 
professional development (Version 3). 2010, University of Oregon, Center for 
Positive Behavioral Interventions and Supports: Eugene, Oregon. 
79. 
Sburlati, E.S., et al., A Model of Therapist Competencies for the Empirically 
Supported Cognitive Behavioral Treatment of Child and Adolescent Anxiety and  
Depressive Disorders. Clinical Child and Family Psychology Review, 2011;14(1). 
80. 
Beidas, R.S., et al., Training school mental health providers to deliver cognitive 
behavioral therapy. School Mental Health, 2012;4:197-206. 

 
  
 
 
 
30 
81. 
Rakovshik, S.G. and F. McManus, Establishing evidence-based training in cognitive 
behavioral therapy: A review of current empirical findings and theoretical guidance. 
Clin Psychol Rev, 201;30(5):496-516. 
82. 
Merriam, S., The changing landscape of adult learning theory, in Review of adult 
learning and literacy: Connecting research, policy and practice, J. Comings, B. 
Garner, and C. Smith, Editors. 2004, Lawrence Erlbaum Associates: Mahwah, NJ. 
199-220. 
83. 
SAS Institute Inc. [computer program]. Cary, NC, USA: 2002-2012. 
84. 
IBM SPSS Statistics for Windows, Version 21.0 [computer program]. Armonk, NY: 
IBM Corp. 
85. 
Hayes AF. Introduction to Mediation, Moderation, and Conditional Process 
Analysis, A Regression-based Approach (second addition). The Gulliford Press, 
New York; London(2018).