Developing a Bullying Prevention Program That Transcends Physical Boundaries
Phase: NA
Conditions: Bullying
Sponsor: Center for Innovative Public Health Research

 
 
STUDY TITLE: BULLYDOWN: DEVELOPING A BULLYING PREVENTION PROGRAM THAT TRANSCENDS 
PHYSICAL BOUNDARIES 
FUNDER: NICHD 
GRANT NUMBER: R21HD104369 
SPONSER: CENTER FOR INNOVATIVE PUBLIC HEALTH RESEARCH (CiPHR) 
PRIMARY INVESTIGATOR: MICHELE YBARRA MPH PHD  
PROTOCOL UPDATED: October 6, 2023 
 
1. RESULTS OF PREVIOUS RESEARCH 
An estimated 20-24% of middle school students bully their peers ‘regularly’ or more often each semester.1 
Bullying involvement is associated with externalizing problems (e.g., alcohol use) for bullies and internalizing behaviors 
(e.g., depressive symptomatology)2-4, and suicidal ideation and behavior for victims.4,5 Efforts to reduce bullying 
perpetration, thereby also reducing victimization, are critical. 
Existing bullying prevention programs have only a modest impact in elementary school settings and even less of 
an impact in middle schools.6-12 Reasons for this include varying implementation fidelity as well as the reliance on 
teachers to deliver proscriptive content, which may result in youth dismissing the messages as a way of rejecting authority 
and exerting control over their social selves.12 Approaches that avoid adult involvement and integrate less proscriptive 
language may have more success.  
In the era of Covid-19, approaches that harness the power of technology are ever more relevant. We believe text 
messaging is one such approach as it provides novel opportunities to “go where youth are”: Nearly all (91%) of the nine in 
ten teens who own cell phones use text messaging,13,14 and usage is high across race, income, and rural vs. urban 
settings,13 Content can be written to be approachable and not sound as if it is being delivered by an authority figure. 
Importantly, emerging evidence suggests that text messaging-based programs can affect behavior change.15-18 
 
2. STUDY PURPOSE AND RATIONALE  
This R21 proposal is an extension of our development work with BullyDown, a text messaging-based bullying 
prevention program designed for middle school students.19,20 Guided by the Social-Emotional Learning (SEL) model,21-25 
the intervention content and program features have been iteratively refined within ongoing feedback from middle school 
youth from across the country. The program was then beta tested with 22 middle school students in Illinois. Nine in 10 
said they liked the program, and all intervention participants said that it equipped them with skills needed to not bully 
others in the future. Although the sample size is too small to be anything but descriptive, rates of bullying dropped from 3 
youth to 1 youth in the intervention while no change was noted in the control group (2 youth at both baseline and follow-
up). 
Building upon our promising findings, we propose to further develop and then pilot test BullyDown in a 
randomized controlled trial (RCT) of middle school students. Our multidisciplinary team includes expertise in text 
messaging-based behavior change intervention development and evaluation (Dr. Ybarra), school-based evaluations of 
bullying prevention programming (Dr. Espelage), and the psychosocial challenges associated with bullying (Drs. Ybarra 
and Espelage). Together, we aim to: 
Specific Aim 1: Finalize the BullyDown program. Feedback from the beta test suggest that some program 
features could be better described and implemented. We also will update program content to acknowledge the growing 
infusion of technology in the classroom, and the online interactions that it requires. Currently, almost all youth in the 
United States are engaging in some type of online schooling experience because of Covid-19 restrictions. Even when 
these are lifted, however, technology will continue to be a central feature of the way education is delivered.  
To finalize the content, we will first conduct focus groups (FGs) to understand how young people are interacting 
with each other online, their opportunities for positive and negative communications, and whether and how teachers fit 
into this picture. Once the intervention content is updated, we will have youth review the messages in Content Advisory 

 
 
Teams to ensure that they are understandable and salient. We also may test the program in a beta test with 15-20 middle 
school youth. 
Specific Aim 2: Pilot test the 7-week BullyDown primary prevention program for feasibility and 
acceptability in an RCT of 150 middle adolescents. Youth will be randomly assigned to either BullyDown, a 4-
message-per-day program, or an attention-matched control group talking about ‘healthy lifestyle’ topics (e.g., exercise). 
Our main outcome measures will be feasibility (e.g., recruitment and enrollment rates) and acceptability (e.g., positive 
program appraisal) at intervention end. The behavioral outcomes we will measure to motivate power analyses for a future 
R01 include: (1) bullying perpetration, (2) other aggressive behavior directed towards peers, and (3) bystander behavior in 
the past 30 days at 3-month post-intervention. We also will examine behavioral indicators at intervention end to observe 
proximal effects. 
Given the limited impact of existing school-based interventions and the time associated with implementing them 
during school time, more efficient and effective intervention methods are needed. Programs that can be delivered to youth 
in middle adolescence, typically of middle school aged, irrespective of whether they are attending school physically or 
virtually may be especially relevant as school boundaries become more fluid. BullyDown addresses this critical need by 
providing a scalable program that could be quickly and cost-effectively disseminated nationally. If efficacious, we will 
test the intervention in an R01-funded RCT. Future endeavors could include adapting the program for Spanish-speaking 
youth and dynamically tailoring content to reported bullying behaviors. 
 
3. RESEARCH PROTOCOL 
3A. DESCRIPTION OF STUDY POPULATION, INCLUSION AND EXCLUSION CRITERIA 
Study Sites. Dr. Espelage has a strong relationship with the Chatham County School District (See Letter of 
Support). There are 8 middle schools in the District, with a combined enrollment of 712 7th graders. The schools are 
diverse in terms of race and income: 47% are racial/ethnic minority (e.g., 31.5% are Black) and 69% are receiving free or 
reduced cost lunch. We will randomly select three schools. Should a school choose to opt out, we will randomly select 
another from the remaining five schools. Note that, while enthusiasm for the program is high, it is against district policy to 
allow principals to provide Letters of Support.  
Should additional sites be needed, we will draw upon local Boys & Girls Clubs. The Boys & Girls Club of 
America (BGCA) includes over 4,000 clubs in the U.S. and serves approximately 4 million young people. Local clubs 
affiliated with the BGCA provide young people with a safe, supervised, and constructive environment during non-school 
hours. The organization serves a diverse population of young people with a large representation from low-income 
backgrounds. The Boys & Girls Clubs of Durham and Orange Counties serve students in the cities of Durham and Chapel 
Hill. The clubs are diverse in terms of race and income: 85% are African American, 6% Hispanic, 3% are White, 3% are 
Asian, 2% are Multiracial, and 1% are American Indian/Alaskan Native. About 62% of students are experiencing 
financial hardship, 1% have disabilities, and 4% are English Language Learners. 
We also may recruit 13-14 year olds using advertisements on social media, such as Facebook and Instagram. 
Youth eligibility criteria. The inclusion criteria for youth matches the anticipated profile of users when the 
program is publicly available in the future. Participants will: be enrolled in one of our partner schools, be in middle school 
(i.e., 7th or 8th grade) or between the ages of 12-14 years of age, be English-speaking, own their own cell phone, intend to 
have the same cell phone number for at least 3 months (this may or may not apply, depending on the number of youth 
who meet this criterion), provide informed written assent, and have parental permission. Non-English speakers and youth 
who do not have the reading ability to complete the screener are not eligible. 
Youth recruited via social media additionally must be 13 years of age or older due to the Children’s Online 
Privacy Protection Act. 
Justification for eligibility criteria. Excluding Spanish youth: Only 11% of students in the Chatham County 
School District are English language learners, suggesting that more than 600 of the 7th graders will be English-speaking.  
Requiring cell phone ownership: If we gave youth phones, findings would not be generalizable (e.g., community-
based organizations are unlikely to provide cell phones). National data suggest that low income youth are as likely as 
higher income youth to have cell phones.13 Our District contact estimates that almost all students have a cell phone. 
Similarly, Dr. Espelage recently conducted FGs with low-income middle school students and most had a cell phone. As 

 
 
such, we are not reifying the digital divide with this criterion.  
3B. SAMPLE SIZE 
We will include ~60 youth in focus groups, ~70 in content advisory teams, 15-20 youth in a beta test if time 
allows, and 150 in a randomized controlled trial. 
For the focus groups, we will enroll about 60 youth to take part in the online discussion. The groups will be 
stratified by characteristics that we posit will contribute to a sense of shared experiences and/or affect the likelihood of 
being a perpetrator of bullying. This may include sex, race/ethnicity, among others. We will endeavor to purposefully 
sample youth so that at least 50% are racial/ethnic minority, at least 50% are receiving free or reduced cost lunch, at least 
30% report bullying perpetration and 30% report bullying victimization (note that there may be overlap such that a youth 
is both a bully and a victim). Actual distributions of youth who screen eligible may result in different ratios. We anticipate 
partnering with about 7 schools and about 2 BCGA Clubs. We will work with school administrators and Club Directors to 
determine the optimal school participation for each research activity.  
3C. DESIGN AND DESCRIPTION OF METHODOLOGY 
Recruitment and enrollment protocol.  
In person recruitment. 
Recruitment and enrollment will be similar for each of the three research activities. Chatham County Public School 
administration has agreed to dedicate classroom time for us to recruit youth from middle schools. We also will recruit 
from local Boys & Girls Clubs and via social media.  
First, research staff will visit classes or Clubs to briefly describe the study activity and encourage youth who are interested 
to complete the screener. If youth are still being taught at home, as they are now in July 2020, the screener will be a brief 
online survey. If youth are back at their brick and mortar school or Club, the screener will be either online or paper and 
pencil. The screener will disguise the qualifying criteria so that those interested in participating will not know the “right” 
answers. Dr. Ybarra’s team has found this to be particularly important in technology-based interventions where students 
are especially keen to take part in the study. 
Next, youth who are eligible will be invited to an enrollment meeting where the study activity will be discussed, any 
questions will be answered, and an enrollment packet will be distributed. Research staff will present the study in small 
groups during times that do not interfere with academic commitments (e.g., during break, lunch). The packet will include 
the parental permission and youth assent forms; it may also include a cover letter from the school principal or the Club 
Director as well, depending on the District supervisor’s choice. 
Based upon previous experience, we anticipate more youth will be eligible than is our target recruitment number for the 
pilot RCT (~50 youth per school). As such, youth will be randomly selected to participate among those who are eligible. 
A list of candidates will be generated by the study software program and students will be contacted sequentially.  
Youth who are interested in taking part in the study will receive an enrollment packet that includes both youth assent and 
parental permission forms. If youth are distance learning, these forms will be available online. If they are back in their 
brick and mortar setting, the forms will be printed. Research staff will ask youth to bring back their signed parental form 
and their signed youth assent form within two weeks. We may also have an online version. 
The benefit of an online screener and enrollment is that youth can complete both activities whenever it is convenient for 
youth. If we conduct the screening and enrollment processes in the schools or Boys & Girls Clubs, we may try to contact 
youth who are absent on the day of the particular research activity up to four times. In the case of enrollment, youth who 
cannot be contacted, as well as those who choose not to participate by either youth or parental choice will be replaced by 
the next student on the random list of candidates. If the school or Boys and Girls Club does not have internet, youth will 
complete a paper-and-pencil screener , which will then be data entered by study staff. 
Once youth assent and parental permission have been obtained, an enrollment confirmation text message will be sent to 
their cell phone to confirm the validity and compatibility of their cell phone number with the intervention software 
program. If challenges arise with receiving the text message, research staff will solve the issue.  
For those who are enrolled through the Boys and Girls Club, we may use a passive parental permission protocol. In this 
case, we will hand the permission forms directly to the parents. 

 
 
Social media recruitment. 
Participants may also be recruited primarily through social media, such as Instagram (IG) and Facebook (FB).  
We will not ask youth to refer their friends because this has the potential to create problems with randomization (i.e., if 
one is randomized to the intervention and the other to the control). 
All youth who view the social media ad and want to learn more will click on it, linking them to the project website. There, 
they will read a description of the study activity and, if interested, complete an eligibility screener. Participants will not be 
required to register to complete the screener; this step will be similar to a contact form.  
We will be requesting a waiver of parental permission. Children who complete an online screener that suggests they 
may be eligible will be contacted by research staff to schedule an enrollment telephone call. Over the telephone, study 
staff will confirm the candidate’s eligibility, explain the study, and obtain verbal assent to participate, including a 
decisional capacity assessment.90 
Research staff may use an external telephone provider, such as Google Voice, to communicate with participants, including 
telephone enrollments. We anticipate transitioning these communications to our internal participant management system 
when human resources allow.   
Enrollment 
For those in the new beta test and RCT, youth will then complete the online baseline survey. They will not be enrolled in 
the study activity until the baseline survey is complete, at which time they will be randomized. To facilitate self-
completion, youth in the RCT will receive an invitation text message with a unique URL to access the survey. This allows 
participants to complete the survey at a time and place convenient for them. 
Other activities 
Focus group participants will have a pre-FG online survey to complete. They will be able to complete it on their own (e.g., 
at home). If they are attending the brick and mortar school or Club, they also will be able to complete the survey at school 
in the computer lab or at the Club on laptops there if they would prefer.  
 
Special cases: 
Focus groups: We will collect assent post-hoc from youth in the focus groups who had parental permission to take part but 
did not provide their own informed assent. To do so, we will send an email and text message to them that reads: 
Hi [Name] – It’s [RA] from BullyDown. Thank you again for taking part in the focus groups! Your feedback was great. 
We forgot to get your permission to take part in the online discussion before you started. We apologize. Please click on 
this link and tell us whether you agree. If you have questions, please email: Michele@innovativepublichealth.org. Thank 
you! 
Those who do not complete an online assent form will be sent a reminder:  
Hi [Name] – It’s [RA] from BullyDown again. Just a quick reminder – please click on this link and tell us whether you 
agree for us to use your answers from the discussion groups. If you have questions, please email: 
Michele@innovativepublichealth.org. Thank you! 
Those who respond, will receive this message: Thank you for letting us know. Have a great rest of your week! 
For those who do not respond, we will return to the schools with paper assents and ask youth to complete them then.  
We will destroy the focus group data from youth who do not provide assent. 
Beta test: Based upon the enrollment experience and our plan to ensure text buddies in the intervention arm are paired 
with people at different schools/settings, it may be difficult to fully randomize all new beta test participants. In this case, 
we may randomize some youth to the intervention or control group, and then purposefully assign other youth to the 
intervention. 
Retention strategies.  
In her 36-school longitudinal study, Dr. Espelage had a retention rate of 88% at 12 months.98 Our BullyDown beta test 

 
 
had a 100% retention rate at one-month post-intervention follow-up. We anticipate a similarly strong retention rate at 3 
months post-intervention end in the current study. To achieve this, we propose: 
Incentives. All youth recruited via schools or Boys and Girls Club via active parental permission will receive $5 
for returned parental permission and youth assent forms irrespective of whether permission was granted. 
Research participants in the FGs and CATs will each receive $30 for their participation. The FG participants will 
receive $15 for each day they participate. CAT participants will receive $15 for returning their feedback on the text 
messages and $15 for taking part in the online discussion. Youth in the beta test will receive $25 after they complete the 
Intervention end survey. Those in the RCT will receive $15 for each survey they complete for a total of up to $45. 
Incentives will be fulfilled with Amazon gift cards because they can be distributed online, monitored to ensure that they 
have been used, and reissued if necessary. These incentives are separate from the incentive provided for the returned 
parental permission form. 
Each school and Club may be given a $500 stipend for their participation each year; all organizations taking part 
in the RCT will be given the stipend. Teachers will not receive any compensation or incentive directly. 
Changed phone numbers. We have found stability of cell phone numbers to be high in Ybarra’s other text 
messaging-based programs. We expect the same will be true for BullyDown. In fact, none of the participants in the beta 
test dropped out because of non-working or changed phone numbers. If participants are non-responsive to program 
prompts, we will reach out to them via text, email, and telephone. If that is unsuccessful, a member of the research team 
will visit the school or Club to check on the participant. During the enrollment phase, we will explain to participants the 
importance of recording the project phone number somewhere convenient for them so they can contact us if they change 
their cell phone number. 
Assertive follow-up. For each data collection period, we will send automated reminders to youth who have not 
completed the survey via text messaging. We will engage in personalized outreach if the survey is not completed within a 
week of the initial invitation. If youth are attending brick and mortar school or BCGA Club, we also will schedule in-
school and in-Club data collection so that youth have multiple opportunities to complete the survey, especially if they 
were absent on a particular day. To maximize data, we also will include youth in the 3-month post-intervention survey 
even if they have missed the intervention end survey.  
Specific Aim 1: Finalize BullyDown content.  
Updating content to reflect current school interactions and realities (n=60). Current messages focus primarily on 
in-person scenarios. We will build upon the development work described above by updating the content so that it better 
reflects online as well as offline peer interactions. To do so, we will conduct FGs to obtain youth voice about their current 
experiences with peers online. While it is unlikely that schools will remain online indefinitely, technology is continually 
being infused into classroom exercises and so we anticipate online interactions to persist even in post-Covid-19 school 
curriculum. 
FGs will be stratified to promote a sense of shared experiences. Stratification characteristics may be sex, 
race/ethnicity, among others. To promote a diversity of experiences, we will purposefully recruit youth with a variety of 
bullying experiences. More information can be found in Recruitment and Retention.  
FGs will be conducted over multiple days on CiPHR’s highly interactive, bulletin board-style, password-protected 
website. This online data collection space has been fully vetted for acceptability and feasibility among adolescents in 
Ybarra’s previous projects with adolescents.51-57 We will ask participants to log on 2-3 times per day at their convenience 
and respond to moderator questions and comments from other participants.  
Example questions include: Now that you are going to school online, how has this changed the way you talk with 
other kids in your class? How does bullying look different now than when you went to school in-person? Share an 
example of how someone was bullied in class this past week. What did other kids do? What did your teacher do? 
Questions will be updated to reflect the academic setting when we are in field. 
Krueger and Casey58 recommend 3-4 FGs per participant category. Online FGs can accommodate more 
participants, such that they typically include about 20 participants. Because one or two FGs are needed to match a 
commensurate sample size as in-person FGs, we plan to conduct about three FGs. We expect coding stability and 
theoretical saturation, the point at which emerging ideas are repetitive and do not substantively alter the themes developed 
from initial interviews, to be achieved after the two FGs. If saturation has not occurred, we will conduct additional FGs. 

 
 
Please see Data Analysis for more information. 
We will monitor the Boards closely for inappropriate behavior but anticipate that the online format will facilitate 
the feeling of a safe place and ability to be open and honest about one’s experiences.  
Dr. Ybarra will review the current content and identify opportunities to update it based upon the FG data. We 
anticipate examples and scenarios will be expanded to refer to online interactions more directly. To remain within the 4 
message per day frequency, an additional week of messaging may be added, based upon how many messages can be 
simply adapted versus added. Once a draft is ready, Dr. Espelage will review and provide feedback. Ybarra will integrate 
the feedback and then send the revised content back to Espelage for review. This process will be iterative until they agree 
that the relevant themes have been well integrated. 
Integrating lessons learned into the content. Several participants in the beta test expressed confusion regarding the 
interactive Text Buddy feature. Further probing revealed that some of this was related to feelings of discomfort about 
discussing bullying with students they went to school with. Nonetheless, beta test participants sent between 0-52 messages 
to their Text Buddy, suggesting its acceptability potential. To improve the experience, we will expand the instructions for 
the Text Buddy feature and include more concrete prompts about what Buddies can talk about with each other (e.g., 
Maybe text your buddy and talk about what you think it would be like if you stood up for a friend being bullied at school 
tomorrow). We also will match youth with someone at a different school/setting and make sure that they are aware of this 
so they do not worry they are talking with a classmate they may know. 
Drafting attention matched control content. In the beta test, the control group received two messages a week 
focused on thanking them for being in the study. As has been done in Dr. Ybarra’s other interventions (e.g., R21 
CA135669; R34MH109296), we will draft attention-matched content to well position this R21 for a competitive R01 
submission. To ensure that youth in both arms receive the same ‘attention,’ content will address healthy lifestyle topics 
unrelated to bullying (e.g., fitness, nutrition).  
Confirming adapted content for acceptability in Content Advisory Teams (CAT; n=~70). The content will next be 
reviewed by a CAT comprised of ~24 students. Each CAT member will have one week to review the messages and 
provide feedback through an online platform. Next, an online group discussion will elicit reactions to the messages and 
program design. Possible questions include: How well do these messages speak to the experiences you and your friends 
have around bullying? How clear do you find the messages? How could we say things differently to make the messages 
more understandable? Once feedback from the CAT has been integrated, a second 24-member CAT of different students 
will confirm that the content has been improved. For instance, in a previous intervention development Ybarra’s team led, 
the second CAT noted that we included too many emojis, which we had added based upon feedback from the first CAT. 
Having two different groups review the content also ensures a wider range of opinions and feedback. The control content 
will be reviewed by a ~24-member CAT.  
Conducting a beta test to confirm the protocol, including the software functionality and surveys. Before rolling 
out the RCT, we may conduct a beta test to pilot the recruitment, enrollment, data collection, and randomization 
procedures, as well as to identify final opportunities to improve the content. In most cases, five is a sufficient number of 
testers to identify any issues. Here, we propose 15-20 youth so that ~10 youth will test the functionality of the intervention 
program and ~5 the functionality of the control program. Usability data will be collected weekly via text messaging from 
participants. We may send questions weekly via text messaging, or by phone for non-responders, about the program 
experience. At the end of the beta test, we may conduct one-on-one interviews to identify any additional experiential 
issues that need to be addressed before the RCT is launched.  
Pilot RCT of BullyDown (n = 150) 
Randomization and group assignment. Once youth complete the baseline survey, they will be randomly assigned 
to either the intervention (n = 76) or the control (n = 74) group. We will use a computerized minimization technique, as 
originally described by Taves59 and more recently reviewed by others,60-62 to balance the sample on youth key indicators. 
These may include sex, school or Club, and bullying perpetration history. Treatment assignment will be generated and 
maintained at a password-protected Internet website monitored by Dr. Ybarra.  
Contamination. Studies across participant populations and behavioral targets report that contamination is 
uncommon and does not affect study outcomes.63,64,65,66 Give this, and to be efficient with R21 resources, students will be 
randomized at the individual rather than school or Club level. As we have done previously,63,64 we will ask questions at 
intervention end that are specific to the BullyDown content. If indicated, we will include the number of correct exposure 

 
 
questions as a proxy for contamination in multivariate models. 
Data collection. Measures will be collected online at baseline, intervention end, and 3 months post-intervention 
end. All participants, regardless of study condition, will complete assessments at each time point. Youth will be able to 
complete the surveys on their own (e.g., at home). If they are attending the brick and mortar school or Club, they also will 
be able to complete the survey at school or Club in the computer lab if they would prefer. To facilitate self-completion, 
youth will receive an invitation text message with a unique URL to access the survey. This allows participants to complete 
the survey at a time and place convenient for them. Alternatively, they can sign up for one of the survey sessions at school 
or Club that the research staff will schedule with the school or Club administration. To avoid unintentional influence that 
school and Club staff may have on youth survey answers, only research staff will be present in the computer lab during 
survey administration.  
 
3D. MEASUREMENTS AND STUDY INSTRUMENTS 
Most of the proposed measures below were used in the beta test, suggesting the survey length can be feasibly 
completed by adolescents in middle school or between the ages of 12-14 year of age. Measures are as follows: 
Table 3. BullyDown assessment measures 
Main 
Outcome 
measures 
(intervention 
end) 
Acceptability 
We will measure intervention acceptability and tolerability67 with a combination of open-ended 
questions (e.g., “What aspect of the program did you like the least?”) and questions that form a 
scale of intervention acceptability (e.g., I liked my Text Buddy; α = .87). Participants will 
estimate the number of messages they received in the past week and percent that they read. The 
program will record the number of weekly level-up questions answered, and the number 
answered correctly, as well as the number of Text Buddy and Forever Friend messages sent.
Feasibility 
We will monitor the number of eligible/ineligible youth, the number of parental permission 
forms received, the number of youth invited but who decline to take part, and the retention rate.
Preliminary 
behavioral 
outcome 
measures  
(Intervention 
end and 3-
month 
follow-up) 
 
Bullying 
perpetration  
Bullying will be queried by type (e.g., relational, physical), mode (e.g., online, in-person), and 
place (e.g., at home, at school).68 Youth who endorse any question at a frequency of 3 times or 
more in the past 30 days (to reflect repetitive behavior) will be coded as engaging in bullying.
Other 
aggression 
Youth who report engaging in the above bullying perpetration items, but less frequently so, 
will be coded as engaging in aggressive acts.68
Bystander 
behavior 
Youth will be asked if, in the past 30 days, they have seen or heard about (e.g., as a rumor) 
someone being bullied. Those who say yes will be asked what they did in response. Options 
will include: nothing, telling an adult, talking to the victim, and standing up to the bully.
Exogenous variables include: 1) SEL components: Communication – youth will be presented with four different 
scenarios and asked to identify whether it reflects passive, aggressive, or assertive communication. Causey & Dubow’s 
Self-report Coping Measure will be used to measure stress coping and problem-solving. Both are 8-item subscales with 
acceptable reliability (α = 0.84 and α = 0.84, respectively). Problem-solving asks youth how often they, for example, 
“Change something so things will work out.” Empathy will be assessed with the 5-item Empathy subscale in the Teen 
Conflict Scale (α = .62).69 Respect for diversity will be measured with the 4-item Support for Cultural Pluralism subscale 
of the Inventory of School Climate (α = 0.68).70 Identifying bullying will be measured by showing participants various 
“features” and asking them to identify those that reflect bullying (i.e., repetitive, with differential power, intent) Attitudes 
towards bullying and aggression will be measured with the 4-item University of Illinois Positive Attitudes toward 
Bullying Scale (α = 0.81).71 Bystanders’ intentions will be measured with the 5-item University of Illinois Willingness to 
Intervene scale (α = .62).72 Anger and hostility will be measured with the University of Illinois Anger Scale (α = .81; e.g., 
“I got in a physical fight because I was angry”).73 Hostility will be measured with the Symptom Checklist-90 subscale, 
which is a 6-item scale (α = .87).74 Impulsivity will be measured with the 4-item Impulsivity subscale from the Teen 
Conflict Survey (α = .62).75,76 2) Peer victimization: Victimization questions that mirror the bullying questions described 
will be asked of all youth. Those who say they have been victimized in at least one way 3 or more times in the past 30 
days, by someone with more power than them, will be coded as being bullied. Those who report less frequent 
victimization and/or an aggressor with equal power will be coded as a victim of other aggressive acts. 3) Exposure to 
domestic violence will be measured using an item from the Juvenile Victimization Questionnaire (JVQ).77 This well-

 
 
validated survey question has been used successfully in Dr. Ybarra’s Growing up with Media survey, which included 
youth as young as 10 years old.78 4) Exposure to community violence: Four JVQ questions will measure exposure to 
community violence (e.g., Have you ever seen someone get attacked or hit on purpose?).77 5) Alcohol use: We will use 
the YRBS question to measure alcohol use in the past year and in the past 30 days.79 6) Literacy will be measured with 
the Comprehension subscale of the Stanford Diagnostic Reading Test80. It was designed to be used by students with low 
literacy and has high internal validity (α = .84-.90)81. And 7) Demographic characteristics: Sex assigned at birth, gender 
identity, sexual identity, age, grade, cell phone number, and number of people living in the home; length of time using text 
messaging, length of time with current cell phone number, and average text messaging frequency. Data analysis is 
described in Statistical Design and Power. 
The reliance on youth self-report bears discussion. We have carefully considered the potential benefits and 
drawbacks of including teacher report. We decided not to because middle adolescents, those who are 12-14 years of age 
and typically in middle school, teachers see their students for one class a day. They are unlikely to know everything that 
occurs during a student’s day and whether they are bullying or being bullied in other classrooms. We decided not to 
collect parent report because parents rarely know about their child’s bullying behavior.82  
Scientific rigor of the experimental design, methods, and analyses. The rigor of the proposed study is high. 
For example, random assignment eliminates the possibility of youth being purposefully assigned to a particular study arm. 
The attention-matched control will reduce the likelihood that behavior change, if detected, is due to the ‘attention’ youth 
received by the daily text messages. Program fidelity will be high as the sending of messages is automated and the same 
for each student.  
3E. STATISTICAL DESIGN AND POWER  
Specific Aim 1: Finalize the BullyDown program.  
Transcripts of the FGs will be entered into Dedoose (formerly EthnoNotes),85 a mixed methods analysis program. 
Codes will be generated based on the first FG, reexamined, and refined through constant comparison with the second 
FG.86 A codebook will be created with codes, brief descriptions, and illustrative quotations.87 Codes will be both 
generated inductively and derived from our initial research questions (e.g., how youth interact with their peers and their 
teacher online). Dr. Ybarra’s project coordinator and Dr. Espelage’s graduate research assistant will code both FGs. To 
assure inter-coder reliability, we will calculate a kappa value and accept a minimum of 0.75 as the criterion for adequate 
agreement between coders.88 After coding is complete, we will extract themes using an iterative process to highlight 
evidence contradictory to our conclusions as well as the modal response.89 The analysis will result in a report that 
describes the findings and advises how to integrate them into program content.  
Specific Aim 2: Pilot test the 7-week BullyDown intervention for feasibility and acceptability in an RCT of 
150 adolescents either in middle school and/or 12-14 years of age.  
The feasibility of the BullyDown intervention will be determined by our recruitment and retention rates. If at 
least 75% of eligible studies assent and 80% complete the survey at 3-months post-intervention, we will deem these data 
points as supportive of a hypothesis of feasibility. 
The acceptability of program features (e.g., badges, Text Buddy) will be measured using 5-point Likert scales; 
average scores of 4 or more will be supportive of acceptability.90 Youth engagement with the program will be examined to 
identify potential trends in greater intervention impact (e.g., for those who use the Text Buddy feature). 
Post-intervention one-on-one interviews to illuminate the intervention experience will be analyzed using a 
thematic approach,91 as our aim will be more descriptive in nature, with a lower level of interpretation needed than is 
afforded by grounded theory or other qualitative analytical techniques.92 
Analyses examining behavioral outcomes will be primarily intended to inform power analyses for the larger 
R01. Data will be treated as intention-to-treat and complete case. We will conduct sensitivity analyses (e.g., missing = 
bullying behavior) to understand how this may affect our findings. Missing data that are not the main outcomes will be 
imputed.  
Baseline covariates will be examined to determine whether randomization performed as intended. Any that are 
significantly imbalanced across study arms will be included in multivariate analyses. 
We will use logistic regression to analyze outcomes (e.g., bullying perpetration, other aggressive behaviors; 
bystander behavior). Once gross estimates of intervention effect have been calculated, covariates will be added to examine 

 
 
whether they appear to mediate the relation between group assignment and the outcome. Three conditions must be met in 
order to claim that mediation is present:93-95 (1) a predictor (e.g., experimental arm) is significantly associated with a 
mediator (e.g., TPP motivation); (2) The mediator is significantly associated with an outcome (e.g., pregnancy); and (3) 
when one controls for the mediator, the effect of the predictor on the outcome becomes non-significant. If this effect 
retains significant but is attenuated, partial mediation is indicated. A 10% or greater change in the point estimate will be 
deemed clinically significant. 96,97 
Sex as a biological variable. It is possible that males will respond to the program differently than females because 
it feels more individual and perhaps, in this way, will resonate more with males.83 Conversely, perhaps because females 
text message at a greater volume than males,36,84 the format of this program may resonate more with them. As such, sex 
will be explicitly examined as a potential moderator. To understand whether the intervention impact varies by sex, we will 
test an interaction term, i.e., experimental group X sex. If the term is statistically significant, then it will suggest that boys 
and girls are impacted differentially by the intervention. 
To examine whether/how the intervention effect attenuates over time, we will use generalized estimation 
equations to analyze the data across follow-up periods using an identity link and exchangeable correlation structure. By 
entering an interaction term for experimental group X time, we can determine if the intervention is differently impactful at 
intervention end versus 3-months post-intervention.  
Should Dr. Ybarra need statistical guidance, we will take advantage of Dr. Espelage’s access to the biostatistics 
‘core’ at the University of North Carolina (please see Budget Justification). 
Power analysis 
Because the behavioral outcomes data in this R21 will be used to inform the power analyses of the future R01, we 
respectfully suggest that a power analysis for this study is premature. 
 
4. PROTECTION OF HUMAN SUBJECTS 
Ethical considerations  
The study will be conducted according to ethical principles stated in the Declaration of Helsinki (2013). IRB 
approval will be obtained before initiating the study. Assent and permission forms will take into consideration the 
well-being, free-will and respect of the participants, including respect of privacy. 
Sources of Information. 
RCT measures will be collected at baseline, intervention end, and 3-months post-intervention end in online surveys. All 
participants, regardless of study condition, will provide data at each time point. FG and CAT data will be collected online.  
4.a. Potential Risks 
The largest risk posed by participation in the study is inadvertent loss of confidentiality of one’s bullying behavior or 
experiences. Confidentiality also may be lost if someone sees the participant’s computer screen or intercepts his/her cell 
phone and reads the text messages. It is possible that survey questions or intervention topics may cause discomfort for 
some youth. It also is possible that conversations between Text Buddies will be harmful (e.g., bullying) in the RCT. 
Finally, it is possible that participants in the focus groups might be in the same online focus group as their bully. 
4.b. Adequacy of Protection against Risks 
The researchers take these potential risks very seriously. The PI will be responsible for all aspects of the project with 
respect to intervention design, data collection, and use of data. All aspects of the study will be approved by Pearl IRB, the 
Center for Innovative Public Health Research (CiPHR)’s OHRP-approved IRB of record prior to commencing data 
collection.  
Informed Youth Assent and Caregiver Permission. 
Youth assent and parental permission forms will receive final approval from the IRB before going into field. The assent 
information will be discussed with the participant by study staff either online, or at school or BCGA Clubs if youth are 
back in person. who are trained in Human Subjects Protections. Participants will be given the opportunity to ask any 
questions. They will be told that they can choose not to participate without any consequence. Given that children are a 

 
 
vulnerable population, we also will be collecting active parental permission.  
Because the study is low-risk, we will be sending the forms home for the parent and child to complete. To ensure that 
youth with language difficulties are able to understand the information they are being provided, the assent form will be 
written below the 6th grade level.  
To further facilitate the ease of recording one’s choice, we may offer the option to indicate one’s acceptance or declination 
of youth assent and parent permission via an online form (i.e., that a clicked “yes” will be sufficient). Accordingly, we are 
requesting a waiver of documentation of youth assent and parent permission for those who complete an online form 
because the study involves minimal risk.  
We will assent youth the same way that we are obtaining parental permission unless it is explicitly stated and documented 
otherwise. 
Phone numbers for Drs. Ybarra and Espelage, and Pearl IRB will be listed in the assent and permission forms in case a 
participant or a parent has a question or would like to discuss the study or any concerns.  
We may employ a passive parental permission protocol for youth recruited from the Boys and Girls Clubs, and also are 
requesting a waiver of parental permission for those recruited via social media. We are requesting this because: 
1. The research involves no more than minimal risk to the subjects. 
2. The waiver or alteration will not adversely affect the rights and welfare of the research participants. 
3. It would not be possible to do this research with active signed parent/guardian permission. In the case of the Boys 
and Girls Clubs, only 50% of active permission forms given to eligible youth were returned. In the case of social 
media, obtaining parental permission would be logistically impracticable. 
4. Due to the longitudinal nature of the research, identifiable private information is required to follow youth over 
time. 
5. Whenever appropriate, the subjects or legally authorized representatives will be provided with additional pertinent 
information after participation 
6. For the Boys and Girls Club youth, participant selection is based on club membership and not exclusionary. 
Youth who are recruited online will provide verbal assent during the enrollment call and again provide assent at the 
beginning of the baseline survey. Youth recruited via Boys and Girls Clubs who are enrolled using a passive parental 
permission protocol will have the opportunity to ask questions when the study activity is initially introduced to them. 
Because the forms will be handed directly to parents, they will have the opportunity to ask questions about the study at 
that time. If either caregivers or youth have questions subsequently, they can utilize the contact information provided on 
the form to reach Drs. Ybarra and Espelage. 
Protection against Risk. 
Reports will only use aggregated data. No report of individual data will be provided. The core research team will have 
monthly meetings, and more frequently when in field; and will utilize real-time Internet technology to monitor the 
progress of the study. 
Research staff will help the participants set their phones to be password protected to limit access by others if they would 
like. We also will provide instruction on how to change their phone settings so that text messages do not scroll across the 
screen as an alert. Youth will be encouraged to only read messages when they are in a private space where they feel like 
they can engage with the content safely. We also will encourage youth to delete any messages that they do not want others 
to read. 
Youth will be encouraged to complete the online surveys in a private space where no one can see their screen. They will 
be able to pause and come back to the survey later if they are not in a safe place.  
Participants will be told in the enrollment meetings and in the assent form that some survey questions and program 
messages may make them feel uncomfortable. They will be encouraged to skip such questions or messages. They will also 
be told that they may withdraw from the program if they choose.  
Because we are recruiting only about 7 youth per school into the male and female focus groups, respectively, we think it is 
very unlikely that a bully victim’s buddy will be in the same focus group. Nonetheless, we will make it clear that 
participants can reach out to us at any time and stop their participation, including if they are uncomfortable with the other 
people who are taking part in the focus group. 

 
 
Monitoring of Text Buddy Communication. Text Buddies are paired intervention participants who will be 
encouraged to text message each other about to discuss program content throughout the intervention. For example: 
“Talking about your bullying experiences with friends can be hard. Now that you’re equipped with mad communication 
skills, try talking with your Buddy about the most recent time you saw someone bullied. What did you do? How did you 
feel? Text your Buddy and share.”  
Buddies will be paired based on sex and school / setting so that they will be at different schools / settings based upon 
findings from the earlier beta test. The software program will identify youth who match on these criteria and automatically 
pair them. Those who are not automatically paired will be flagged so that the research team can manually pair youth. 
We posit that social support from a Text Buddy will positively influence youth over time. However, balancing this with 
the privacy needs of our participants, who likely do not want to exchange contact information with a stranger, is 
important. As such, all Text Buddy messages are routed through the study server, replacing the need for this exchange. 
Participant cell phone numbers will never be distributed to other participants. To send a message, participants will simply 
text the dedicated BuddyLine, the program phone number that participants will use to communicate with their Buddy. 
This message is sent directly to the program server, which identifies the intended recipient, and routes the message to the 
assigned Text Buddy. This methodology will also allow us to measure the amount of interaction between Buddies 
accurately; if participants texted each other directly, we would have to rely on self-report to measure frequency of contact. 
For analysis, we will keep a count record to reflect the number of messages sent by each participant.  
Prior to the program start, we ask intervention participants to agree to a Text Buddy Code of Conduct that sets clear 
expectations for healthy interaction and support. We have closely monitored the interactions between Buddies during the 
beta test and did not see any inappropriate interactions. We also will block messages that contain specific key words (e.g., 
racial epithets, drug names, participant contact information such as social networking profile names) for review before 
they are sent through to the Buddy. Study staff closely monitor the Buddy interactions (16 hours/day) while in field. 
Concerning message content is elevated to the PI within 24 hours. Study staff are assigned to monitor the messages in 
real time from 8am – 10pm pacific standard time. Staff are able to receive text message alerts on their phone when a 
Buddy message is detected should they be away from their computer during their monitoring session. We note that across 
three projects that Ybarra’s team has implemented this feature, never has there been a case of on buddy bullying another. 
Providing resources and referrals. This is not a mental health intervention. Nonetheless, we will provide resources 
and referrals for youth. For example, we will have a resource page on the intervention website for youth who are upset or 
need help. The 24-hour crisis hotline, 1-800-SUICIDE, will be listed in case youth require immediate intervention. Those 
who are experiencing thoughts or feelings that are upsetting will be referred to the National Information Mental 
Information Center for assistance identifying a treatment provider. The National Mental Health Information Center is 
funded by the Center for Mental Health Services within the Substance Abuse and Mental Health Services Administration 
(SAMHSA). As one of SAMHSA’s clearinghouses, the Center provides mental health information and treatment referrals 
to a varied audience throughout the United States and its territories, via the toll-free number (800-789-2647) and website 
(www.mentalhealth.samhsa.gov) that offers a service locator for finding local resources.  
If someone articulates a plan to hurt themselves or others when monitoring Text Buddy messages or other participant 
program interactions, we will advise them to call 911 or go to their nearest emergency room. We also will provide the 
suicide crisis hotline. If crisis language is observed during monitoring, youth will be referred to Christina Patts, LICSW, 
for assessment and referral. Ms. Patts has extensive experience implementing this type of crisis contact for youth taking 
part in research studies. She is the clinician on call for Dr. Ybarra’s Girl2Girl and Growing up with Media studies. She 
has also served as consultant for studies measuring child sexual and physical abuse. 
Protecting privacy. Dr. Ybarra’s team has conducted several online national surveys with very sensitive questions. 
There are many benefits to this methodology, including increased self-disclosure compared to interviewer-based data 
collection, and convenience and increased safety for the participant as s/he can decide when and where to best answer the 
survey. Participants will be given the option to complete the survey on their own, or at school or the Club if they are back 
in brick and mortar settings. Participants will receive a personalized link which they can click on to take them directly into 
the survey. To further ensure privacy, we also ask participants to complete the survey in a private place – and then ask 
again at the beginning of the survey to see if they are. If they indicate they are not, they are encouraged to pause the 
survey and come back to it later. 
No names will be collected except on youth assent and parent permission forms. Study materials will be 
safeguarded as follows: informed parent permission and youth assent forms will be kept under double lock in filing 

 
 
cabinets in separate physical locations. Secure UNC SOE server if not physically collected. 
Secure electronic transmissions and storage of data. All data are password protected with strong encrypted 
passwords and is transmitted securely using SSL (TLS) 128-bit encryption across the Internet (HTTP). SSL provides 
front-end users with the assurance of access to a valid, “non-spoofed” site and prevents data interception or tampering 
with sensitive information. The 128-bit encryption is the preferred security level of government and financial institutions. 
To ensure against the remote possibility that an intruder gains access to stored data, all data stored are protected with 
strong passwords that are also encrypted, making use of any acquired data nearly impossible. Any Personally Identifiable 
Information (PII) is securely encrypted and stored separately from study data. All access to participant data is limited to 
access via a secure VPN network, making it impossible to access otherwise. 
Data are located on our dedicated server in Liquid Web's secure SSAE-16 Compliant datacenter in Lansing, Michigan. 
Dedicated servers eliminate security issues involved with shared hosting environments where hundreds of websites and 
users reside on one shared web server, as well as ensuring both physical and network security. 
Dedicated server facility security includes: 
 
24/7/365 Manned Facilities 
 
CCTV Security Cameras Covering Inside, Outside and All Entrances of Data Centers 
 
Site Entrances Controlled By Electronic Perimeter Access Card System 
 
Sites Remotely Monitored By 3rd Party Security Company 
 
Entrances Secured by Mantraps with Interlocking Doors 
 
SSAE-16 & HIPAA Compliant, Safe Harbor Certified 
Having a backup copy offsite is critical for protecting against data loss due to a catastrophic event (e.g., an earthquake). 
The datacenter is equipped with redundant tier 1 bandwidth, ensuring minimal latency and fast connections to all points of 
the global Internet. Datacenter access is strictly limited to technical staff. Electronic security systems control datacenter 
access and are accompanied by a full complement of motion-detecting security cameras that monitor the entire facility.  
Our servers are equipped with daily backups to an external hard drive, as well as extensive server-hardening, 
firewall protection, brute force detection and evasion, Apache denial-of-service attack prevention/protection, and email 
virus filtering. In addition, the server is monitored by Liquid Web 24/7 to ensure 100% uptime and receives daily security 
audits and monthly vulnerability scans. 
A description of the text messaging platform. To further promote data security, the BullyDown program is 
delivered through CiPHR’s proprietary platform that has been developed and improved upon over several research 
projects (e.g., R01 TW007918, R21 CA135669, R01MH096660 R34MH109296, R34MH108781, R01 HD095648). The 
system was developed using PHP, Javascript and Jquery programming languages and utilizes MySQL Databases running 
on a Linux (LAMP) based server. The system makes us of the Twilio gateway service's API to both send and receive text 
messages. 
4.c. Potential Benefits of the Proposed Research to the Participants and Others  
Extant research has demonstrated the link between incidence of school bullying and resulting adverse outcomes such as 
depression, anxiety, physical illness, school dropout, and suicide. Research also has identified a link between school 
bullying behaviors and high-intensity violence in schools (e.g., school shootings). To the extent that the BullyDown text 
messaging program is effective in reducing bullying in schools, we anticipate positive long-term societal effects for youth, 
including decreased need for delinquency and mental health services, enhanced positive youth development, improved 
academic achievement, and societal cost savings associated with these outcomes.  
4.d. Importance of the Knowledge to be Gained  
BullyDown will contribute to scientific knowledge in several important ways: 1) BullyDown will be delivered outside of 
school. By engaging with the content in a non-academic setting, youth may be more likely to apply their new behaviors 
across contexts, including bullying scenarios that take place online and other places where youth congregate. This also has 
the advantage of giving youth the opportunity to learn SEL components and interact with someone else in the program 
(i.e., a Text Buddy) without a potential perpetrator sitting in a desk nearby. It also bypasses situations when school-based 
programming is not viewed as "cool" by some students, resulting in their under-engagement in program activities, which 

 
 
can implicitly reinforce negative social norms. 2) In school-based intervention programs, school personnel are often called 
upon to implement the content. To do so, they need to be extensively trained to maximize efficacy. Increasingly, financial 
resources for training are simply not available in US public schools. BullyDown is administered through text messaging, 
bypassing training and competition with professional development time. 3) Compared to in-person interventions which 
may be vulnerable to variable implementation fidelity, all youth in the program receive the same content in the same 
order, thereby ensuring fidelity. Also compared to “apps” that require a smart phone and data, all cell phones are text 
messaging-capable. 4) The meta-analytic results described above reflect the best-case scenario. Although all states require 
schools to implement bullying prevention programming, there is significant variation in what is mandated, resulting in a 
wide spectrum of programming offered in schools across the US. In the FGs we conducted in the BullyDown 
development process, very few youth talked about a comprehensive bullying prevention curriculum in their school. 
BullyDown can be delivered as a “booster” outside of school time that enhances whatever programming is being offered 
by schools. In doing so, one of the benefits of BullyDown is that it helps to ensure all youth are exposed to the basic tenets 
of a bullying prevention curriculum. 5) Given the focus on victimization that most of the current prevention programs 
have, our focus on perpetration is innovative. 
4.e. Data and Safety Monitoring Plan 
The PI will be responsible for all aspects of the project with respect to intervention design, data collection, and use of 
data. Pearl IRB will approve all aspects of the study prior to commencing data collection. Informed youth assent and 
parental permission will be obtained from all participants. Participants will have access to 24-hour referral information as 
well as phone numbers for the study administrators (i.e., PI and IRB). 
 
Data Monitoring 
The PI will be responsible for monitoring data to ensure the safety of participants. Additionally, the IRBs will be 
responsible for data and safety monitoring.  
In order to ensure privacy for participants and to minimize risk, we will be using computer- and text messaging-based data 
collection procedures. This reduces the number of people who view the data and increases self-disclosure on sensitive 
topics. Access to the data will be password protected. 
Each participant will be given a random unique identifier in the data set which will be stripped of all personal information 
to protect confidentiality. Datasets used for analysis will contain project identification numbers but neither names nor 
other identifying information such as email addresses. Identification information will be retained by CiPHR for the 
duration of the study and stored separately from the responses provided by subjects.  
Collaborators will receive data stripped of personal identifiers. To ensure complete confidentiality, access to the key 
linking personal identifiers to usernames and passwords will be restricted to key personnel. Reports will not identify 
individual adolescents. Dr. Ybarra will oversee the data storage and reporting procedures. Upon completion of the 
analysis of the data collected in this study, records will be destroyed. Reports will only use aggregated data. The core 
research team will have weekly meetings utilizing real-time Internet technology to monitor the progress of the study.  
 
Adverse Events 
Dr. Ybarra will review the progress of the project and data being collected to ensure that potential adverse effects are 
identified, if they occur, and reported to both IRBs. This includes unanticipated problems involving risks to subjects or 
others. Any action recommended by the IRBs to the PI will be implemented immediately in order to minimize further risk. 
The adverse event report will further be communicated to other entities (e.g., NIH) in a timely manner as appropriate 
based on their policies. Dr. Ybarra will also be responsible for reporting to the project officer if the IRBs or any other 
entity temporarily or permanently suspends an NIH-funded trial. All notifications will be done via email immediately 
followed with a certified letter. 
 
 

 
 
References  
 
1. 
Nansel TR, Overpeck M, Pilla RS, Ruan WJ, Simons-Morton B, Scheidt P. Bullying behaviors among US youth: 
Prevalence and association with psychosocial adjustment. Journal of the American Medical Association. 
2001;285(16):2094-2100. 
2. 
Sourander A, Helstela L, Helenius H, Piha J. Persistence of bullying from childhood to adolescence--a 
longitudinal 8-year follow-up study. Child Abuse & Neglect. 2000;24(7):873-881. 
3. 
Bond L, Carlin JB, Thomas L, Rubin K, Patton G. Does bullying cause emotional problems? A prospective study 
of young teenagers. BMJ. 2001;323(7311):480-484. 
4. 
Arseneault L, Bowes L, Shakoor S. Bullying victimization in youths and mental health problems: ‘Much ado 
about nothing’? Psychological Medicine. 2010;40(5):717-729. 
5. 
Brunstein Klomek A, Sourander A, Gould M. The association of suicide and bullying in childhood to young 
adulthood: a review of cross-sectional and longitudinal research findings. Canadian Journal of Psychiatry. 
2010;55(5):282-288. 
6. 
Merrell KW, Gueldner BA, Ross SW, Isava DM. How effective are school bullying intervention programs? A 
meta-analysis of intervention research. School Psychology Quarterly. 2008;23(1):26-42. 
7. 
Ttofi MM, Farrington DP. Effectiveness of school-based programs to reduce bullying: A systematic and meta-
analytic review. Journal of Experimental Criminology. 2011;7(1):27-56. 
8. 
Ferguson CJ, San Miguel C, Kilburn JC, Sanchez P. The effectiveness of school-based anti-bullying programs: A 
meta-analytic review. Criminal Justice Review. 2007;32(4):401-414. 
9. 
Smith JD, Schneider BH, Smith PK, Ananiadou K. The effectiveness of whole-school antibullying programs: A 
synthesis of evaluation research. School Psychology Review. 2004;33(4):547-560. 
10. 
Vreeman RC, Carroll AE. A systematic review of school-based interventions to prevent bullying. Archives of 
Pediatrics & Adolescent Medicine. 2007;161(1):78-88. 
11. 
Ttofi MM, Farrington DP. Bullying prevention programs: The importance of peer intervention, disciplinary 
methods and age variations. Journal of Experimental Criminology. 2012;8(4):443-462. 
12. 
Yeager DS, Fong CJ, Lee HY, Espelage DL. Declines in efficacy of anti-bullying programs among older 
adolescents: Theory and a three-level meta-analysis. Journal of Applied Developmental Psychology. 2015;37:36-
51. 
13. 
Lenhart A. Teens, social media, & technology overview 2015. Washington, DC: Pew Internet & American Life 
Project;2015. 
14. 
The Common Sense Census: Media use by tweens and teens. Common Sense. 
https://www.commonsensemedia.org/Media-use-by-tweens-and-teens-2019-infographic. Published 2019. 
Accessed March 16, 2020. 
15. 
Ybarra ML, Jiang Y, Free C, Abroms LC, Whittaker R. Participant-level meta-analysis of mobile phone-based 
interventions for smoking cessation across different countries. Preventive Medicine. 2016;89:90-97. 
16. 
Preston KE, Walhart TA, O'Sullivan AL. Prompting healthy behavior via text messaging in adolescents and 
young adults. American Journal of Lifestyle Medicine. 2011;5(3):247-252. 
17. 
Wei J, Hollin I, Kachnowski S. A review of the use of mobile phone text messaging in clinical and healthy 
behaviour interventions. Journal of Telemedicine and Telecare. 2011;17(1):41-48. 
18. 
Cole-Lewis H, Kershaw T. Text messaging as a tool for behavior change in disease prevention and management. 
Epidemiologic Reviews. 2010;32(1):56-69. 
19. 
Ybarra ML, Prescott TL, Espelage DL. Stepwise development a text messaging-based bullying prevention 
program for middle school students (BullyDown). JMIR mHealth uHealth. 2016;4(2):e60. 

 
 
20. 
Ybarra ML, Espelage DL, Valido A, Hong JS, Prescott TL. Perceptions of middle school youth about school 
bullying. J Adolesc. 2019;75:175-187. 
21. 
Belfield C, Bowden B, Klapp A, Levin H, Shand R, Zander S. The Economic Value of Social and Emotional 
Learning. New York, NY: Center for Benefit-Cost Studies in Education, Teachers College, Columbia 
University;2015. 
22. 
Ragozzino K, Utne O’Brien M. Social and emotional learning and bullying prevention. Chicago, IL: 
Collaborative for Academic, Social, and Emotional Learning (CASEL);2009. 
23. 
Guerra NG, Bradshaw CP. Linking the prevention of problem behaviors and positive youth development: Core 
competencies for positive youth development and risk prevention. New Directions for Child and Adolescent 
Development. 2008;2008(122):1-17. 
24. 
Greenberg M, Weissberg R, O'Brien M, et al. Enhancing school-based prevention and youth development through 
coordinated social, emotional, and academic learning. American Psychologist. 2003;58(6-7):466-474. 
25. 
Hawkins JD, Smith BH, Catalano RF. Social development and social and emotional learning. In: Zins JE, 
Weissberg RP, Wang MC, Walberg HJ, eds. Building academic success on social and emotional learning: What 
does the research say? New York, NY: Teachers College Press; 2004:135-150. 
26. 
Klomek AB, Kleinman M, Altschuler E, Marrocco F, Amakawa L, Gould MS. High school bullying as a risk for 
later depression and suicidality. Suicide and Life-Threatening Behavior. 2011;41(5):501-516. 
27. 
Ttofi MM, Farrington DP, Lösel F, Loeber R. Do the victims of school bullies tend to become depressed later in 
life? A systematic review and meta-analysis of longitudinal studies. Journal of Aggression, Conflict and Peace 
Research. 2011;3(2):63-73. 
28. 
Ttofi MM, Farrington DP, Losel F, Loeber R. The predictive efficiency of school bullying versus later offending: 
a systematic/meta-analytic review of longitudinal studies. Criminal Behaviour and Mental Health. 2011;21(2):80-
89. 
29. 
Kumpulainen K, Räsänen E, Puura K. Psychiatric disorders and the use of mental health services among children 
involved in bullying. Aggressive Behavior. 2001;27(2):102-110. 
30. 
Rivers I, Poteat VP, Noret N, Ashurst N. Observing bullying at school: The mental health implications of witness 
status. School Psychology Quarterly. 2009;24(4):211-223. 
31. 
Miller S, Williams J, Cutbush S, Gibbs D, Clinton-Sherrod M, Jones S. Dating Violence, Bullying, and Sexual 
Harassment: Longitudinal Profiles and Transitions Over Time. Journal of Youth & Adolescence. 2013;42(4):607-
618. 
32. 
Ryoo JH, Cixin W, Swearer SM. Examination of the change in latent statuses in bullying behaviors across time. 
School psychology quarterly : the official journal of the Division of School Psychology, American Psychological 
Association. 2015;30(1):105-122. 
33. 
Kärnä A, Voeten M, Little TD, Alanen E, Poskiparta E, Salmivalli C. Effectiveness of the KiVa Antibullying 
Program: Grades 1–3 and 7–9. Journal of Educational Psychology. 2013;105(2):535-551. 
34. 
Bull S, Ezeanochie N. From Foucault to Freire through Facebook: Toward an integrated theory of mHealth. 
Health Education & Behavior. 2015. 
35. 
Sherman SE, Takahaski N, Kalra P, et al. Care coordination to increase referrals to smoking cessation telephone 
counseling: A demonstration project. American Journal of Managed Care. 2008;14(3):141-148. 
36. 
Lenhart A, Ling R, Campbell S, Purcell K. Teens and mobile phones. Washington, DC: Pew Internet & American 
Life Project;2010. 
37. 
Aitken M. Patient adoption of mHealth: Use, evidence and remaining barriers to mainstream acceptance. 
http://www.vcbeat.net/wp-content/uploads/2015/10/IIHI_Patient_Adoption_of_mHealth.pdf. Published 2015. 
Accessed April 1, 2016. 
38. 
Lattie EG, Schueller SM, Sargent E, et al. Uptake and usage of IntelliCare: A publicly available suite of mental 
health and well-being apps. Internet Interventions. 2016;4(2):152-158. 

 
 
39. 
Whittaker R, McRobbie H, Bullen C, Borland R, Rodgers A, Gu Y. Mobile phone-based interventions for 
smoking cessation. Cochrane Database of Systematic Reviews. 2012(11):CD006611. 
40. 
Garofalo R, Kuhns LM, Reisner SL, Mimiaga MJ. Behavioral interventions to prevent HIV transmission and 
acquisition for transgender women: A critical review. Journal of acquired immune deficiency syndromes (1999). 
2016;72. 
41. 
Ybarra ML, Prescott TL, Phillips II GL, Bull SS, Parsons JT, Mustanski B. Pilot RCT results of an mHealth HIV 
prevention program for sexual minority male adolescents. Pediatrics. 2017;140(1). 
42. 
Espelage DL, Low S, Polanin JR, Brown EC. Clinical trial of Second Step© middle-school program: Impact on 
aggression & victimization. Journal of Applied Developmental Psychology. 2015;37(0):52-63. 
43. 
Espelage DL, Van Ryzin M, Low S, Polanin J, Brown E. Clinical trial of Second Step© middle-school program: 
Impact on bullying, cyberbullying, homophobic teasing, & sexual harassment perpetration. School Psychology 
Review. 2015;37:52-63. 
44. 
Elias MJ, Zins JE, Weissberg RP, et al. Promoting social and emotional learning: Guidelines for educators. 
Alexandria, VA: Association for Supervision & Curriculum Development; 1997. 
45. 
Huesmann LR. An information processing model for the development of aggression. Aggressive Behavior. 
1988;14(1):13-24. 
46. 
DeWall CN, Anderson CA, Bushman BJ. The general aggression model: Theoretical extensions to violence. 
Psychology of Violence. 2011;1(3):245-258. 
47. 
Bushman BJ. Violent media exposure and hostile appraisals: A meta-analytic review. Aggressive Behavior. 
2016;42(6):605-613. 
48. 
Anderson CA, Bushman BJ. Human aggression. Annual Review of Psychology. 2002;53(1):27-51. 
49. 
Bushman BJ, Anderson CA, Bartholow BD, et al. Screen violence and youth behavior. Pediatrics. (in press). 
50. 
Readable. Readability Formulas. https://readable.com/features/readability-formulas/. Published 2020. Accessed. 
51. 
Ybarra ML. Identifying positive aspects of youth Internet use: The next step in prevention (5R01HD057191-03). 
In. Santa Ana, CA: Eunice Kennedy Shriver National Institute of Child Health & Human Development; 2009-
2013. 
52. 
Ybarra ML. Mapping etiological pathways to SV perpetration from childhood to young adulthood 
(5R01CE001543-03). In. Santa Ana, CA: Centers for Disease Control and Prevention, National Center for Injury 
Prevention and Control; 2009-2012. 
53. 
Ybarra ML. Course and Prediction of Sexual Perpetration in Adolescence through Young Adulthood 
(R01HD083072). In. San Clemente, CA: National Institute of Child Health & Human Development; 2015-2020. 
54. 
Ybarra ML. Mental health effects of Internet-mediated violence (1-U49-CE000206-01). In. Irvine, CA: Centers 
for Disease Control and Prevention, National Center for Injury Prevention and Control; 2004-2007. 
55. 
Ybarra ML. Harnessing text messaging to reduce teen pregnancy among LGB girls (1 TP2AH000035-01-00). In. 
San Clemente, CA: Office of Adolescent Health; 2015-2020. 
56. 
Ybarra ML, Mustanski B. Harnessing the power of text messaging to invigorate AMSM HIV preventive behavior 
(R01MH096660). In. Center for Innovative Public Health Research, San Clemente, CA: National Institute of 
Mental Health; 2012-2015. 
57. 
Ybarra ML, Espelage D. Texting 4 Bullying Prevention. In. San Clemente, CA: HopeLab Foundation; 2014. 
58. 
Krueger RA, Casey MA. Focus groups: A practical guide for applied research. 4th ed. Thousand Oaks, CA: Sage; 
2009. 
59. 
Taves DR. Minimization: A new method of assigning patients to treatment and control groups. Clinical 
Pharmacology and Therapeutics. 1974;15(5):443-453. 
60. 
Therneau TM. How many stratification factors are too many to use in a randomization plan? Control Clin Trials. 

 
 
1993;14(2):98-108. 
61. 
McEntegart DJ. The pursuit of balance using stratified and dynamic randomization techniques: An overview. 
Drug Information Journal. 2003;37(3):293-308. 
62. 
Scott NW, McPherson GC, Ramsay CR, Campbell MK. The method of minimization for allocation to clinical 
trials: A review. Controlled Clinical Trials. 2002;23(6):662-674. 
63. 
Fritz K, McFarland W, Wyrod R, et al. Evaluation of a peer network-based sexual risk reduction intervention for 
men in beer halls in Zimbabwe: Results from a randomized controlled trial. AIDS and Behavior. 
2011;15(8):1732-1744. 
64. 
Ybarra ML, Bull SS, Prescott TL, Korchmaros JD, Bangsberg DR, Kiwanuka JP. Adolescent abstinence and 
unprotected sex in CyberSenga, an Internet-based HIV prevention program: Randomized clinical trial of efficacy. 
Public Library of Science: ONE. 2013;8(8):e70083. 
65. 
Simmons N, Donnell D, Ou SS, et al. Assessment of contamination and misclassification biases in a randomized 
controlled trial of a social network peer education intervention to reduce HIV risk behaviors among drug users 
and risk partners in Philadelphia, PA and Chiang Mai, Thailand. AIDS Behav. 2015;19(10):1818-1827. 
66. 
Lang DL, DiClemente RJ, Hardin JW, Crosby RA, Salazar LF, Hertzberg VS. Threats of cross-contamination on 
effects of a sexual risk reduction intervention: fact or fiction. Prev Sci. 2009;10(3):270-275. 
67. 
Center on AIDS and Community Health. Diffusion of Effective Behavioral Interventions (DEBI). 
http://www.effectiveinterventions.org/. Published 2010. Accessed. 
68. 
Ybarra ML, Espelage DL, Mitchell KJ. Differentiating youth who are bullied from other victims of peer-
aggression: The importance of differential power and repetition. Journal of Adolescent Health. 2014;55(293-
300):Advance online publication. 
69. 
Bosworth K, Espelage D. Teen conflict survey. Bloomington, IN: Center for Adolescent Studies, Indiana 
University; 1995. 
70. 
Brand S, Felner RD, Shim MS, Seitsinger A, Dumas T. Middle school improvement and reform: Development 
and validation of a school-level assessment of climate, cultural pluralism, and school safety. Journal of 
Educational Psychology. 2003;95(3):570-588. 
71. 
Espelage D, Bosworth K, Simon T. Short-term stability and prospective correlates of bullying in middle-school 
students: an examination of potential demographic, psychosocial, and environmental influences. Violence and 
Victims. 2001;16(4):411-426. 
72. 
Espelage D, Green H, Polanin J. Willingness to intervene in bullying episodes among middle school students: 
Individual and peer-group influences. Journal of Early Adolescence. 2012;32(6):776-801. 
73. 
Espelage DL, Stein ND. The University of Illinois and Wellesley College: Student Behavior Survey. Champaign, 
IL: University of Illinois at Urbana-Champaign, Department of Educational Psychology; 2006. 
74. 
Derogatis LR, Rickels K, Rock AF. The SCL-90 and the MMPI: A step in the validation of a new self-report 
scale. British Journal of Psychiatry. 1976;128:280-289. 
75. 
Espelage DL, Bosworth K, Simon TR. Examining the social context of bullying behaviors in early adolescence. 
Journal of Counseling and Development. 2000;78(3):326-333. 
76. 
Bosworth K, Espelage DL, Simon T. Factors associated with bullying behavior in middle school students. Journal 
of Early Adolescence. 1999;19:341-362. 
77. 
Finkelhor D, Hamby SL, Ormond R, Turner H. The Juvenile Victimization Questionnaire: Reliability, validity, 
and national norms. Child Abuse & Neglect. 2005;29(4):383-412. 
78. 
Ybarra ML, Diener-West M, Markow D, Leaf P, Hamburger M, Boxer P. Linkages between Internet and other 
media violence with seriously violent behavior by youth. Pediatrics. 2008;122(5):929-937. 
79. 
Centers for Disease Control and Prevention. Youth Risk Behavior Surveillance --- United States, 2013. Atlanta, 
GA: Centers for Disease Control and Prevention;2014. 

 
 
80. 
Karlsen B, Gardner EF, eds. Adult Basic Learning Examination (ABLE).: Form E. . Harcourt Brace Jovanovich; 
1986. 
81. 
Nagy W, Berninger VW, Abbott RD. Contributions of morphology beyond phonology to literacy outcomes of 
upper elementary and middle-school students. Journal of Educational Psychology. 2006;98(1):134–147. 
82. 
Unever J, Cornell D. Middle school victims of bullying: Who reports being bullied? Aggressive Behaviour. 
2004(30):373-388. 
83. 
Orr AJ. Gendered Capital: Childhood Socialization and the “Boy Crisis” in Education. Sex Roles. 
2011;65(3):271-284. 
84. 
Lenhart A. Teens, smartphones & texting. Washington, DC: Pew Internet & American Life Project; Mar 19 2012. 
85. 
Lieber E, Weisner TS, Presley M. EthnoNotes: An Internet-based field note management tool. Field Methods. 
2003;15(4):405-425. 
86. 
Taylor SJ, Bogdan R. Introduction to qualitative research methods. 3rd ed. New York: John Wiley; 1998. 
87. 
MacQueen K, McLellan E, Kay K, Milstein B. Codebook development for team-based qualitative research. 
Cultural Anthropology Methods Journal. 1998;10(2):31-36. 
88. 
Hruschka DJ, Schwartz D, St.John DC, Picone-Decaro E, Jenkins RA, Carey JW. Reliability in coding open-
ended data: Lessons learned from HIV behavioral research. Field Methods. 2004;16(3):307-331. 
89. 
Ryan GW, Bernard HR. Techniques to identify themes. Field Methods. 2003;15(1):85-109. 
90. 
Tarnowski KJ, Simonian SJ. Assessing treatment acceptance: the Abbreviated Acceptability Rating Profile. 
Journal of Behavior Therapy and Experimental Psychiatry. 1992;23(2):101-106. 
91. 
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006;3(2):77-
101. 
92. 
Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: Implications for conducting a 
qualitative descriptive study. Nursing & Health Sciences. 2013;15(3):398-405. 
93. 
Little TD, Card NA, Bovaird JA, Preacher KJ, Crandall CS. Structural equation modeling of mediation and 
moderation with contextual factors. In: Little TD, Bovaird JA, Card NA, eds. Modeling Contextual Effects in 
Longitudinal Studies. Mahwah, NJ: Lawrence Erlbaum Associates; 2007:207–230. 
94. 
Baron RM, Kenny DA. The moderator-mediator variable distinction in social psychological research: Conceptual, 
strategic, and statistical considerations. Journal of Personality and Social Psychology. 1986;51:1173-1182. 
95. 
Preacher K, Rucker D, AF H. Addressing moderated mediation hypotheses: Theory, methods, and prescriptions. 
Multivariate Behavioral Research. 2007;42(1):185-227. 
96. 
Ybarra ML, Mitchell KJ, Palmer NA, Reisner SL. Online social support as a buffer against online and offline peer 
and sexual victimization among U.S. LGBT and non-LGBT youth. Child Abuse & Neglect. 2015;39:123-136. 
97. 
Ybarra ML, Boyd D. Can clans protect adolescent players of massively multiplayer online games from violent 
behaviors? International Journal of Public Health. 2015;60(2):267-276. 
98. 
Espelage DL, Low S, Polanin J, Brown E. The impact of a middle-school program to reduce aggression, 
victimization, and sexual violence. Journal of Adolescent Health. 2013;53(2):180-186.