Randomized Controlled Trial of an Online Machine Learning-Driven Risk Assessment and Intervention Platform for Increasing the Use of Crisis Services
Phase: NA
Conditions: Crisis Intervention, Suicide and Depression
Sponsor: Harvard University

31 August 2017 
Sponsor: Harvard University 
Investigator: Adam Jaroszewski, A.M. 
 
 
 
Randomized Controlled Trial of an Online Machine Learning-Driven Risk 
Assessment and Intervention Platform for Increasing the Use of Crisis Services 
 
 
 
 
Study Protocol 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
1 
Instructions: Complete all of the sections below (type an x in a Yes/No box or provide an 
answer).   
 
GENERAL INFORMATION 
Protocol # (if assigned): IRB17-1303 
Version Number or Date: August 31, 2017 
Principal Investigator Name: Matthew K. Nock, Ph.D. 
     Faculty         Graduate student         Post-doc          Undergraduate        Extension school student   
     Junior Fellow         Staff         Visiting Scholar        Other (specify):  
Faculty Sponsor (if Principal Investigator is not faculty):  
Other Advisor Name (if applicable):  
     Supervising lecturer         Instructor         Graduate student         Thesis advisor        Other 
(
if )
Protocol Title: Analysis of a Peer-to-Peer Support Social Media Platform 
 
1. Background 
1.1.    Provide the scientific background, rationale for the study, and importance 
in adding to existing knowledge.  
 
There is a lot of speculation about the nature of online communications regarding suicidal 
and self-injurious behavior and whether such behavior is harmful or helpful. A recent 
survey of suicide prevention on social media platforms identified the following unique 
benefits of social media: it allows for communication with those facing similar hardships, 
provides opportunity to share feelings, grants opportunity to gain emotional support from 
others, and provides ability to help others (Robinson, Rodrigues, Fisher, Bailey, & Herrman, 
2015). Although there also are potential risks, overall, the benefits of social media in suicide 
prevention may outweigh the risks (Robinson, Rodrigues, et al., 2015). Winkel and 
colleagues (2005) found that increased social support from suicide forums relates to 
decreased suicide ideation (Winkel et al., 2005). Furthermore, research suggests that peer 
support may be preferable to professional mental health treatment (e.g., Barton, Hirsch, & 
Lovejoy, 2012).  
 
With the ever-growing use of social media, it is essential to be able to detect when users are 
under significant stress so that resources may be provided.  The purpose of the current 
study is to analyze anonymous behavioral data from a popular social media platform (Koko; 
www.itskoko.com) to determine how to best detect this distress and the most effective 
ways through which help may be provided, with a long-term goal of better preventing self-
harm and suicide ideation. 
 
References 
 
Barton, A. L., Hirsch, J. K., & Lovejoy, M. C. (2012). Peer Response to Messages of Distress. 
Crisis, 34, 183-191.  
 
Robinson, J., Rodrigues, M., Fisher, S., Bailey, E., & Herrman, H. (2015). Social media and 
suicide prevention: findings from a stakeholder survey. Shanghai Arch Psychiatry, 27(1), 27-
35. doi: 10.11919/j.issn.1002-0829.214133. 
 
Winkel, S., Groen, G., & Petermann, F. (2005). [Social support in suicide forums]. Prax 
Kinderpsychol Kinderpsychiatr, 54(9), 714-727.  
 
x 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
2 
 
2. Study Design 
2.1. Provide a thorough description of all study procedures.  
 
We plan to work with the company “Koko,” which provides users with two forms of 
support: 
 
• 
Scalable support for emotional distress.  From bullying and harassment to suicide and 
self-harm, Koko helps people in all states of distress, providing evidence-based support 
while referring high-risk users to international lifelines for immediate help. 
 
• 
Crisis detection and abusive content moderation.  Koko’s APIs help social networks 
automatically detect crisis and abusive content in text, protecting users and supplementing 
human moderation efforts with deep-learning artificial intelligence APIs. 
 
Koko has implemented a quality improvement program in which they are testing out 
different ways of referring people to help; they have collected data on the different ways 
(without retaining any identifying information) and would like to share the data/results 
with us so that we may assist with the analysis/tailoring of techniques in the future.   
 
Koko collects and retains data on user behavior such as how they respond to questions to 
determine if they might be at risk for suicide, whether they are referred to crisis services 
(e.g., suicide prevention hotlines), and whether they accept that recommendation and 
actually call the hotline. Koko tries different strategies as part of their quality improvement 
process (using random assignment to different conditions that vary based on the 
language/strategy used to refer people to crisis services) to determine the most effective 
way to try to get people to access crisis services (e.g., call the hotline to which they are 
referred). We will work with Koko to analyze their data and write them up in scientific 
papers. 
 
We view this as an exercise in quality improvement, but do intend to publish the findings as 
well as suggestions for methods to use across social media platforms in the future. The data 
do not contain any identifiers (and in fact does not even contain items such as age and 
gender); we are submitting this application because the Harvard Office of Sponsored 
Programs has requested an IRB determination before they will review our submission of a 
Data Use Agreement to be put in place with Koko. 
 
 
 
2.2. Indicate the duration of a participant’s involvement.  
 
Koko will send Harvard data on user behavior (as described above) that it has collected 
following its own quality improvement processes (e.g., testing different ways of referring 
users to crisis services). We also will collaborate with them in the future: Our role would be 
to advise them on what strategies/language to use to try to increase the percentage of users 
in need of crisis services who go on to use those services. We will provide advice to Koko, 
they will collect data on the effectiveness of the strategies/language we help to create, and 
they will then send data (with no identifiers) for analysis and we will write reports 
regarding which referral strategies/language were most effective. 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
3 
2.3. Indicate the estimated number of participants, by subgroup if applicable. If it 
is not possible to estimate the number of participants (e.g., open online survey), 
please indicate that it is not possible and provide an explanation of why it is not 
possible.  
 
Data are collected across all users on the platform.  Subgroups are not available because 
identifying information, gender, and age are not collected. 
 
2.4. Are you audio or video recording any participants? 
      No       Yes:  If yes, describe your procedures and what provisions are in 
place for notifying participants of this recording. 
 
2.5. List inclusion and exclusion criteria and describe any screening process.  
 
Data are collected across all users on the platform.  Subgroups are not available because 
identifying information, gender, and age are not collected. 
 
 
2.6. Are there any potentially vulnerable populations or individuals proposed 
for involvement in the research (check all that apply)? 
    
 
Children 
   
Wards of the State 
 
Prisoners/Detainees 
 
Pregnant Women 
 
Adults Not Competent to Consent 
 
Employees or Students of Harvard University 
 
Other (please describe):   
 
Demographic information is not collected with the user data, so we do not have 
any way of knowing if these populations are involved.  We are not targeting any 
certain populations, as the data will be anonymous data provided by Koko. 
 
3. Recruitment Methods 
3.1. Will potential participants be provided with information about the study? 
     No: 
If no, skip to 4.1. 
     Yes: If yes, indicate how, when, where, and by whom participants will be 
recruited.  
If you are recruiting from a Harvard University study pool, describe how you meet 
their requirements.  
 
Koko has informed its users about the extent of data use. We are attaching their privacy 
policy. We are performing secondary analyses on an anonymous data set. 
 
3.2. Are there any materials that will be used to recruit participants (e.g., emails, 
posters, oral scripts)? 
      No       Yes:  If yes, list the materials by document name here, and be sure 
to attach final copies to the “Consent and Recruitment Materials” Smartform.  
 
x 
 
 
x 
 
x 
 
 
 
 
 
 
x 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
4 
3.3. Will participants receive reimbursement or compensation in the form of 
money, gifts, incentives, or raffles?  
     No       Yes:  If yes, specify the amount, method, and timing of disbursement.  
Please refer to the Harvard University Financial Policy on Human Subject Payments.  
 
4. Study Setting 
4.1. Is any of the research conducted outside the United States?  
     No        Yes:  If yes, describe how you are ensuring that the research is 
appropriate considering local laws, regulations, and customs.  
This should be either a formal review by a local ethics board, Ministry of Health, etc., 
or a statement that a formal review is not required along with your source of 
information that the proposed research is in accordance with local laws, regulations, 
and customs.  
 
Koko users are not geographically limited; they may be from anywhere.  User location is not 
included in the anonymous dataset. 
 
4.2. Are there any permissions that must be obtained from cooperating 
institutions, community leaders, government officials, or other individuals, 
including approval from an IRB or research ethics committee?  
     No        Yes:  If yes, list the permission(s) by document name and be sure to 
attach copies to the “Supporting Documents” Smartform.  
 
5. Available Resources 
5.1. Describe the experience of the investigator with the proposed research 
procedures and population.  
Dr. Matthew Nock is a Professor of Psychology and Director of the Laboratory for Clinical and 
Developmental Research in the Department of Psychology at Harvard. Dr. Nock has extensive 
experience in clinical research, and has conducted and supervised numerous research projects 
with undergraduate and graduate students. Professor Nock received his Ph.D. in psychology from 
Yale University (2003) and completed his clinical internship at Bellevue Hospital and the New 
York University Child Study Center (2003). Nock’s research is aimed at advancing the 
understanding why people behave in ways that are harmful to them, with an emphasis on suicide 
and other forms of self-harm. His research is multi-disciplinary in nature and uses a range of 
methodological approaches (e.g., epidemiologic surveys, laboratory-based experiments, and 
clinic-based studies) to better understand how these behaviors develop, how to predict them, and 
how to prevent their occurrence. His work was funded by grants from the National Institutes of 
Health and several private foundations, and been published in over 200 scientific papers and book 
chapters. Nock’s work was recognized through the receipt of four early career awards from the 
American Psychological Association, the Association for Behavioral and Cognitive Therapies, 
the American Association of Suicidology, and in 2011 he was named a MacArthur Fellow. In 
addition to conducting research, Nock was a consultant/scientific advisor to the National 
Institutes of Health, the World Health Organization’s World Mental Health Survey Initiative, the 
American Psychological Association, and the American Psychiatric Association DSM-5 
Childhood and Adolescent Disorder Work Group. At Harvard, Professor Nock teaches courses on 
statistics, research methods, self-destructive behaviors, developmental psychopathology, and 
cultural diversity—for which he has received several teaching awards including the Roslyn 
Abramson Teaching Award and the Petra Shattuck Prize.  
 
x 
  
x 
 
x 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
5 
 
5.2. Are there any additional study team members whose role in the research 
requires special qualifications in addition to ethics training (e.g., licensed 
clinical psychologist)?  
     No       Yes:  If yes, describe.  
 
5.3. Are provisions needed for medical and/or psychosocial support resources 
(e.g., in the event of research-related distress or incidental findings)? 
     No       Yes:  If yes, describe the provisions and their availability.  
 
This is a secondary analysis on an anonymous dataset. 
 
6. Consent Process 
6.1. Will participants be asked to agree to be in the study? 
      No: If no, explain why not, then skip to 7.1. 
      Yes: If yes, describe the consenting process.  
If the study includes minors or others who cannot consent for themselves, describe 
how you will obtain their assent and the permission of their parent or guardian. Be 
sure to attach copies of appropriate documents to the “Consent and Recruitment 
Materials” Smartform.  
 
Koko has informed its users about the extent of data use. We are attaching their privacy 
policy. We are performing secondary analyses on an anonymous data set. 
 
6.2. Will the consenting process involve obtaining a signature?  
     Yes      No:  If no, explain why not.  
 
6.3. Will participants be offered a copy of the consenting information? 
     Yes      No:  If no, explain why not.  
 
6.4. Are you recruiting any participants who are not fluent in English? 
     No       Yes:  If yes, describe provisions for communicating information 
needed for consent.  
  
6.5. Does the study involve (a) deception (providing false information) or (b) 
incomplete disclosure (withholding information about some or all aspects of the 
research purpose or procedures in order to maintain the scientific integrity of the 
study)?  
     No        Yes:  If yes, explain the rationale and plans for protecting participants 
(e.g., debriefing).  
Be sure to attach any debriefing materials to the “Supporting Documents” 
Smartform.  
 
7. Risks  
7.1. Are there any reasonably foreseeable risks or discomforts to participants 
and/or groups/communities? 
     No       Yes:  If yes, describe the risks and outline proposed provisions to 
minimize risk.  
Risks may be physical, psychological, social, reputational, legal, and/or economic.  
 
x 
 
x 
 
x 
 
 
 
 
x 
 
 
 
 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
6 
 
We are performing secondary analyses on an anonymous data set. No identifying 
information is collected, so therefore the risk to privacy has been eliminated. 
 
 
8. Benefits  
8.1. Describe any potential benefits to study participants and to society.  
 
The purpose of this project is to improve the detection and referral of people in distress. We 
hope the methods developed will better serve users of Koko (and social media in general) in 
the future. 
 
9. Participant Privacy 
9.1. Describe provisions to protect participants’ privacy (their ability to control 
access to information about themselves or their person, e.g., the use of a 
private interview room) and to minimize the intrusiveness of study 
questions or procedures.  
 
Koko has informed its users about the extent of data use. We are attaching their privacy 
policy. We are performing secondary analyses on an anonymous data set. 
 
10. Data Confidentiality 
10.1. Which category of information best describes the data you will be 
recording?  
Please refer to the Harvard University Data Security Policy. 
 
 LEVEL 1 - Data that is publicly available or not identifiable. Examples: 
• 
Research data that has been de-identified in accordance with applicable 
rules; 
• 
Published research data; published information about the University; 
• 
Course catalogs; 
• 
Directory information about students who have not requested a FERPA 
block; 
• 
Faculty and staff directory information. 
  
 LEVEL 2 - Information the University has chosen to keep confidential 
but the disclosure of which would not cause material harm. Examples:  
• 
Research data that is identifiable but is not considered sensitive; 
• 
Patent applications and work papers, drafts of research papers; 
• 
Building plans and information about the University physical plant. 
 
LEVEL 3 - Information that could cause risk of material harm to 
individuals or the University if disclosed. Examples: 
• 
Information protected by the Family Educational Rights and Privacy Act 
(FERPA) to the extent it is not covered under Level 4 including non-
directory student information and directory information about students 
who have requested a FERPA block; 
• 
HUIDs associated with names or any other information that could 
identify individuals; 
x 
 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
7 
• 
Harvard personnel records (employees may discuss terms and 
conditions of employment with each other and third parties); 
• 
Institutional financial records; 
• 
Individual donor information; 
• 
Other personal information protected under state, federal and foreign 
privacy laws not classified as Level 4 or 5. 
 
LEVEL 4 - Information that would likely cause serious harm to 
individuals or the University if disclosed. Examples: 
• 
High Risk Confidential Information (HRCI) and research information 
classified as Level 4 by an IRB; 
• 
Personally identifiable financial or medical information; 
• 
Information commonly used to establish identity that is protected by 
state, federal, or foreign privacy laws and regulations; 
• 
Individually identifiable genetic information that is not Level 5; 
• 
National security information (subject to specific government 
requirements); 
• 
Passwords and Harvard PINs that can be used to access confidential 
information. 
 
LEVEL 5 - Information that would cause severe harm to individuals or 
the University if disclosed. Examples: 
• 
Research information classified as Level 5 by an IRB or otherwise 
required to be stored or processed in a high security environment and 
on a computer not connected to the Harvard data networks; 
• 
Certain individually identifiable medical records and genetic 
information, categorized as extremely sensitive. 
 
10.2. Will the data be physically housed at Harvard or stored remotely under the 
management of Harvard researchers?   
      
      No        Yes 
 
10.3. In what format (electronic, paper, etc.) will the research data be collected 
and stored? 
 
The data will be stored electronically.  
 
10.4. Explain where research data will be stored (e.g., personal laptop, thumb 
drive, departmental   
computer server, office file cabinet, etc.). 
 
The data will be stored on password-protected computers for analysis and archived on our 
Harvard-managed shared drive. 
 
10.5. Describe i) plans for any transmission of data (identifiable or non-
identifiable); ii) how long and with what protections data will be stored; and 
iii) plans for the data at the end of the storage period.  
 
No identifiable data is in the dataset.  The data will be transferred using email (if the file is 
too large, we will use a secure transfer service such as Accellion. This is a unique and 
 
 
 
x 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
8 
valuable dataset, so even though the data do not contain identifying information, we will 
treat it and store it as if it does. The data will be stored under password protection and/or 
on our Harvard-owned shared drive. The data will be stored indefinitely. If it comes time to 
destroy the data, it will be deleted from each computer and the shared drive, and the trash 
will be emptied. 
 
11. Sharing Study Results 
11.1. Is there a plan to share study results with individual participants, partnering 
organizations, and/or the participant community? 
     No       Yes:  If yes, describe the plan.  
 
Findings will be shared with Koko so that they may use them to improve their detection and 
referral process, and these findings/suggestions for improvements are intended to be published to 
help the social media administration community at large. Users of Koko may hear about these 
findings via publication or in the media, but we will not be able to share with them directly 
because identifying information is not collected and we do not have a way of knowing which 
users’ data is included. 
 
12. Multi-site Study Management  
12.1. Are one or more sites conducting this study in addition to sites overseen by 
the Harvard PI? 
     No       Yes:  If yes, indicate whether there is a coordinating research site 
and describe plans for communication among sites regarding unanticipated 
problems involving risks to subjects or other individuals, interim results, 
protocol modifications, monitoring of data, etc.  
 
13. Devices 
13.1. Does this study involve the use of a device subject to FDA regulations? 
      No: If no, skip to 14.1 
      Yes, and the device is being used according to its labeled indication: Skip 
to 14.1 
      Yes, and the device is an Investigational Device: Describe why this is a 
non-significant risk device study and why it qualifies either for an 
abbreviated IDE determination or for exemption from the IDE requirements.  
 
 
14. HIPAA Privacy Protections 
14.1. Are HIPAA privacy protections required? Mark Yes only if the investigator is at 
Harvard University Health Services or data will be obtained from a hospital, health 
center, or health insurance plan. 
 No       Yes:  If yes, either describe plans for obtaining authorization to 
access protected health information or provide an explanation of why this is 
not possible.  
 
15. Establishing a Data or Specimen Bank 
15.1. Does the study include establishing a repository for sharing data or 
specimens with other researchers? This does not include contributing de-
identified data to an existing repository.  
      No: If no, skip the remaining questions. 
x 
 
 
x 
 
x 
x 
x 
 
 

 
CUHS Research Protocol – for use by University-Area investigators 
 
Research Protocol Template | HRP-503-CUHS | Version date: April 2017 |  
 
9 
      Yes: If yes, identify what data or specimens will be collected and stored, 
and what information will be associated with the specimens. Please know that 
a separate IRB submission may be needed if a data or specimen repository will be 
created.  
 
15.2. Describe where and how long the data/specimens will be stored and 
whether participants’ permission will be obtained to use the 
data/specimens in other future research projects.  
 
15.3. Identify who may access and use data/specimens and how.  
 
15.4. Will specimens and/or data be sent to research collaborators outside of 
Harvard? 
     No       Yes:  If yes, describe the plan, and be sure to attach copies of any 
agreements to the “Supporting Documents” Smartform.  
 
15.5. Will specimens and/or data be received from collaborators outside of 
Harvard? 
     No       Yes:  If yes, describe the plan, and be sure to attach copies of any 
agreements to the “Supporting Documents” Smartform.