SlowGuess/NAACL_2022_OCR huggingface dataset:
Luis F. Guzman-Nateras<sup>1</sup>, Viet Dac Lai<sup>1</sup>, Amir Pouran Ben Veyseh<sup>1</sup>, Franck Dernoncourt<sup>2</sup>, and Thien Huu Nguyen<sup>1</sup>
2 Adobe Research, Seattle, WA, USA
{lfguzman, vietl, apouranb, thien}@cs.uoregon.edu,
Suicide is a serious problem in every society. Understanding life events of a potential patient is essential for successful suicide-risk assessment and prevention. In this work, we focus on the Event Detection (ED) task to identify event trigger words of suicide-related events in public posts of discussion forums. In particular, we introduce SuicideED: a new dataset for the ED task that features seven suicidal event types to comprehensively capture suicide actions and ideation, and general risk and protective factors. Our experiments with current state-of-the-art ED systems suggest that this domain poses meaningful challenges as there is significant room for improvement of ED models. We publicly release SuicideED to support future research in this important area.
Suicide is a serious and growing problem in our society<sup>1</sup>. The most common procedure for suicide risk assessment is for clinicians to set up clinical interviews with potential patients that will provide rating scales based on a list of preset questions (Ross et al., 2012). However, interviews and similar activities require the willingness of potential patients to participate. Given the associated mental states, such participation can be challenging to obtain for patients with high suicidal risks.
In the meantime, people are increasingly spending more of their time on social networks, sharing inner thoughts and daily activities. This collection of social posts might draw a comprehensive picture of the patient's life that can be used to support the diagnosis of suicidal conditions. In fact, moderators of some social networks (e.g., Reddit, Reachout) use social posts to monitor suicide and apply immediate intervention if necessary. To assist with the processing of the large amount of posts, there have been a few methods and tools
for automatically analyzing online posts to detect suicidal intent (Ji et al., 2018; Shing et al., 2018; Coppersmith et al., 2015; Milne et al., 2016). However, these studies mainly focus on assessing the patients' susceptibility to suicide and fail to consider contributory life events that cause/lead to such conditions. In this paper, we argue that recognizing suicide-related events is also critical to suicide understanding, identification and prevention, and natural language processing (NLP) methods are necessary to support automatic identification of such events from the vast and growing number of social media posts.
This work aims to advance the ultimate goal of creating NLP methods for suicide understanding by exploring the novel task of Event Detection (ED) for suicide-related events. ED is an important task in Information Extraction (IE) whose purpose is to identify event trigger words/mentions text data(Ahn, 2006; Ji and Grishman, 2008). Take, for instance, the following paragraph:
I don't date anyone and never will. It's a reason why I have no friends and never will. I wanna be funny and have a personality and be desirable but I'm not that. I know its depression that causes it.
Adapted to our interest in suicide-related events, an ED system should be able to recognize "date" and "have" as trigger words for deteriorated personal relationship events (i.e., risk factors); "wanna", "have", and "desirable" as triggers for protective factor events, and "depression" as a trigger for a health-related risk factor event.
The vast majority of advanced methods for ED are based on training deep neural networks on large labeled corpora (Nguyen and Grishman, 2015; Chen et al., 2015). As such, to facilitate research in ED for suicide prevention, a key requirement is to have a benchmark dataset to standardize the development and evaluation of ED models. Unfortunately, a large amount of existing suicide-related datasets are protected due to sensitive pri
<sup>1</sup>https://www.nimh.nih.gov/health/statistics/suicide
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1952 - 1961 July 10-15, 2022 ©2022 Association for Computational Linguistics
vacy concerns and, thus, fail to support the larger research community (Coppersmith et al., 2015; Vioules et al., 2018; Bhat and Goldman-Mellor, 2017). Moreover, these existing datasets are created to detect potential suicidal attempts based on text classification (Vioules et al., 2018; Bhat and Goldman-Mellor, 2017; Shing et al., 2018), which does not provide event trigger annotations of suicide-related events for ED.
To overcome such challenges, this paper introduces SuicideED, a new dataset for suicidal event detection that is manually annotated for seven distinct event types to comprehensively characterize suicide-related events regarding suicidal actions, thoughts, and risk and protective factors. To enable data sharing, our dataset is based on public posts from Reddit where personal information is not presented to avoid privacy issues. The SuicideED dataset is challenging as it involves informal texts, and requires event factuality and affected entity reasoning. Our experiments show that the performance of current state-of-the-art ED models on SuicideED lags behind their performance on other general-purpose ED datasets, thus calling for more research effort for suicide-specific ED. To facilitate future research in this area, SuicideED is released publicly² for the research community.
2 Ontology Design
An important, previously unexplored, question is what constitutes relevant events that can provide useful insights for clinicians to better understand and recognize suicide-related incidents. Accordingly, we consult specialized literature related to suicidal-behavior identification and treatment (Gutierrez, 2006; de Ruiter and Nicholls, 2011; O'Connor et al., 2013) to define the event categories for our dataset. As such, we design the event types to be exclusive to avoid type overlapping, and sufficiently comprehensive to cover relevant/impactful suicide-related events in the data. Eventually, we select the following seven event types that capture suicide-related actions, thoughts, and risk/protective factors.
The first two event types are concerned with statements to indicate suicidal attempts or intentions. In particular, the ACTION event type is dedicated to the direct expressions for actual suicidal attempts/actions, e.g., "I've started cutting
myself again $^{3}$ . In contrast, the second event type, IDEATION, represents suicidal inner thoughts, feelings, or desires, where no real action present, e.g., "I'm going to kill myself soon". These two types directly integrate factuality differentiation into the event types to better address the uniqueness of the data where hypothetical events are prevalent and understanding the factuality of events is critical to suicide intervention and prevention.
The second group of event types focuses on external events that increase a subject's susceptibility to suicidal behaviours, i.e., risk factors (RF) (Gutierrez, 2006). Given the diverse nature of RF, four event types are proposed. RF-LIFE events include mentions of a death of a close/loved entity, e.g., "My dog just died". RF-RELATIONSHIP concerns events related to social isolation, family breakdowns, or any mention of deteriorated interpersonal relationship, e.g., "My dad kicked me out of the house". Events for RF-HEALTH cover mentions of physical diseases, mental illness, and behaviors that directly affect the subject's health, e.g., "I feel depressed". Finally, RF-OTHER incorporates every other RF event that cannot be assigned to life, relationship, or health issues but still qualify as RF, including financial issues, chronic abuse, and general quality-of-life problems.
The final type, PROTECTIVE, captures events that drive an individual towards a better mental-health state, involving a broad range of positive activities, such as receiving effective medication or being motivated by social connections, e.g., "The medication seems to be helping". A detailed description and representative examples for each event type are presented in Appendix B.
3 Data Collection and Annotation
The documents for SuicideED are collected from publicly available posts from reddit.com. In particular, we focus on three subreddits (subgroups) that contain a high percentage of suicide-related posts: r/SuicideWatch, r/depression, and r/mentalhealth. Each original post is considered as a separate document and only posts with more than 50 words are considered to increase the probability of an event being present.
Given the event types described in section 2, an annotation job posting is created on upwork.com and seven freelance annotators with previous experience in mental health and psychology, such
$^{2}$ https://github.com/nlp-uoregon/suicideED
3In our examples, event trigger words are highlighted.
Train	Dev	Test
#Event triggers	33,055	1,925	1,998
#Documents	2,214	130	109
#Sentences	20,677	1,178	1,176
#Words	378,435	20,301	21,541
Table 1: Data statistics for SuicideED.
Label	Count
RF-OTHER	15,343
PROTECTIVE	7,389
IDEATION	6,645
RF-RELATION	3,890
RF-HEALTH	2,408
ACTION	1,084
RF-LIFE	219
Table 2: Label distribution of SuicideED.
as physicians and psychology graduates, are recruited. They are provided with a comprehensive guideline document<sup>4</sup> with thorough annotation instructions and numerous detailed examples for training. The annotators are instructed to select a single word for each event trigger (i.e., the most important) that clearly evokes the event, following the practices of prior ED work (Nguyen and Grishman, 2015). Overall, we annotate 2,300 documents for the seven event types from which the proportions of documents taken from the subreddits r/SuicideWatch, r/depression, and r/mentalhealth are  $50\%$ ,  $30\%$ , and  $20\%$ , respectively. We select  $20\%$  of the documents to be used for co-annotation, leading to a Fleiss' Kappa score of 0.8 (i.e., close to the almost perfect agreement range of  $[0.81 - 1.0]$ ). The remaining  $80\%$  of documents are distributed among the annotators for individual annotation. To facilitate and standardize future research, we divide SuicideED into three different portions for training, test, and development purposes. Table 1 presents some statistics for the different data portions while Table 2 shows the event type distribution.
4 Dataset Challenges
Compared to existing, general purpose, ED datasets, e.g., ACE-05 (Walker et al., 2006), MAVEN (Wang et al., 2020), and CySecED (Trong et al., 2020), our SuicideED dataset features at least three unique challenges for ED models.
First, as its documents are obtained from Reddit posts, SuicideED involves texts where infor
mal words (e.g., "wanna", "gonna") are prevalent, sentences might not follow well-structured grammar rules, and first-person point of view is the main writing style. This is in contrast to existing ED datasets where documents are often retrieved from news outlets or reports with formal and well-structured texts.
Second, in addition to the relevance to suicide, the event types in SuicideED sometimes require models to simultaneously consider event factuality to accurately determine the types. This is clear for differentiating ACTION and IDEATION where the key distinction concerns event factuality. Another example involves the potential confusions between PROTECTIVE and RF where different event factuality might lead to different event types for the similar expressions. For instance, in the sentence "I have a lot of friends", the event trigger "have", belongs to the PROTECTIVE type given that it reveals a positive environment for the subject. On the contrary, in the sentence "I had a lot of friends", the trigger "had" should be considered as a RF-RELATIONSHIP type as it might instead imply current deteriorated social connections.
Furthermore, the event type determination in SuicideED also necessitates appropriate identification of the entity that should be considered for the effect of an event. For instance, in the sentence "My sister killed herself," the trigger word "killed" should have the ACTION type if the entity of consideration is "sister". However, if we consider the event from the point of view of the poster/speaker, "killed" should be a RF-LIFE event. In SuicideED, the annotators are instructed to take the first person point of view (i.e., the poster) in the annotation decision. As such, ED models are expected to learn this feature from the data to achieve good performance.
Finally, some triggers in general-purpose datasets such as ACE05 can be type-indicative to a great extent. For instance, the trigger word "bought" is almost certainly indicative for the Transaction: Transfer-Ownership type. In contrast, Figure 1 illustrates the ambiguity of the event triggers in SuicideED by presenting the label distribution of the top 5 most frequent event trigger words. As can be seen, there is no particular dominant label for even the most frequent words. Hence, a ED model must effectively capture surrounding context of triggers to perform classification.
<sup>4</sup>https://github.com/nlp-uoregon/suicideED/guidelines.pdf
Figure 1: Label distribution of common trigger words.
To assess the complexity of the ED task in SuicideED, we evaluate the performance of the following state-of-the-art ED models: CNN: a convolutional neural network for ED (Nguyen and Grishman, 2015); DMBERT: a dynamic multipooling model based on BERT (Wang et al., 2019); BERTED: a BERT-based model augmented with multi-layer perception (Yang et al., 2019); BERTGCN: a graph convolutional network (GCN) based on dependency trees (Nguyen and Grishman, 2018); GatedGCN: a GCN model using BERT and trigger-aware gating mechanism (Lai et al., 2020); and EEGCN: a GCN model that exploits syntactic structure and typed dependency information (Cui et al., 2020). All of these models leverage the pre-trained BERT model to obtain representation vectors. The hyperparameters of the models are fine-tuned over the development data.
Additionally, we further finetune the pre-trained BERT model over unlabeled Reddit posts from the same three subreddits (i.e., about 40K posts) using masked language modeling (Devlin et al., 2019). We report the model performance when the finetuned BERT replaces the original pre-trained BERT to explore the effectiveness of domain customization of BERT for the informal texts in Reddit.
Model	BERT-base-cased	Finetuned BERT
P	R	F	P	R	F
CNN	47.5	44.9	46.2	48.6	46.7	47.6
DMBERT	51.7	62.1	56.4	52.1	64.1	57.5
BERTED	47.8	66.3	55.5	48.8	65.3	55.8
BERTGCN	56.0	61.9	58.8	55.5	63.5	59.2
GatedGCN	54.6	64.1	59.0	54.2	65.1	59.2
EEGCN	54.6	65.5	59.5	53.7	66.7	59.5
Table 3: Performance of the models on the SuicideED test set using BERT and finetuned BERT embeddings.
Table 3 presents the performance of the mod-
els on the SuicideED test set. Our first observation is that fine-tuning BERT over Reddit posts can successfully improve the performance of all ED models. This improvement, though, is less pronounced for the more recent and advanced ED models, i.e., GatedGCN and EEGCN. Second, the performance of the graph-based models (e.g., GatedGCN and EEGCN) is significantly better than those for non-graph-based models (i.e., CNN, DMBERT, and BERTED). As such, despite the informal nature of texts that can hinder the performance for dependency parsing, dependency trees are still helpful for the representation learning of ED models in SuicideED. Finally and most importantly, we find that the performance of existing ED models on SuicideED is substantially below than the typical performance of such models general purpose ED datasets (e.g.,  $77.6\%$  on ACE-05 with GatedGCN and EEGCN) (Lai et al., 2020). These results then suggest the unique challenges of SuicideED for ED models and highlight the need for further, domain-specific research to improve ED for suicide-related events.
Suicide detection and prevention using NLP methods has caught the attention of many researchers. Due to the privacy restrictions associated with clinical databases, researchers have used publicly-available data from social media with manual annotations of recognizable signals of mental health issues (Coppersmith et al., 2015; Shing et al., 2018). The majority of methods, however, focus on detecting suicidal attempts or assessing suicide propensity of users based on social media posts (Coppersmith et al., 2015; Bhat and Goldman-Mellor, 2017; Shing et al., 2018; Zirikly et al., 2019). As such, these prior works have only explored the setting of overall text classification which fails to explore fine-grained analysis/classification at word level required to reveal suicide-related events as this paper does.
Prior research efforts for ED, in general-purpose settings, have introduced various methodologies to address such task, including feature engineering (Ahn, 2006; Ji and Grishman, 2008; Li et al., 2013) and deep learning (Nguyen and Grishman, 2015; Chen et al., 2015; Wang et al., 2019; Cui et al., 2020; Ngo Trung et al., 2021; Pouran Ben Veyseh et al., 2021b,a) models. However, such prior work mainly utilizes the ED datasets with general
event types and formal texts, i.e., ACE-05 (Walker et al., 2006), that might not be helpful for particular domains with unique requirements such as the one addressed in this work. Recently, there have been some effort on creating new datasets for ED in more specific domains, including biomedical texts (Kim et al., 2009), literary texts (Sims et al., 2019), cybersecurity texts (Satyapanich et al., 2020; Trong et al., 2020), fine-grained event types (Le and Nguyen, 2021), and historical texts (Lai et al., 2021). However, none of existing ED datasets explore suicide-related events in social media texts.
We present SuicideED, the first dataset focused on the event detection task for suicide-related events. SuicideED is manually annotated for seven event types and provides enough training examples to develop large-scale deep learning models. We perform extensive evaluations of state-of-the-art ED models that demonstrate the challenges entailed by this difficult domain and call for further efforts to improve performance. In the future, we plan to extend SuicideED to annotate event arguments and other event properties to better support event analysis and understanding for suicide.
8 Statement of Ethics and Human Subject Research
Working with sensitive data such as mental health information from human subjects requires taking special care. This becomes particularly relevant in this case as our main objective is to provide a dataset for general public use. Benton et al. (2017) discuss, however, that research with human subjects information is exempted from the required full Institutional Review Board (IRB) review if the data is already available from public sources or if the identity of the subjects cannot be recovered.
By design, Reddit is a platform where users remain anonymous and make their posts available to the general public. Nonetheless, additional privacy measures were taken by removing any username mentions from the documents as they can sometimes include identifiable information. Furthermore, unlike previous works where the main objective is to assess suicidal risk at the user level (Coppersmith et al., 2015; Bhat and Goldman-Mellor, 2017; Shing et al., 2018; Zirikly et al., 2019), this work focuses on sentence-level ED. As such, our dataset does not include any user-level information
that could be used to identify individual subjects. Hence, this work is considered exempt from review by our University's IRB as the documents used are already publicly available and the original posters are impossible to identify.
Minimizing impact on the annotators: All prospective annotators were informed beforehand about the nature of the related text material and were made aware of its potential impact on their mental health. All chosen annotators had background knowledge/training on the subject at hand and were either clinicians or psychology graduates. Any candidates who reported suffering from, or having a history of, mental health-related issues were not considered out of concern for their health.
David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events, pages 1-8.
Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing.
Harish S Bhat and Sidra J Goldman-Mellor. 2017. Predicting adolescent suicide attempts with neural networks. NIPS 2017 Workshop on Machine Learning for Health.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 167-176.
Glen Coppersmith, Ryan Leary, Eric Whyne, and Tony Wood. 2015. Quantifying suicidal ideation via language usage on social media. In *Joint Statistics Meetings Proceedings*, Statistical Computing Section, JSM, volume 110.
Shiyao Cui, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Xuebin Wang, and Jinqiao Shi. 2020. Edge-enhanced graph convolution networks for event detection with syntactic relation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2329-2339.
Corine de Ruiter and Tonia Nicholls. 2011. Protective factors in forensic mental health. In International Journal of Forensic Mental Health.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Peter M. Gutierrez. 2006. Integratively assessing risk and protective factors for adolescent suicide. In *Suicide and Life-Threatening Behavior*.
Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL-08: HLT, pages 254-262.
Shaoxiong Ji, Celina Ping Yu, Sai-fu Fung, Shirui Pan, and Guodong Long. 2018. Supervised learning for suicidal ideation detection in online user content. Complexity, 2018.
Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun'ichi Tsujii. 2009. Overview of BioNLP'09 shared task on event extraction. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task.
Viet Lai, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021. Event extraction from historical texts: A new dataset for black rebellions. In