SlowGuess/NAACL_2021_OCR huggingface dataset:
Ramit Sawhney†‡* and Harshit Joshi¶* and Rajiv Ratn Shah‡ and Lucie Flek†
† Conversational AI and Social Analytics (CAISA) Lab
‡ Multimodal Digital Media Analysis (MIDAS) Lab
Department of Computer Science and Engineering, IIIT Delhi
http://midas.iitd.edu.in
$\mathbb{Q}$  Cluster Innovation Centre, University of Delhi
Recent psychological studies indicate that individuals exhibiting suicidal ideation increasingly turn to social media rather than mental health practitioners. Personally contextualizing the buildup of such ideation is critical for accurate identification of users at risk. In this work, we propose a framework jointly leveraging a user's emotional history and social information from a user's neighborhood in a network to contextualize the interpretation of the latest tweet of a user on Twitter. Reflecting upon the scale-free nature of social network relationships, we propose the use of Hyperbolic Graph Convolution Networks, in combination with the Hawkes process to learn the historical emotional spectrum of a user in a time-sensitive manner. Our system significantly outperforms state-of-the-art methods on this task, showing the benefits of both socially and personally contextualized representations.
Every 40 seconds, a person dies by suicide (Roth et al., 2018). Despite the success of psychoclinical methods, such as the Suicide Probability Scale (Bagge and Osman, 1998) and Suicide Ideation Questionnaire (wa Fu et al., 2007), the suicide rate in the U.S. has risen by  $35\%$  in the last 20 years (Hedegaard et al., 2020). While these methods are professional (Pestian et al., 2017), they have limited efficacy and may even impact participants negatively (Harris and Goh, 2017). Their limitations include barriers such as social stigma (Crisp et al., 2000), low literacy (Batterham et al., 2013), low motivation to seek hel (Essau, 2005), and finances (Czyz et al., 2013). Tragically,  $80\%$  of patients do not undergo clinical treatment, and  $60\%$  of those who died by suicide denied having suicidal thoughts to practitioners (McHugh et al., 2019).
Contrarily, people turn to social media to express suicidal thoughts (Luxton et al., 2012; Coppersmith et al., 2014; Robinson et al., 2016), with 8 of 10 people disclosing their suicidal plans (Golden et al., 2009). Consequently, a growing body of work has shown that natural language processing can complement social media analysis to identify risk markers in online user behavior to aid suicide risk assessment (McCarthy, 2010; De Choudhury et al., 2016; Reger et al., 2020; Shing et al., 2018). However, analyzing individual user posts is not always sufficient to infer user's mental state and the associated suicide risk (Harris, 2010; Sisask et al., 2008).
Studies suggest that suicide can be influenced by social factors (Masuda et al., 2013; Gvion and Apter, 2012), and is a contagious phenomenon (Mann, 2002). If a user is inclined to suicide ideation, a neighbor in the social network also often exhibits suicidal behavior (Wray et al., 2011). Further, social media cultivates safe spaces that encourage users to share thoughts with those who appear similar to themselves (Bak et al., 2012; McPherson et al., 2001; Franklin et al., 2017). Analyzing such social context along with historical activity, as in Figure 1, can help further ascertain suicidal risk (Van Heeringen and Marusic, 2003).
According to psychosocial research, there exists an uneven distribution of power and influence on social media (Avin et al., 2018). People exhibiting suicidal ideation form social clusters (Robertson et al., 2012) and preferentially copy the behavior of popular users, manifesting social learning of suicide-related behavior such as the "copycat suicide" (Mesoudi, 2009; Henrich and Gil-White, 2001) (Figure 1). These social networks present a hierarchical structure of ideation propagation, characteristic for Scale-free networks (Barabási and Bonabeau, 2003). In a scale-free network, most nodes have very few links, whereas a handful of in
* Both authors contributed equally
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2176-2190 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Illustration of social influence and context, specifically copycat suicidal ideation, in a scale-free network setting. Such social and temporal context can contextualize a user's state for a more accurate suicide risk assessment. We paraphrase all examples in this paper as per the moderate disguise scheme (Bruckman, 2002) to protect user privacy (Chancellor et al., 2019b).
fluential nodes have a large number of connections, creating social hubs, further amplifying phenomena such as the "Werther effect" (Fahey et al., 2018).
Social networks with scale-free structure are subjects to major distortions when embedded into the Euclidean representation space (Chen et al., 2013; Aparicio et al., 2015) by ordinary graph neural networks. To overcome this limitation, we propose to model the social relations using graph convolutions over hyperbolic space (Chami et al., 2019).
(i) We present the first deep graph neural framework to identify suicide ideation on social media by explicitly modeling users' social and temporal emotional context jointly (§3).
(ii) Motivated by psychological studies and the scale-free nature of social networks, we propose the use of Hyperbolic Graph Convolutions (§3.4).
(iii) We propose a mechanism leveraging Hawkes process to learn the historic emotional spectrum of a user in a time-sensitive manner from their historical posts (§3.3).
(iv) Through a series of experiments (§5), we show that our framework significantly outperforms existing methods (§6.1) on this task, as well as standard Graph Neural Networks (§6.2).
(v) Finally, we analyze the contributions of Hyper-SOS's individual components to assess sui
cidal intent (§6.2, §6.3, §6.4) and demonstrate practical applicability through a qualitative analysis (§6.5).
Aware of the sensitive nature of this work, we dedicate a standalone section (§7) to the ethical considerations and applicability of this work.
2.1 Suicide Ideation Detection
Early efforts in leveraging NLP for suicide ideation detection on social media (De Choudhury et al., 2013, 2016; Shing et al., 2018; Sawhney et al., 2018) combine general features such as n-grams and POS tags with lexicons like LIWC (Pennebaker et al., 2001). Deep learning models like CNNs (Naderi et al., 2019) and LSTMs (Coppersmith et al., 2018) have improved suicide ideation detection (Ji et al., 2020) thanks to a more robust semantic context to interpret the tweet in question, however, lacking user-level context, are often unable to ascertain suicide risk (Sisak et al., 2008). The best performing models (Matero et al., 2019; Naderi et al., 2019) at the CLPsych (Zirikly et al., 2019) and CLEF e-Risk (Losada et al., 2019) exemplify the promising yet underexplored direction of user context modeling (Flek, 2020) for suicide ideation detection. Although recent studies (Shing et al., 2020; Sawhney et al., 2020) explore the personal historical context of users, community-based social context has rarely been explored for this task. One of the few attempts includes SNAPBATNET (Sinha et al., 2019), a shallow embedding model to extract network structural features.
While graph neural networks (GNNs) have made advances in enhancing NLP models for various tasks (Mishra et al., 2019a; Del Tredici et al., 2019; Lu and Li, 2020), two broad shortcomings limit their effectiveness for suicide ideation detection. First, these methods do not capture the personal historical and social network context together, both of which are strongly correlated to risk assessment on social media (Yang and Eisenstein, 2017). Second, studies have shown that users exhibiting suicide ideation tend to form social networks with scale-free characteristics (Jonas, 1992; Mesoudi, 2009), which regular GNNs are unable to accurately capture (Chami et al., 2019) in learnt social representations. We build on these limitations by combining historical and social contexts in the hy
Figure 2: An overview of Hyper-SOS: We first extract the emotional representation of the tweet to be assessed and the historic emotional spectrum of a user via the HEAT mechanism to initialize tweet nodes and user nodes in the heterogeneous social graph, respectively. A Hyperbolic GCN is then used to aggregate features from neighboring nodes to learn social and historic representation, which we use to assess the presence of suicidal intent.
perbolic space to further contextualize and improve suicide ideation detection on social media.
3 Hyper-SOS: Formulation and Design
In this section we present the architecture of the Hyper-SOS framework (Hyperbolic Graph Convolutional Network for Suicide assessment On Social media) shown in Figure 3, designed to identify suicide ideation on social media by explicitly modeling user's social and temporal emotional context.
We formulate suicidal intent (SI) detection as a binary classification task to predict the presence of suicidal intent  $y_{i}$  for a tweet  $t_i$ , where,  $y_{i} \in \{\mathrm{SI~present,~SI~absent}\}$ . We denote the tweet to be assessed for the presence of suicidal intent as  $t_i \in T = \{t_1,t_2,\dots ,t_N\}$ , authored by a user  $u_{j} \in U = \{u_{1},u_{2},\dots ,u_{M}\}$ , posted at time  $\tau_{curr}^{i}$ . Each tweet  $t_i$  is associated with history  $H_{i}^{j} = [(h_{1}^{i},\tau_{1}^{i}),(h_{2}^{i},\tau_{2}^{i}),\dots ,(h_{L}^{i},\tau_{L}^{i})]$  where  $h_k^i$  is a historic tweet authored by user  $u_j$  posted at time  $\tau_k^i$  with  $\tau_1^i < \tau_2^i < \dots < \tau_L^i < \tau_{curr}^i$ . Moreover, two users are connected if they interact with each other's tweets on Twitter. We acknowledge that modeling suicidal intent as a binary classification task is a strong simplification.
3.2 Encoding Tweets
We build on previous studies which show that the linguistic styles (De Choudhury et al., 2013, 2016) and emotions expressed in suicidal tweets
play an important role in assessing suicidal behavior (Sueki, 2015; Zhang et al., 2017; Spates et al., 2018). Thus, building on this correlation between emotions and suicidal ideation, we fine-tune BERT on EmoNet (Abdul-Mageed and Ungar, 2017) for capturing fine-grained (Plutchik-based) emotions (Plutchik, 1980; Sawhney et al., 2020).
Tweet to be assessed: We utilize the final 768-dimension hidden state corresponding to the [CLS] token as the aggregate representation of emotions in a tweet. Formally, we encode each tweet to be assessed  $(t_i)$  to an emotion representation vector  $\mathbf{T}_{\mathbf{i}}^{\prime} = \mathrm{BERT}_{\mathrm{finetuned}}(t_i); \mathbf{T}_{\mathbf{i}}^{\prime} \in \mathbb{R}^{768}$ .
Historical Tweets: We encode user's historical tweets  $h_k^i$  using our fine-tuned BERT to learn representations of a user's emotional spectrum over time as  $\mathbf{e}_{\mathbf{k}}^{\mathbf{i}} = \mathrm{BERT}_{\mathrm{finetuned}}(h_k^i)$ ,  $\mathbf{e}_{\mathbf{k}}^{\mathbf{i}} \in \mathbb{R}^{768}$ . These representations can be indicative of a user's mental state and emotion buildup over time (Aragón et al., 2019; Tarrier et al., 2007), and better contextualize temporal behavior to ascertain suicidal intent (Links et al., 2008; Palmier-Claus et al., 2012).
3.3 Modeling Personal Historical Context
To model historical emotions of a user and factor in the natural irregularities in posting time of historical tweets (Lei et al., 2018; Wojcik and Hughes, 2019), we propose the HEAT mechanism: Hawkes temporal Emotion AggregatIon. HEAT leverages Hawkes Process (Hawkes, 1971), a self-exciting temporal point process to model the in
tensity of emotions whenever a tweet is posted in the past (Guo et al., 2019). Intuitively, it assumes that emotions exhibited in different historic tweets can influence one another. To obtain the final historic representation  $(\mathbf{E}_{\mathbf{j}}^{\mathbf{i}} \in \mathbb{R}^{768})$  of the tweet to be assessed  $t_i$ , HEAT aggregates encoded historical emotions  $\mathbf{e}_{\mathbf{k}}^{\mathbf{i}}$  using an exponential kernel as:
$$
\mathbf {E _ {j} ^ {i}} = \sum_ {k: \Delta \tau_ {k} \geq 0} \left(\mathbf {e _ {k} ^ {i}} + \epsilon \mathbf {e _ {k} ^ {i ^ {\prime}}} e ^ {- \beta \Delta \tau_ {k}}\right), \mathbf {e _ {k} ^ {i ^ {\prime}}} = \max (\mathbf {e _ {k} ^ {i}}, 0) (1)
$$
where,  $\Delta \tau_{k}$  is the time gap between a historical tweet and the tweet to be assessed (current tweet) posted at time  $\tau_{k}$  and  $\tau_{curr}$ , respectively.  $\epsilon$  and  $\beta$  are hyperparameters such that  $\epsilon < \beta$ .
3.4 Modeling Social Network Context
Studies show that users' emotions (Hill et al., 2010, 2015), depressive behavior (Rosenquist, 2011), and loneliness (Cacioppo et al., 2009) can be transmitted through social connections. Hence, leveraging social relationships between users can contextualize potential suicidal intent (Mueller and Abrutyn, 2015; Burnap et al., 2015; Colombo et al., 2016).
We model such relationships as a graph  $\mathcal{G} = (V, E)$ , where each edge  $e^U \in E$  represents one of three types of interaction between two users  $u_x, u_y \in U$ : i) User  $u_x$  quotes (retweets) a tweet  $t_i$ , posted by user  $u_y$ , ii) User  $u_x$  mentions user  $u_y$  in a tweet  $t_i$ , iii) User  $u_x$  replies to user  $u_y$ , by posting a tweet  $t_i$ . We further extend the social graph  $\mathcal{G}$  by introducing tweet nodes  $t \in T$ , which represent labeled tweets to be assessed. Each tweet node  $t$  is connected to its author (user) node  $u$  by a user-tweet interaction edge  $e^T \in E$ . The constructed social graph  $\mathcal{G}$  is heterogeneous, having two types (users and tweets) of nodes  $V = \{U \cup T\}$ , and two types (user-user and user-tweet) of edges  $E = \{e^T \cup e^U\}$ , as shown in Figure 2. Note that the tweet nodes  $t$  are labeled for the presence of suicidal intent, while the user nodes  $u$  are unlabeled.
3.5 Hyperbolic Graph Neural Network
To augment language and historical context-based features, we leverage GNNs to learn representations of the constructed social graph  $\mathcal{G}$ . However, most GNNs such as Graph Convolution Networks (GCNs) operate in the Euclidean space, and often do not generalize well to the kind of hierarchical, tree-like networks users on social media, particularly those exhibiting suicidal behavior (Chen et al., 2013). Sociological studies (Bild et al.,
Figure 3: Hyperbolic feature transformation  $(F^{E} \rightarrow F^{H})$  via projection on the Poincaré ball manifold to better represent the scale-free social network (left). Neighborhood-based node feature updation via hyperbolic linear transformation followed by Frechet Mean aggregation (right) to enrich user and tweet features.
2015; Aparicio et al., 2015), show that such networks show scale-free characteristics (Scatà et al., 2018), which follow the power law, i.e., the degree distribution of nodes decreases exponentially with a few nodes having a large number of connections (Ravasz and Barabási, 2003). To capture such hierarchical and scale-free structural properties in the social network graph, we propose the use of a Hyperbolic Graph Convolution Network (HGCN) (Chami et al., 2019). HGCNs project language and historical feature embeddings in the hyperbolic space to minimize distortions and learn a better representation of the underlying scale-free nature of social networks (Krioukov et al., 2010; Papadopoulos et al., 2012).
Initialization: Our proposed HGCN aggregates features from neighboring nodes based on graph convolutions in the hyperbolic space to enrich learned language and historical emotion features. We initialize user nodes with their historical emotional spectrum  $\mathbf{E}_{\mathbf{j}}^{\mathrm{i}}$  obtained through the HEAT mechanism, and tweet nodes with their emotional representation  $\mathbf{T}_{\mathbf{i}}^{\prime}$ . Hyper-SOS then performs hyperbolic graph convolutions on these user and tweet features on the social graph  $\mathcal{G}$  with  $|U|$  user nodes and  $|T|$  tweet nodes, which can also be represented by: its adjacency matrix  $\mathbf{A} \in \mathbb{R}^{(|U| + |T|) \times (|U| + |T|)}$ , a diagonal degree matrix  $\mathbf{D}$ , where  $\mathbf{D}_{ii} = \sum_{j} \mathbf{A}_{ij}$  and a feature matrix  $\mathbf{F}^{E} \in \mathbb{R}^{(|U| + |T|) \times 768}$  in the Euclidean space (denoted by  $^E$ ), containing the 768-dimensional representation of each node.
Feature Aggregation by Hyperbolic Graph Convolutions: To capture the network's hierarchical structure, Hyper-SOS first uses Poincaré ball manifold  $(\exp_{o}^{K})$  with a sectional curvature  $-1 / K$ , to map the features  $\mathbf{F}^E$  to the hyperbolic space (denoted by  $H$ )  $\mathbf{F}^H = \exp_{o}^{K}(\mathbf{F}^{E})$  as shown in Figure 3. Next, we perform a linear transformation to capture macroscopic neighborhood structures on the Poincaré ball manifold, followed by a Frechet Mean operation (Frechet, 1948) denoted by FM. Owing to the trainable curvature  $K$ , Hyper-SOS utilizes a hyperbolic non-linear activation with varying curvature  $(\sigma^{\otimes K_{i - 1},K_i})$  to allow a different curvature at each HGCN layer.  $\otimes$  is the Möbius transformation operator. Formally, the feature aggregation-based update rule at the  $i^{th}$  HGCN layer is:
$$
\mathbf {O} ^ {(i)} = \sigma^ {\otimes K _ {i - 1}, K _ {i}} \left(\mathrm {F M} \left(\tilde {\mathbf {A}} \mathbf {O} ^ {(i - 1)} \mathbf {W} ^ {(i)}\right)\right) \tag {2}
$$
where  $-1 / K_{i - 1}$  and  $-1 / K_{i}$  are the hyperbolic curvatures at layer  $i - 1$  and  $i$ , respectively.  $\tilde{\mathbf{A}} = \mathbf{D}^{-\frac{1}{2}}\mathbf{AD}^{-\frac{1}{2}}$  is the degree normalized adjacency matrix and  $\mathbf{W}$  is a trainable network parameter.
Finally, Hyper-SOS applies a dense layer with Rectified Linear Unit (ReLU) to get a prediction vector, followed by softmax to output the probabilities for the presence of SI  $(\hat{y})$  as:
$$
\hat {\mathbf {y}} = \operatorname {s o f t m a x} \left(R e L U \left(\mathbf {W} _ {y} \left(\mathbf {O} ^ {(2)} + \mathbf {b} _ {y}\right)\right) \quad (3) \right.
$$
where,  $\{\mathbf{W}_y,\mathbf{b}_y\}$  are network parameters.  $\mathbf{O}^{(2)}$  is the output of two stacked convolutions (Equation 2), with input  $\mathbf{O}^{(0)}$  set as the initial features  $F^H$
3.6 Hyper-SOS Training and Optimization
Tweets with SI present form a very small proportion of the data (Ji et al., 2019). To address this problem of class imbalance (the imbalance is much greater in the real world), we train HGCN using Class-Balanced Focal Loss (Lin et al., 2017; Cui et al., 2019). This loss function re-weights loss inversely with the effective number of samples per class, thereby yielding a class-balanced loss  $\mathcal{L}$  as:
$$
\mathcal {L} = \mathrm {C B} _ {\text {f o c a l}} \left(\hat {\mathbf {y}} _ {\mathbf {i}}, y _ {i}; \beta_ {c b}, \gamma\right) \tag {4}
$$
where  $\mathrm{CB}_{\text {focal }}$  is class-balanced focal loss,  $\hat{\mathbf{y}}_{\mathbf{i}}$  is the predicted label and  $y_{i}$  is the label of the tweet to be assessed.  $\beta_{cb}$  and  $\gamma$  are hyperparameters.
4 Dataset Properties
4.1 Data description
We use an existing Twitter dataset curated by Mishra et al. (2019b). The dataset contains Twitter
timelines of 32,558 unique users, spanning over ten years of historical tweets from 2009 to 2019, summing up to 2.3M unlabeled tweets. The users were selected based on a seed lexicon of 143 suicidal phrases (e.g., "wanting to die", "last day"), which identified 34,306 tweets potentially containing suicide ideation. Two psychology students then annotated these tweets under the supervision of a professional psychologist, achieving Cohen's  $\kappa$  of 0.72, under the below guidelines:
SI Present: Tweets where suicide ideation or attempts are discussed in a somber, non-flippant tone.  
SI Absent: Tweets with no evidence for risk of suicide, e.g., song lyrics, condolences, news.
3984 of the annotated tweets were identified as truly containing suicidal ideation. We feed all the 2.3M tweets to the HEAT mechanism to build user representations (§3.3). The number of historical tweets per user  $(748 \pm 789)$  and the time difference between consecutive tweets  $(2 \pm 24$  days) are indicative of large variations across users. 4070 users were found to have no historical tweets.
4.2 Data Split
We perform a stratified temporal 70:10:20 split, such that the train, validation, and test sets consist of 24014, 3431, and 6861 labeled tweets, respectively, and ensure that there is no overlap between users in these sets.
4.3 Network Analysis
In Table 1, we outline quantitative analyses of the social network  $\mathcal{G}$  and report Gromov's  $\delta$ -hyperbolicity of the graph (Jonckheere et al., 2008). A lower hyperbolicity  $\delta$  indicates a scale-free graph, for trees  $\delta = 0$ . Based on the low hyperbolicity (Chami et al., 2019), values of the power law coefficients  $x_{min}, \alpha$  (Clauset et al., 2009) of the graph  $\mathcal{G}$ , and the frequency distribution of node degrees in Figure 4, we note that the social network graph  $\mathcal{G}$  shows scale-free characteristics. These observations validate our experimental design, and are in line with social network analysis on the structure of social media (Gonçalves et al., 2011), particularly Twitter (Bakshy et al., 2011).
We reimplement and compare the following previous works to Hyper-SOS on temporal split (§4.2):
Figure 4: Node degree frequency distribution
Property	Value
Hyperbolicity δ	1.5
Max. Node Degree	2,452
Median Node Degree	1.0
Node Density	1.9e-4
Power Law	p(x) = Cx-α
xmin	14.0
α	2.97
$\mathbf{RF} + \mathbf{TF}$  (Sawhney et al., 2018): Feeds features such as statistical, LIWC (Pennebaker et al., 2001), n-grams, and POS counts from the tweet to a Random Forest (RF) classifier.
LSTM (Coppersmith et al., 2018): A deep neural network model that uses an LSTM for sequentially encoding GloVe embedding of tweets.
C-CNN (Gaur et al., 2019): Utilizes GloVe encoded tweets as a bag of tweets that are then concatenated and fed non-sequentially to a Contextual Convolutional Neural Network (Shin et al., 2018).
Suicide Detection Model (SDM) (Cao et al., 2019): Applies LSTM + Attention over fine-tuned FastText embeddings of historical tweets, followed by concatenation with tweet to be assessed.
DualContextBert (Matero et al., 2019): Best performing model at CLPsych 2019 (Zirikly et al., 2019). BERT embeddings of each historical tweet are sequentially fed to an attention-based RNN.
STATENet (Sawhney et al., 2020): A deep neural network model. Uses T-LSTM (Baytas et al., 2017) which applies a monotonically decreasing function of elapsed time to weight historical tweets and utilizes BERT fine-tuned on Plutchik-based emotions for the tweet to be assessed.
SNAP-BATNET (Mishra et al., 2019b): Encodes social graph structure using Node2Vec (Grover and Leskovec, 2016) embeddings concatenated with GloVe embeddings for the tweet to be assessed. They report weighted F1.
We evaluate Hyper-SOS using macro F1 score and recall for the SI class. We set hyperparameters for all models based on the validation macro F1 score. We use Grid search to explore: Hidden dimension  $H^{d} \in \{128, 256, \dots, 1024\}$ , Dropout  $\delta \in \{0.0, 0.1, \dots, 0.7\}$ . For the HEAT:  $\beta \in \{1e^{-3}, \dots, 1e^{-1}\}$  and  $\epsilon \in \{1e^{-3}, \dots, 1e^{-1}\}$ .  $\beta_{cb} \in \{0.999, 0.9999, \dots, 0.999999\}$  and  $\gamma \in \{2.0, 2.5, \dots, 4.0\}$ , learning rate  $I_{lr} \in \{1e^{-6}, \dots, 1e^{-3}\}$ ,
weight decay  $w_{d} \in \{1e^{-6}, \dots, 1e^{-3}\}$ . We find the optimal hyperparameters as:  $H_{d} = 512$ ,  $\delta = 0.2$ ,  $\beta = 1e^{-3}$ ,  $\epsilon = 1e^{-2}$ ,  $\beta_{cb} = 0.9999$ ,  $\gamma = 3.0$ ,  $I_{lr} = 1e^{-4}$ ,  $w_{d} = 5e^{-4}$ . We use PyTorch for all models, optimize Hyper-SOS using Adam for 5,000 epochs and apply early stopping with a patience of 100 epochs in 1,260s on a Tesla K80 GPU.
6.1 Comparisons with Prior Work