SlowGuess/ACL_2020_OCR huggingface dataset:
Han-Chin Shing
Computer Science University of Maryland
shing@cs.umd.edu
Philip Resnik
Linguistic/UMIACS
Ilege Park, MD
resnik@umd.edu
Douglas W. Oard
iSchool/UMIACS
oard@umd.edu
We reframe suicide risk assessment from social media as a ranking problem whose goal is maximizing detection of severely at-risk individuals given the time available. Building on measures developed for resource-bounded document retrieval, we introduce a well founded evaluation paradigm, and demonstrate using an expert-annotated test collection that meaningful improvements over plausible cascade model baselines can be achieved using an approach that jointly ranks individuals and their social media posts.
Mental illness is one of the most significant problems in healthcare: in economic terms alone, by 2030 mental illness worldwide is projected to cost more than cardiovascular disease, and more than cancer, chronic respiratory diseases, and diabetes combined (Bloom et al., 2012). Suicide takes a terrible toll: in 2016 it became the second leading cause of death in the U.S. among those aged 10-34, fourth among those aged 35-54 (Hedegaard et al., 2018). Prevalence statistics suggest that roughly 141 of the 3,283 people who attended ACL 2019 have since had serious thoughts of suicide, 42 have made a plan, and 19 have actually made attempts.<sup>1</sup>
The good news is that NLP and machine learning are showing strong promise for impact in mental health, just as they are having large impacts everywhere else. Traditional methods for predicting suicidal thoughts and behaviors have failed to make progress for fifty years (Franklin et al., 2017), but with the advent of machine learning approaches (Linthicum et al., 2019), including text analysis methods for psychology (Chung and Pennebaker, 2007) and the rise of research on mental
health using social media (Choudhury, 2013), algorithmic classification has reached the point where it can now dramatically outstrip performance of prior, more traditional prediction methods (Linthicum et al., 2019; Coppersmith et al., 2018). Further progress is on the way as the community shows increasing awareness and enthusiasm in this problem space (e.g., Milne et al., 2016; Losada et al., 2020; Zirikly et al., 2019).
The bad news is that moving these methods from the lab into practice will create a major new challenge: identifying larger numbers of people who may require clinical assessment and intervention will increase stress on a severely resource-limited mental health ecosystem that cannot easily scale up. This motivates a reformulation of the technological problem from classification to prioritization of individuals who might be at risk, for clinicians or other suitably trained staff as downstream users.
Perhaps the most basic way to do prioritization is with a single priority queue that the user scans from top to bottom. This "ranked retrieval" paradigm is common for Information Retrieval (IR) tasks such as document retrieval. The same approach has been applied to ranking people based on their expertise (Balog et al., 2012), or more generally to ranking entities based on their characteristics (Balog, 2018). Rather than evaluating categorical accuracy, ranked retrieval systems are typically evaluated by some measure of search quality that rewards placing desired items closer to the top (Voorhees, 2001). Most such measures use only item position, but we find it important to also model the time it takes to recognize desired items, since in our setting the time of qualified users is the most limited resource.
In this paper, we do so by building on Time-
2120M Americans live in areas with mental healthcare provider shortages (Bureau of Health Workforce, 2020). That number reflects an increase of about 7 million people between September 30, 2019 and March 31, 2020.
<sup>1</sup>Approximately: ACL is international, but these figures use prevalence statistics for U.S. adults (SAMHSA, 2019).
8124
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8124-8137 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Illustration of an assessment framework in which individuals are ranked by predicted suicide risk based on social media posts, posts are ranked by expected usefulness for downstream review by a clinician, and word-attention highlighting helps foreground important information for risk assessment. Real Reddit posts, obfuscated and altered for privacy.
Biased Gain (TBG, Smucker and Clarke, 2012), an IR evaluation measure that models the expected number of relevant items a user can find in a ranked list given a time budget. We observe that in many risk assessment settings (e.g., Yates et al. (2017); Coppersmith et al. (2018); Zirikly et al. (2019)), the available information comprises a (possibly large and/or longitudinal) set of documents, e.g. social media posts, associated with each individual, of which possibly only a small number contain a relevant signal. $^{3}$  This gives rise to a formulation of our scenario as a nested, or hierarchical, ranking problem, in which individuals are ordered by priority, but each individual's documents must also be ranked (Figure 1). Accordingly, we introduce hierarchical Time-Biased Gain (hTBG), a variant of TBG in which individuals are the top level ranked items, and expected reading time is modeled for the ranked list of documents that provides evidence for each individual's assessment. In addition, we introduce a prioritization model that uses a three-level hierarchical attention network to jointly optimize the nested ranking task; this model also addresses the fact that in our scenario, as in many other healthcare-related scenarios, relevance obtains at the level of individuals rather than individual documents (cf. Shing et al., 2019). Using a test collection of Reddit-posting individuals who have been assessed for suicide risk by clinicians based on their posts (Shing et al., 2018), we use hTBG to model prioritization of individuals and demonstrate that our joint model substantially outperforms cascade model baselines in which the nested rankings are produced independently.
NLP for Risk Assessment. Calvo et al. (2017) survey NLP for mental health applications using non-clinical texts such as social media. Several recent studies and shared tasks focus on risk assessment of individuals in social media using a multi-level scale (Milne et al., 2016; Yates et al., 2017; Losada et al., 2020). Shing et al. (2018) introduce the dataset we use, and Zirikly et al. (2019) describe a shared task in which 11 teams tackled the individual-level classification that feeds into our prioritization model (their Task B). Our work contributes by modeling the downstream users' prioritization task as taking a key step closer to the real-world problem.
Hierarchical Attention Attention, especially in the context of NLP, has two main advantages: it allows the network to attend to likely-relevant parts of the input (either words or sentences), often leading to improved performance, and it provides insight into which parts of the input are being used to make the prediction. These characteristics have made attention mechanisms a popular choice for deep learning that requires human investigation, such as automatic clinical coding (Baumel et al., 2018; Mullenbach et al., 2018; Shing et al., 2019). Although concerns about using attention for interpretation exist (Jain and Wallace, 2019; Wegreiffe and Pinter, 2019; Wallace, 2019), Shing et al. (2019) show hierarchical document attention can align well with human-provided ground truth.
Our prediction model, 3HAN, is a variant of Hierarchical Attention Networks (HAN, Yang et al., 2016). Yang et al. use a two-level attention mechanism that learns to pay attention to specific words in a sentence to form a sentence representation, and at the next higher level to weight specific sentences in
<sup>3</sup>Our dataset, for example, has one severe risk individual with 1,326 postings, of which only two are "signal" posts identified by the experts. See Table 2 for detailed statistics.
8125
a document in forming a document representation. Adapting this approach to suicide assessment of at-risk individuals, our model moves a level up the representational hierarchy, learning also to weight documents to form representations of individuals. This allows us to jointly model ranking individuals and ranking their documents as potentially relevant evidence, without document-level annotations.
Evaluating rankings. There is an extensive IR literature on quality measures for ranked lists (Järvelin and Kekäläinen, 2002; Chapelle et al., 2009; Smucker and Clarke, 2012; Sakai, 2019), which generally reward placing highly relevant items near the top of the list, and are often relatively insensitive to mistakes made near the bottom.
In the setting of suicidality risk assessment, we care about how much gain (number of at-risk individuals found) can be achieved for a given time budget. Time-biased gain (TBG, Smucker and Clarke, 2012) measures this by assuming a determined user working down a ranked list, with the discount being a function of the time it takes to reach that position. However, neither TBG nor other ranking measures, to the best of our knowledge, can measure the hierarchical ranking found in the scenario that motivates our work: ranking items (i.e. individuals) when each item itself contains a ranked list of potential evidence (their posts). In this paper, we design a new metric, hierarchical time-biased gain (hTBG), to measure the hierarchical ranking by incorporating the cascading user model found in Expected Reciprocal Rank (ERR, Chapelle et al., 2009) into TBG.
3 A Measure for Risk Prioritization
Section 1 argued for formulating risk assessment as a prioritization process where the assessor has a limited time budget. This leads to four desired properties in an evaluation measure:4
- Risk-based: Individuals with high risk should be ranked above others.
- Head-weighted: Ranking quality near the top of the list, where assessors are more likely to assess, should matter more than near the bottom.
- Speed-biased: For equally at-risk individuals, the measure should reward ranking the one who can be assessed more quickly closer to
Figure 2: User model for Time-Biased Gain (TBG)
the top, so that more people at risk can be identified within a given time budget.
- Interpretable: The evaluation score assigned to a system should be meaningful to assessors.
Among many rank-based measures that satisfy the risk-based and head-weighted criteria, TBG directly accounts for assessment time in a way that also satisfies the speed-biased criterion (see Theorem 3.1). Furthermore, the numeric value of TBG is a lower bound on the expected number of relevant items — in our case, high-risk individuals — found in a given time budget (Smucker and Clarke, 2012), making it interpretable. After introducing TBG, in Section 3.2 we develop hierarchical Time-Biased Gain (hTBG), an extension of TBG, to account for specific properties of risk assessment using social media posts.[5]
3.1 Time-Biased Gain
TBG was originally developed in IR for the case of a user seeking to find a relevant document, but here we frame it in the context of risk assessment (Figure 2). TBG assumes a determined user (say a clinician) examining a ranked list of individuals in the order presented by the system. For each individual, the clinician first examines a summary and then decides whether to check relevance via more detailed examination, or to move on. Checking requires more time to make an assessment of whether the individual is indeed at-risk. TBG is a weighted sum of gain,  $g_{k}$ , and discount,  $D(\cdot)$ , a function of time:
$$
\mathrm {T B G} = \sum_ {k = 1} ^ {\infty} g _ {k} D (T (k)). \tag {1}
$$
4 Throughout, assessor or user signify a clinician or other human assessor, and individual is someone being assessed.
${}^{5}$  TBG and hTBG code: https://github.com/sidenver/hTBG
8126
Parameter	Description	Value
Pcheck(reli)	Prob. to check, given the relevance of summary	0.64, if reli = 1
0.39, if reli = 0
Flag(reli)	Prob. to flag, given the relevance of individual	0.77, if reli = 1
0.27, if reli = 0
Ts	Seconds to evaluate a summary	4.4
TαW + Tβ	Seconds to judge W words	0.018W + 7.8
Table 1: Parameters used for TBG and hierarchical TBG.
$T(k)$  is the expected amount of time it takes a user to reach position  $k$ :
$$
T (k) = \sum_ {i = 1} ^ {k - 1} t (i) \tag {2}
$$
$$
t (i) = T _ {s} + P _ {\text {c h e c k}} \left(\operatorname {r e l} _ {i}\right) E _ {i} \tag {3}
$$
where  $t(i)$  is expected time spent at position  $i$ . Breaking down  $t(i)$ ,  $T_{s}$  is the time it takes to read a summary and decide whether to check the individual; if yes (probability  $P_{\mathrm{check}}(\mathrm{rel}_i)$ ),  $E_{i}$  is expected time for detailed assessment, calculated as a function of the individual's total word count  $W_{i}$ :
$$
E _ {i} = T _ {\alpha} W _ {i} + T _ {\beta} \tag {4}
$$
where  $T_{\alpha}$  and  $T_{\beta}$  scales words to time. The discount function  $D(t)$  decays exponentially with half-life  $h$ :
$$
D (t) = 2 ^ {- \frac {t}{h}} \tag {5}
$$
where  $h$  is the time at which half of the clinicians will stop, on average. The expected stop time (or mean-life) is  $\frac{h}{\ln(2)}$ . Finally, the gain,  $g_{k}$  is:
$$
g _ {k} = P _ {\text {c h e c k}} \left(\operatorname {r e l} _ {k}\right) P _ {\text {f l a g}} \left(\operatorname {r e l} _ {k}\right) \mathbb {1} _ {\left[ r e l _ {k} = 1 \right]} \tag {6}
$$
where  $P_{\mathrm{check}}(\mathrm{rel}_k)$  is the probability of checking the individual after reading the summary at position  $k$ , and  $P_{\mathrm{flag}}(\mathrm{rel}_k)$  is the probability of then flagging that individual as high risk. Gain thus accrues only if a clinician actually finds a high-risk individual.
The decay function in Equation 5 monotonically decreases with increasing time (and thus rank), so TBG satisfies the head-weighted criterion. Table 1 shows the parameters used in Smucker and Clarke (2012), which were estimated from user studies using data from TREC 2005 Robust track.
Particularly of interest in a time-limited assessment, we can prove that TBG is speed-biased:
Theorem 3.1 (TGB satisfies the speed-biased criterion). Swapping an at-risk individual of longer
Figure 3: hTBG's model for calculating expected assessment time for an individual, replacing shaded box in Figure 2.
assessment time ranked at  $k$  with an equally attrisk individual of shorter assessment time ranked at  $k + r$ , where  $r > 0$ , always increases TBG.
Proof. See Appendix B.1
3.2 Hierarchical Time-Biased Gain
TBG assumes that detailed assessment involves looking at all available evidence (Equation 4). However, in our setting, an individual may have a large or even overwhelming number of social media posts. One severe risk individual in the SuicideWatch dataset, for example, has 1,326 posts in Reddit, the vast majority of which would provide the assessor with no useful information. Therefore we need to prioritize the documents to be read, and a way of estimating when the user will have read enough to make a decision.
In general, clinicians engage in a sensemaking process as they examine evidence, and modeling the full complexity of that process would be difficult. We therefore make two simplifying assumptions: (1) that there is a high-signal document that suffices, once read, to support a positive relevance judgment, and (2) that the clinician will not read more than some maximum number of documents. These assumptions align well with those of Expected Reciprocal Rank (ERR), whose cascading user model assumes that as the user works down a ranked list (in our case, the ranked documents posted by a single individual), they are more likely to stop after viewing a highly relevant document than after viewing an irrelevant one, as their information need is more likely to have been satisfied (Chapelle et al., 2009). This results in a cascade model of user behavior:  $\mathrm{ERR} = \sum_{k=1}^{\infty} \frac{1}{k} P(\text{stop at k})$ , in which  $P(\text{stop at k}) = R_k \prod_{i=1}^{k-1} (1 - R_i)$ , where  $R_k = f(rel_k)$  is the probability of stopping at position  $k$  as a function of relevance.
8127
This suggests replacing Equation 4 with the following expected time estimate for detailed assessment of an individual:
$$
E _ {i} = T _ {\alpha} \sum_ {l = 1} ^ {L} \left(W _ {i, l} \prod_ {m = 1} ^ {l - 1} (1 - R _ {i, m})\right) + T _ {\beta} (7)
$$
where  $R_{i,l}$  is the probability of stopping at the  $l$ -th document for individual  $i$ , and  $W_{i,l} > 0$  is the cost (in our case, word count) of reading the  $l$ -th document for individual  $i$ . Note that for the special case of  $\forall i, l \in N, R_{i,l} = 0$ , hTBG reduces to TBG. See Figure 3 for an illustration of  $E_i$  of hTBG. For derivation of Equation 7 from ERR's cascading user model, see Appendix B.3.
3.3 Optimal Values for TBG and hTBG
Calculation of the optimal value for a measure is often important for normalization, though not always easy; in some cases it can be NP-hard (Agrawal et al., 2009, ERR-IA). Another popular approach is to normalize by calculating the metric with an ideal collection. For example, Smucker and Clarke (2012) calculate the normalization factor of TBG by assuming a collection with an infinite number of relevant documents, each of which lack any content. In our case, however, we are actually interested in an optimal value achievable for a given test collection: the optimal values of TBG and hTBG are properties of the bottleneck that occurs due to the user's limited time-budget. We find that:
Theorem 3.2 (Optimal TBG). The optimal value of TBG under binary relevance is obtained if and only if (1) all at-risk individuals are ranked above not-at-risk individuals, and (2) within the at-risk individuals, they are sorted based on time spent in ascending order.
Theorem 3.2 makes sense, as any time spent on assessing a not-at-risk individual is time not spent on assessing other potentially at-risk individuals. Preference in assessing individuals with shorter assessment time also increased the chance of assessing more individuals in the given time budget.
Minimum Individual Assessment Time. To calculate optimal hTBG, we need to minimize individual assessment time. A natural question to ask, then, is whether a result similar to Theorem 3.2 holds for the individual assessment time of hTBG
in Equation 7. By swapping paired documents, we can use proof by contradiction to show that:
Theorem 3.3. Minimum individual assessment time is obtained if the documents are sorted in descending order by  $\frac{R_{i,l}}{W_{i,l}}$ .
Proof. See Appendix B.2
Theorem 3.3 shows a surprisingly intuitive tradeoff between how relevant a document might be, and how much time (proportional to word counts) the expert needs to take to read it: highly relevant documents with short reading time are preferred.
Observe that Theorem 3.1 (speed-biased criterion) and Theorem 3.2 both apply to hTBG, as the two theorems only concern the ranking of individuals, not documents, and hTBG is an extension of TBG to measure the document ranking. Using Theorem 3.3 and Theorem 3.2, calculation of optimal TBG and hTBG values is simply a matter of sorting. For TBG, time complexity is  $O(n \log(n))$ , where  $n \leq K$  is the number of at-risk individuals in the test collection. For hTBG, worst-case time complexity is  $O(n \log(n) + nm \log(m))$ , where  $m \leq L$  is the maximum number of relevant documents per individual.
4 Classification Model
We began by motivating risk assessment via social media as a person-centered, time-limited prioritization problem, in which the technological goal is to support downstream clinicians or other assessors in identifying as many people at risk as possible. This led to the conclusion that systems should not only rank individuals but, for each individual, rank their posts, and we introduced an evaluation framework that involves an abstraction of the user's process of identifying people at risk given a nested ranking.
Next, we need a system that can produce such nested rankings of individuals and their posts. Ideally such a system should be able to train on only individual-level, not document-level, labels, since suicide risk is a property of individuals, not documents, and document labels are more difficult to obtain. In addition, such a system should ideally produce additional information to help the downstream user — if not justification of its output, then at least highlighting potentially useful information.
To address this need, we introduce 3HAN, a hierarchical attention network (Yang et al., 2016) that extends up to the level of individuals, who are
8128
represented as sequences of documents. This architecture is similar to the network we proposed in Shing et al. (2019) for coding clinical encounters; it obtained good predictive performance and we also showed that, despite concerns about the interpretation of network attention (Jain and Wallace, 2019), hierarchical document-level attention succeeded in identifying documents containing relevant evidence. The architecture here differs in that it builds representations hierarchically from the word level, as opposed to pre-extracted conceptual features, and takes document ordering into account using a bi-directional GRU (Bahdanau et al., 2015).
Specifically, our model has five layers (Figure 4). The first is a word-embedding layer that turns a one-hot word vector into a dense vector. The second to fourth layers are three Seq2Vec layers with attention that learn to aggregate, respectively, a sequence of word vectors into a sentence vector, a sequence of sentence vectors into a document vector, and a sequence of document vectors into an individual vector (hence 3HAN). The final layer is a fully connected layer followed by softmax.
We detail our Seq2Vec layer in the context of aggregating a sequence of document vectors to an individual's vector, though the three Seq2Vec layers are the same. See Figure 4b for an illustration. Document vectors  $\{d_{i,j}\}_{j = 1}^{m}$  are first passed through a bi-directional GRU layer. The outputs, after passing through a fully-connected layer and a non-linear layer, are then compared to a learnable attention vector,  $v_{\text {attention }}$ . Specifically,
$$
g _ {i, j} = \operatorname {B i - G R U} \left(d _ {i, j}\right) \tag {8}
$$
$$
r _ {i, j} = \tanh  \left(W g _ {i, j} + b\right) \tag {9}
$$
$$
a _ {i, j} = \frac {e ^ {r _ {i , j} ^ {\top} v _ {\text {a t t e n t i o n}}}}{\sum_ {j ^ {\prime} = 1} ^ {m} e ^ {r _ {i , j ^ {\prime}} ^ {\top} v _ {\text {a t t e n t i o n}}}} \tag {10}
$$
$$
u _ {i} = \sum_ {j = 1} ^ {m} a _ {i, j} g _ {i, j} \tag {11}
$$
where  $a_{i,j}$  is the normalized document attention score for the  $j$ -th vector, and  $u_i$  is the final aggregated individual vector. As shown in Equation 10, the transformed vector  $r_{i,j}$  is compared with the learnable attention vector  $v_{\text{attention}}$  using a dot product, and further normalized for the weighted averaging step in Equation 11.
Once we have the individual vector  $u_{i}$ , we can predict the risk label of the individual by passing it through a fully-connected layer and a softmax.
$$
P \left(\hat {y} _ {i}\right) = \text {s o f t m a x} \left(W _ {F C} u _ {i} + b _ {F C}\right) \tag {12}
$$
Finally, we compare with the ground truth label  $y_{i}$  of individual  $i$  using negative log-likelihood to calculate a loss:
$$
\operatorname {l o s s} _ {i} = - \log (P (\hat {y} _ {i} = y _ {i})). \tag {13}
$$
We first introduce the test collection and then show how we can evaluate 3HAN and the cascade model baselines on the test collection using hTBG.
To demonstrate the effectiveness of the 3HAN model, which jointly learns to rank individuals and, within each individual, their posts as evidence, we compare it with different combinations of individual-level rankers and document-level rankers. Training details for all the models can be found in Appendix C.
5.1 Test Collection