SlowGuess/NAACL_2022_OCR huggingface dataset:
Bhanu Pratap Singh Rawat $^{1,\ast}$ , Samuel Kovaly $^{1,\#}$ , Wilfred R. Pigeon $^{2,3,\dagger}$ , Hong Yu $^{1,3,4,\ddagger}$
$^{1}$ CICS, UMass-Amherst,  $^{2}$ University of Rochester,  $^{3}$ U.S. Department of Veterans Affairs,
<sup>4</sup>Center of Biomedical and Health Research in Data Sciences
$^{*}$ brawat@umass.edu,  $\#$ skovaly@umass.edu,
$^{\dagger}$ wilfred_pigeon@urmc.rochester.edu,  $^{\ddagger}$ hong_yu@uml.edu
Suicide is an important public health concern and one of the leading causes of death worldwide. Suicidal behaviors, including suicide attempts (SA) and suicide ideations (SI), are leading risk factors for death by suicide. Information related to patients' previous and current SA and SI are frequently documented in the electronic health record (EHR) notes. Accurate detection of such documentation may help improve surveillance and predictions of patients' suicidal behaviors and alert medical professionals for suicide prevention efforts. In this study, we first built Suicide Attempt and Ideation Events (ScAN) dataset, a subset of the publicly available MIMIC III dataset spanning over  $12k+$  EHR notes with  $19k+$  annotated SA and SI events information. The annotations also contain attributes such as method of suicide attempt. We also provide a strong baseline model ScANER (Suicide Attempt and Ideation Events Retreiver), a multi-task RoBERTab-based model with a retrieval module to extract all the relevant suicidal behavioral evidences from EHR notes of an hospital-stay and, and a prediction module to identify the type of suicidal behavior (SA and SI) concluded during the patient's stay at the hospital. ScANER achieved a macro-weighted F1-score of 0.83 for identifying suicidal behavioral evidences and a macro F1-score of 0.78 and 0.60 for classification of SA and SI for the patient's hospital-stay, respectively. ScAN and ScANER are publicly available<sup>1</sup>.
For decades, suicide has been one of the leading causes of death (CBHSQ, 2020). The suicide rate in the United States increased from 10.5 per 100,000 in 1999 to 14.2 in 2018, a  $35\%$  increase (Hedegaard et al., 2020). Globally, 740,000 people
[*Name*] is a 22 year old male with a history of angry and impulsive behavior who is transferred from an outside hospital s/p Tylenol overdose. [*Name*] reports that he and his girlfriend broke up last Wednesday, and that he subsequently went on an alcohol and cocaine binge lasting from Thursday to Saturday. He has used alcohol and cocaine regularly in the past, but he denies having had a binge of this quantity or duration before. On Saturday night, [*Name**] told his father that he had tried to hang himself at a nearby park, but the rope had broken.
Annotations:
Two instances of suicide attempts are annotated in the paragraph.
1.  $[^{*}\text{Name}^{*}]$  ... overdose: Annotated for suicide attempt and is assigned 'unsure' category as there is no definite documentation that it is a suicide attempt.
2. On Sat ... broken: This is also annotated for suicide attempt and is assigned the category: 'T71' (asphyxiation, hanging).
Figure 1: An example of positive and unsure evidence annotations for SA in an EHR note.
commit suicide each year. The rates of suicidal behaviors, suicide attempt (SA) and suicide ideation (SI), are much higher (WHO, 2021).
A prior study shows that a large proportion of suicide victims sought care well before their death (Kessler et al., 2020). Suicidal behaviors, including SA and SI are recorded by clinicians in electronic health records (EHRs). This knowledge can in turn help clinicians assess risk of suicide and make prevention efforts (Jensen et al., 2012). The diagnostic ICD codes include suicidality codes for both SA and SI. However a study has shown that ICD codes can only capture  $3\%$  SI events, while  $97\%$  of SI events are described in notes (Anderson et al., 2015). In addition, of patients described with SA in their EHR notes, only  $19\%$  had the corresponding ICD codes (Anderson et al., 2015). Therefore, it is important to develop natural language process
<sup>1</sup>The annotations, code and the models are available at https://github.com/bsinghpratap/ScAN.
Human Language Technologies, pages 1029 - 1040
ing (NLP) approaches to capture such important suicidality information.
Researchers have developed NLP approaches to detect SA and SI from EHR notes (Metzger et al., 2017; Downs et al., 2017; Fernandes et al., 2018; Cusick et al., 2021). These studies either used rule-based approaches (Downs et al., 2017; Fernandes et al., 2018; Cusick et al., 2021) or built the SA and SI identification models on a small set (Metzger et al., 2017) or private set (Bhat and Goldman-Mellor, 2017; Tran et al., 2013; Haerian et al., 2012) of EHR notes. It is also difficult to compare the results of those studies as they varied in EHR data, data curation, as well as NLP models, which were not made available to the public.
In this study, we present ScAN: Suicide Attempt and Ideation Events Dataset, a publicly available EHR dataset that is a subset of the MIMIC III data (Johnson et al., 2016). ScAN contains 19,690 expert-annotated SA and SI events with their attributes (e.g., methods for SA) over 12,759 EHR notes. Specifically, experts annotated suicidality evidence or sentences relevant to SA and SI events during a patient's stay at the healthcare facility, an example of SA annotations is shown in Fig 1. The evidences were put together to assess whether the patient has an SA or SI event.
We also present ScANER (Suicide Attempt and Ideation Events Retriever), a RoBERTa-based NLP model that is built on a multi-task learning framework for retrieving evidences from the EHRs and then predicting a patient's SA or SI event using the complete set of EHR notes from the hospital stay using a multi-head attention model. We focus on the prediction of SA and SI using all the EHR notes during a patient's stay because for the whole duration, multiple EHR notes and note types are generated, including admission notes, nursing notes, and discharge summary notes. Suicidal information are described in multiple notes throughout the stay. For example, a patient was admitted to the hospital with opioid overdose. It was documented initially in the admission note as an SA, but later dismissed as an accident after physician's evaluation. In another example, an opioid overdose admission was first documented as an accident on admission, but later documented to be an SA event after clinical assessment. Both ScAN and ScANER capture SA and SI information at the hospital-stay level. ScANER is able to retrieve suicidal evidences from EHR notes with a macro
weighted F1-score of 0.83 and is able to predict SA and SI with a macro F1-score of 0.78 and 0.60, respectively. Our annotation guidelines, ScAN, and ScANER system will be made publicly available, making ScAN a benchmark EHR dataset for SA and SI events detection. We will release the training and evaluations splits used in this study for benchmarking new models.
Efforts on detecting SA and SI within EHRs have been explored in recent years. Most work used rule-based or traditional machine learning-based approaches. In one study, experts created handcrafted rules from mentions of suicidality (both SA and SI) and then used the rules to identify suicidality as positive, negative, or unknown in a document (Downs et al., 2017). The rule-based approaches are limited by their scalability. In another study, structured and unstructured EHRs were used to classify at the hospital-stay level as SA, SI, or no mention of suicidal behavior (Metzger et al., 2017). The training data consisted of only 112 SA, 49 SI and 322 unrelated examples. In contrast, ScAN comprises of 697 hospital-stays with more than 19,000 suicidal event examples over 12,759 clinical notes. Only traditional machine learning models such as random forest (Breiman, 2001) were explored. In contrast, ScANER was built on the state-of-the-art self-attention based model.
Hybrid approaches have also been developed to identify SA at the hospital-stay level (Fernandes et al., 2018). In that study, a post-processing heuristic rule-based filter (e.g., removing negated events) was applied to the machine-learning-based classifier (a SVM (Cortes and Vapnik, 1995) classifier) to reduce false positives. Training and evaluation were done also on relatively small datasets (500 for training and 500 for evaluation).
Finally, weakly supervised approaches have been developed to identify SI from EHRs (Cusick et al., 2021). In that study, authors used ICD codes to identify 200 patients with SI and then obtained EHR notes of those patients (6, 588). This EHR note dataset was then used as the 'current' SI training data. The remaining 400 patients were labelled as 'potential' SI and their 12,227 EHRs were also labelled the same. Authors used multiple statistical machine learning models and one deep learning model: convolutional neural network. (Bhat and Goldman-Mellor, 2017) also used
feed-forward neural networks to predict suicide attempts over 500k unique patients but the EHR data for this study is not publicly available. (Ji et al., 2020) surveyed multiple studies where the researchers worked on private datasets (Tran et al., 2013; Haerian et al., 2012) for suicide attempt and ideation prediction. Whereas in our study, in contrast to using the ICD codes which has considerable errors, domain experts chart-reviewed a large, publicly available set of EHRs for SI and SA, along with their attributes (e.g., positive or negative SA, SI and the type of self-harm such as asphyxiation and overdose).
In this section, we introduce ScAN (Suicide Attempt and Ideation Events Dataset) and describe it's data collection and annotation process. We also discuss some examples from ScAN along with basic dataset statistics.
For annotation, we selected the notes from the MIMIC-III (Johnson et al., 2016) dataset, which consists of the de-identified EHR data of patients admitted to the Beth Israel Deaconess Medical Center in Boston, Massachusetts from 2001 to 2012 (Johnson et al., 2016). The data includes notes, diagnostic codes, medical history, demographics, lab measurements among many other record types. We chose MIMIC-III because it is publicly available under a data use agreement and allows clinical studies to be easily reproduced and compared.
The diagnostic ICD codes for the patients are provided at hospital-stay level in MIMIC with admission identification numbers (HADM_ID in MIMIC database). We first filtered the hospital stays that had ICD codes associated with suicide and overdose. This resulted in 697 hospital-stays for 669 unique patients. For each stay, multiple de-identified notes such as nursing notes, physician notes, and discharge summaries are available. For the selected 697 hospital-stays we extracted a total of 12,759 notes. Each medical note contains multiple sections about a patient such as family and medical history, assessment and plan, and discharge instructions. We extracted different sections from these clinical notes using MedSpaCy's2 clinical_sectionizer and filtered the relevant sections from these clinical notes for annota
tion. The extensive list of these sections is provided in Appendix A.
3.2 Annotation Process
The aim was to annotate all instances of SA and SI documented in the medical notes as defined by Center of Disease Control and Prevention (CDC) (Hedegaard et al., 2020). The filtered 12,759 notes were annotated by a trained annotator under the supervision of a senior physician. Each note consisted of instances of SA, SI, both or none. The senior physician randomly annotated 330 notes and had a  $100\%$  agreement with the annotator on hospital-stay level annotation and  $85\%$  agreement on sentence-level annotations. After adjudication between the senior physician and the annotator, the disagreements were discussed and adjusted by the annotator.
Tylenol toxicity - Initially treated with NAC protocol. After her history became more clear, it was felt that this was an accidental overdose. Psychiatry was consulted, who agreed there was no evidence of intention to harm. She should be instructed to limit her acetaminophen use in the future to 2gm/day.
Annotation:
1. After ... overdose.: Negative suicide attempt.
2. Psychiatry ... harm: Negative suicide ideation.
Figure 2: An example with negative SA and negative SI annotations.
Suicide Attempt (SA): The annotator labelled all the sentences with a mention of SA. Some hospital stays could represent multiple types of SA, such as in Fig. 1, where 'tried to hang himself' is labelled as a positive SA and 'Tylenol overdose' is labelled as unsure since the overdose was never confirmed as an SA event elsewhere in the medical notes of the patient's hospital-stay. The label unsure is used when it is not clearly documented if a self-harm was an SA event or not. The negative instance, example shown in Fig. 2, is a sentence that confirms that the self-harm, an "accidental overdose", responsible for the patient's hospital-stay is not an SA event. In this work, we only focused on suicidal self-harm and not non-suicidal self-harm (Crosby et al., 2011).
Further sub-categories are also provided for an SA annotation in the form of the ICD label group: a.) T36-T50: Poisoning by drugs, medications
$^{2}$ https://github.com/medspacy/medspacy
and biological substances b.) T51-T65: Toxic effects on non-medical substances c.) T71: Asphyxiation or suffocation and d.) X71-X83: Drowning, firearm, explosive material, jumping from a high place, crashing motor vehicles, other specified means.
Suicide Ideation (SI): SI is defined as any mention and/or indication of wanting to take one's own life or harm oneself. Similar to SA, any sentence with a mention of SI was labelled within the patient's notes. A SI annotation could be labeled as positive or negative, an example for each label is shown in Fig. 3.
She continued to be tearful and extremely upset that she was still alive, and was refusing medications, radiology, and blood draws. She was maintained on a 1:1 sitter and suicide precautions.
1. She cont .... draws: Positive suicide ideation.
Pts mental status cleared and was discharged with normal mental status. Psych evaluated pt and did not see any signs of suicidality. Did not recommend antidepressants.
1. Psych ... suicidality: Negative suicide ideation.
Figure 3: Examples of positive and negative SI annotations.
A sentence without SA or SI annotation would be considered as a neutral-SA or neutral-SI sentence respectively. Sentence level annotations provide more visibility to a medical expert for the hospital-stay level annotation.
ScAN consists of 19,690 unique evidence annotations for the suicide relevant sections of 12,759 EHRs of 697 patient hospital-stays. There are a total of 17,723 annotations for SA events and 1,967 annotations for SI events. The distribution for both SA and SI events is provided in Table 1.
In this section, we introduce ScANER (Suicide Attempt and Ideation Events Retreiver): a strong baseline model for our dataset. ScANER consists of two sub-modules: (1) An evidence retriever module that extracts the evidences related to SA and
General Statistics	Patients 669	Hospital-stays 697	Notes 12,759
Suicide Attempt	Positive 14,815	Negative 170	Unsure 2,738
Suicide Ideation	Positive 1,167	Negative 800
SI and (2) A predictor module that predicts SA or SI label for the patient's hospital-stay using the evidences extracted by the retriever module.
Table 1: Distribution of unique annotations at the patient, hospital-stay and notes level in ScAN.
Evidence	Train	Validation	Test
Yes	9,880	1,803	3,038
No	30,133	4,864	7,836
Suicide Attempt (SA)
Positive	7,597	1,474	2,433
Negative	136	36	20
Unsure	1,607	216	431
Neutral-SA	30,673	4,941	7,990
Suicide Ideation (SI)
Positive	928	153	331
Negative	654	107	189
Neutral-SI	38,431	6,407	10,354
Table 2: Distribution of evidences at paragraph level in ScAN for train, validation and test sets. A paragraph was considered an evidence, labeled as Yes, if it had at least one sentence annotated as SA or SI. A No evidence paragraph was both Neutral-SA and Neutral-SI.
4.1 Evidence Retriever
Problem Formulation: Given an input clinical note, the model extracts the evidences (one or more sentences) related to SA or SI (SA-SI) from the note. This is a binary classification problem where given a text snippet the model predicts whether it has an evidence for SA-SI or not. We learn this task at paragraph level where the input is a set of 20 consecutive sentences because the local surrounding context provides additional important information (Yang et al., 2021; Rawat et al., 2019). A paragraph was labeled as evidence no, if all the sentences in that paragraph are neutral-SA and neutral-SI. If there was at least one SA-SI sentence, it was con
2These splits would be released as part of ScAN for benchmarking models.
sidered an evidence yes. As the number of non-evidence sentences significantly outsized the evidence sentences, we decided to use an overlapping window of 5 sentences between the paragraphs to build more evidence paragraphs. The distribution of the paragraphs, across all evidence, SA and SI labels for train, validation, and test set is provided in Table 2. We segregated the train and test set such that any patient observed by the retriever module during training was not seen in the test set. This is important as there are patients who had multiple hospital-stays in ScAN.
Proposed Model: Transformer (Vaswani et al., 2017) based language models (Devlin et al., 2018; Liu et al., 2019) have shown great performance for a broad range of NLP classification tasks. Hence, to extract the evidence paragraphs we trained a RoBERTa (Liu et al., 2019) based model. It has been previously shown that the domain-adapted versions of the pre-trained language models, such as clinicalBERT (Alsentzer et al., 2019) or BioBERT (Lee et al., 2020), work better than their base versions. So, we further pre-trained the RoBERTa-base model over the MIMIC dataset to create a clinical version of RoBERTa model, hereby referenced as medRoBERTa. During our initial exploration, we experimented with clinicalBERT and BioBERT but found that medRoBERTa consistently outperformed both models. medRoBERTa achieved an overall F1-score of 0.88 whereas both clinicalBERT and BioBERT achieved an overall F1-score of 0.85. Our hospital-level SA and SI predictor would work with any encoder-based evidence retriever model.
Multi-task Learning: We trained medRoBERTa in a multi-task learning setting where along with learning the evidence classification task, the model also learns two auxiliary tasks: (a.) Identifying the label for SA between positive, negative, unsure and neutral-SA and, (b.) Identifying the label for SI between positive, negative and neutral-SI. The training loss  $(L(\theta))$  for our evidence retriever model was formulated as:
$$
L (\theta) = L _ {e v i} + \alpha * L _ {S A} + \beta * L _ {S I} \tag {1}
$$
Where  $L_{evi}$  is the negative log likelihood loss for evidence classification,  $L_{SA}$  and  $L_{SI}$  are SA and SI prediction losses respectively, and  $\alpha$  and  $\beta$  are the weights for the auxiliary tasks' losses. The distribution of labels across all the three tasks is highly skewed, hence, we applied the following
Suicide Attempt	Positive	Neg_Unsure	Neutral-SA
Train	377	54	1,381
Val	50	10	189
Test	91	19	326
Suicide Ideation	Positive	Negative	Neutral-SI
Train	377	214	1,521
Val	45	28	208
Test	44	35	357
Table 3: Distribution of SA and SI at hospital-stay level in training, validation and testing set.
techniques to learn an efficient and robust model.
- Weighted log loss was used in both main task and auxiliary tasks. The total loss for each task was calculated as the weighted sum of loss according to the label of the input paragraph. Log weighing helps smooth the weights for highly unbalanced classes. The weight for each class was calculated using:
$$
w _ {l, t} = \left\{ \begin{array}{l l} 1. 0 & \text {i f} (w _ {l, t} <   1. 0) \\ \log (\gamma * N _ {t} / N _ {l, t}) & \end{array} \right.
$$
Where  $N_{t}$  is the count of all training paragraphs for the task  $t$  and  $N_{l,t}$  is the count of paragraphs with label  $l$  for the task  $t$  and  $w_{l,t}$  is the calculated weight for those paragraphs. We tuned  $\gamma$  as a hyper-parameter. All training hyper-parameters for our best model are provided in Appendix B.
- We also employed different sampling techniques (Youssef, 1999), up and down sampling, to help our model learn from an imbalanced dataset. After sweeping for different sampling combinations as hyper-parameters, we found that down-sampling the no-evidence paragraphs by  $10\%$  resulted in the best performance.
- The negative label of SA is severely underrepresented in ScAN making it difficult for the model to learn useful patterns from such instances, refer Table 2. After discussion with the experts, we decided to group the instances of negative and unsure together and label them as neg_unsure because for both groups the general psych outcome is to let the patient leave after the hospital-stay as there is no solid evidence defining whether the self-harm was a SA event.
Trained in a Multi-task Setting
Figure 4: ScANER (Suicide Attempt and Ideation Events Retreiver) consists of two sub-modules: (a.) Evidence retriever module extracts evidence paragraphs from all EHR notes. We trained the module using all annotated paragraphs from ScAN. (b.) Prediction module predicts the SA and SI label for a patient using the evidence paragraphs extracted by the retriever module from EHR notes during the patient's hospital-stay.
4.2 Hospital-stay level SA and SI Predictor
Problem Formulation Given all the clinical notes of a patient during the hospital stay, the model predicts the label for SA (positive, neg_unsure and neutral-SA) and SI (positive, negative and neutral-SI). The prediction module uses the evidence paragraphs extracted by the retriever module.
Robust Finetuning The retriever module is not perfect and can extract false positives. This results in extracting irrelevant paragraphs, with evidence label No, along with evidence paragraphs for a hospital-stay with SA or SI and extracting irrelevant paragraphs as evidences for a hospital-stay with both SA and SI marked as neutral. To tackle such situations and train a robust model, we applied three techniques:
- For a hospital-stay with a non-neutral label for SA or SI, during training we added some noise in the form of irrelevant paragraphs (a paragraph with no SA or SI annotation) from the
notes to the set of actual evidence paragraphs for the input. An irrelevant paragraph from a clinical note was sampled with a probability of 0.05. This forced the predictor module to learn effectively even with noisy inputs.
- For a neutral hospital-stay with no evidence paragraphs, we randomly chose  $X$  unique irrelevant paragraphs from the notes.  $X$  was sampled from the distribution of number of evidence paragraphs of the non-neutral hospital-stays. This prevented the leaking of any information to the prediction module during training by keeping the distribution of number of input paragraphs the same across neutral and non-neutral instances.
- Since these hospital-stays were extracted using the ICD codes related to suicide and overdose, the data is quite skewed with only 102 neutral events from a total of 697 hospital-stays. Whereas in a real-world scenario, neutral hospital-stays would be much higher than non-neutral ones. Hence, to facilitate a bal
anced learning of the predictor module we introduced 1,800 neutral hospital-stays from the MIMIC dataset. The distribution for SA and SI at hospital-stay level is provided in Table 3.
Proposed Model The paragraphs extracted using the retriever module for a patient's hospital-stay were provided as an input to the predictor module. We used a multi-head attention model to predict the SA and SI label for a hospital-stay as self-attention based models have proved to be quite effective for a lot of prediction tasks in machine learning (Devlin et al., 2018; Cao et al., 2020; Hoogi et al., 2019).