Venue: NAACL 2022

# ScAN: Suicide Attempt and Ideation Events Dataset

Bhanu Pratap Singh Rawat\({}^{1,*}\), Samuel Kovaly\({}^{1,\divide}\), Wilfred R. Pigeon\({}^{2,3,\dagger}\), Hong Yu\({}^{1,3,4,\ddagger}\)

\({}^{1}\)CICS, UMass-Amherst, \({}^{2}\)University of Rochester, \({}^{3}\)U.S. Department of Veterans Affairs,

\({}^{4}\)Center of Biomedical and Health Research in Data Sciences

\({}^{*}\)brawat@umass.edu, \({}^{\dagger}\)skovaly@umass.edu,

\({}^{\dagger}\)wilfred_pigeon@urmc.rochester.edu, \({}^{\ddagger}\)hong_yu@uml.edu

###### Abstract

Suicide is an important public health concern and one of the leading causes of death worldwide. Suicidal behaviors, including suicide attempts (SA) and suicide ideations (SI), are leading risk factors for death by suicide. Information related to patients' previous and current SA and SI are frequently documented in the electronic health record (EHR) notes. Accurate detection of such documentation may help improve surveillance and predictions of patients' suicidal behaviors and alert medical professionals for suicide prevention efforts. In this study, we first built Suicide Attempt and Ideation Events (ScAN) dataset, a subset of the publicly available MIMIC III dataset spanning over \(12k+\) EHR notes with \(19k+\) annotated SA and SI events information. The annotations also contain attributes such as method of suicide attempt. We also provide a strong baseline model ScANER (**S**ui c**H**tempt and Ideation **E**vents **R**etr)ever, a multi-task RoBERTa-based model with a _retrieval module_ to extract all the relevant suicidal behavioral evidences from EHR notes of an hospital-stay and, and a _prediction module_ to identify the type of suicidal behavior (SA and SI) concluded during the patient's stay at the hospital. ScANER achieved a macro-weighted F1-score of \(0.83\) for identifying suicidal behavioral evidences and a macro F1-score of \(0.78\) and \(0.60\) for classification of SA and SI for the patient's hospital-stay, respectively. ScAN and ScANER are publicly available1.

Footnote 1: The annotations, code and the models are availble at [https://github.com/bsingbrpatrap/ScAN](https://github.com/bsingbrpatrap/ScAN).

## 1 Introduction

For decades, suicide has been one of the leading causes of death [5]. The suicide rate in the United States increased from \(10.5\) per \(100,000\) in \(1999\) to \(14.2\) in \(2018\), a \(35\%\) increase [1]. Globally, \(740,000\) people commit suicide each year. The rates of suicidal behaviors, suicide attempt (SA) and suicide ideation (SI), are much higher [23].

A prior study shows that a large proportion of suicide victims sought care well before their death [10]. Suicidal behaviors, including SA and SI are recorded by clinicians in electronic health records (EHRs). This knowledge can in turn help clinicians assess risk of suicide and make prevention efforts [13]. The diagnostic ICD codes include suicidality codes for both SA and SI. However a study has shown that ICD codes can only capture 3% SI events, while 97% of SI events are described in notes [1]. In addition, of patients described with SA in their EHR notes, only 19% had the corresponding ICD codes [1]. Therefore, it is important to develop natural language process

Figure 1: An example of _positive_ and _unsure_ evidence annotations for SA in an EHR note.



ing (NLP) approaches to capture such important sucidality information.

Researchers have developed NLP approaches to detect SA and SI from EHR notes Metzger et al. (2017); Downs et al. (2017); Fernandes et al. (2018); Cusick et al. (2021). These studies either used rule-based approaches Downs et al. (2017); Fernandes et al. (2018); Cusick et al. (2021) or built the SA and SI identification models on a small set Metzgerer et al. (2017) or private set Bhat and Goldman-Mellor (2017); Tran et al. (2013); Haerian et al. (2012) of EHR notes. It is also difficult to compare the results of those studies as they varied in EHR data, data curation, as well as NLP models, which were not made available to the public.

In this study, we present ScAN: **S**uicide **A**ttempt and Ideation Events Dataset, a publicly available EHR dataset that is a subset of the MIMIC III d Johnson et al. (2016). ScAN contains \(19,690\) expert-annotated SA and SI events with their attributes (e.g., methods for SA) over \(12,759\) EHR notes. Specifically, experts annotated suicidality evidence or sentences relevant to SA and SI events during a patient's stay at the healthcare facility, an example of SA annotations is shown in Fig 1. The evidences were put together to assess whether the patient has an SA or SI event.

We also present ScANER (**S**uicide **A**ttempt and Ideation **E**vents **R**etriever), a RoBERTa-based NLP model that is built on a multi-task learning framework for retrieving evidences from the EHRs and then predicting a patient's SA or SI event using the complete set of EHR notes from the hospital stay using a multi-head attention model. We focus on the prediction of SA and SI using all the EHR notes during a patient's stay because for the whole duration, multiple EHR notes and note types are generated, including _admission_ notes, _nursing_ notes, and _discharge summary_ notes. Suicidal information are described in multiple notes throughout the stay. For example, a patient was admitted to the hospital with opioid overdose. It was documented initially in the admission note as an SA, but later dismissed as an accident after physician's evaluation. In another example, an opioid overdose admission was first documented as an accident on admission, but later documented to be an SA event after clinical assessment. Both ScAN and ScANER capture SA and SI information at the hospital-stay level. ScANER is able to retrieve suicidal evidences from EHR notes with a macro-weighted F1-score of \(0.83\) and is able to predict SA and SI with a macro F1-score of \(0.78\) and \(0.60\), respectively. Our annotation guidelines, ScAN, and ScANER system will be made publicly available, making ScAN a benchmark EHR dataset for SA and SI events detection. We will release the training and evaluations splits used in this study for benchmarking new models.

## 2 Related Works

Efforts on detecting SA and SI within EHRs have been explored in recent years. Most work used rule-based or traditional machine learning-based approaches. In one study, experts created handcrafted rules from mentions of suicidality (both SA and SI) and then used the rules to identify suicidality as positive, negative, or unknown in a document Downs et al. (2017). The rule-based approaches are limited by their scalability. In another study, structured and unstructured EHRs were used to classify at the hospital-stay level as SA, SI, or no mention of suicidal behavior Metzger et al. (2017). The training data consisted of only 112 SA, 49 SI and 322 unrelated examples. In contrast, ScAN comprises of 697 hospital-stays with more than 19,000 suicidal event examples over 12,759 clinical notes. Only traditional machine learning models such as random forest Breiman (2001) were explored. In contrast, ScANER was built on the state-of-the-art self-attention based model.

Hybrid approaches have also been developed to identify SA at the hospital-stay level Fernandes et al. (2018). In that study, a post-processing heuristic rule-based filter (e.g., removing negated events) was applied to the machine-learning-based classifier (a SVM Cortes and Vapnik (1995) classifier) to reduce false positives. Training and evaluation were done also on relatively small datasets (500 for training and 500 for evaluation).

Finally, weakly supervised approaches have been developed to identify SI from EHRs Cusick et al. (2021). In that study, authors used ICD codes to identify 200 patients with SI and then obtained EHR notes of those patients (\(6,588\)). This EHR note dataset was then used as the 'current' SI training data. The remaining 400 patients were labelled as 'potential' SI and their \(12,227\) EHRs were also labelled the same. Authors used multiple statistical machine learning models and one deep learning model: convolutional neural network Bhat and Goldman-Mellor (2017) also used 

feed-forward neural networks to predict suicide attempts over 500k unique patients but the EHR data for this study is not publicly available. Ji2020 surveyed multiple studies where the researchers worked on private datasets Tran2013,Haerian2012 for suicide attempt and ideation prediction. Whereas in our study, in contrast to using the ICD codes which has considerable errors, domain experts chart-reviewed a large, publicly available set of EHRs for SI and SA, along with their attributes (e.g., positive or negative SA, SI and the type of self-harm such as asphyxiation and overdose).

## 3 Dataset

In this section, we introduce ScAN (**S**uieide **A**ttempt and Ideation **E**vents **Dataset**) and describe it's data collection and annotation process. We also discuss some examples from ScAN along with basic dataset statistics.

### Dataset collection

For annotation, we selected the notes from the MIMIC-III Johnson et al. (2016) dataset, which consists of the de-identified EHR data of patients admitted to the Beth Israel Deaconess Medical Center in Boston, Massachusetts from 2001 to 2012 Johnson et al. (2016). The data includes notes, diagnostic codes, medical history, demographics, lab measurements among many other record types. We chose MIMIC-III because it is publicly available under a data use agreement and allows clinical studies to be easily reproduced and compared.

The diagnostic ICD codes for the patients are provided at hospital-stay level in MIMIC with admission identification numbers (HADM_ID in MIMIC database). We first filtered the hospital stays that had ICD codes associated with suicide and overdose. This resulted in \(697\) hospital-stays for \(669\) unique patients. For each stay, multiple de-identified notes such as nursing notes, physician notes, and discharge summaries are available. For the selected \(697\) hospital-stays we extracted a total of \(12,759\) notes. Each medical note contains multiple sections about a patient such as family and medical history, assessment and plan, and discharge instructions. We extracted different sections from these clinical notes using MedSpaCy's2clinical_sectionizer and filtered the relevant sections from these clinical notes for annotation. The extensive list of these sections is provided in Appendix A.

Footnote 2: [https://github.com/medspacy/medspace](https://github.com/medspacy/medspace)

### Annotation Process

The aim was to annotate all instances of SA and SI documented in the medical notes as defined by Center of Disease Control and Prevention (CDC) Hedegaard2020. The filtered \(12,759\) notes were annotated by a trained annotator under the supervision of a senior physician. Each note consisted of instances of SA, SI, both or none. The senior physician randomly annotated \(330\) notes and had a 100% agreement with the annotator on hospital-stay level annotation and \(85\)% agreement on sentence-level annotations. After adjudication between the senior physician and the annotator, the disagreements were discussed and adjusted by the annotator.

Suicide Attempt (SA):The annotator labelled all the sentences with a mention of SA. Some hospital stays could represent multiple types of SA, such as in Fig. 1, where 'tried to hang himself' is labelled as a _positive_ SA and 'Tylenol overdose' is labelled as _unsure_ since the overdose was never confirmed as an SA event elsewhere in the medical notes of the patient's hospital-stay. The label _unsure_ is used when it is not clearly documented if a self-harm was an SA event or not. The _negative_ instance, example shown in Fig. 2, is a sentence that confirms that the self-harm, an "accidental overdose", responsible for the patient's hospital-stay is not an SA event. In this work, we only focused on suicidal self-harm and not non-suicidal self-harm Crosby2011.

Further sub-categories are also provided for an SA annotation in the form of the ICD label group: a.) T36-T50: Poisoning by drugs, medications

Figure 2: An example with _negative_ SA and _negative_ SI annotations.

 

and biological substances b.) T51-T65: Toxic effects on non-medical substances c.) 771: Asphyxiation or suffocation and d.) X71-X83: Drowning, firearm, explosive material, jumping from a high place, crashing motor vehicles, other specified means.

Suicide Ideation (SI):SI is defined as any mention and/or indication of wanting to take one's own life or harm oneself. Similar to SA, any sentence with a mention of SI was labelled within the patient's notes. A SI annotation could be labeled as _positive_ or _negative_, an example for each label is shown in Fig. 3.

A sentence without SA or SI annotation would be considered as a _neutral-SA_ or _neutral-SI_ sentence respectively. Sentence level annotations provide more visibility to a medical expert for the hospital-stay level annotation.

### Dataset statistics

ScAN consists of \(19,690\) unique evidence annotations for the suicide relevant sections of \(12,759\) EHRs of \(697\) patient hospital-stays. There are a total of \(17,723\) annotations for SA events and \(1,967\) annotations for SI events. The distribution for both SA and SI events is provided in Table 1.

## 4 Methodology

In this section, we introduce _ScANER_ (S Suicide **A**ttempt and Ideation **E**vents **R**etreiver): a strong baseline model for our dataset. ScANER consists of two sub-modules: (1) An _evidence retriever module_ that extracts the evidences related to SA and SI and (2) A _predictor module_ that predicts SA or SI label for the patient's hospital-stay using the evidences extracted by the retriever module.

### Evidence Retriever

Problem Formulation:Given an input clinical note, the model extracts the evidences (one or more sentences) related to SA or SI (SA-SI) from the note. This is a binary classification problem where given a text snippet the model predicts whether it has an evidence for SA-SI or not. We learn this task at paragraph level where the input is a set of \(20\) consecutive sentences because the local surrounding context provides additional important information Yang et al. (2021); Rawat et al. (2019). A paragraph was labeled as _evidence no_, if all the sentences in that paragraph are _neutral-SA_ and _neutral-SI_. If there was at least one SA-SI sentence, it was con

Figure 3: Examples of _positive_ and _negative_ SI annotations.

\begin{table}
\begin{tabular}{l l l l} \hline \hline \multirow{2}{*}{_General Statistics_} & Patients & Hospital-stays & Notes \\  & \(669\) & \(697\) & \(12,759\) \\ \hline \multirow{2}{*}{_Suicide Attempt_} & Positive & Negative & Unsurc \\  & \(14,815\) & \(170\) & \(2,738\) \\ \hline \multirow{2}{*}{_Suicide Ideation_} & Positive & Negative & \\  & \(1,167\) & \(800\) & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Distribution of unique annotations at the patient, hospital-stay and notes level in ScAN.

 

sidered an _evidence yes_. As the number of non-evidence sentences significantly outsized the evidence sentences, we decided to use an overlapping window of \(5\) sentences between the paragraphs to build more evidence paragraphs. The distribution of the paragraphs, across all evidence, SA and SI labels for train, validation, and test set is provided in Table 2. We segregated the train and test set such that any patient observed by the _retriever module_ during training was not seen in the test set. This is important as there are patients who had multiple hospital-stays in ScAN.

**Proposed Model:** Transformer Vaswani et al. (2017) based language models Devlin et al. (2018); Liu et al. (2019) have shown great performance for a broad range of NLP classification tasks. Hence, to extract the evidence paragraphs we trained a RobertaLiu et al. (2019) based model. It has been previously shown that the domain-adapted versions of the pre-trained language models, such as clinicalbertAlsentzer et al. (2019) or BiobertLee et al. (2020), work better than their base versions. So, we further pre-trained the Robertabase model over the MIMIC dataset to create a clinical version of Roberta model, hereby referenced as medRxiv. During our initial exploration, we experimented with clinicalBERT and BioBERT but found that medRoBERTa consistently outperformed both models. medRoBERTa achieved an overall F1-score of \(0.88\) whereas both clinicalBERT and BioBERT achieved an overall F1-score of \(0.85\). Our hospital-level SA and SI predictor would work with any encoder-based evidence retriever model.

**Multi-task Learning:** We trained medRxiv in a multi-task learning setting where along with learning the evidence classification task, the model also learns two auxiliary tasks: (a.) Identifying the label for SA between _positive_, _negative_, _unsure_ and _neutral-SA_ and, (b.) Identifying the label for SI between _positive_, _negative_ and _neutral-SI_. The training loss (\(L(\theta)\)) for our evidence retriever model was formulated as:

\[L(\theta)=L_{evi}+\alpha*L_{SA}+\beta*L_{SI} \tag{1}\]

Where \(L_{evi}\) is the negative log likelihood loss for evidence classification, \(L_{SA}\) and \(L_{SI}\) are SA and SI prediction losses respectively, and \(\alpha\) and \(\beta\) are the weights for the auxiliary tasks' losses. The distribution of labels across all the three tasks is highly skewed, hence, we applied the following techniques to learn an efficient and robust model.

* Weighted log loss was used in both main task and auxiliary tasks. The total loss for each task was calculated as the weighted sum of loss according to the _label_ of the input paragraph. Log weighing helps smooth the weights for highly unbalanced classes. The weight for each class was calculated using: \[w_{l,t}=\left\{\begin{array}{ll}1.0&\text{if}(w_{l,t}<1.0)\\ log(\gamma*N_{t}/N_{l,t})\end{array}\right.\] Where \(N_{t}\) is the count of all training paragraphs for the task \(t\) and \(N_{l,t}\) is the count of paragraphs with label \(l\) for the task \(t\) and \(w_{l,t}\) is the calculated weight for those paragraphs. We tuned \(\gamma\) as a hyper-parameter. All training hyper-parameters for our best model are provided in Appendix B.
* We also employed different sampling techniques Youssef (1999), up and down sampling, to help our model learn from an imbalanced dataset. After sweeping for different sampling combinations as hyper-parameters, we found that down-sampling the _no_-evidence paragraphs by \(10\%\) resulted in the best performance.
* The _negative_ label of SA is severely underrepresented in ScAN making it difficult for the model to learn useful patterns from such instances, refer Table 2. After discussion with the experts, we decided to group the instances of _negative_ and _unsure_ together and label them as _neg_unsure_ because for both groups the general psych outcome is to let the patient leave after the hospital-stay as there is no solid evidence defining whether the self-harm was a SA event.

\begin{table}
\begin{tabular}{p{43.4pt} p{65.4pt} p{65.4pt} p{65.4pt}} \hline \hline _Suicide Attempt_ & Positive & Neg\_Unsure & Neutral-SA \\ \hline Train Val & \(377\) & \(54\) & \(1,381\) \\ Test & \(50\) & \(10\) & \(189\) \\ \hline _S Suicide Ideation_ & Positive & Negative & Neutral-SI \\ \hline Train Val & \(377\) & \(214\) & \(1,521\) \\ Validation Test & \(45\) & \(28\) & \(208\) \\ Test & \(44\) & \(35\) & \(357\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Distribution of SA and SI at hospital-stay level in training, validation and testing set.

 



### Hospital-stay level SA and SI Predictor

Problem FormulationGiven all the clinical notes of a patient during the the hospital stay, the model predicts the label for SA (_positive_, _neg_unsure_ and _neutral-SA_) and SI (_positive_, _negative_ and _neutral-SI_). The _prediction module_ uses the evidence paragraphs extracted by the _retriever module_.

Robust FinetuningThe retriever module is not perfect and can extract false positives. This results in extracting irrelevant paragraphs, with evidence label _No_, along with evidence paragraphs for a hospital-stay with SA or SI and extracting irrelevant paragraphs as evidences for a hospital-stay with both SA and SI marked as _neutral_. To tackle such situations and train a robust model, we applied three techniques:

* For a hospital-stay with a non-_neutral_ label for SA or SI, during training we added some noise in the form of irrelevant paragraphs (a paragraph with no SA or SI annotation) from the notes to the set of actual evidence paragraphs for the input. An irrelevant paragraph from a clinical note was sampled with a probability of \(0.05\). This forced the predictor module to learn effectively even with noisy inputs.
* For a _neutral_ hospital-stay with no evidence paragraphs, we randomly chose \(X\) unique irrelevant paragraphs from the notes. \(X\) was sampled from the distribution of number of evidence paragraphs of the non-_neutral_ hospitals- stays. This prevented the leaking of any information to the prediction module during training by keeping the distribution of number of input paragraphs the same across _neutral_ and non-_neutral_ instances.
* Since these hospital-stays were extracted using the ICD codes related to suicide and overdose, the data is quite skewed with only \(102\)_neutral_ events from a total of \(697\) hospital-stays. Whereas in a real-world scenario, _neutral_ hospital-stays would be much higher than non-_neutral_ ones. Hence, to facilitate a bal

Figure 4: ScANER (**S Suicide Attempt and Ideation Events R**etreiver) consists of two sub-modules: (a.) Evidence retriever module extracts evidence paragraphs from all EHR notes. We trained the module using all annotated paragraphs from ScAN. (b.) Prediction module predicts the SA and SI label for a patient using the evidence paragraphs extracted by the _retriever module_ from EHR notes during the patient’s hospital-stay.



anced learning of the predictor module we introduced \(1,800\)_neutral_ hospital-stays from the MIMIC dataset. The distribution for SA and SI at hospital-stay level is provided in Table 3.

Proposed ModelThe paragraphs extracted using the _retriever module_ for a patient's hospital-stay were provided as an input to the predictor module. We used a multi-head attention model to predict the SA and SI label for a hospital-stay as self-attention based models have proved to be quite effective for a lot of prediction tasks in machine learning Devlin et al. (2018); Cao et al. (2020); Hoogi et al. (2019).

We encoded the extracted paragraphs (\([p_{1},p_{2}....p_{n}]\)) using the retriever module, medRoBERTa, to get a vector representation of \(768\) dimensions for each of the paragraphs (\([v_{1},v_{2}...v_{n}]\)). Training the _retriever module_ on auxiliary tasks of predicting SA and SI helped align these paragraph representations for SA and SI prediction. Then, we added a prediction vector (\(v_{0}\)) along with all the vector representations of the paragraphs to get \(\mathcal{V}=[v_{0},v_{1},v_{2}...v_{n}]\). We passed \(\mathcal{V}\) through our multi-head attention model to get the hidden representations \(\mathcal{H}=[h_{0},h_{1}...h_{n}]\). We then passed \(h_{0}\) through a SA inference layer and SI inference layer to predict the labels. During the whole training process, the weights of the retriever module were frozen whereas \(v_{0}\) was a learnable vector initialised as an embedding in the multi-head attention model. We used a separate \(v_{0}\) prediction vector so that it could retain the information from all the other paragraph representations for hospital-stay level prediction similar to how [CLS] is utilized in different transformer-based models for sequence prediction Devlin et al. (2018); Liu et al. (2019). We tuned the number of layers and number of attention heads of our prediction module as hyper-parameters and achieved the best performance using a 2-layer and 3-attention head model. Our complete ScANER model is illustrated in Fig 4.

## 5 Results and Discussion

Since the labels for both the retriever and prediction task are imbalanced, we used macro-weighted precision, recall, and F1-score to evaluate the overall performance of our models. Macro-weighted metrics provide better model insights across all labels.

Evidence Retriever PerformanceOur multi-task learning model achieved a F1-score of \(0.83\) for extracting positive evidence paragraphs and an F1-score of \(0.88\) overall. The retriever model has higher recall than precision for the positive evidence paragraphs (\(0.87>0.79\)), SA (\(0.74>0.71\)), and SI (\(0.62>0.46\)) events, as shown in Table 4. In healthcare, there is an incentive to maximize recall over precision Watson and McKinstry (2009). As mentioned in SS4.2, ScANER was trained with added noisy paragraphs and is therefore robust to the extracted evidence paragraphs if they contain some false positives.

The _retriever module_ achieves an overall F1-score of \(0.63\) for SA prediction and \(0.64\) for SI prediction at paragraph-level. The performance for positive SA and SI evidence is much higher than the performance for _neg_unsure_ SA and _negative_ SI. We looked at the confusion matrices for SA and SI paragraph-level prediction and found that largely ScANER made mistakes between _positive_ and _neg_unsure_ labels for SA prediction and between _positive_ and _negative_ labels for SI prediction (refer Appendix C). The poor performance in SA for _neg_unsure_ evidence prediction is mainly due to data sparsity where the _neg_unsure_ cases are only \(1743\); in contrast, the positive cases are 4-fold higher. Similarly, for SI the _positive_ cases are \(1.4\) times higher than the _negative_ cases.

#### Hospital-stay level Prediction Performance

Our multi-head attention model is able to achieve an overall macro F1-score of \(0.78\) for SA prediction and \(0.60\) for SI prediction, as shown in Table 5. For SA, the _prediction module_ achieves a recall of \(0.93\) for the positive label. After analysing the confusion matrix, the model largely predicts a _positive_ label for the visits with _neg_unsure_ label, as shown in Table 6. The poor performance for _neg_unsure_ is largely because of its small representation in the training set of ScAN, \(54\) negative cases as compared to \(377\) positive and \(1,381\) neutral instances. In our future work, we plan to expand ScAN with more instances of _negative_ SA events.

For SI, the prediction module achieves an overall F1-score of \(0.60\) with a precision of \(0.63\) and recall of \(0.66\). The model has a high recall for _neutral-SI_ and _positive_ but the _positive_ label has a low precision of \(0.49\). After analysing the test set, we observed that a lot of patient hospital-stays with _negative_ labels are getting wrongly predicted as _positive_, as shown in Table 6. After doing error 

analysis for hospital-stays with _negative_ labels, we observed that a lot of extracted evidence paragraphs contain information that suggests that the patient had SI before the SA but does not have SI anymore during the hospital-stay. As shown in the example in Fig 5, the past SI is an explanation for the SA but then the patient does not have any further SI during the hospital-stay. This suggests that period assertions for these annotations are quite important and we aim to add period assertion property in our future work by further annotating ScAN.

**Context: Depression: On further questioning, the pt reported that the cocaine ingestion was a suicide attempt. He stated that he was having financial difficulty over the past few months and thought that if he attempted suicide, his family would be able to receive his life insurance; A 1:i sitter was ordered and psychiatry consult was called. He denied any futher suicidal ideations while in the hospital. # Fever: Pt developed fever to 101 with leukocytosis on [**2189-7-20*]. Annotations: \(1\). On further, insurance: Positive SA \(2\). He denied...hospital: Negative SI

## 6 Conclusion

In this paper, we introduce ScAN: a publicly available suicide attempt (SA) and ideation (SI) events dataset that consists of \(12,759\) EHR notes with \(19,960\) unique evidence annotations for suicidal behavior. To our knowledge, this is the largest and publicly available dataset for SA and SI, an important resource for suicidal behaviors research. We also provide a strong Roberta baseline model for the dataset: ScANER (SA and SI retriever) which consists of two sub-modules: (a.) an _evidence retriever module_ that extracts all the relevant evidence paragraphs from the patient's notes and (b.) a _prediction module_ that evaluates the extracted evidence paragraphs and predicts the SA and SI event label for the patient's stay at the hospital. ScAN and ScANER could help extract suicidal behavior

Figure 5: An instance for which ScANER incorrectly predicted a _negative_ hospital-level SI as _positive_.

\begin{table}
\begin{tabular}{l|c c c} \hline \multicolumn{4}{c}{_Hospital-stay SA Prediction_} \\ \hline  & Positive & Neg\_Unsure & Neutral-SA \\ \hline Positive & 85 & 4 & 2 \\ Neg\_Unsure & 5 & 11 & 3 \\ Neutral-SA & 15 & 8 & 303 \\ \hline \multicolumn{4}{c}{_Hospital-stay SI Prediction_} \\ \hline  & Positive & Negative & Neutral-SI \\ \hline Positive & 41 & 2 & 1 \\ Negative & 27 & 4 & 4 \\ Neutral-SI & 15 & 4 & 338 \\ \hline \end{tabular}
\end{table}
Table 6: Confusion matrices for SA and SI prediction at hospital-stay level.

 

in patients for suicide surveillance and predictions, leading to potentially early intervention and prevention efforts by medical professionals.

## References

* Asentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical bert embeddings. _arXiv preprint arXiv:1904.03323_.
* Anderson et al. (2015) Heather D Anderson, Wilson D Pace, Elias Brandt, Rodney D Nielsen, Richard R Allen, Anne M Libby, David R West, and Robert J Valuck. 2015. Monitoring suicidal patients in primary care using electronic health records. _The Journal of the American Board of Family Medicine_, 28(1):65-71.
* Bhat and Goldman-Mellor (2017) Harish S Bhat and Sidra J Goldman-Mellor. 2017. Predicting adolescent suicide attempts with neural networks. _arXiv preprint arXiv:1711.10057_.
* Breiman (2001) Leo Breiman. 2001. Random forests. _Machine learning_, 45(1):5-32.
* Cao et al. (2020) Ran Cao, Leyuan Fang, Ting Lu, and Nanjun He. 2020. Self-attention-based deep feature fusion for remote sensing scene classification. _IEEE Geoscience and Remote Sensing Letters_, 18(1):43-47.
* Center for Behavioral Health Statistics Quality CBHSQ. 2020. 2019 national survey on drug use and health (nsduh): Methodological summary and definitions. _Substance Abuse and Mental Health Services Administration_.
* Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995. Supportvector networks. _Machine learning_, 20(3):273-297.
* Crosby et al. (2011) Alex Crosby, LaVonne Ortega, and Cindi Melanson. 2011. Self-directed violence surveillance; uniform definitions and recommended data elements.
* Cusick et al. (2021) Marika Cusick, Prakash Adekkanattu, Thomas R Campion Jr, Evan T Sholle, Annie Myers, Samprit Banerjee, George Alexopoulos, Yanshan Wang, and Jyotishman Pathak. 2021. Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation. _Journal of psychiatric research_, 136:95-102.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Downs et al. (2017) Johnny Downs, Sumithra Velupillai, Gkotsis George, Rachel Holden, Maxim Kikoler, Harry Dean, Andrea Fernandes, and Rina Dutta. 2017. Detection of suicidality in adolescents with autism spectrum disorders: developing a natural language processing approach for use in electronic health records. In _AMIA annual symposium proceedings_, volume 2017, page 641. American Medical Informatics Association.
* Fernandes et al. (2018) Andrea C Fernandes, Rina Dutta, Sumithra Velupillai, Jyoti Sanyal, Robert Stewart, and David Chandran. 2018. Identifying suicide ideation and suicidal attempts in a psychiatric clinical research database using natural language processing. _Scientific reports_, 8(1):1-10.
* Haerian et al. (2012) Krystl Haerian, Hojjat Salmasian, and Carol Friedman. 2012. Methods for identifying suicide or suicidal ideation in ehrs. In _AMIA annual symposium proceedings_, volume 2012, page 1244. American Medical Informatics Association.
* Hedegaard et al. (2020) Holly Hedegaard, Sally C Curtin, and Margaret Warner. 2020. Increase in suicide mortality in the united states, 1999-2018.
* Hogoi et al. (2019) Assaf Hoogi, Brian Wilcox, Yachee Gupta, and Daniel L Rubin. 2019. Self-attention capsule networks for image classification. _arXiv preprint arXiv:1904.12483_.
* Jensen et al. (2012) Peter B Jensen, Lars J Jensen, and Soren Brunak. 2012. Mining electronic health records: towards better research applications and clinical care. _Nature Reviews Genetics_, 13(6):395-405.
* Ji et al. (2020) Shaoxiong Ji, Shirui Pan, Xue Li, Erik Cambria, Guodong Long, and Zi Huang. 2020. Suicidal ideation detection: A review of machine learning methods and applications. _IEEE Transactions on Computational Social Systems_.
* Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimici-iii, a freely accessible critical care database. _Scientific data_, 3(1):1-9.
* Kessler et al. (2020) Ronald C Kessler, Robert M Bossarte, Alex Luedtke, Alan M Zaslavsky, and Jose R Zubizarreta. 2020. Suicide prediction models: a critical review of recent research with recommendations for the way forward. _Molecular psychiatry_, 25(1):168-179.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Metzger et al. (2017) Marie-Helene Metzger, Nastassia TVardik, Quentin Gicquel, Come Bouvry, Emmanuel Poulet, and Veronique Potinet-Pagliaroli. 2017. Use of emergency department electronic medical records for automated epidemiological surveillance of suicide attempts: a french pilot study. _International journal of methods in psychiatric research_, 26(2):e1522 

Bhanu Pratap Singh Rawat, Fei Li, and Hong Yu. 2019. Naranjo question answering using end-to-end multitask learning model. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2547-2555.
* Tran et al. (2013) Truyen Tran, Dinh Phung, Wei Luo, Richard Harvey, Michael Berk, and Svetha Venkatesh. 2013. An integrated framework for suicide risk prediction. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1410-1418.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Watson and McKinstry (2009) Philip WB Watson and Brian McKinstry. 2009. A systematic review of interventions to improve recall of medical advice in healthcare consultations. _Journal of the Royal Society of Medicine_, 102(6):235-243.
* WHO (2021) WHO. 2021. Suicide fact sheet: [https://www.who.int/news-room/fact-sheets/detail/suicide](https://www.who.int/news-room/fact-sheets/detail/suicide).
* Yang et al. (2021) Baosong Yang, Longyue Wang, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2021. Context-aware self-attention networks for natural language processing. _Neurocomputing_, 458:157-169.
* Youssef (1999) Abdou Youssef. 1999. Image downsampling and upsampling methods. _National Institute of Standards and Technology_.

 

Selected Clinical Sections

The sections selected for annotations after using clinical_sectionizer are enumerated below:

1. Allergy
2. Case Management
3. Consult
4. Discharge Summary
5. Family history
6. General
7. HIV Screening
8. Labs and Studies
9. Medication
10. Nursing
11. Nursing/other
12. Nutrition
13. Observation and Plan
14. Past Medical History
15. Patient Instructions
16. Physical Exam
17. Physician
18. Present Illness
19. Problem List
20. Radiology
21. Rehab Services
22. Respiratory
23. Sexual and Social History
24. Social Work

## Appendix B Hyper-parameter Settings

All the hyper-parameter settings for both modules of ScANER are provided in Table 7.

 



## Appendix C Confusion matrices

The confusion matrices for SA and SI prediction at paragraph level is provided in Table 8.

[TABLE:A1.T7][E

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & _Evidence Retrieval Module_ & \\ \hline Learning Rate & Warmup steps & Optimizer & Adam \(\epsilon\) \\
2e-5 & 2,000 & Adam & 1e-8 \\ \hline \(\gamma\) & \(\alpha\) & \(\beta\) & \\
2.5 & 1.1 & 1.5 & \\ \hline \multicolumn{4}{c}{_Hospital Stay SCa-SI Prediction Module_} \\ \hline Attention Heads & Attention Layers & Learning Rate & Warmup steps \\
3 & 2 & 1e-3 & 1,200 \\ \hline Optimizer & Adam \(\epsilon\) & & \\ Adam & 1e-8 & & \\ \hline \hline \end{tabular}
\end{table}
Table 7: Hyper-parameter setting for both retriever and prediction module of ScANER.

\begin{table}
\begin{tabular}{l|r r r} \hline \hline \multicolumn{4}{c}{_Paragraph SA Prediction_} \\ \hline  & Positive & Neg\_Unsure & Neutral-SA \\ \hline Positive & 1,804 & 285 & 344 \\ Neg\_Unsure & 253 & 118 & 80 \\ Neutral-SA & 472 & 204 & 7,314 \\ \hline \multicolumn{4}{c}{_Pargraph SI Prediction_} \\ \hline  & Positive & Negative & Neutral-SI \\ \hline Positive & 206 & 69 & 56 \\ Negative & 71 & 87 & 31 \\ Neutral-SI & 170 & 73 & 10,111 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Confusion matrices for the predictions on the test set of evidence retriever.